In their ever evolving attempt to help webmasters get their deep level & dynamic pages indexed the major search engines (namely Google & Yahoo) have been reworking the way they view and follow Robots.txt files over the past year or so. Earlier this month Yahoo announced some additions to the types tags they will now support, intorducing support for X-Robots-Tag directives which can be added to http headers allowing webmasters to tag PDF,s Word Documents, PowerPoints, videos etc! This gives webmasters control over the indexing of file types that previously we could only dream of.
December 5th, 2007
Yahoo! Search Support for X-Robots-Tag Directive to Simplify Webmaster’s Control and Weather Update
Today we’re announcing support for tags that give webmasters even more flexibility over which pages and documents are crawled and indexed by Yahoo! Search. Specifically, we’re extending our support of page level exclusion tags — NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW — to provide additional control for archiving and summarization of ANY file type…
Previously, these page level tags could only be expressed within html pages through the META directive (for e.g.
<META NAME="Slurp" CONTENT="NOARCHIVE">), but based on feedback from our webmasters, Yahoo! now enables these tags to be expressed through X-Robots-Tag directive in the http header, giving webmasters the flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process…Here are a few examples:
- X-Robots-Tag: NOINDEX — If you don’t want to show the URL in the Yahoo! Search results.Note: We’ll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.
- X-Robots-Tag: NOARCHIVE — If you don’t want to display cache link in the search results page.
- X-Robots-Tag: NOSNIPPET — If you don’t want to display summary in the search results page.
- X-Robots-Tag: NOFOLLOW — If you don’t want Yahoo! to crawl links in the page.
- Orginally posted at:
http://www.ysearchblog.com/archives/000508.html
You’ll recognize the tags since they are the same tags used in traditional meta tagging, but news that these tags can be applied to alternate file types and that Yahoo will be recognizing these tags has many webmasters sighing great gasps of relief and sliding glances over at Google to see if they’ll continue their cooperative effort to standardize some long chaotic indexing controls.
Heres to hoping!
Loading ...
