You may remember when Google took a step towards transparency in early 2007 when they started laying out tips on the 'how' & 'why' of using a robots.txt file in ...
In their ever evolving attempt to help webmasters get their deep level & dynamic pages indexed the major search engines (namely Google & Yahoo) have been reworking the way they ...