| Re: Do not want some of my web pagfes listed in yahoo sercah dir -
06-02-2007, 08:52 PM
"EMAIL REMOVED" <EMAIL REMOVED> wrote in
news:EMAIL REMOVED ups.com:
> Hi,
>
> I would like to have a folder of html files that are not indexed by
> yahoo, will this work?????
>
> User-agent: *yahoo*
> Disallow: privatefolder
>
> I could not fine the name of the yahoo bot..
>
>
It may!
In the old days when the major SE's crawled from a single bot and/or IP
range, even then there oc***ional malfunctions.
Today with all the major bots utilizing a variety of bots and IP ranges
to support a variety of tools the likelihood of runaway bots and
compliance is more open to abuse.
A more effective solution for what you desire would be to place your
files in the private and NOT mention the folder in robots.txt (request),
while simultaneously implementing htaccess restrictions which will NOT
permit any bots access. |