Re: Spider Traps (Was Re: Google lists vulnerable sites.)

[Darryl Luff <darryl () snakegully nu>]

I think whether you have a robots.txt or not is irrelevant. The type of 
content that causes the problems shouldn't be available to web users in 
the first place. The directories containing databases shouldn't have to 
be web-visible at all?

On the vuln-dev side, I

> RD> > GOOGLE is an administration tool? hmm... anyway, you're right about the fact that yes it's not google's 
> responsibility, admins could have a better robots.txt in their webfolders too.
> RD>
> RD> Not all web-robots are well trained enough to pay attention to robots.txt
> RD> files though...
>
> That's right.. and those bots should be stopped in the first place.