No means no, even for Perplexity · Victor Wynne

When a website puts up a robots.txt file, it’s basically putting up a “No Solicitors” sign on their digital front door. It doesn’t matter if you’re selling Girl Scout cookies, collecting for charity, or claim you’re there to help the homeowner. The sign says no, and that should mean no. Period.

I like this analogy. People argue that robots.txt doesn’t necessarily have to be respected. So technically you can ignore it all you want, but it kinda makes you an asshole and a parasite to the open web if you choose to go that route.