Online Place for Reference Materials

jayrod · **Posted:** Thu Oct 27, 2022 1:03 pm

Ed Kapuscinski wrote:

jayrod wrote:

Ed Kapuscinski wrote:

In those situations you include some logic for user agent in your authentication.

If it's a bot you like, you let them in.

This thread switched over to the wrong siding belonging more in a beginner web development class.

Anyway, to clear up any confusion for those very few who care to know:
Put simply, bots, just like the web browser software on your device, use User Agent to identify what they are to the server. As a refinement, you can disallow specific bots access to your server via their User Agent in the robots.txt file versus the blanket disallow all. All others not restricted would get a pass by default. That’s how you pick and choose. It’s not a matter of defining which to allow, but which to exclude.

That's a dangerous inaccuracy in the way robots.txt works.
It only asks bots not to do whatever you tell it. They still can, there's nothing stopping them, it only works if they follow the protocol.

When you're dealing with well known above board ones, functionally there's no difference, but don't expect robots.txt to defend you from nefarious actors.

Sigh…. I know. From my previous post:

Quote:

Robots.txt can only restrict a bot from indexing a folder or file and the restriction isn’t necessarily honored by all bots.

Emphasis added.

Please note that I’ve attempted to not go into any more excruciating, mind numbing detail than than necessary to correct a misconception.

jayrod · **Posted:** Thu Oct 27, 2022 1:59 pm

I’m happy to say that HRA has graciously offered to host the cross reference. I have a few items to verify and an editing of the introductory pages to do in an update and then I’ll send the latest version to them. I’ll also note that in the sticky above and delete the references to my site when done.

If anyone wants to add any research and verifiable paint data to the cross reference in the future, let me know and I’ll make sure it finds its way in there.

Overmod · **Joined:** Thu May 24, 2012 1:37 pm **Posts:** 2492

Thank you, Eric, for having provided this resource (and for caring enough to see it maintained)