this post was submitted on 29 Jan 2025
608 points (96.8% liked)
Technology
61227 readers
4281 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
They're framing it as "AI haters" instead of what it actually is, which is people who do not like that robots have been programmed to completely ignore the robots.txt files on a website.
No AI system in the world would get stuck in this if it simply obeyed the robots.txt files.
The disingenuous phrasing is like "pro life" instead of what it is, "anti-choice"
The internet being what it is, I'd be more surprised if there wasn't already a website set up somewhere with a malicious robots.txt file to screw over ANY crawler regardless of providence.
Waiting for Apache or Nginx to import a robots.txt and send crawlers down a rabbit hole instead of trusting them.