Ask HN: Why Reddit blocks all automated access but has .json for all URLs?

Reddit is aggressively blocking all automated access (see their robots.txt) and uses a lot of heuristics to block crawlers that do not honor it.

However, at the same time all Reddit URLs can be made machine readable by adding a .json to the end.

Can anyone explain what's the point of that?

5 points | by ksajadi 21 hours ago

6 comments

  • jerlam 13 hours ago
    Reddit locked down their API three years ago, purportedly to monetize their content for AI. This also killed all the free third party apps.

    https://arstechnica.com/gadgets/2023/10/reddit-may-block-sea...

  • dlcarrier 7 hours ago
    JSON is a popular way to send data around, and the site was probably built around it long ago, using third-party libraries that are difficult to customize.

    It does make one wonder for a site so easy to scrape, why there aren't any popular third-party clients that use scraped data, like FreeTube and NewPipe do with YouTube.

  • downbad_ 20 hours ago
    I just want to know why Reddit keeps banning people for no good reason,while letting bots roam free.
    [-]
    • ksajadi 20 hours ago
      It's turned into a bizarre place. Today I asked the same question on /r/meta and my question was immediately removed without explanation. I tried to message the mods and got bounced with a "you cannot send a message to that user"
      [-]
      • downbad_ 16 hours ago
        It was much better years ago.
  • brudgers 16 hours ago
    My guess is they see correlation between patterns of automated access and problematic behaviors.

    In other words, the current state is the result of hard won experience not syllogistic reasoning.

  • PaulHoule 21 hours ago
    Circa 2009 or so I was interested in automated link building systems, there were some sites that had no defenses, but I saw enough going on around Reddit that I just didn't want to mess with it.
  • maheenaslam 17 hours ago
    I guess it's Reddit saying we don’t want just any bot crawling around, but if you need the data in a simple format, here’s an easy way to get it.