If you are using LLMs to interact with sites like GitLab and GitHub, and you have the option to use a GraphQL API, you should jump on it immediately.
GraphQL is absolutely terrible for human developers to interact with, but it's like Facebook could see into the future back in 2012. I cannot imagine a more perfect API surface for agents. With the REST API on GitHub, you can consume maybe 10 issue JSON blobs before your context window is blown out. With GraphQL constraining the results you can easily read hundreds in the same token budget.
Additionally, the # of requests your agents need to make can be reduced in many cases since GraphQL can join across types whereas REST APIs cannot. You essentially get savings in two dimensions here. Quota and raw token volume per logical response.
Providing kickbacks to the repos being scraped would be a good way to help fund open source projects and pay creators like streaming services do. Seems like they're headed in this direction - it would be a massive product differentiator over GH
My first reaction was that I really like this idea.
If we had a system where people who access projects pay and popular FOSS developers get paid for it we'd have much better alignment.
My second thought was that bots would immediately try to circumvent such a plan. They'd probably spam Gitlab with fake repos to try to harvest those payouts.
The guidance given seems to hurt open source projects, not help.
> Make the project private if the traffic is not coming from the audience you built it for, which stops anonymous callers reaching it at all. Or upgrade to Premium or Ultimate for much higher limits.
Three minutes after kickbacks were announced there would be a flurry of new repos being created with bots repeatedly scraping them just to get those kickbacks.
Making requests is inherently cheaper than delivering responses, even with caches. Efficiency improvements can buy a little time on a given resources but won't solve the problem of bot saturation now that everybody can spawn a custom bot in about 12 seconds and is being encouraged to do so.
Rate limits, blocking, and pay-per-use are the only roads out and even those might not last as models get better at hacking and masquerading.
The internet we want to use LLM's with is simply not one that can support LLM's, and with LLM's not going anywhere, the whole experience of the internet is going to be forced into some radically less open and more expensive paradigm.
Policies like this just represent the beginning of the transition.
We need a new non-commercial version of the internet. Free of bots, free of ads, ... you pay to access social media optimized to be interesting enough to be worth paying instead of addictive enough to keep you scrolling to show you more ads.
It may look impossible right now. But what is impossible for real is to continue as we are. The damage that internet does to society is increasing by the day while its value is reduced (economic value, social value).
> We need a new non-commercial version of the internet.
> you pay to access social media optimized to be interesting...
So is it commercial or non commercial?
Nothing is stopping you from creating a social network that is pay gated. Go build it. If you can't get anyone to sign up perhaps you'll realize it's not so easy as scapegoating addictive social media.
There is a whole cottage industry of people who legitimately make their living criticizing Facebook. It's a consumer software product. Yet few of these people seem to have their conviction extend to building an alternative that ever catches an audience. Why is that? Because addiction? Any other excuses?
You'd need a force strong enough to prevent it from falling prey to this tragedy of the commons, and that force would need to be stronger than the incentives to commercialize it. And that's where plenty of contemporary scraping-based salaries lay.
That's this internet. Nothing precludes you from dropping big tech and solely engaging with the "indie web"[0][1] if you so choose. You're free to build your own website with its own RSS feed, join webrings[2] of like-minded people, and engage organically.
[0] One of many similar initiatives, I'm sure. Not an endorsement.
Careful what you wish for. The ticket to entry will (have to) be a proof of identity, the ultimate nail in the coffin of privacy and anonymity on the web.Outside of that, chaos.
I'm not so certain. I used to buy a broadsheet newspaper, which was full of ads. Now I pay a few hundred a year, and get the same newspaper online, with no ads at all.
So, the precursor to online media has already gone through this paradigm shift.
I mean, what you're saying is "If I pay for a product ads go away"
Which is partially true, but it only shifts the distribution of the problem. Once your service gains enough popularity network effects cause it to gain value. You have to worry about high priced buyouts of the entire service (great for the site owner, terrible for the users).
It's not, it just requires creativity. One example I can think of is limiting connectivity by distance between nodes. Something like Meshtastic seems unlikely to ever be commercialized in the same way that the Internet has been. Sure, you would lose some useful applications of the Internet, but you would gain other things.
I would spend thousands of dollars for gitlab in terms of:
1) better UX for admin panel, I'm not sure what I've enabled and what not. Several buttons do not disable the rest of the settings, leaving me with some doubts (e.g. if I disabled grafana, why is there a setting that talks about where/how I store?)
2) a minimal version of gitlab without all the AI
Average, yes, but the way they phrase it, it could be a token bucket or similar, where you can do 60 quick requests and then be blocked for a bit while the bucket refills
I think it's because people are building agentic flows, reducing the amount of developer seats needed. It's the first step towards usage based pricing.
Getting that so I do not know exactly what they are doing. From the title I am guessing they are restricting or throttling if downloads exceeds some value.
I noticed that recently Github.com has some kind of weird bot detection on public repos. I have a browser extension for switching User Agents for a specific legacy site, sometimes i forget to turn it off and Github will require me to login to view public repos.
All of this is most likely due to mass scraping by LLMs. Welcome to the total shitification of the web.
GraphQL is absolutely terrible for human developers to interact with, but it's like Facebook could see into the future back in 2012. I cannot imagine a more perfect API surface for agents. With the REST API on GitHub, you can consume maybe 10 issue JSON blobs before your context window is blown out. With GraphQL constraining the results you can easily read hundreds in the same token budget.
Additionally, the # of requests your agents need to make can be reduced in many cases since GraphQL can join across types whereas REST APIs cannot. You essentially get savings in two dimensions here. Quota and raw token volume per logical response.
If we had a system where people who access projects pay and popular FOSS developers get paid for it we'd have much better alignment.
My second thought was that bots would immediately try to circumvent such a plan. They'd probably spam Gitlab with fake repos to try to harvest those payouts.
The guidance given seems to hurt open source projects, not help.
> Make the project private if the traffic is not coming from the audience you built it for, which stops anonymous callers reaching it at all. Or upgrade to Premium or Ultimate for much higher limits.
Rate limits, blocking, and pay-per-use are the only roads out and even those might not last as models get better at hacking and masquerading.
The internet we want to use LLM's with is simply not one that can support LLM's, and with LLM's not going anywhere, the whole experience of the internet is going to be forced into some radically less open and more expensive paradigm.
Policies like this just represent the beginning of the transition.
It may look impossible right now. But what is impossible for real is to continue as we are. The damage that internet does to society is increasing by the day while its value is reduced (economic value, social value).
> you pay to access social media optimized to be interesting...
So is it commercial or non commercial?
Nothing is stopping you from creating a social network that is pay gated. Go build it. If you can't get anyone to sign up perhaps you'll realize it's not so easy as scapegoating addictive social media.
There is a whole cottage industry of people who legitimately make their living criticizing Facebook. It's a consumer software product. Yet few of these people seem to have their conviction extend to building an alternative that ever catches an audience. Why is that? Because addiction? Any other excuses?
Except it's non-commercial, therefore valuable, therefore commercialized, therefore commercial.
You'd need a force strong enough to prevent it from falling prey to this tragedy of the commons, and that force would need to be stronger than the incentives to commercialize it. And that's where plenty of contemporary scraping-based salaries lay.
[0] One of many similar initiatives, I'm sure. Not an endorsement.
[1] https://indieweb.org/
[2] https://en.wikipedia.org/wiki/Webring
Your human engagement will attract said predators because it's a unique information signal.
Gating everything behind paid (but with no ads) likely would hurt a significant amount of lower income users.
So, the precursor to online media has already gone through this paradigm shift.
Which is partially true, but it only shifts the distribution of the problem. Once your service gains enough popularity network effects cause it to gain value. You have to worry about high priced buyouts of the entire service (great for the site owner, terrible for the users).
One request per minute.
Browsing open issues or reviewing a few PRs will easily use more than one request per minute.
They say they based the limits on the breathe user but I wonder if the overwhelming “average” interaction is to view a readme and bounce.
I don’t know that putting a paywall up to learn from or even consider contributing to public projects is a good thing.
Gitlab must pay a fortune to bot traffic, most of which is malicious or garbage at best.
Getting that so I do not know exactly what they are doing. From the title I am guessing they are restricting or throttling if downloads exceeds some value.
All of this is most likely due to mass scraping by LLMs. Welcome to the total shitification of the web.