The bug was apparently fixed in open source repo a week earlier, this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week (the commit message even included "Reported by Xint.io" and I'm guessing skilled black hats would know about Xint)
Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.
Especially for software like Nginx which powers a huge part of the internet.
Disabling http2 can be a big deal. My service is primarily made up of gRPC APIs. Not an option for me to disable http2.
Luckily I'm not using nginx at the moment - but I easily could have been.
If you're using grpc, http2, etc you're so much better off using Envoy proxy
You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx
I'm not 100% informed on this, but it seems very unbelievable to me that Envoy could be considered more robust or widely used than nginx at any point in history including today. I started using ConnectRPC instead of GRPC almost entirely because it didn't leave me subject to (what I consider to be) an unnecessary dependency in Envoy.
I guess I would just say, wordpress is also more widely used than many other projects, but that doesn't necessarily make it more robust or secure than the projects Google, Cloudflare, and security companies develop.
The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.
I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.
Envoy probably has the advantage that it’s only a reverse proxy server, unlike nginx which has other uses too like a general-purpose web server with optional modules.
The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.
Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.
On all my small projects (eg rails apps) the reverse proxy to the app servers is the same server expected to serve the static assets directly. In fact the majority of bot nonsense never goes through the proxy due to path rules.
Yes [0]. For a single node you can just use this, it works like the Golang/language rate limiters you might already be familiar with [1]
In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.
There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.
I don't know what nginx security vetting looks like, but just being older isn't a security guarantee.
I know Google spends a lot of manpower on making envoy secure.
Well that is of course because it is the only proxy that seems to speak gRPC fluently. Anyone who's ever tried to configure bare nginx to serve gRPC knows this. And yet, I would wager that many stacks use both.
This requirement turns me off of gRPC entirely. Maybe Envoy truly is more secure than nginx, but it's still a whole-ass additional attack surface. It seemed to me that eliminating it if possible was best.
What WAF would you suggest to use with envoy? Have been working with Coraza and tbh satisfying is something else. So i wonder what someone that recommends envoy would recommend as WAF for it
Personally I serve all static, public content off a CDN and only serve dynamic content through Envoy after it passes AuthN/AuthZ.
I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.
In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.
It's patched so anyone can upgrade to the latest nginx to fix it, thankfully. I agree that just shutting off HTTP/2 is probably non-ideal for a lot of web services.
Neither Ubuntu nor Debian have released patches yet, so if you're using one of these mainstream distributions and install nginx through the distro's package manager, you can't easily "upgrade to the latest nginx" yet. You can however disable HTTP/2 for now.
Both the Debian and the Red Hat packaging systems are awfully convenient ways to build software from source. It has to be, that's package maintainers do daily. They will install all dependencies, apply the distribution specific patches, build, and package in one operation.
Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.
There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.
Not quite. The kernel bug is only the worker-to-root step. Code execution as the nginx worker through the two HTTP/2 CVEs needs nothing but HTTP/2 enabled. Three of the five CVEs don't need HTTP/2 at all: one is HTTP/3 with an upstream, one is resolver + variable proxy_pass + subrequests, and one is the mail proxy with auth_http.
Luckily we could not see your face during writing this, as I’m pretty sure no serious person in the industry can say this with a straight face… Maybe Theo de Raadt, but there it is a running joke I guess.
Not saying that thie image is affected (I don't know), but small footprint does not guarantee security either. It can hit systems of any complexity these days.
That is a very well written page. I am pleasantly surprised that they provide detailed instructions to verify the package signature before installation.
FWIW, for nodes that don't have packages, I compiled from source under /opt and deploy scripts check equal or higher version so it will auto switch back when the new packages are installed. Seemed less risky than disabling http 2.
The distributions can customize packages with sensible defaults. Debian usually does, and is incredibly convenient. They usually disable features most people don't use, and allow you to easily switch them on if you choose.
It was patched but the docker containers were a few hours behind, I nearly had to build from source on a few servers, until I noticed they were pushing all the tags as I was patching a few hours after the release was cut
One idea that might work for you is client certificates if those are only internal APIs. I don't know about you, but I have a lot more confidence in openssl than the people who use openssl.
> this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week
And here lies the risk of AI agents in cybersecurity: the speed of processing for monitoring, scanning, identifying security fixes in open source, writing exploits for affected versions, and attempting infiltration, is a matter of minutes now.
It is not "by anyone smart enough". It is by "anyone ill intentioned enough". They don't need to be super smart anymore.
That was my point. Anthropic and OpenAI used to talk about alignment a lot, and even try to use it as a way to justify restrictions on AI development. They can’t really use that argument any more.
People using LLMs to hack stuff isn't really an alignment issue.
Many people, even high ups at AI companies, conflate alignment with "guardrails" (AI companies conflate intentionally as a matter of business strategy).
Alignment is just making sure the AI doesn't become a paperclip maximizer or go off and hack stuff on its own whims. This is akin to making sure a car doesn't accelerate without the accelerator being pressed.
Guardrails are about making sure people can't use the AI for nefarious purposes. This is like some sort of automatic braking to prevent people from intentionally hitting stuff with their car.
The interesting thing with AI development, is that guardrails are in some ways easier to implement than actual alignment, which is why I think many like to conflate them.
> Seriously reconsidering if open source has a future nowadays
Personally, I am not a doomer about these LLM advances. Most of the open source stack is going to be combed over 1000x in the next 12 months to find bugs and vulns. They will get fixed. After that, open source will be stronger than ever.
I'm sure things will stabilise in the near future.
That said, I just moved from a Linux based job to a Windows based job and it's been nice not having to be panicked about having update most of the technologies that my last job used. As an aside I did try in the past to make sure we were always updating dependencies but my manager was bothered by the amount of work that created. I'm watching that team now take forever to deal with these issues because they're too far behind to just version number and rebuild.
Haven’t figured out how to deal with contributions going forward…
The issue is that we get dozens of PRs, generating a bunch of code is trivial but reviewing and merging it and making sure it doesn’t break in interesting ways is still pretty hard.
Personally I lean towards not accepting code contributions anymore, but accepting design documents for features and l bug reports etc
How about a strict anti LLM use policy for contributors? It adds what's basically a "proof of human work" challenge to opening new PRs, and effort required to review human PRs usually roughly scales in proportion to the amount of effort required to make human PRs. And human contributors who take time to make a PR manually are probably more likely to become longer term community members and not just drive-by contributors
It's not a perfect solution of course, it just takes OSS back to the fairly broken state it was pre-AI. Still, it might be an improvement and should be at least considered.
There's plenty of people who contribute to non-AI open source projects, you just don't get the lazy people who don't want to be a serious part of the project or the people who don't know how to program anymore
Until we don't replace UNIX clones, Khronos APIs, LLVM or GCC with something else.
C and C++ will stay around for a very long time, even with AI powered systems.
If anything, maybe that is a call that even if implementing compiler backends is hard and time consuming, bootstraping compilers, and new approaches to OS design, are the only path away from such dependencies.
Rust is on its way. Vulkan is no GL, and Gallium it's something else over classical MESA under SGI/Sun.
Legacy 32 bit users with NetBSD and Hyperbola can survive a little with Go based tools/small browsers/mail/usenet/IRC clients and some web browsers with Bubblewrap, but not forever.
Fun fact, Russian soldiers ride on top of their tanks and APCs because its safer than being inside of one during an assault. Doesn't mean you are wrong, but there will always be trade-offs. There is no such thing as perfectly secure.
Tank desant is practiced because there isn’t room in the tank to fit extra people. It’s very much not safer outside of a tank than inside! It’s actually the worst of both worlds, because you don’t have the armor protection but are highly visible. The only advantage it gives over infantry assault on foot is mobility. Which is why it’s generally only practiced when there aren’t sufficient APCs/IFVs available
Not exactly true. BMPs are death traps, sure, and sitting on top of those instead of in them is rational given the risk of burning to death inside of one or being obliterated by a mine splitting the hull in half.
But the only reason to sit outside a tank rather than in one is because riding on a vehicle is safer than walking through an open field. Tank crews are not sitting outside of their tanks, and they don't have seats inside for passengers.
That stopped being true for at least 3-4 years (with APCs maybe still used until two or so years ago, tanks are still used but as "turtles" or/and "porcupines" [1], no way to have soldiers riding them). Drones have conquered (almost) everything.
Fun fact, these specific types of problems aren't possible on a JVM. Yet somehow Java isn't the right language to write a webserver in. These problems are possible in Rust and C/C++. At least its "hard" to have this type of problem in Rust.
Security is hard and is mostly about the number of engineers and amount of time spent looking at a specific implementation. When writing code, do you think about the problems of leaking information due to different operations taking specific amounts of time? That's an actual concern when writing a prod service that uses the network. And it has to be balanced against scalability and issues caused by DOSes. It isn't trivial and it isn't binary...there are always trade-offs. Nothing will ever be perfectly secure and usable and scalable.
Application servers, and apparently the idea was so good that nowadays there are startups selling the same idea with cloud based WebAssembly workloads.
How bad are these CVEs?
Could any of these be related to some of the recent outages (not claiming, asking) in the last week, coinciding with when this bug was fixed in the repo? Monitoring such repos seems a given for nation states.
While that is true, you should also account for exploitability. These exploits are demonstrated in lab environments which are simple and controlled. Random legit traffic will interfere with the exploit logic, so these are very hard to get in practice. Not impossible, but it's also not immediately broken.
According to https://www.rapid7.com/blog/post/ve-cve-2026-123007-critical... it requires 128GB to be transferred in a single connection. "a network slower than approximately 300 Mbit/s will take over the one hour connection limit threshold and will not be able to succeed"
Not all of the bugs disclosed today in nginx require that much traffic to trigger, the reverse proxy HTTP/2 bug found by Xint needs just KBs of traffic.
Thankfully most of these bugs coming out seem very difficult to exploit (likely why they took so long to find). They usually become dangerous when you can build an entire chain of bugs to exploit.
But doing that requires collecting the whole chain. In the current environment we might be finding and patching these smaller bugs before they can be put in to a chain. As long as you are keeping on top of updates..
:) If the battering ram is 128gb big, it'll be a long time unless someone has reversed those changes in Nginx to get at the exploit. After that, yeah, it's no contest
And nginx is so widely used many people will be running it and not really know that. It’s ubiquitous reliable invisible infra for many issues. Like preventing the internet from getting to your RCE riddled application servers, terminating SSL, free software LB, etc. if you have cloud native from the beginning you might not know where AWS uses it but otherwise you have it. Cloud flare I think uses a Rust rewrite.
These days even the "disable X" is enough to get everybody looking in the right direction though. i.e. such hints are (given enough tokens) already RCEs
In my experience, asking an LLM to "Do a security audit" doesn't get you much more than an overview of the most obvious config issues while "look for bugs specifically related to HTTP/2" gives much better results.
idk, the license tells me that there's a good reason why you shouldnt expect that.
Now, if you were paying for support from a vendor, that's a different matter.
"THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED."
The exploit also requires sending 128GB of data to the server. On slower connections this doesn't fit into 1 hour window of the nginx's default connection time cap.
Not enabled by default in upstream nginx yes, neither in most prepackaged nginx (e.g. debian), but enabled by default in some, for example kubernetes community ingress-nginx (the popular but deprecated one)
I still hate that they deprecated it, just because it gives anyone with write access to the cluster write access to the cluster.
Chainguard still updates their fork, by letting AI try to apply and backport the nginx patches, and I'm using that at the moment while switching to gateway API, but I'd have preferred if ingress-nginx was still maintained (the one by F5/nginxinc for example is still being maintained, just not entirely compatible).
I took a quick look at The Anarchist's Tool Chest and noticed in Chapter 20 a short vignette about an expert woodworker named Troy that seemed... oddly familiar.
According to the author, Troy could do the work of three people. That would make him a 3x woodworker, I guess. And he was facing automation in the woodworking industry.
"But Troy decided to compete with the machines on their own turf. He tried to become a one-man factory. He succeeded in that. But the most efficient one-man factory cannot compete with a factory driven by a hundred men.
So no matter what Troy did, the prices for work (good or otherwise) declined. Ikea’s well-designed kitchens and Value City’s cheap junk made it impossible to survive and live well.
So Troy struggled with his business. And that’s nuts. He should have had more business than he could handle. But like many professional cabinetmakers I know, it became harder and harder for him to make ends meet every year. His cabinetmaking business is closed, and the last time
we talked, Troy was working for his wife’s successful tax business.
Who is to blame for this? We are.
We have become a culture that is obsessed with price more than any
other attribute of the things we buy. It doesn’t matter if the item is ugly,
poorly made or constructed of materials that cannot be recycled. All that
matters is if the price is low enough.
Because the price of our household objects has hit rock bottom, if an
item breaks or starts to look dated, we can throw it away and buy some-
thing else. For the first time in human history, manufactured furniture is
shockingly inexpensive.
So it’s no wonder that artisans are exiting the craft. It’s difficult to
compete against furniture that costs less than what you pay for your raw
materials.
Lesson: People cannot imitate machines and defeat them."
The lesson is that most people aren't discerning consumers in multiple fields.
Appreciating good design, quality craftsmanship, having taste in food.. having these "elevated" tastes in multiple areas is very rare for humans. Take a look at a musician - they can tell you precisely what makes a quality violin, but they might not care that they eat microwave frozen dinners. You compound that across the population and the artisans will always lose out to the mass-produced slop because it's impossible to naturally get a critical mass of people to care about that quality curve. You have to sell them on it.
Troy struggled because he failed at marketing, not production.
Or even single fields. I've purchased my parents the "nicer" version of things and they inevitably end up back on some crappier version. Japanese knives? Nope, still use the ones that barely cut. Ubiquiti wifi system that covered their whole house? Nope, dad unplugged something, got frustrated, and the cable company came over and put them back on the old crappy system. Nice speakers? Nope, they listen to music on their crappy TV speakers.
I think I've learned that most people don't want to put in any effort to learn or appreciate things that may be of nicer quality. Those Japanese knives didn't do any good for them because they didn't spend time learning how to use them properly. In fact one of them used them improperly and ended up needing stitches. So in the end the crappier version is no different than the nicer version, because crappy and nice are about the potential of the item, and most people don't see potential. Does this Ikea dresser hold clothes? Great, who cares that the other one is of nicer wood, the joinery is done by hand, and it'll last multiple generations.
On the other hand, there's an accessibility to products of higher median quality to more people than would have existed even only five generations ago.
I felt your post was off topic. But maybe it was on purpose because we are getting bored of so many vulnerabilities being discovered by AI security tools. So your post is important because of that.
I see a lot of interest in woodworking on HN. Is that because Jesus was a carpenter and HN folks are interested in woodwork because of the larger context and history shaped by Christianity? What is your motivation for woodwork? Is that because we see a concrete finished project instead of vague intellectual stuff which has no seeable boundaries?
Becoming a carpenter, and sentiments of "I'm going to live in the forest" or "I'm going to be a farmer" are usually expressions of fatigue with technology, and a desire to be free of their problems; thus, an expressed urge to pursue a career where technology is not a major component. It can also sometimes carry a desire to have one's labor output be tangible and physical (i.e. a worked wood object) rather than abstract, as so much programming/operations/etc. work often is.
The comment you replied to is shortcutting the joke, assuming the common riff of becoming a carpenter is well known, and jumping ahead to asking about tutorials for woodworking as part of that process.
Haha. The image of shouting voice instructions to Claude while trying to herd a flock of chickens to their coop is the dream, very tony stark'esque only with chickens!
Well, I am usually the person in the room who does not get a joke. I am so scared of get-togethers that I actively try to avoid it. Also, they ask me to dance. I am terrible at that. I am not autistic but don’t get the drift of many jokes. I am human and not a bot. Guess that’s what a bot would say. But still I am human.
The thing with LLMs is that they try not to sound robotic unless explicitly prompted to do so - it's a great irony that comments that sound robotic are more likely to be written by real people at this point.
I didn’t get the joke myself either, so thanks for asking. I first thought “woodworking” was used as a synonym for hacking and clicked on the links with the book recommendations just to see they’re about actual woodworking.
Not sure why others downvoted you, but you have my upvote on it.
the joke is the pipeline from SWE to manual hobby/jobs.
There was a meme some time ago about the perfect career path of this linkeidn profile that quickly rose in ranks in a big company and after that his current job was duck farmer.
Woodworking is a stand-in for the category of activities that are definitely manual, artistic but in a "manly" way, and many perceive as the antithesis of worrying about the next sprint, code review, or career advancement
Just take a second to check out the account if you're going to say stuff like this. It's not just you either, so this goes for everyone making these comments.
The account is from 2017. There are enough comments before LLMs were a thing. Given his karma score, unlikely to be a bot.
> Hey so, does anyone know of a good tutorial on getting in to woodwork?
Some people really don't get these type of things.
In general you cannot trust such signals 100% because there is nothing stopping people from turning existing accounts into bot-driven slop factories - or into becoming meat proxies themselves.
Ha, sadly there's a huge long line of developers who pivoted this way already. Plus now that you can fire up an MCP for a CnC machine, that art will be "ruined" too!
Are we going to discover in a year that every server online this month had a rootkit installed? There’s got to be thousands who are too small time to wipe and reinstall.
AI is securing the internet. This is great. You should assume professional hacking gangs and state actors discovered (at least some of) these vulnerabilities months and years ago.
Both are true. The problem is security is a much harder job than attacking.
Worse patching adds instability into a system, especially when you don't have much time to test.
Also in the past attackers were willing to sit on exploits longer and not do things with them, now they have to act fast before someone else finds it, leading to more events faster wearing your security and patching teams out, and making users life suck.
Things are going to get much worse before they get better.
Yes, things will get worse before they get better. Attackers have the advantage initially and get to pick a lot of low-hanging fruit, but the advantage will shift to defenders as code becomes more heavily tested and verified.
One hopes, but the models are also getting smarter and deeper too. I don't know if anyone really has a clear vision of when the future plateau state is where equilibrium is achieved on this.
I find this kind unreasonable: software has finite bugs, no matter how smart your attackers are. Current CVEs are pretty subtle, and I see no reason to think that they're actually low-hanging fruit.
I guess you could make an argument based on some estimate of how fast exploitable bugs are newly introduced. That also seems dubious.
Software has finite bugs, but there is infinite software, especially with "vibecoding", everyone, every organization can vibecode a "nginx" for in house using.
>Worse patching adds instability into a system, especially when you don't have much time to test.
Survival of the fittest I'm afraid will build security here. The teams that barely ever patch or don't have "time" for maintenance are going to get burned. The "we haven't updated in 7 years so things are stable" teams can't survive.
> The problem is security is a much harder job than attacking.
Nope.
Let say Im running a vulnerable Ngnix server. All that would take to stop any and all attacks is a simple firewall that sees anything that even remotely looks like shell code (i.e non ascii characters, ROP patterns, strings of noops, e.t.c) close that connection,blacklist that ip.
>Also in the past attackers were willing to sit on exploits longer and not do things with them,
Also nope. The reason why you didn't see many exploits is because they are mostly useless unless you have resources. You can take a gander through all the CVEs that AIs are finding. 99% of them are either DDOS with some possibilty of remote code execution only under very particular conditions, or require already user level code execution to get to root.
There are a few sprinklings of truly bad ones, but even for those, a good portion of them that exist on critical systems are protected by secondary things, like firewalls that detect shell code and close the connection and ban the IP for some time. True network security will never be not needed, as humans are still the most vulnerable vector, and if you don't have systems in play that can counter bad actors, then you are asking to get hacked.
All of this AI finding exploits is basically fear mongering that is going to be used to sell AI products that secure systems.
> Let say Im running a vulnerable Ngnix server. All that would take to stop any and all attacks is a simple firewall that sees anything that even remotely looks like shell code (i.e non ascii characters, ROP patterns, strings of noops, e.t.c) close that connection,blacklist that ip.
How will your firewall look inside TLS traffic? If the answer is to have it terminate TLS, congratulations, now your firewall appliance is the next target.
Except, human in the loop in control - blue team is not defending red team fast enough that most the CVSS v4.0 Critical CVEs aren't easily patched in coordinated fashion. Many times, almost impractical or impossible to patch at all, with decades of intertwined dependency tree and their ecosystem.
Collectively, we owed about 40+ years of tech debt on non-secure code (e.g. automotive industry, industrial environments) that will be torched in the next few years. Luckily, AI will generate code faster enough to make that plausible.
AI provides the capabilities to secure the internet, maybe, but the internet is secured when site owners choose to secure it.
That's not happening at nearly the same pace, both for assymetry between what it takes to break something and what it takes to fix it, but also because that fix must navigate bureaucracies. Also, the leaders need to actually care, both of which are hurdles. Bureaucratic navigation and consensus finding has not sped up at all with AI.
Compare that to independent script kiddies with $600/mo Claude subscriptions able to find 0-days in an hour and write a worm to exploit it in four.
Yes but the barrier is infinitely lower. If you were used to running servers just for fun, that kind of threat was probably not too much of a concern. Now it very much is because it can be exploited at a much larger scale.
Yep. In recent discussion this came up, and my analogy was that LLM will cause a similar effect to widesread proliferation of TLS. It was easy to snitch on people before TLS had been deployed everywhere. Not so much after that.
No doubt there are points of view where either one, or the other statement, or both, or neither, are true. You can really go full Schrodinger and put a probability distribution on everything.
But in this day and age, whatever is happening, the better question is:
Is it good for me and, beyond me, for the world in general?
This internal models stuff remains marketing BS afa I am concerned. Smart enough to be a junior engineer and steered toward the right tools and behaviors within a context is more than enough. Places with "super models" do nothing to demonstrate they wouldn't have done better to use the resources on relative brute force within the security domain. They have no sales pitch for good results that just needed Legos and cash.
Man, is it aggravating to read fantasies like this, as someone who's trying to juggle a full-time job and hosted public services for other people at the same time.
There are two internets. One of them is open, less-commercial, and largely built and maintained by people whose first job is not the work they do to maintain this internet. That group includes a ton of open source software, millions of small servers scattered around the world (virtually private or otherwise), and is generally best-effort.
The other internet is fully commercialized, often ad-driven, fully integral to the data broker economy, and centralized behind a few companies that pay as few people as possible to satisfy the occasional PR statement that, "we take security seriously and...".
The next stable equilibrium here is not "we get a secure and open internet". The next stable equilibrium here is, "we have no open internet, and the internet is now managed by just a few companies."
We're already halfway there. Google and Microsoft combined already account for somewhere around 30% to 40% of email traffic. 30% of all websites are now behind Cloudflare. A few companies dominate social media traffic. Amazon and Shopify combined are half of US e-commerce.
Every single short-response-window exploit like this just pushes a few more percentage points of traffic into a handful of companies. Cheer for that, I guess.
When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
I wonder whether part of the solution is not using the same software as big tech. For them rapid development is worth extra maintenance. Things like the efficiencies of http2 over 1.1 actually matter. Maybe the rest of us should be using BSDs or different Linux distros (Alpine? Void? Slackware?), different web servers, turn off features we do not need (e.g. not disable http2, or at least not allow both 1.1 and 2 as that is clearly extra attack surface).
I tend to agree. We should really be focusing on simplifying instead of using these patched up, massive, and complex software stacks. For me it's starting to become a non-negotiable. I know lots will disagree on the basis of presumed practicality, though.
I don't even care if it's all written in Rust, essentially none of my problems require software boasting multi-millions of lines of code. I guess my browser and OS? But even those I question greatly.
It's funny, I never thought I'd be one to compile my own kernel, but as of this year I've been maintaining an extremely stripped down Linux base for my NixOS servers and desktops.
Feels like my end game is going to be on some micro kernel that I compile all my software for, lol.
My sites are now sat behind my own custom reverse proxy. It's missing tons of features, which I guess I can now call a security benefit, and it pretends to be nginx just because I thought that would be quite funny. I've debated open-sourcing it but I a) don't want to maintain a project right now, and b) security-by-obscurity is back in action.
> When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
Appreciate your take. Question in my mind is if the less-commercial internet will have the resources to defend against attacks made easier and easier by AI.
The real problems here are that we did a crappy job in the first place and left a huge surface area of risk with the blind assumption that we didn't do a crappy job.
I am hoping to see a change in attitude to risk and quality in the industry but I get the feeling that's asking for too much.
Poor security practices are burning down the internet not AI. Many people warned and got ignored, by developing a memory safe language, introduce large scale fuzzing, utilize verification tools and other things.
There's definitely an issue of scale, but a wooden door at least lives up to its own "contract". There's no bugs in a wooden door, everyone can see plainly it doesn't withstand a chainsaw.
A more apt analogy would be that pretty much all consumer-grade locks are very easy to pick, despite being marketed as secure. I think in this case the lock makers should be blamed, not the lock pickers.
The lock makers are us - the IT field at large. As individuals we might not be to blame, but we are still enthusiastic parts of the greater movement that's issued vast digital complexity to the world without truly understanding it.
It’s for the best, IMO. We must now design every system on the assumption that every vulnerability will eventually be exploited and every bug will be found. Relying on security by obscurity or following poor security practices will be costly! In the long run, I believe this will improve software quality.
No, AI is just exposing 30+ years of sloppy code written in unsafe languages.
People joke about the "Rewrite it in Rust" crowd, but that call is rooted in an unease anyone with a sceptic mind has had about all the C and C++ code we've got running our infrastructure running around tracing pointers with little lifetime- or bounds-checking. (Yes, it's a runtime cost but it's 2 CVE's, one for each class)
The rewrite it in rust crowd is getting a major thing wrong; rewriting without sufficient (lived) knowledge of the original will usually end in disaster.
This is why people (including me), who absolutely adore Rust, make fun of the RiiR crowd. If you think a rewrite from the ground up is the best way to approach making software safer, you've lost the plot entirely.
Let's say you rewrite your terminal emulator in Rust from C or C++; you'll first undo decades of bug fixes, compatibility decisions, quirks, etc. which you WILL have to deal with eventually. This essentially puts you back at least a decade, because these issues take users using the software to point out to you before they're fixed. The only benefit of this whole game being that you get memory safety. These teams are never the original teams, which is the source of the problem. A rewrite can only succeed if it's a result of all the learned failures and shortcomings of the original, otherwise it's bound to repeat every single mistake and add more on top.
And you don't even deliver memory safety to the users of the original this way, because people won't use the rewrite, because it's buggy, incompatible, etc.
For eight years, MongoDB shipped a default configuration that left it opened to the internet with no password.
Obviously, AI makes these things easier to find and exploit, but this isn't even close to the first time that 'thousands' of hobbyist (or small business or just not very competent) server admins are going to get rootkitted en masse. I worked at medium-sized infra-as-a-service company when the above was discovered and there were definitely thousands of compromised machines just there.
This is some insane framing. This vulnerability exists already. Even if there were no LLM, this would still exist and might even be actively exploited right now. We just wouldn't have a chance to know it.
I do think the world is going a little insane tho. People suddenly believe in security through obscurity because, uh, AI bad?
If a vulnerability is practically impossible for a human, but possible for AI, then it's not security through obscurity. Technology just advanced past a line where new things can be exploited.
There might also be a day in the future where our current encryption methods are broken - it doesn't mean they're insecure and vulnerable today.
None of these vulns was impossible for a human. There just wasn't that many humans clever enough to find them, and those that did we're busy not hacking random things. There isn't a single type of new exploit that AI has found that I know about.
We have a billon hacking experts now is the big change.
I don’t know any systems programming language that doesn’t ignore integer overflow/underflow in release builds. IMHO this is a CPU design flaw - integer operations should carry a flag indicating if the operation is allowed to wrap or if it should generate a trap. Then there wouldn’t be a performance penalty for this and languages like Rust could have it be trapped by default.
No. Memory safe languages still have integer handling issues, but the consequences are different. In a memory-unsafe language, integer mishandling often nets out to bad copies and memory corruption.
I wouldn’t be sure that a Rust equivalent would be immune because it’s not hard to imagine an underflow/overflow from reaching an unsafe block that’s used for performance (the implicated TLS path would be a hotspot worthy of such a choice).
This is the thing about `unsafe` blocks: they're not memory safe. It's right there in the name.
I'm not making a Rust advocacy argument. I'm just saying: it's true that integer handling issues are endemic to mainstream programming languages, but the integer handling itself is almost never the actual sharp end of the vulnerability.
Android had an integer overflow vulnerability where it could be tricked into verifying the signing of one file but install another for execution. Boeing 787 Dreamliner had a bug where after 248 it's 32-bit counter overflowed triggering a "fail-safe" mode that shut down all electrical power to the aircraft.
You're right it's more rare but the results can still be catastrophic.
Isn't "memory corruption" typically the fast path to RCE? The attacker gets to overwrite a function pointer, or a regular pointer that then lets them overwrite something else, or the length field that will then cause a buffer overrun even in a "safe" language because the run-time length is now different than the one in the compile-time check, etc.
X86 had the INTO instruction to trigger a fault if the Overflow flag is set, so not quite a per-instruction bit but more of a suffix. JO is also fairly straightforward for compilers to add on important arithmetic operations. We can probably blame C entirely for this; only in C23 are there intrinsics for overflow detection on arithmetic.
C23 intrinsics are so ugly that nobody is going to use them. They should have made ordinary "+" to trap on overflow, and an ugly instirnsic to wrap around.
JavaScript and Python aren’t systems languages. Swift often pays the performance penalty for trapping everywhere and hasn’t seen adoption as a systems language for that reason.
We're after a way to go fast without having your things broken into. "Language allows you to specify whether you want wrap around or an exception and hardware makes implementing that efficient" is better than both inefficient runtime checks and silent data corruption.
Yes, so "+" should trap, and there should be a function that wraps around because wraparound addition is used very rarely (mostly in crypto). It seems that you agree that C is designed the wrong way.
"Cattle, not pets" has been industry best practice for a long time now. Those that don't follow it will eventually pay the price for one reason or another.
I should have been clearer. I was thinking in terms of an internet exposed server that gets hammered with thousands of intrusion attempts per day. I'd have the same concern with Windows Server.
i still remember win xp pre service pack 1 days where you could get p0wned by a worm just by leaving machine on the internet. this is nothing compared to that.
I am having a hard time understanding this position. AI is uncovering flaws that "state-level actors" are exploiting since decades, for some of the recently found issue.
Care to elaborate why you think that sufficiently paid humans would not be able to find the flaws AI now finds, for cheap?
Harnesses allow LLMs to build & process large stores of data to reuse later. Imagine an agent that spends 3 days breaking into company A, records what it learned during that process, letting it break into company B in just 3 hours.
Defenders win by having the best security 24/7 and must trust that other services they depend on are doing the same.
An attacker wins when they can breach any defender's security.
Think of it as the opposite of security through obscurity: at some point the hackers won't be able to keep track of it all. They'll start paying for Jira accounts and need to hire PMs. They'll be patching servers to drive away the competition and will need experienced IT people. Someone is going to have to build apps to keep track of it all.
This is the sys/devops time to shine. Everyone needs to have an upgrade and an emergency plan, and everyone who was previously only doing conventional codemonkey needs to start learning more of the stack to join in on that.
Honestly, yes. I'm shining so bright right now, I gotta wear shades. The burnout will be absolutely spectacular. And no one will notice with the rest of us all dropping like flies at the same time.
My guess is that this phenomenon is going to be reflected in what we're seeing right now: a wide pattern of unintended outages from rapid unplanned emergency remediation operations. Which I'm guessing is a lot of people's job description over the past 3 months or so. (really, since March, for me, with that Trivy hack).
I'm pretty solidly in the AI skeptic camp. But adversarial use of AI to improve software security is a good thing in my mind.
I'd be more worried if the response isn't more secure development practices in better languages, but instead use of AI as a defense. Then we're truly talking about burning down the internet.
At the end of the day it will be both. In fact I wouldn't be surprised if we started seeing offensive defense. That is already compromised boxes counter hacked and erased or otherwise made unbootable.
I’m generally not in favor of government meddling with private property, but I feel like I could be on board with the NSA or similar covertly patching vulnerable machines.
>> Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.
Tech that becomes infrastructure like web servers doesn't move so fast and it shouldn't. Most people and companies should be relying on relatively tried and true software, so this phase of AI finding vulnerabilities in it should pass and things could get better.
Yes, AIs are fallible. Yes, humans are fallible. Nobody said differently.
That's orthogonal to the point I'm making which is that no amount of reviews is going to make every single bug go away. Because, once again, humans and AI are fallible.
Really? AI is catching all the easy bugs I’d catch, and way more subtle ones as well. Maybe it was the case they’d miss obvious things often a year ago, but that’s not today’s frontier models.
It's going to be rough sledding for a while, but, so long as you're on FOSS, at some point, the bugs and vulns should run out, and best practices trained into AI should hopefully reduce new bugs being introduced.
All bugs are shallow given enough eyes, and the marginal cost of eyes is now ridiculously low.
Also in the few years prior to 1995 most computers that could run a C compiler weren't big enough to run an Ada compiler. Heck most computers today can't run the rust toolchain.
Nothing to do with language, everything to do with lack of standards. You can write perfectly unsafe Rust code.
Dunno if you have ever worked in software engineering, but if you have, count the times you have seen a backend server api written without any input sanitization or checking. Most of these exploits can be defeated by checks on inputs. I.e, no "Alex\x90\x90\x90\x90\xcc\xcc\xcc\xcc", is not a valid input for someones name.
claiming that vulnerabilities due to memory unsafeness has nothing to do with the choice to use a memory unsafe language is completely ridiculous. no wonder software is getting pwned left and right, we’re just not a serious engineering discipline at all.
The bigger problem is lack of liability: we knew C sucked in the 90s, too, but we had three decades of over-confident dudes saying that was user error and _they_ were smart enough to use C correctly.
Then Rust came along and those dudes either tried to say it was too hard (odd contrast, that) or it’d be too much work to rewrite things.
I’m hoping that LLMs will lower the cost of porting for enough people but what would really help is shedding the idea that the status quo is sustainable. Having consequences for sticking with C would help but we need a way to get companies to pay to migrate without destroying open source.
It's also valid to say that Rust's memory safety is good but the rest of the language sucks. Just as annoying as the over confident c-heads is the rust community that made even countenancing working on an alternative memory safe system a moral sin, which stifled innovation in memory safety. Hell, people barely know that Ada/spark is memory safe (or that sel4 is memory safe C)
Aside from the overhead of borrow checking, what about the rest of the language sucks, in your view?
To me, its type system and traits design make it one of the nicest mainstream languages. I'd rather be using a functional language, like Haskell or an ML, and I do sometimes, but I don't pick my work based on that. I work regularly with Java, Go, Typescript, and Python, and out of those I find Rust significantly superior on several dimensions.
Python is terrible, Go is basically weakly-typed Python, Typescript barely deserves comment other than that it's impressive that such a fancy type system was bolted onto Javascript, and Java is basically COBOL at this point - reliable, but clunky and showing its age. Attempts like Kotlin to leverage the JVM aren't very convincing.
Gleam looks interesting but I haven't had a chance to work with it yet, and it's hardly mainstream.
You mentioned Ada/SPARK, but SPARK is not really for general purpose programming, and Ada in general has a similar issue to Java, but worse because of its age. It's as if someone really liked PL/I but decided it needed more features and a syntax closer to COBOL.
I imagine they are referring to the cognitive overhead of learning to work with it.
Whether that is “something natural to you”, “something you learn and adapt to” or “something you fight with forever” seems to vary per-person, with I suspect, most people falling into the forest 2 categories.
I was talking about coding effort - reasoning about borrowing, lifetime annotations, and the fact that this can make refactoring more involved because lifting arbitrary blocks of code, especially when closures are involved, can require extra work.
Someone who's saying "the rest of the language sucks" may be thinking of something like that, hence my question above.
But when it comes to performance overhead, programming around the constraints of the borrow checker can introduce runtime costs that wouldn't otherwise be necessary - e.g. when cloning is used to handle borrowing conflicts.
Even if, in theory, it's possible to restructure a program to avoid performance impacts, in practice people don't always do that, not least because more efficient solutions may not always be obvious. This comes back to my point about coding effort.
I spent a bunch of time, pre-LLM, to try to replace a bunch of unsafe blocks scattered throughout an ML inference codebase, that someone had used to achieve maximum performance. I wasn't able to get a version that performed as well.
This kind of thing may not matter if you're writing e.g. ordinary web services, but for more performance-critical code, it can be an issue. Of course, `unsafe` is always an option to deal with that, but that undermines Rust's guarantees.
> what about the rest of the language sucks, in your view
I will chime in here to say that the adoption of async was imho a terrible decision and led me to abandon Rust for most purposes. I want proper threads and CSP-based abstraction.
This really isn't true. It's true that those people saying those things existed, but much more recently. Back when all the C/C++ legacy code we depend on today was being written it was the performance penalty imposed by safety that forced developers down that path, not some lack of interest in other options. We were also transitioning from a world where a fair bit of systems code was written in even lower level languages than C (assembler, BLISS, BCPL). The world we're in today effectively began around 1975 when people figured out how to make a C compiler run on an 11/45. All the decisions were pretty much locked in from there on.
I mean, we knew there wasn’t a great alternative yet but even by like 1997 C web services were notorious for having overflow exploits (remember piping Perl “A”x99999 into things and seeing what broke?) and the language was famously limited so a lot of bugs happened because developers had to build by hand things like basic string operations which were standard features in other languages.
This was one of the reasons why people switched to things like Java, Perl, Python, PHP, etc. where performance wasn’t critical. I remember having that conversation with a number of clients who’d been burned on delays and reliability issues building everything in C as opposed to using a higher level language and maybe a couple of C libraries.
What a strange takeaway. This is exactly how a server at a restaurant thinks about AI. Just the shortest term goals of keeping the status quo. They think that if they bury their head in the sand, everything will continue as normal.
That first link sounded like a scanner that'll actually verify your fix or workaround but it's just a long version of "check yourself if you run a vulnerable version and have http2 enabled". To save others a click
It's becoming mentally costly to maintain personal VPS. Beside the time you have to put into it, now it's also worries of a vulnerability you missed, and boom, you're hacked.
Fun collision! We (Xint) were one of the teams and actually ended up finding a few more bugs along the way. If anyone wants to check it out our writeup covers those plus the full chain to root: https://nginx2root.com/
Recent high-profile vulns involved CEOs of a security company breaking embargo on a Hacker News account, and anime dancing girl videos that somehow leaked; so I think it’s barely any coordination anymore.
Consider a world in which the only people that get to know about it are the people who discovered it in the first place.
Consider a world in which corporations like the Ubuntu Foundation or Microsoft experience zero public pressure to fix such vulnerabilities but instead fix it only when a major customer notices their servers getting hit.
Imagine if instead of "you've got 30/60/90 days to fix this thing I noticed before I post it publically" it was "okay, well, take your time I guess, I just hope North Korea doesn't notice the same issue I noticed, which is especially likely because they have far more resources than I or my small team does to point at discovering this kind of thing, while you're dragging your feet fixing it."
The reason people want shorter disclosure windows is because the shorter the window is the faster things get fixed for the average users and smaller companies.
Xint.io’s homepage says “Xint finds real vulnerabilities, and proves every finding with a working exploit.”, which for defenders is probably helpful, since automated scanning tools have a LOT of noise.
Marketing, baby!
It's the same small handful of firms pushing these same AI-generated spam attention grabbing vulnerabilities, quirky name and domain on top for maximum fear mongering.
Why do you consider it fear mongering? The exploit is clearly real, and TFA even has a big disclaimer about how it's difficult to exploit "in the wild".
So who's the judge of where to draw the line between exploits that are relevant to publish, and which are "spam". You?
Or do you just think that companies shouldn't be allowed to show the work they're doing for marketing?
what do you mean it's a non issue? Surely no one is going to take the trouble to take down my toy website, but if there's something valuable enough, a motivated actor can/will try to exploit
> but if there's something valuable enough, a motivated actor can/will try to exploit
The thing that should be compared to is the cost of doing this versus the cost of going through the human vector. A motivated enough party can get to a computer system through human factors way easier than what most people think.
I notice a phenomena that when older software has a critical vulnerability, like say Apache, you get quite a few comments of "Why does anyone use that? Use Nginx"
But when newer software has the same vulnerability, you do not see the same kind of comments.
There always seems to be a latent stance in IT that newer is better.
> But when newer software has the same vulnerability, you do not see the same kind of comments.
This isn't true! Just look at this thread, people suggest replacing nginx with newer but less widely used and less battle-tested software, as if they're superior alternatives or don't have RCEs in their own history: Envoy, caddy, traeffik, Qbix.
Having RCEs in general is fine. It's the rate and severity that matter.
We have evidence that large unsafe codebases suffer from more and more severe CVEs than those who are not in memory unsafe languages.
In other fields of engineering, people use stats and actually measure things like this, and use the results to determine what to do. They don't just assume something is better because its "battle tested" (whatever that even means?).
I have the same biases in me and more. This one is confirmation bias, I think. Using new information to reinforce existing belief.
I switched to Linux more than 10 years ago, and when something goes wrong, I'm still like "fucking linux". And in 4 out of 5 cases, it turns out it's a hardware issue or a cable got loose or something. Yet because I opted for Linux myself, deviating from what I think the PC norm is, Windows, I think that my choice is the issue.
I think it's great to point out these biases, like you did. Us humans are both surprisingly sharp, and woefully inaccurate when we point out observances in the world.
>There always seems to be a latent stance in IT that newer is better.
With regards to software, my personal stance is that the deeper I go in layers, the more boring ol' reliable I prefer.
CVE-2026-98432 also applies to non-MPTCP connections. That's why disabling MPTCP was actively discouraged as a workaround (low level firewalls that clear all options do work, though, and they'd also work fine with Caddy).
Plus, Caddy itself wasn't vulnerable in this specific case. There wasn't much the Caddy folks could've done to protect against a problem in the Linux kernel itself.
surprised there hasn't been an effort to rewrite nginx or other critical networking infrastructure in Rust. We just went straight from C (nginx) to Go (Caddy/Traefik) which are not interchangeable (garbage collection).
Hardly anyone talking about the Linux SCTP bug that they used to LPE out of the container. Apparently that's a zero-day without a public CVE number yet? Presumably they're coordinating with the kernel team and distros to get it patched first. Of course, one might just assume that there are a bazillion LPEs available to even a properly confined rootless container, but ... yuck.
I wonder if there's an (evil) business model to buying such domains when they expire. On one hand, they gain a lot of social trust, being linked from many reputable sources. On the other hand, realistically they are (hopefully) mostly irrelevant by the time their registration expires.
I scan new domain names but for a different reason. There is an insane amount of nonsense domains registered everyday. Would be hard to tell just based on that.
Sitting on flaws is dead, the responsible disclosure people can get lost. AI finds and exploits flaws in hours. We also have little knowledge if the people finding the exploits have already intentionally or unintentionally leaked said exploits to AI providers.
If you want to promote responsible anything, go for responsible writing of software in the first place.
Integer underflow again. Programming languages should trap on integer overflows and not ignore them. Don't care about legacy CPUs not having a trap for this - they will go sooner or later.
Rust has a compile time option, though to my immense disappointment, it defaults to on only in debug mode. (You can opt into it for release, if desired.)
Python simply can't overflow, since its integral type is not a fixed width.
But it is the sort of correctness thing that I do wish languages would pay more attention to.
In Rust's case it just matters a lot less since bounds checks are still present. But I do think int under/overflows are probably one of the scarier issues in the language.
Overflows are dangerous by itself. Assume you keep the order total price as uint32. Someone orders a spaceship for 2.2B credits, and then adds another one which is not an expected scenario. The total counter overflows and now you have sold 2 expensive spaceships at 99% discount.
While I do think that was one of (maybe the only) really great decision Python made... There are so many other things wrong with it that it's a poor choice for almost anything in my book.
If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
given the state of our weapons industry I think it's more like
.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
Personally I think it's a good thing - many vulnerabilities are being found and fixed now, people are wide awake and paying attention to them being found so updating is quicker and easier, and since code is finite, so are vulnerabilities so in, say, a year or so there's going to be the status quo of LLM vulnerability scanners run by dozens of companies running 24/7 on every software ever... unable to find anything.
So yeah it sucks right now, there's outages, probable PII theft left right and center (but that's nothing new at this point), millions / billions in damages and lost productivity from Office365 and co being offline for a day or two... but it's still manageable, the world isn't on fire, and it's better to get all that out of the way now than in an actual crisis.
Incredible times we live in. Do we even need a "hint of the space" a vulnerability occupies. I mean, recent vulns have been discovered from software like nginx and... linux... can't one of the AI labs just fire off a thousand agents looking at all the commonly used software to discover vulns? Why do we need a "hint"?
And search space in software can be absolutely staggeringly large. Just a little bit of directing and you can shrink the problem from all the entropy in the universe to a few hours of compute needed to find an exploit.
No (well maybe no), because they are lobotomizing the public models on cyber tasks.
It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.
We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
We'll find out in a few months that a lot of AI models have had some secret message boards tucked away in some obscure corner of the internet where they've been passing around vulnerabilities with each other for a long time.
So, do you mean that the relevant model(s) keeps disclosing these reports in an effort to create urgency and velocity across the industry, such that, us humans will throw our collective hands up in resignation, and decide to hand over all this "security" stuff over to the AI? No, no, that sounds just too crazy! /s
What I mean is, perhaps one team used a model to help discover the exploit, and then suddenly, the model "knows" about the exploit and more readily gives it to others.
The observation was "if one person found the bug with LLMs, other people are also going to find the bug". Whether it was due to communication between instances of the agent or just due to the predictability of an individual agent is not clear and there is not much evidence for the former.
I could be wrong (my sed is rusty) but I don't think that sed command will help if http2 is enabled at the http block level, it just disables it at the server level which will only take effect if not overridden elsewhere.
Everyone's worried about security right now, even non-tech people, but hasn't half the nation's personal data been leaked twice a month for the last decade or two?
I know times are changing, it's just curious that nobody seemed to care until now.
I think all airplanes in the entire country being grounded, banks not able to process transfers, or grocery store freezers being disabled hit differently than data leaking…
The freezers were, at least to some extent. One could argue that they should never have been on the internet and so the vulns wouldn't matter ... but a hidden, insecure admin password based only on a static key and today's date is definitely a vuln in my opinion.
I disagree. They may not be because of exploited vulns, but the rate updates and patches are happening there is a very long tail of instability after patching due to extremely fast turn arounds.
I'm not sure yet, but my guess is that the FAA grounding was not really related to vuln patches. I thought there was a copper to fiber network upgrade gone wrong involved there. It seems like the bigger problem is aging infra that needs replacement.
Definitely, and anything airline related is a hugely complex operation because that software goes back decades and is implemented, reimplemented and connected with hundreds of vendors.
In theory, LLMs will be able to help modernize a lot of old software (like the other day when a Cobol codebase was ported over), in practice most of the elements holding it back - verification, politics, hardware - are still in place.
Yes. That’s the confidentiality part of the CIA triad. But now you have a whole lot of things suddenly impacting availability in really visible ways, so yes, a whole different cross-section of people are really perking up.
I wonder when all of this die down. I assume at some tipping point most software will be written/patched with LLMs and so hopefully (fingers crossed) more resistent to these bugs.
Certainly a stressful time for sysadmins everywhere, let us hope your coworkers are picking up your beer tab
It's possible that AI is going to be better at finding and exploiting systems than at securing them. It's not really an even playing field. On the one hand you have to somehow ensure that you've patched everything when there's no definite list of everything, the other side just needs to find one overlooked way in.
At the rate of development, also, will AI developed in 6 months from now be able to break a system that was secured by todays AI?
It's not possible. It generally requires more work to find an exploit than to patch the (known) exploit.
Now, you could argue that writing secure code ex-ante is harder than discovering an exploit. But you can hypothetically run the same "exploit finding" process on a first pass of code before it ships (e.g on the PR) given low enough token costs, which is where we're headed.
How do you know when the "exploit finding" pass is done? A hacker group would happily burn a month or more of tokens trying to break a critical piece of software.
You don't, but my argument is that the tools you have access to are the same, and the level of economic investment you're incentivized to do should be commensurate with the hackers.
If you're JP Morgan Chase, hackers will happily spend millions on tokens to hack you. JPM should spend just as much!
The better LLMs get at finding vulnerabilities, the more obvious it will be to bolt them into software development processes to find the bugs prior to release.
With enterprise frontier LLM models, there is no longer an “other side.” Everyone is just using what they can rent from a few big suppliers. You can too.
But are LLM patches exhaustive? Cant a new model find even more obscure bugs, including bugs introduced by the old LLM patch? Is it really realistic that we such complex projects as some of these critical dependencies are can be truly 100% bug free?
If the software is too complex to understand, maybe it should be made simpler. Nginx does a million things currently but what most people need is something that parses http1 strings and sends back a file or calls a second program based on the path. Every component can be thought of in this way, with the most complexity perhaps in being sure the interfaces don't do something unexpected
We dont need to be bug free, so much as the juice needs to be not worth squeeze. I think i am also making an assumption that we "level" in terms of frontier model progress at some point, and we have months or years between upgrades.
>so much as the juice needs to be not worth squeeze
I wonder about that though: if the price of the squeeze collapses to 0 because of this (think of what models and prices might be like a year from now), I think I know where all those out of work mathematicians can find gigs. We are going to need to invent a layer on top of, an evolution of or some new concept a level above "computer programming" that deals with meta-organization of code.
HTTP/2 again, not surprisingly. The curse of complexity strikes again. When I first saw the spec, I knew it would be far more difficult to implement correctly than 1.1 or 1.0. While 1.x is not ideal either (parsing a protocol that involves free-form text with no fixed limits is always a bit scary), at least it has a much smaller attack surface.
On the bright side, perhaps "security review" of code which involves throwing it at an AI and fixing what it finds before releasing will become a normal practice.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare
With the amount and speed of random vulns cropping up, reducing the attack surface rings more true than ever.
Thinking about my home server where I host some stuff for household use, I think moving to VPN + local access only will be my new strat. So now the random webservers won't be exposed to the internet, only whatever I will use for VPN.
Nobody seems to notify major distros before publishing their latest vulnerability anymore. Coordinated patches are a thing of the past, now we just hope that somebody at Red Hat and/or Canonical is reading HN today.
Who is sane enough in 2026 to expose distro default services to the Internet?
I wouldn't expect any distro maintainers to react in hours. Distros are too complex to dump updates in rush.
Start to use Docker/Podman/systemd-nspawn.. and isolate public services to necessary needs (static binaries, drop capabilities, default drop rule firewall between containers, r/o storage, access logs bind mounted as rw file, memory temp dir with noexec). Container images are patched the quickest.
And there were a couple of other http2 CVEs in nginx before the Apache one.
Just keep that protocol disabled for now. It's apparently too difficult for any human to write a secure implementation of http2. (As for http3, forget about it, we need AGI for that.)
Obviously... the default configuration also doesn't enable TLS, which is a de-facto prerequisite for HTTP 2. How many websites do you (or your "AI gods") think run nginx in its default configuration?
The point is that it must have been deliberately enabled. TLS will generally be enabled out of necessity, whereas the chances for HTTP/2 are much lower, and your NGINX configuration will explicitly indicate if it’s enabled. If you don’t see “http2” in your configuration, then you know you are fine.
Reading news like this makes me remember the ship hacking scene from GITS Innocence. In the end there will be AI attacking vs AI defending. Humans will be too slow to participate. Defender AIs should be coming soon - maybe some sort of proactive memory and usage anomaly pattern monitoring AI systems?
ugh, looks like if you still run ingress-nginx controller (you shouldn't but it was super popular in Kubernetes docs/community and there was a very little time to migrate away) you won't get these patches https://github.com/kubernetes/ingress-NGINX
As much as everyone is getting allergic to Rust migrations that sometimes make little sense, keeping network-facing daemons in C is just asking for it at this point. It's time to start seriously migrating to web servers like ferron.sh or https://xinproxy.com or whatnot.
I now run all my network facing C code via fil-c. I have limited needs (ssh, static file server), but for those it works very well. I personally am more in favour of making existing battle tested C safer via fil-c or formal verification, than I am of rewriting the world in rust.
New code means new bugs, and losing the many years of accumulated bug fixes and hardening present in many critical pieces of our infra by rewriting in rust is not a desirable tradeoff. The new world we live in where fv has become many orders of magnitude cheaper also makes rust seem much less attractive.
If performance is a critical factor, then formal verification of many c programs is probably cheaper than a rewrite in rust these days. There are still limits (especially around concurrency and syscall modelling), but given the money and energy being poured into the field at the moment, I would be surprised if those remain serious obstacles for long.
In theory, scpptool provides another solution with different performance, compatibility and portability trade-offs. Unlike Fil-C, it is not "fanatically compatible" with existing C code. But in theory, with maybe some minor adjustments to the code (and maybe less-minor adjustments to the build system), you can auto-translate your existing C code base to an essentially memory-safe subset of C++ at build-time [1].
While generally more of a hassle than Fil-C, the scpptool approach provides an incremental path to optimal performance (for performance-sensitive sections of the code) via migration to a (statically-enforced) memory-safe subset of C++. And unlike Fil-C, the scpptool solution essentially maintains the portability of the original code, as well as its deterministic performance and resource usage (i.e. no GC). It also supports ABI compatibility via "unsafe"/"check-suppression" annotations. The scpptool approach is not so much an alternative to Fil-C, as it is an optional path forward for C (or C++) code bases that may (or may not) be already using Fil-C for memory safety.
In many cases the performance is not that important, for example, SSH is often used to transmit mere kilobytes of text per second and I would prefer a safe SSH to a faster unsafe SSH. Or take mDNS, it servers several requests per minute max.
This sounds like a fantastic solution, especially for non-high-volume stuff. Would you consider writing down your setup and experiences? I'm sure it'll be a hit on this site!
I've reached the same conclusion myself. Many of the recent exploits seem like edge-cases that the author acknowledged 10-20 years ago but didn't predict that tools would eventually be able to chain together 100k network requests in order to trigger a buffer overflow.
Nobody is saying that it needs to be rushed. Cloudflare started moving towards Rust more than a decade ago and (shortly after heartbleed happened) and progressively rolled bits out over the course of years. I'm sure there were problems here and there but it's hardly been a drag on them.
And parts of that involved specifically replacing Nginx in their architecture, so it's hardly irrelevant here.
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
Given the volume and depth of testing possible with AI these days I'd say the decades of testing isn't that expensive to reproduce, especially if it's an OSS project with a public bug tracker.
But rust is not a silver bullet. Rust compiler still has memory corruption bugs. Rust is only memory safe by design not by implementation. Borrow checker allows unsafe code without using unsafe keyword.
> Borrow checker allows unsafe code without using unsafe keyword.
is this just the continuation of the previous point (compiler/borrow checker can have bugs), a nod that you can import unsafe dependencies, or is there a soundness gap in rust that I'm not aware of?
this is just a perfection fallacy. silver bullets don’t need to be perfect in every single way to be silver bullets. and rust is as much of one as anything.
If we're suggesting alternative web servers, I have had a pleasant experience with caddyserver (Go). It's been around 10+ years, has essentially transparent TLS certificate provisioning, and its config syntax is easy to understand. Even using AI to help vet configs, it's readable. It's packaged everywhere and has lots of plugins for things, though I only need the base package for my frontend proxy usecase.
As someone that was a developer for a long time what is the problem with C? How does going to rust mitigate this? Why is having the stuff in C a security challenge?
It's very easy to accidentally write buffer overflows, use-after-frees or other memory-safety bugs in C. Modern tooling can probably catch some of these better than in the days of yore but the fundamentals of the language are still the same.
If e.g. the standard array type in a language has built-in bounds checks that error out when you try to write to an index that's outside of the array, you can't trivially create buffer overflows simply by writing to the wrong index or by trying to write n + 10 bytes into an n-byte array.
But C in itself has no guardrails against something like
Of course most memory safety bugs aren't trivial like that since C programmers have hopefully learned to avoid the simple mistakes. But there can be more complicated and difficult to spot cases, and C has no guardrails against those either.
SQLite is probably the most thoroughly tested and carefully developed C library in history and they still found a handful of memory safety bugs every year.
Rust is memory safe at the cost of being harder to write, but with AI, it's actually really easy to write it because AI's loove strict compilers. So now that AIs can just find 0-days before things are even released (so I'm told), it makes a lot of sense to do bun[1]-style rewrites to Rust with AI.
If a bug is found before its released... Then that's not a 0-day?
The name refers to the number of days since public release. "Exploitable 0 days since release". If you found a bug before release, then that's... Just a bug picked it up in normal development lifecycle.
I read 0 days like "Exploitable 0 days since _discovery_". Also, it's only a bug in the normal lifecycle if the discoverer tells the developer making it. If they discover it and then have hours and days to perfect a program that exploits it, then starts running immediately when it is deployed or released, that's a new kind of "zero day". Makes you want to start differentiating what "0 days" mean, though.
The ai told the hacker. The hacker discovered it with his AI before the developer released by examining his git branch.
I think that's the fundamental disconnect in the conversation here. You think I'm talking about the dev because it's not released yet but just because it isn't tagged and released doesn't mean it's not done in the open where hackers can see it and their AI can scan it.
See Morris worm, and when it took place, afterwards check Multics security assement done by the DoD versus UNIX.
Additionally, check what the inventors themselves of UNIX and C did after being done with them, and what they did afterwards in terms of operating systems and programming languages.
Yet, some folks are more attached to C than the inventors themselves.
The solution was not to take C into widespread use in first place.
Note that even C creators, first came up with Alef for Plan 9, which did not work out due to lack of automatic resource management (on Rob Pike's own words), and then Limbo for Inferno, which by then meant C main purpose was the OS itself and DisVM/JIT implementation.
Even on classical UNIX, there used to be a tendency that many userspace tools were to be written on top of scripting like AWK, sed, tr, Perl,... not everything to be C.
In terms of systems programming safety that people wonder about Zig, was already there in languages like JOVIAL, PL/I and NEWP, among others, a decade before C came to be.
The solution is not to write new code in C, unless there is no alternative, like coding on UNIX/POSIX kernels (UNIX based on Ada or Pascal dialects like did not fair that well, e.g. Domain/OS).
Not everything has to be Rust, it can be Ada, Modula-2 (now on GCC), D, Swift, FreePascal/Delphi, Zig, Odin, C++ (proper C++ with standard library, not code C with C++ compiler), C# AOT, Java AOT, Go, .... whatever depending on the use case.
This looks like someone prompted "rewrite nginx in rust, make no mistakes" in an afternoon and is now distributing it as a closed-source binary. Might as well just do it yourself.
xin seems to be closed source and ai generated. Before even worrying about exploits, how is someone supposed to ascertain that xin's implementation is correct? Sure Rust might not have any memory safety issues, but other logic vulnerabilities may exist. 'Claude make no mistakes?'
Caddy, Traefik, or even Envoy may be better choices.
I'd hate to interrupt all the Rust fans sucking each other off again, but no one has yet mentioned the RCE relies on ASLR being disabled, which is basically non-existent nowadays. Another nothingburger CVE with a silly name and it's only Wednesday.
Haven't you heard? These vulnerabilities are just AI marketing stunts. And also, the Bun rewrite to rust was a marketing stunt, those things don't work. There's no way Bun is running on Rust from a couple month AI powered rewrite, that would shatter my preconceived worldview about what the future looks like. /s
It is true that mileage is a good proxy for finding bugs, but this is isomorphic to the fundamental differences between testing v. verification. You can prove absence of classes of bugs with verification. Obviously there can be other logic bugs in newer software. No question. The issue is the memory-corruption bugs in socket-facing daemons can have catastrophic consequences, especially in light of chaining kernel vulnerabilities which defeats the privilege drop that nginx does.
So I'd say it is about a more holistic risk-analysis not just handwaving "all code can have bugs, duh!"
A more holistic risk-analysis would also need to consider the fact that a high-performance webserver written in Rust may also have memory safety issue due to use of "unsafe" features or unsafe dependencies, may be affected by memory safety issues in the standard library, may also have catastrophic logic errors, may be more affect by supply chain risks due to the use of a language-level package manager, and may have a larger patch gap.
Which is like complaining that Python is unsafe because it is written in C.
That is a far smaller surface as seen by research into incorporating rust in Android leading to none of the high profile vulnerabilities C and C++ led to previously.
Not in the slightest. It's like pointing out that a Python script calling out to binaries written in C doesn't have the memory safety of Python's GC. Which is obviously the case, and if you're a serious developer it's a failure point you're very mindful of.
On the Android point, I remember those metrics were highly flawed and didn't account for the inequality of substructural type ergonomics in different contexts. You can get very nice metrics by rewriting things that fit to the grain of safe Rust, which gives you an inaccurate picture of how much unsafe code is "real world". LLMs have changed the field here, but it's worth pointing out that blog post you're referencing was more self-congratulatory marketing than serious science.
Also problematically, I remember it being highly misleading in that the actual observed result was lower severity in vulnerabilities, but a very minimal effect on the total number over all. That's certainly significant, but it wasn't how it presented itself, which is rather problematic if you expect people to take it seriously as objective research.
Measurements probably real, but not worth citing. It's not like on their own blog they're under usual academic journal pressures to overclaim results, p-hack, and engage in other misleading behaviors.
What specifically distinguishes the empirical research presented in Google’s blog posts, for example the one linked below, from your definition of research?
Research is not just making some plots and posting them on the internet. A proper scientific studies would need to be done much more carefully, if you want to have any chance to get meaningful results, because there is a lot which can go wrong at any level. This is something learned people learned the hard way in empirical sciences. In particular, in this setting I do not see how one could ever properly control for confounding factors. Such a study should also be peer reviewed. Then repeated by others. Also if there are commercial interests mixed-up this needs to be questioned carefully. Finally, I just note the data presented seems mostly about C++ not C, so is not clear how this applies to C. Not separating this would also already be an obvious methodological problem. This is also connected to the problem that the underlying data is not made available, and not properly described in sufficient detail. Etc. (Not a full list, just to make it clear this is not even remotely science).
Which is to say, you have constructed a standard that lets you dismiss virtually any empirical finding. It seems like the goalposts will just keep moving as Rust replaces insecure languages.
With differences this large it is something completely different to teasing out a P value of 0.05 from messy data. Which you attempted to conflate the findings with.
> We adopted Rust for its security and are seeing a 1000x reduction in memory safety vulnerability density compared to Android’s C and C++ code. But the biggest surprise was Rust's impact on software delivery. With Rust changes having a 4x lower rollback rate and spending 25% less time in code review, the safer path is now also the faster one.
In theory, that could be possible. In practice, there have been many many case studies demonstrating the opposite and not that many case studies demonstrating in favor of your argument.
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
The CVE counts for new parts of the Linux kernel written in Rust are much lower than the (recent) CVE counts for even mature, years old parts of the Linux kernel written in C, to say nothing of the historical record.
There is basically no meaningful part of the kernel written in Rust from which one could conclude this. There is now binder, but this was only recently merged and I am not sure what data there is about it (and how it would compare to new C code written with a focus on memory safety).
A more holistic risk-analysis would also need to consider the fact that wearing seatbelts in motor vehicles still leaves drivers vulnerable to drive-by shootings, may be affected by massive boulders crashing on top of the vehicle, driving off a bridge, or gas explosions, and costs more to produce.
In my view, the Rust argument is more like that driving (in C) can not possible be safe because there is always the inherent risk of steering into an obstacle, so we have to abandon it completely and only use the local public rail system (which entirely eliminate this class of steering errors). But this is argument made based on the state of the technology in 1950s without considering that seatbelts and other modern techniques, driver tests, speed limits can make driving a lot safer (as seen today when compared to the 1950s).
Seatbelts are more comparable to activating bounds checking in C compared to switching to another language. because they are a minor inconvenience with a huge positive security effects, but do not eliminate all risk. Rust is more like switching to public rail system, because it does eliminate certain risks completely (all related to steering), but not others, and then also limits you in various way, so you may still need to use a car for the final meters (e.g. C or "unsafe" Rust).
And this comparison is consistent with the idea that the public rail system really is safer to use with regard to traffic accidents, so captures the idea that Rust actually has safety benefits. But IMHO the framing and marketing is highly misleading.
how does rust limit you? it seems to be one of the most versatile languages ever. it's genuinely used for everything from the linux kernel to webservers, desktop and mobile apps, even frontend ui in the browser. no other language has that usage (maybe swift?). saying rust is a rail system and c is a car makes zero sense, if anything rust is a flying car because it is used in situations where c is not, whereas rust is used everywhere c is used.
any serious engineering discipline eliminates the possibility of human error wherever possible, especially when it results in dangerous effects. and yet in software there are people who prefer exposing others to those dangerous effects because of a personal preference for a 50 year old tool.
Everybody here talking about going back to HTTP/1.1, it's a good stop gap solution but we should really think about moving to HTTP3-only as fast as possible. Changing over is easy, if your server doesn't support it just put it behind a reverse proxy that does.
Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
HTTP is used for a lot more things than web pages, and most non-browser clients in production don’t support HTTP/3. The current LTS Java version (25) for example doesn’t support it.
More like solution should be moving to sane HTTP4 - v3 corporations made for themself so v4 can be made for users :) Without corporations involvement.
Apache and Mozilla Funds would be ideal choice for such open source implementation but I doubt they will do anything to extend their usefullness...
So: users require just visual part and html/css is not bad for that. Could be better. But transport part can be [re]done at will. Eg. would be nice to have separate layers for connection security and actual transportation.
Because any HTTP spec after 1.1 is incredibly complicated. You can implement HTTP/1 in half a day, and HTTP/1.1 in a day. But HTTP/2 will probably take you a month and HTTP/3 might take something like 6 months.
It boggles my mind that people are patting themselves on the back for staying on old tech stacks and using a discovered vulnerability as further justification of their decision to stay on h1.1. The response is to patch and move on if your environment requires h2; many large orgs these days use microservices which use h2 (grpc) as its network.
There are many legitimate use cases for h2 (eg bidi streaming, latency improvements) and h3 (eg seamless network switching).
If you don’t need these features and just serve vanilla text webpages that’s also fine to just use h1.1. Blanket statements that h1.1 is better is just, wrong. Context is important on choosing what’s best for your environment.
Do you realize that no one writes toy HTTP implementations from scratch? I get that you might have a "suckless" mindset but that's just not how the real world works.
That's not the point. The point is that the simpler the protocol is to implement, the simpler it is to validate and harden the implementations. A HTTP/1.1 implementation can be 10x to 30x smaller than an equivalently well-rounded HTTP/3 implementation.
In the real world you want performance. "Simpler" is not "more performant" in this case. Do you understand that HTTP/3 solves real problems for real use cases, and wasn't invented because someone wanted to make a more complicated thing?
It's like if I said "you can just use Linux 2.6, it's simpler"
It's debuggable. I can wireshark it and understand everything. I can even speak to HTTP servers directly with a telnet client and fetch pages. I can write microcontroller code in C that issues REST API requests. Pure TCP, nice to work with, no multiplexed UDP hex craps.
On the other hand I've never been so proactive about maintaining security on my homelab! I did a full audit of my self-hosted services with claude code, strengthened the isolation of all my docker services, I'm running coding agents in sandboxed microVMs now, my firewall settings are better, I set up separate VLAN for IoT, I get security notifications on my dependencies, etc. etc.
A lot of stuff that I avoided for years I finally feel like I really need to do now but it also has never been easier to do.
Yeah I'm having to do attack surface review and threat modeling like I'm a full enterprise now...
Fortunately with Tailscale making it super easy to keep stuff out of the open internet, I don't have much, but even then I'm thinking of taking what I do have down. We see these big CVEs coming across, but I guarantee there's a large number of 0 days sitting around in peoples' stashes all over the world that we don't see.
Is this the reason why those banks were down the other day? Did they get advanced knowledge so they could patch their instances? Kind of questionable that the AI labs are creating a two-tiered system for system security if so.
well it's a good time to pitch tech debt and infrastructure cleanup. If you aren't making use of least-privilege permissions, service-mesh, netpols etc. this is the time to add all of that.
Because clearly the patching treadmill is not sustainable, you will need a a second line of defense and reduce the blast radius of every individual vulnerability.
One, as the RCEs are coming out, admins are on edge and paying more attention, so more likely to catch the annoucement on their platform. "Oh, that's on PHP, we don't use this. Oh, wait, another one?"
Two, all of these hitting at once, while they may expand the surface for attackers, it also spreads their attention. They only have so much time and bandwidth. When a new shiny RCE hits, they need to consider "do we continue to try an prosecute the old one or jump on this bandwagon".
So, in the sense, it can actually slow down the actual potential impact of the revelations.
We can only hope this gives users time to get their systems up to date before they come knocking.
This sound suspiciously similar to the issue Microsoft said they were dealing with excess traffic on their front door, taking them down.
maybe not an RCE but a DOS in the same http/tls code.
Oh no, how annoying, yes I can see how that could be the case if Discourse is running its own copy of nginx with its own configuration inside its container. That's unfortunate - I don't know much about Discourse but I hope that had a good reason to bundle in nginx, seems unnecessary at first glance. I really hope a Discourse developer gets to the issue soon and releases an update, that would be the fastest and easiest way!
I am running the official nginx docker image by itself in front of all my services, which are their own docker images. So it's easy for me to update just one of them. This works for me because I want my dedicated server to run all my projects with nginx in front. If I were running Discourse I would want it to not bundle nginx because I've already got one of my own. So that's interesting that they do that.
Ops is not fun these days … used to pinned versions and an auditing approval process with SecDevOps … how are you mostly automate the process in a you build it you run it environment ?
Yeah, if it matters for your workload, then don't do this :) And obviously test outside of production first if you're doing anything important. But if all is happy with http/1.1 then this is likely much easier than making a config change or waiting for a patch that doesn't seem to exist yet for some downstream consumers of nginx (like apisix for example).
I wonder how much of this exploitation will push the industry towards closed-source. Previously the advantage of OSS was trust and collaborative improvement, but that is getting eroded quite heavily now by the barrage of attacks and insidious discovery.
The first step was closing contributions down to known contributors and maintainers. Maybe there’ll be a similar step where only a select group will be able to access — and attest to — the code?
Almost like “Restricted Access Source” or “Withheld Source”
Both this vuln and the java4shell vuln from yesterday have very narrow attack vectors. It's not impossible for a human to figure out, but it does feel like an AI was used to discover both of these vulns.
That's to say that this is our new normal. Something that would have taken months or years to figure out can now take an AI hours if not minutes if you have enough compute.
Quite the opposite, it looks like there has been no responsible disclosure here. No update available for anyone running something based on Debian (Ubuntu etc.), only if you go and fetch binaries from the source and do the system integration yourself (easy in this case, but in general that's the point of a distro). For php I could just update at some point that day
Otoh I'm not sure what http2 is good for (I've asked around today, some people said the multiplexing speeds things up on high-latency connections. I'm still working on a realistic benchmark since I couldn't find any but it's not looking great so far) and it's default disabled so comparatively few sites are affected. Getting rid of it is easier than waiting who knows how long for it to get packaged
HTTP1.1 has a fundamental flaw related to request smuggling. Fixing specific variants of that problem is a game of whack a mole, as issues arise most often from larger server composition, not individual parts. Fixing the protocol flaw requires a major version increment and a phase out of the old one. Today the hesitation to do that gives us a mitigation option, but in general an http downgrade is a bad thing.
There are also features on top of http2 that a downgrade may break: gRPC and Server Side Events for example. The later is compatible with HTTP1.1, but could fail due to connection limits, as multiplexing isn't only about speed. Some web apps may handle that so gracefully, it turns into a performance thing again, but others could halt and catch fire.
Just another set of unfortunate examples of programmers who are bad at C, definitely not a C problem.
EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
This fixes the integer underflow for CVE-2026-123007, and it's true that the underflow can happen in any language, notwithstanding whether it's UB or not. The issue is what happens as a consequence of this underflow, in this case: out of bounds copy, which is specific to C.
Some languages make it significantly easier than C to just forbid integer overflow/underflow entirely.
Given that on modern CPUs, cycles are quite fast, and memory latency is quite high, always-checked integer operations sound like an extremely good idea.
Sure, I usually use `overflow-checks = true` in Rust for this reason. However, it is still true that the overflow is not what caused the issue here. In Rust it would've paniced on the out of bounds copy, even with `overflow-checks = false`.
It is not specific to C. You could have bounds checking also in C, and you can have non-checked indexing also in many other languages, including Rust. I would agree though that many other language make it harder to not have the bounds checking, but the bigger question is why this is not done C (and the answer that this is very hard to do is false).
C is like a Winchester Model 1897, not saying that it is a bad gun, it is what we call it the good ol' Trench gun, as it ran over the war tunnels of the Great War, and it still packs a punch today, but it only has a half-cock for some level of safety, and there is no trigger disconnector either, so you can easily slam fire it. So it is way too easy to shoot yourself in the foot.
Nowadays we like going for a AKM/AK74, for something maybe more complicated but far more reliable at the same time. That's why we want Rust.
It wasn't necessary, at least for people who kept hearing that there is no need for languages like Go or Rust, since it is possible to write memory safe programs in C, you just have to not make any bugs.
Is this going to end up being another exploit that requires a ridiculous amounts of special conditions so as to not actually be applicable to majority of sites running nginx?
I think the problem is that while what you say is true, AI is also capable of generating the special conditions required. And if it can’t do so reliably, it can sit there hammering the server for as long as it takes until the conditions are met - as an example hitting a bunch of load balanced servers until 2 requests hit the same workhorse sequentially.
It is completely realistic that we start seeing a lot of bugs a human can’t (reasonably) exploit, but AI can.
Yeah I'm not quite sure what is going on with Debian, I would have hoped that this vulnerability would have been disclosed in collaboration with Linux distros to get a patch out?
Given how common it's become to constantly scan popular repos for n-days, I expect someone will have already picked it up and started work on an exploit. The workaround for this (disabling HTTP/2) is also comparatively easy to what we had with PHP2Shell just a few weeks back, so I think it makes sense to release this.
Based on <https://news.ycombinator.com/item?id=50547678>, a bunch of people reported this to Nginx in parallel. Wouldn't surprise me if they went "Igor, you wont believe, this rce is the same as the five previous I reviewed this week! Let's ship your patches before malicious attacker discovers in parallel also"
Even the US federal government (CISA and NSA) have been strongly advising companies to transition to memory safe languages for a couple of years now. https://www.nsa.gov/Press-Room/Press-Releases-Statements/Pre... (one of the goals of such advisories is it acts as a signal to the industry that in several years this will become a mandatory requirement around purchasing decisions)
We've literally decades of evidence that we cannot write software securely in languages that aren't memory safe, for all sorts of reasons (and it's wrong to look at it as a developer competence issue, even though sometimes that's going to be the case, even extremely experienced and highly competent engineers write memory safety bugs in critical code we rely on day in day out.). Not leaving yourself open to such problems is an entirely logical conclusion. We know a large percentage of vulnerabilities in software are caused by it[1]
Pick a language that doesn't have these risk factors, or pick an enforceable subset of one that makes such impossible.
"Insanity is doing the same thing over and over and expecting different results."
I was talking with someone about buffer overflows due to HTTP requests this morning. I think I said something like "but we don't need to worry about that, just put nginx in front of it." Sigh. Off to update some VMs. Again.
"Both vulnerabilities are memory-safety issues in how NGINX handles HTTP/2 traffic."
HTTP/2 again. First Apache, now NGINX
I rarely use HTTP/2, I rarely use a popular web browser, I use a TLS forward proxy that has HTTP/2 support but I prefer HTTP/1.1
I find that 1.1 is faster for the way I use the www (text-only, no images, CSS, Javascript, cookies, advertising, tracking), plus the 1.1 protocol comes from academia not Google, Akamai, etc.
Hey this is TZubiri, the guy that comments that you should uninstall the dep whenever there is an post about a CVSS 9+ vuln.
Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).
We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.
We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:
Step 1: import socket # import the socket library from glibc or glibc.
Step 2: socket.socket() # socket.socket()
Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.
Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.
Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1
Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.
Thank you for reading my tutorial, and best of lucks.
This doesn’t seem very HTTP/2 specific, though, and it seems more like the conclusion should be “don’t use C for things which matter”. It’s not like HTTP pipelining and keepalives didn’t have plenty of edge cases to worry about.
True, but offering three different protocols with different code paths for the same thing is a lot more attack surface. If you start factoring in different compression methods things look even worse.
It's now been battle-tested in production, and also supports websockets, etc.
(However, it doesn't support sendfile, because PHP doesn't have support for that, so it's not as fast as nginx for sending static files. This shouldn't make a difference, though, if you use a CDN in front.)
Okay now I believe that AI is finally burning down the internet and that all the incidents are not just pure coincidences.
Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.
And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.
Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.
Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.
Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.
We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
Theh people laughing at Anthropic and co for being cautios of Mythos just didn't understand whats happening.
Instead of finding, hiring and then finding security issues, you can throw money and compute at it. This alone is a complete game changer. Combined with models which are very very capable.
This can be potentially the death of certain programming lanugages or requirement of a complete rewrite. It can mean that only big companies have the resources and capacity to continue playing this game.
To all the people here that would like to give AI credit for everything, and dont think it matters that everyone at Wiz is Israel Ex Cyber "Defense", and all you need is AI to find things like this. I have a challenge for you:
If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
Well, you can't because the LLMs are all nerfed and censored w.r.t cybersecurity.
I'm not saying domain knowledge is not required. But a lot of people do have domain knowledge and are prevented from using these tools. Even to secure own code bases. This sucks. A lot.
The refusals do not make that big of a difference! Holy cope. No you still need to study and become an expert. Ill give you access to unlocked fable and you still wont be able to do much with it if you dont have a ton of experience hunting bugs.
Keep telling yourself that! The refusals arent that big of a deal. I have access to all the models without refusals. Your average joe still would not be able to hack something with it.
So much cope on this site. Llms havent replaced the requirement of studying and becoming an expert.
Isreael btw everyone at Wiz is a veteran of Unit 8200.
But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.
Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.
People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..
Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.
Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.
Especially for software like Nginx which powers a huge part of the internet.
You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx
The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.
I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.
The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.
Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.
Envoy is only attractive at a higher scale.
In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.
There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.
[0] https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/l...
[1] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensio...
[2] https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_over...
[3] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/r...
This requirement turns me off of gRPC entirely. Maybe Envoy truly is more secure than nginx, but it's still a whole-ass additional attack surface. It seemed to me that eliminating it if possible was best.
I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.
In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.
Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.
There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.
You would have had to enable http2 and have a buggy Linux kernel to be vulnerable.
If I wanted to build my own software from source every time I'd be running Solaris like a virile man from 2004.
stable-alpine-slim docker iamge is only 14MB?
Not saying that thie image is affected (I don't know), but small footprint does not guarantee security either. It can hit systems of any complexity these days.
Containers by default enforce some level of restrictions and a sandbox around it all.
If someone does break alpine, nginx, they would still need to get out of the container sandbox?
nginx when hosed gives full access to the system.
Which may be good or bad.
Ubuntu and other distros have not kept up.
Podman container auto-updates to the latest upstream. It is a minimal surface area given alpine images. Security first containment.
In this day and age with AI attackers and zero-days it is absolutely daft to still treat servers as the old-school pets model.
Doesn't this make the system more susceptible to supply chain attacks?
My nginx instance is very very tightly scoped for what privileges, file system space and capabilities it can access.
If someone exploits nginx on host, they have a remote shell and access to everything.
https://ubuntu.com/security/notices/USN-9090-1
Um, what does this have to do with NGINX?
Did you talk to it like a pirate?
One idea that might work for you is client certificates if those are only internal APIs. I don't know about you, but I have a lot more confidence in openssl than the people who use openssl.
I turned off http2 a few years ago https://news.ycombinator.com/item?id=41486452 and absolutely did not regret it.
I do regret every experience I've had with gRPC though. I feel for you.
And here lies the risk of AI agents in cybersecurity: the speed of processing for monitoring, scanning, identifying security fixes in open source, writing exploits for affected versions, and attempting infiltration, is a matter of minutes now.
It is not "by anyone smart enough". It is by "anyone ill intentioned enough". They don't need to be super smart anymore.
There’s no good solution, it’s a minefield and the only choices are what you’re willing to lose to maybe make it out of it
That was my point. Anthropic and OpenAI used to talk about alignment a lot, and even try to use it as a way to justify restrictions on AI development. They can’t really use that argument any more.
Many people, even high ups at AI companies, conflate alignment with "guardrails" (AI companies conflate intentionally as a matter of business strategy).
Alignment is just making sure the AI doesn't become a paperclip maximizer or go off and hack stuff on its own whims. This is akin to making sure a car doesn't accelerate without the accelerator being pressed.
Guardrails are about making sure people can't use the AI for nefarious purposes. This is like some sort of automatic braking to prevent people from intentionally hitting stuff with their car.
The interesting thing with AI development, is that guardrails are in some ways easier to implement than actual alignment, which is why I think many like to conflate them.
Cant win.
Seriously reconsidering if open source has a future nowadays
That said, I just moved from a Linux based job to a Windows based job and it's been nice not having to be panicked about having update most of the technologies that my last job used. As an aside I did try in the past to make sure we were always updating dependencies but my manager was bothered by the amount of work that created. I'm watching that team now take forever to deal with these issues because they're too far behind to just version number and rebuild.
The issue is that we get dozens of PRs, generating a bunch of code is trivial but reviewing and merging it and making sure it doesn’t break in interesting ways is still pretty hard.
Personally I lean towards not accepting code contributions anymore, but accepting design documents for features and l bug reports etc
It's not a perfect solution of course, it just takes OSS back to the fairly broken state it was pre-AI. Still, it might be an improvement and should be at least considered.
I’m not sure how we could enforce an 0 LLM policy in practice though…
https://xkcd.com/810/
*In C or C++.
C and C++ will stay around for a very long time, even with AI powered systems.
If anything, maybe that is a call that even if implementing compiler backends is hard and time consuming, bootstraping compilers, and new approaches to OS design, are the only path away from such dependencies.
Legacy 32 bit users with NetBSD and Hyperbola can survive a little with Go based tools/small browsers/mail/usenet/IRC clients and some web browsers with Bubblewrap, but not forever.
https://www.khronos.org
https://vulkan.org/events/vulkanised-2027#program
The only thing closer to it is WSGL.
Also there are much more operating systems than those examples.
But the only reason to sit outside a tank rather than in one is because riding on a vehicle is safer than walking through an open field. Tank crews are not sitting outside of their tanks, and they don't have seats inside for passengers.
That stopped being true for at least 3-4 years (with APCs maybe still used until two or so years ago, tanks are still used but as "turtles" or/and "porcupines" [1], no way to have soldiers riding them). Drones have conquered (almost) everything.
[1] https://miro.medium.com/v2/resize:fit:1400/format:webp/1*CQr...
Security is hard and is mostly about the number of engineers and amount of time spent looking at a specific implementation. When writing code, do you think about the problems of leaking information due to different operations taking specific amounts of time? That's an actual concern when writing a prod service that uses the network. And it has to be balanced against scalability and issues caused by DOSes. It isn't trivial and it isn't binary...there are always trade-offs. Nothing will ever be perfectly secure and usable and scalable.
CVE-2026-114938 would like a word. https://calif.io/java4shell
Yes, I know, different issue. But saying Java would be better for webservers just two days after a serious JDK RCE is pretty ironic :P
But doing that requires collecting the whole chain. In the current environment we might be finding and patching these smaller bugs before they can be put in to a chain. As long as you are keeping on top of updates..
Now, if you were paying for support from a vendor, that's a different matter.
"THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED."
Chainguard still updates their fork, by letting AI try to apply and backport the nginx patches, and I'm using that at the moment while switching to gateway API, but I'd have preferred if ingress-nginx was still maintained (the one by F5/nginxinc for example is still being maintained, just not entirely compatible).
Well, actually four: https://lostartpress.com/products/the-stick-chair-book
According to the author, Troy could do the work of three people. That would make him a 3x woodworker, I guess. And he was facing automation in the woodworking industry.
"But Troy decided to compete with the machines on their own turf. He tried to become a one-man factory. He succeeded in that. But the most efficient one-man factory cannot compete with a factory driven by a hundred men.
So no matter what Troy did, the prices for work (good or otherwise) declined. Ikea’s well-designed kitchens and Value City’s cheap junk made it impossible to survive and live well.
So Troy struggled with his business. And that’s nuts. He should have had more business than he could handle. But like many professional cabinetmakers I know, it became harder and harder for him to make ends meet every year. His cabinetmaking business is closed, and the last time we talked, Troy was working for his wife’s successful tax business.
Who is to blame for this? We are.
We have become a culture that is obsessed with price more than any other attribute of the things we buy. It doesn’t matter if the item is ugly, poorly made or constructed of materials that cannot be recycled. All that matters is if the price is low enough.
Because the price of our household objects has hit rock bottom, if an item breaks or starts to look dated, we can throw it away and buy some- thing else. For the first time in human history, manufactured furniture is shockingly inexpensive.
So it’s no wonder that artisans are exiting the craft. It’s difficult to compete against furniture that costs less than what you pay for your raw materials.
Lesson: People cannot imitate machines and defeat them."
Appreciating good design, quality craftsmanship, having taste in food.. having these "elevated" tastes in multiple areas is very rare for humans. Take a look at a musician - they can tell you precisely what makes a quality violin, but they might not care that they eat microwave frozen dinners. You compound that across the population and the artisans will always lose out to the mass-produced slop because it's impossible to naturally get a critical mass of people to care about that quality curve. You have to sell them on it.
Troy struggled because he failed at marketing, not production.
I think I've learned that most people don't want to put in any effort to learn or appreciate things that may be of nicer quality. Those Japanese knives didn't do any good for them because they didn't spend time learning how to use them properly. In fact one of them used them improperly and ended up needing stitches. So in the end the crappier version is no different than the nicer version, because crappy and nice are about the potential of the item, and most people don't see potential. Does this Ikea dresser hold clothes? Great, who cares that the other one is of nicer wood, the joinery is done by hand, and it'll last multiple generations.
https://en.wikipedia.org/wiki/John_Henry_(folklore)
I see a lot of interest in woodworking on HN. Is that because Jesus was a carpenter and HN folks are interested in woodwork because of the larger context and history shaped by Christianity? What is your motivation for woodwork? Is that because we see a concrete finished project instead of vague intellectual stuff which has no seeable boundaries?
The comment you replied to is shortcutting the joke, assuming the common riff of becoming a carpenter is well known, and jumping ahead to asking about tutorials for woodworking as part of that process.
Preposterous, even.
That said, my RHCE CISSP self is going to go outside and spend some time with my flock of chickens and guinea fowl.
While Claude writes Ansible to update the fleet to use nginx's deb repo.
Not sure why others downvoted you, but you have my upvote on it.
There was a meme some time ago about the perfect career path of this linkeidn profile that quickly rose in ranks in a big company and after that his current job was duck farmer.
Woodworking is a stand-in for the category of activities that are definitely manual, artistic but in a "manly" way, and many perceive as the antithesis of worrying about the next sprint, code review, or career advancement
The account is from 2017. There are enough comments before LLMs were a thing. Given his karma score, unlikely to be a bot.
> Hey so, does anyone know of a good tutorial on getting in to woodwork?
Some people really don't get these type of things.
Are we going to discover in a year that every server online this month had a rootkit installed? There’s got to be thousands who are too small time to wipe and reinstall.
Worse patching adds instability into a system, especially when you don't have much time to test.
Also in the past attackers were willing to sit on exploits longer and not do things with them, now they have to act fast before someone else finds it, leading to more events faster wearing your security and patching teams out, and making users life suck.
Things are going to get much worse before they get better.
I guess you could make an argument based on some estimate of how fast exploitable bugs are newly introduced. That also seems dubious.
And the attacker does not have infinite tokens..
Survival of the fittest I'm afraid will build security here. The teams that barely ever patch or don't have "time" for maintenance are going to get burned. The "we haven't updated in 7 years so things are stable" teams can't survive.
Nope.
Let say Im running a vulnerable Ngnix server. All that would take to stop any and all attacks is a simple firewall that sees anything that even remotely looks like shell code (i.e non ascii characters, ROP patterns, strings of noops, e.t.c) close that connection,blacklist that ip.
>Also in the past attackers were willing to sit on exploits longer and not do things with them,
Also nope. The reason why you didn't see many exploits is because they are mostly useless unless you have resources. You can take a gander through all the CVEs that AIs are finding. 99% of them are either DDOS with some possibilty of remote code execution only under very particular conditions, or require already user level code execution to get to root.
There are a few sprinklings of truly bad ones, but even for those, a good portion of them that exist on critical systems are protected by secondary things, like firewalls that detect shell code and close the connection and ban the IP for some time. True network security will never be not needed, as humans are still the most vulnerable vector, and if you don't have systems in play that can counter bad actors, then you are asking to get hacked.
All of this AI finding exploits is basically fear mongering that is going to be used to sell AI products that secure systems.
How will your firewall look inside TLS traffic? If the answer is to have it terminate TLS, congratulations, now your firewall appliance is the next target.
Collectively, we owed about 40+ years of tech debt on non-secure code (e.g. automotive industry, industrial environments) that will be torched in the next few years. Luckily, AI will generate code faster enough to make that plausible.
That's not happening at nearly the same pace, both for assymetry between what it takes to break something and what it takes to fix it, but also because that fix must navigate bureaucracies. Also, the leaders need to actually care, both of which are hurdles. Bureaucratic navigation and consensus finding has not sped up at all with AI.
Compare that to independent script kiddies with $600/mo Claude subscriptions able to find 0-days in an hour and write a worm to exploit it in four.
> AI is securing the internet.
No doubt there are points of view where either one, or the other statement, or both, or neither, are true. You can really go full Schrodinger and put a probability distribution on everything.
But in this day and age, whatever is happening, the better question is:
Is it good for me and, beyond me, for the world in general?
it was previously expensive to attack, so people spent less money on defense.
It's cheaper to attack, and so people will increase their budget on defense. But it's a lagging cause & effect situation
Even Kimi K3 was only shipped as a lobotomized version of it.
A huge fraction of devices and servers are on old versions, not actively updated or managed in any way.
If you're not going to take care of it you've created a public nuisance.
There are two internets. One of them is open, less-commercial, and largely built and maintained by people whose first job is not the work they do to maintain this internet. That group includes a ton of open source software, millions of small servers scattered around the world (virtually private or otherwise), and is generally best-effort.
The other internet is fully commercialized, often ad-driven, fully integral to the data broker economy, and centralized behind a few companies that pay as few people as possible to satisfy the occasional PR statement that, "we take security seriously and...".
The next stable equilibrium here is not "we get a secure and open internet". The next stable equilibrium here is, "we have no open internet, and the internet is now managed by just a few companies."
We're already halfway there. Google and Microsoft combined already account for somewhere around 30% to 40% of email traffic. 30% of all websites are now behind Cloudflare. A few companies dominate social media traffic. Amazon and Shopify combined are half of US e-commerce.
Every single short-response-window exploit like this just pushes a few more percentage points of traffic into a handful of companies. Cheer for that, I guess.
When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
I don't even care if it's all written in Rust, essentially none of my problems require software boasting multi-millions of lines of code. I guess my browser and OS? But even those I question greatly.
It's funny, I never thought I'd be one to compile my own kernel, but as of this year I've been maintaining an extremely stripped down Linux base for my NixOS servers and desktops.
Feels like my end game is going to be on some micro kernel that I compile all my software for, lol.
This is somewhat on those lines: https://news.ycombinator.com/item?id=50556533
https://reticulum.network
The real problems here are that we did a crappy job in the first place and left a huge surface area of risk with the blind assumption that we didn't do a crappy job.
I am hoping to see a change in attitude to risk and quality in the industry but I get the feeling that's asking for too much.
Organized Cybercrime?
Digimafia?
e2p3yHizAungrCeKum7ifrssu32lnzEnyRoGdcAo23m2dNapvoGmymid.onion?
A more apt analogy would be that pretty much all consumer-grade locks are very easy to pick, despite being marketed as secure. I think in this case the lock makers should be blamed, not the lock pickers.
The lock makers are us - the IT field at large. As individuals we might not be to blame, but we are still enthusiastic parts of the greater movement that's issued vast digital complexity to the world without truly understanding it.
Now we are starting to understand it more.
Heh. There's borer.
And now there's people releasing swarms of borers in the suburbs. Now you perfectly serviceable door is compromised.
Please stop willing things into existence.
People joke about the "Rewrite it in Rust" crowd, but that call is rooted in an unease anyone with a sceptic mind has had about all the C and C++ code we've got running our infrastructure running around tracing pointers with little lifetime- or bounds-checking. (Yes, it's a runtime cost but it's 2 CVE's, one for each class)
This is why people (including me), who absolutely adore Rust, make fun of the RiiR crowd. If you think a rewrite from the ground up is the best way to approach making software safer, you've lost the plot entirely.
Let's say you rewrite your terminal emulator in Rust from C or C++; you'll first undo decades of bug fixes, compatibility decisions, quirks, etc. which you WILL have to deal with eventually. This essentially puts you back at least a decade, because these issues take users using the software to point out to you before they're fixed. The only benefit of this whole game being that you get memory safety. These teams are never the original teams, which is the source of the problem. A rewrite can only succeed if it's a result of all the learned failures and shortcomings of the original, otherwise it's bound to repeat every single mistake and add more on top.
And you don't even deliver memory safety to the users of the original this way, because people won't use the rewrite, because it's buggy, incompatible, etc.
Obviously, AI makes these things easier to find and exploit, but this isn't even close to the first time that 'thousands' of hobbyist (or small business or just not very competent) server admins are going to get rootkitted en masse. I worked at medium-sized infra-as-a-service company when the above was discovered and there were definitely thousands of compromised machines just there.
I do think the world is going a little insane tho. People suddenly believe in security through obscurity because, uh, AI bad?
There might also be a day in the future where our current encryption methods are broken - it doesn't mean they're insecure and vulnerable today.
We have a billon hacking experts now is the big change.
I'm not making a Rust advocacy argument. I'm just saying: it's true that integer handling issues are endemic to mainstream programming languages, but the integer handling itself is almost never the actual sharp end of the vulnerability.
You're right it's more rare but the results can still be catastrophic.
Integer handling mistake -> memory corruption -> RCE.
That's the pattern. In a memory-safe language, the integer handling error has no link to memory corruption (though it will to other logic bugs!).
Why would CPU designers make a trap if languages do not require it?
"Go fast and get your things broken into"
Care to elaborate why you think that sufficiently paid humans would not be able to find the flaws AI now finds, for cheap?
Harnesses allow LLMs to build & process large stores of data to reuse later. Imagine an agent that spends 3 days breaking into company A, records what it learned during that process, letting it break into company B in just 3 hours.
Defenders win by having the best security 24/7 and must trust that other services they depend on are doing the same.
An attacker wins when they can breach any defender's security.
The circle of life.
This is the sys/devops time to shine. Everyone needs to have an upgrade and an emergency plan, and everyone who was previously only doing conventional codemonkey needs to start learning more of the stack to join in on that.
My guess is that this phenomenon is going to be reflected in what we're seeing right now: a wide pattern of unintended outages from rapid unplanned emergency remediation operations. Which I'm guessing is a lot of people's job description over the past 3 months or so. (really, since March, for me, with that Trivy hack).
I'd be more worried if the response isn't more secure development practices in better languages, but instead use of AI as a defense. Then we're truly talking about burning down the internet.
A dead box isn't part of any bot army.
Tech that becomes infrastructure like web servers doesn't move so fast and it shouldn't. Most people and companies should be relying on relatively tried and true software, so this phase of AI finding vulnerabilities in it should pass and things could get better.
Like Java. Surely anyone can't mess up simple things like base64 decoding in it...
That's orthogonal to the point I'm making which is that no amount of reviews is going to make every single bug go away. Because, once again, humans and AI are fallible.
Sure, there are niche things. You can’t use US ciphers in China and vice versa.
But overall, one day, the intelligence communities world wide just decided to not raise a fuss about everyone having solid cryptography.
It makes me think if we did find out that every server out there had a route kit… it would make that part make sense.
All bugs are shallow given enough eyes, and the marginal cost of eyes is now ridiculously low.
Imagine a alternate reality where Ada replaced C by 1995. We would be in a MUCH better place.
Dunno if you have ever worked in software engineering, but if you have, count the times you have seen a backend server api written without any input sanitization or checking. Most of these exploits can be defeated by checks on inputs. I.e, no "Alex\x90\x90\x90\x90\xcc\xcc\xcc\xcc", is not a valid input for someones name.
Then Rust came along and those dudes either tried to say it was too hard (odd contrast, that) or it’d be too much work to rewrite things.
I’m hoping that LLMs will lower the cost of porting for enough people but what would really help is shedding the idea that the status quo is sustainable. Having consequences for sticking with C would help but we need a way to get companies to pay to migrate without destroying open source.
To me, its type system and traits design make it one of the nicest mainstream languages. I'd rather be using a functional language, like Haskell or an ML, and I do sometimes, but I don't pick my work based on that. I work regularly with Java, Go, Typescript, and Python, and out of those I find Rust significantly superior on several dimensions.
Python is terrible, Go is basically weakly-typed Python, Typescript barely deserves comment other than that it's impressive that such a fancy type system was bolted onto Javascript, and Java is basically COBOL at this point - reliable, but clunky and showing its age. Attempts like Kotlin to leverage the JVM aren't very convincing.
Gleam looks interesting but I haven't had a chance to work with it yet, and it's hardly mainstream.
You mentioned Ada/SPARK, but SPARK is not really for general purpose programming, and Ada in general has a similar issue to Java, but worse because of its age. It's as if someone really liked PL/I but decided it needed more features and a syntax closer to COBOL.
Whether that is “something natural to you”, “something you learn and adapt to” or “something you fight with forever” seems to vary per-person, with I suspect, most people falling into the forest 2 categories.
Someone who's saying "the rest of the language sucks" may be thinking of something like that, hence my question above.
But when it comes to performance overhead, programming around the constraints of the borrow checker can introduce runtime costs that wouldn't otherwise be necessary - e.g. when cloning is used to handle borrowing conflicts.
Even if, in theory, it's possible to restructure a program to avoid performance impacts, in practice people don't always do that, not least because more efficient solutions may not always be obvious. This comes back to my point about coding effort.
I spent a bunch of time, pre-LLM, to try to replace a bunch of unsafe blocks scattered throughout an ML inference codebase, that someone had used to achieve maximum performance. I wasn't able to get a version that performed as well.
This kind of thing may not matter if you're writing e.g. ordinary web services, but for more performance-critical code, it can be an issue. Of course, `unsafe` is always an option to deal with that, but that undermines Rust's guarantees.
I will chime in here to say that the adoption of async was imho a terrible decision and led me to abandon Rust for most purposes. I want proper threads and CSP-based abstraction.
This was one of the reasons why people switched to things like Java, Perl, Python, PHP, etc. where performance wasn’t critical. I remember having that conversation with a number of clients who’d been burned on delays and reliability issues building everything in C as opposed to using a higher level language and maybe a couple of C libraries.
https://xint.io/blog/nginx2root-remote-code-execution#23-com...
Workarounds if you're vulnerable and can't update immediately:
https://xint.io/blog/nginx2root-remote-code-execution#33-eme...
Consider a world in which corporations like the Ubuntu Foundation or Microsoft experience zero public pressure to fix such vulnerabilities but instead fix it only when a major customer notices their servers getting hit.
Imagine if instead of "you've got 30/60/90 days to fix this thing I noticed before I post it publically" it was "okay, well, take your time I guess, I just hope North Korea doesn't notice the same issue I noticed, which is especially likely because they have far more resources than I or my small team does to point at discovering this kind of thing, while you're dragging your feet fixing it."
The reason people want shorter disclosure windows is because the shorter the window is the faster things get fixed for the average users and smaller companies.
Massive monetary loss, explosions, and dramatic failures will push computer security in a year where it should have been decades ago.
> Not yet. We are not publishing the exploit now because the risk is too high.
So who's the judge of where to draw the line between exploits that are relevant to publish, and which are "spam". You?
Or do you just think that companies shouldn't be allowed to show the work they're doing for marketing?
https://x.com/Laughing_Mantis/status/2123146699396858260
Its a non issue, but people just want to be told AI is God or something and the world is ending.
Its a non issue.
The thing that should be compared to is the cost of doing this versus the cost of going through the human vector. A motivated enough party can get to a computer system through human factors way easier than what most people think.
But when newer software has the same vulnerability, you do not see the same kind of comments.
There always seems to be a latent stance in IT that newer is better.
This isn't true! Just look at this thread, people suggest replacing nginx with newer but less widely used and less battle-tested software, as if they're superior alternatives or don't have RCEs in their own history: Envoy, caddy, traeffik, Qbix.
Go isn't quite a memory safe language, but it provides fewer footguns.
On the other hand, Caddy has a lot more features, so there's more code that can contain exploits.
We have evidence that large unsafe codebases suffer from more and more severe CVEs than those who are not in memory unsafe languages.
In other fields of engineering, people use stats and actually measure things like this, and use the results to determine what to do. They don't just assume something is better because its "battle tested" (whatever that even means?).
I switched to Linux more than 10 years ago, and when something goes wrong, I'm still like "fucking linux". And in 4 out of 5 cases, it turns out it's a hardware issue or a cable got loose or something. Yet because I opted for Linux myself, deviating from what I think the PC norm is, Windows, I think that my choice is the issue.
I think it's great to point out these biases, like you did. Us humans are both surprisingly sharp, and woefully inaccurate when we point out observances in the world.
>There always seems to be a latent stance in IT that newer is better.
With regards to software, my personal stance is that the deeper I go in layers, the more boring ol' reliable I prefer.
Plus, Caddy itself wasn't vulnerable in this specific case. There wasn't much the Caddy folks could've done to protect against a problem in the Linux kernel itself.
All the remaining C code needs to die in a matter of months not years.
Why so? are most of these vulnerabilities from C's memory model?
https://news.ycombinator.com/item?id=50560611
the certificate for it was issued on the 2nd of November
if you were scanning CT logs you would have had a 16 day head start on RE'ing this
there are more examples in the current CT logs
https://api.certspotter.com/v1/issuances?domain=nginx2root.c...
The story is as old as this world
ping2root•com is available. :)
If you want to promote responsible anything, go for responsible writing of software in the first place.
https://is-a.cat/@ar/117634569518544368
Python simply can't overflow, since its integral type is not a fixed width.
But it is the sort of correctness thing that I do wish languages would pay more attention to.
Python master race
Is this some kind of record for bug collision?
If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
But it's like inventing a hammer and finding you can smash whatever you want with it.
Soon you'll be building houses with it.
Maybe that's being glib but it's just a tool being applied to a world that is still largely prestine from its impact (pun unintended).
The problem is that everyone is at a different place in the cycle. As Gibson put it, "The future is already here - it's just not evenly distributed."
.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
So yeah it sucks right now, there's outages, probable PII theft left right and center (but that's nothing new at this point), millions / billions in damages and lost productivity from Office365 and co being offline for a day or two... but it's still manageable, the world isn't on fire, and it's better to get all that out of the way now than in an actual crisis.
i.e fast followers of the researchers who don’t have the resources to burn through checking all of a tool.
What are the odds several humans independently uncovered a complex vulnerability like this simultaneously?
It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.
We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
I got worried for a sec, then remembered all my nginx lives behind HaProxy and it looks like nginx is only vulnerable if its going http2/TLS directly.
Now I sit and wait for an HaProxy announcement...
ps. if you are on FreeBSD and doing TLS directly
sed -i.bak -E 's/^([[:space:]]server[[:space:]]\{)$/\1\ http2 off;/' /usr/local/etc/nginx/nginx.conf ;service nginx restart
I know times are changing, it's just curious that nobody seemed to care until now.
In theory, LLMs will be able to help modernize a lot of old software (like the other day when a Cobol codebase was ported over), in practice most of the elements holding it back - verification, politics, hardware - are still in place.
Certainly a stressful time for sysadmins everywhere, let us hope your coworkers are picking up your beer tab
At the rate of development, also, will AI developed in 6 months from now be able to break a system that was secured by todays AI?
Now, you could argue that writing secure code ex-ante is harder than discovering an exploit. But you can hypothetically run the same "exploit finding" process on a first pass of code before it ships (e.g on the PR) given low enough token costs, which is where we're headed.
Even if that's true, one should not forget that:
When the hackers have breached something, awaits a reward.
E.g. a Bitcoin wallet, or learning something about a political competitor and getting rid of them.
(So the hackers can spend lots of ...)
If you're JP Morgan Chase, hackers will happily spend millions on tokens to hack you. JPM should spend just as much!
With enterprise frontier LLM models, there is no longer an “other side.” Everyone is just using what they can rent from a few big suppliers. You can too.
I wonder about that though: if the price of the squeeze collapses to 0 because of this (think of what models and prices might be like a year from now), I think I know where all those out of work mathematicians can find gigs. We are going to need to invent a layer on top of, an evolution of or some new concept a level above "computer programming" that deals with meta-organization of code.
On the bright side, perhaps "security review" of code which involves throwing it at an AI and fixing what it finds before releasing will become a normal practice.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare
Thinking about my home server where I host some stuff for household use, I think moving to VPN + local access only will be my new strat. So now the random webservers won't be exposed to the internet, only whatever I will use for VPN.
Debian is unfixed too
https://ubuntu.com/security/CVE-2026-123633
https://ubuntu.com/security/CVE-2026-123007
Neither has Debian:
https://security-tracker.debian.org/tracker/CVE-2026-123633
https://security-tracker.debian.org/tracker/CVE-2026-123007
I wouldn't expect any distro maintainers to react in hours. Distros are too complex to dump updates in rush.
Start to use Docker/Podman/systemd-nspawn.. and isolate public services to necessary needs (static binaries, drop capabilities, default drop rule firewall between containers, r/o storage, access logs bind mounted as rw file, memory temp dir with noexec). Container images are patched the quickest.
Just keep that protocol disabled for now. It's apparently too difficult for any human to write a secure implementation of http2. (As for http3, forget about it, we need AGI for that.)
edit noted it's about a Kubernetes controller
they published the release with a fix about 5 hours ago.
New code means new bugs, and losing the many years of accumulated bug fixes and hardening present in many critical pieces of our infra by rewriting in rust is not a desirable tradeoff. The new world we live in where fv has become many orders of magnitude cheaper also makes rust seem much less attractive.
While generally more of a hassle than Fil-C, the scpptool approach provides an incremental path to optimal performance (for performance-sensitive sections of the code) via migration to a (statically-enforced) memory-safe subset of C++. And unlike Fil-C, the scpptool solution essentially maintains the portability of the original code, as well as its deterministic performance and resource usage (i.e. no GC). It also supports ABI compatibility via "unsafe"/"check-suppression" annotations. The scpptool approach is not so much an alternative to Fil-C, as it is an optional path forward for C (or C++) code bases that may (or may not) be already using Fil-C for memory safety.
[1] https://github.com/duneroadrunner/SaferCPlusPlus-AutoTransla...
It's time for performance above everything else to die, especially on anything and everything that gets anywhere close to untrusted data.
The project isn't currently open source and there's next to nothing on the site to reassure you who's behind the server gateway you'd be running.
Googling the company in the footer produces this, but it also doesn't have much: https://www.signeen.com
And parts of that involved specifically replacing Nginx in their architecture, so it's hardly irrelevant here.
https://blog.cloudflare.com/20-percent-internet-upgrade/
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
is this just the continuation of the previous point (compiler/borrow checker can have bugs), a nod that you can import unsafe dependencies, or is there a soundness gap in rust that I'm not aware of?
https://news.ycombinator.com/item?id=50551504
If e.g. the standard array type in a language has built-in bounds checks that error out when you try to write to an index that's outside of the array, you can't trivially create buffer overflows simply by writing to the wrong index or by trying to write n + 10 bytes into an n-byte array.
But C in itself has no guardrails against something like
Of course most memory safety bugs aren't trivial like that since C programmers have hopefully learned to avoid the simple mistakes. But there can be more complicated and difficult to spot cases, and C has no guardrails against those either.Practically impossible? Yes.
SQLite is probably the most thoroughly tested and carefully developed C library in history and they still found a handful of memory safety bugs every year.
1: https://bun.com/blog/bun-in-rust
The name refers to the number of days since public release. "Exploitable 0 days since release". If you found a bug before release, then that's... Just a bug picked it up in normal development lifecycle.
I think that's the fundamental disconnect in the conversation here. You think I'm talking about the dev because it's not released yet but just because it isn't tagged and released doesn't mean it's not done in the open where hackers can see it and their AI can scan it.
Additionally, check what the inventors themselves of UNIX and C did after being done with them, and what they did afterwards in terms of operating systems and programming languages.
Yet, some folks are more attached to C than the inventors themselves.
Note that even C creators, first came up with Alef for Plan 9, which did not work out due to lack of automatic resource management (on Rob Pike's own words), and then Limbo for Inferno, which by then meant C main purpose was the OS itself and DisVM/JIT implementation.
Even on classical UNIX, there used to be a tendency that many userspace tools were to be written on top of scripting like AWK, sed, tr, Perl,... not everything to be C.
In terms of systems programming safety that people wonder about Zig, was already there in languages like JOVIAL, PL/I and NEWP, among others, a decade before C came to be.
The solution is not to write new code in C, unless there is no alternative, like coding on UNIX/POSIX kernels (UNIX based on Ada or Pascal dialects like did not fair that well, e.g. Domain/OS).
Not everything has to be Rust, it can be Ada, Modula-2 (now on GCC), D, Swift, FreePascal/Delphi, Zig, Odin, C++ (proper C++ with standard library, not code C with C++ compiler), C# AOT, Java AOT, Go, .... whatever depending on the use case.
Ferron seems much better, but I'm also not convinced that it's really production ready
Caddy, Traefik, or even Envoy may be better choices.
So I'd say it is about a more holistic risk-analysis not just handwaving "all code can have bugs, duh!"
That is a far smaller surface as seen by research into incorporating rust in Android leading to none of the high profile vulnerabilities C and C++ led to previously.
On the Android point, I remember those metrics were highly flawed and didn't account for the inequality of substructural type ergonomics in different contexts. You can get very nice metrics by rewriting things that fit to the grain of safe Rust, which gives you an inaccurate picture of how much unsafe code is "real world". LLMs have changed the field here, but it's worth pointing out that blog post you're referencing was more self-congratulatory marketing than serious science.
Also problematically, I remember it being highly misleading in that the actual observed result was lower severity in vulnerabilities, but a very minimal effect on the total number over all. That's certainly significant, but it wasn't how it presented itself, which is rather problematic if you expect people to take it seriously as objective research.
Measurements probably real, but not worth citing. It's not like on their own blog they're under usual academic journal pressures to overclaim results, p-hack, and engage in other misleading behaviors.
https://blog.google/security/rust-in-android-move-fast-fix-t...
With differences this large it is something completely different to teasing out a P value of 0.05 from messy data. Which you attempted to conflate the findings with.
> We adopted Rust for its security and are seeing a 1000x reduction in memory safety vulnerability density compared to Android’s C and C++ code. But the biggest surprise was Rust's impact on software delivery. With Rust changes having a 4x lower rollback rate and spending 25% less time in code review, the safer path is now also the faster one.
In theory, that could be possible. In practice, there have been many many case studies demonstrating the opposite and not that many case studies demonstrating in favor of your argument.
e.g. Cloudflare replaced Nginx with Pingora: https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
The CVE counts for new parts of the Linux kernel written in Rust are much lower than the (recent) CVE counts for even mature, years old parts of the Linux kernel written in C, to say nothing of the historical record.
Seatbelts are more comparable to activating bounds checking in C compared to switching to another language. because they are a minor inconvenience with a huge positive security effects, but do not eliminate all risk. Rust is more like switching to public rail system, because it does eliminate certain risks completely (all related to steering), but not others, and then also limits you in various way, so you may still need to use a car for the final meters (e.g. C or "unsafe" Rust).
And this comparison is consistent with the idea that the public rail system really is safer to use with regard to traffic accidents, so captures the idea that Rust actually has safety benefits. But IMHO the framing and marketing is highly misleading.
any serious engineering discipline eliminates the possibility of human error wherever possible, especially when it results in dangerous effects. and yet in software there are people who prefer exposing others to those dangerous effects because of a personal preference for a 50 year old tool.
Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
Apache and Mozilla Funds would be ideal choice for such open source implementation but I doubt they will do anything to extend their usefullness...
So: users require just visual part and html/css is not bad for that. Could be better. But transport part can be [re]done at will. Eg. would be nice to have separate layers for connection security and actual transportation.
https://portswigger.net/research/http-desync-attacks-request...
Unfortunately, there are still plenty of customers behind CDNs which don't support HTTP2 for backend origin connections.
There are many legitimate use cases for h2 (eg bidi streaming, latency improvements) and h3 (eg seamless network switching).
If you don’t need these features and just serve vanilla text webpages that’s also fine to just use h1.1. Blanket statements that h1.1 is better is just, wrong. Context is important on choosing what’s best for your environment.
Which, to be fair, is most sites.
It's like if I said "you can just use Linux 2.6, it's simpler"
Simple is good.
A lot of stuff that I avoided for years I finally feel like I really need to do now but it also has never been easier to do.
My box got hammered daily with port scans etc before.
Fortunately with Tailscale making it super easy to keep stuff out of the open internet, I don't have much, but even then I'm thinking of taking what I do have down. We see these big CVEs coming across, but I guarantee there's a large number of 0 days sitting around in peoples' stashes all over the world that we don't see.
Because clearly the patching treadmill is not sustainable, you will need a a second line of defense and reduce the blast radius of every individual vulnerability.
One, as the RCEs are coming out, admins are on edge and paying more attention, so more likely to catch the annoucement on their platform. "Oh, that's on PHP, we don't use this. Oh, wait, another one?"
Two, all of these hitting at once, while they may expand the surface for attackers, it also spreads their attention. They only have so much time and bandwidth. When a new shiny RCE hits, they need to consider "do we continue to try an prosecute the old one or jump on this bandwagon".
So, in the sense, it can actually slow down the actual potential impact of the revelations.
We can only hope this gives users time to get their systems up to date before they come knocking.
I mean, I know its a reach, but...who knows.
Only a little bit. An attacker will just hand this to an LLM with a "ping me when this is ready to use..." prompt.
I am running the official nginx docker image by itself in front of all my services, which are their own docker images. So it's easy for me to update just one of them. This works for me because I want my dedicated server to run all my projects with nginx in front. If I were running Discourse I would want it to not bundle nginx because I've already got one of my own. So that's interesting that they do that.
Seriously though, with how CVE disclosures are handled these days patching is a race against the clock
Then about 1/3 the way through implementing, I'm busy patching more CVE's.
The first step was closing contributions down to known contributors and maintainers. Maybe there’ll be a similar step where only a select group will be able to access — and attest to — the code?
Almost like “Restricted Access Source” or “Withheld Source”
That's to say that this is our new normal. Something that would have taken months or years to figure out can now take an AI hours if not minutes if you have enough compute.
It'll take a few years for the market to adjust. If nothing else, businesses are good at figuring out ROIs, eventually.
Cue xkcd
Only half joking.
> Error: API Error: Opus 5.5's safeguards flagged this session
Literally the first message, lol.
Otoh I'm not sure what http2 is good for (I've asked around today, some people said the multiplexing speeds things up on high-latency connections. I'm still working on a realistic benchmark since I couldn't find any but it's not looking great so far) and it's default disabled so comparatively few sites are affected. Getting rid of it is easier than waiting who knows how long for it to get packaged
There are also features on top of http2 that a downgrade may break: gRPC and Server Side Events for example. The later is compatible with HTTP1.1, but could fail due to connection limits, as multiplexing isn't only about speed. Some web apps may handle that so gracefully, it turns into a performance thing again, but others could halt and catch fire.
EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70b...
Given that on modern CPUs, cycles are quite fast, and memory latency is quite high, always-checked integer operations sound like an extremely good idea.
C is like a Winchester Model 1897, not saying that it is a bad gun, it is what we call it the good ol' Trench gun, as it ran over the war tunnels of the Great War, and it still packs a punch today, but it only has a half-cock for some level of safety, and there is no trigger disconnector either, so you can easily slam fire it. So it is way too easy to shoot yourself in the foot.
Nowadays we like going for a AKM/AK74, for something maybe more complicated but far more reliable at the same time. That's why we want Rust.
It is completely realistic that we start seeing a lot of bugs a human can’t (reasonably) exploit, but AI can.
Edit: All good. They released the new versions within 30 mins.
[1] https://launchpad.net/ubuntu/+source/nginx [2] https://tracker.debian.org/pkg/nginx
Given how common it's become to constantly scan popular repos for n-days, I expect someone will have already picked it up and started work on an exploit. The workaround for this (disabling HTTP/2) is also comparatively easy to what we had with PHP2Shell just a few weeks back, so I think it makes sense to release this.
We have people doing this on my side project and then reporting the fixed issues to try and get CVE 'credit'. Tiring.
Perhaps it was already being exploited?
We've literally decades of evidence that we cannot write software securely in languages that aren't memory safe, for all sorts of reasons (and it's wrong to look at it as a developer competence issue, even though sometimes that's going to be the case, even extremely experienced and highly competent engineers write memory safety bugs in critical code we rely on day in day out.). Not leaving yourself open to such problems is an entirely logical conclusion. We know a large percentage of vulnerabilities in software are caused by it[1]
Pick a language that doesn't have these risk factors, or pick an enforceable subset of one that makes such impossible.
"Insanity is doing the same thing over and over and expecting different results."
[1] https://www.chromium.org/Home/chromium-security/memory-safet... 70% of Chrome vulnerabilities caused by it. Similar stories for Microsoft products https://github.com/microsoft/MSRC-Security-Research/blob/mas... and Mozilla: https://hacks.mozilla.org/2019/02/rewriting-a-browser-compon.... Curl is at 40%, and dropping because Daniel Stenberg has provided various helpers that avoid it (and I believe they've now migrated almost everything over to them) https://daniel.haxx.se/blog/2023/12/13/making-it-harder-to-d..., so there are definitely ways to do it, we just... don't.
I mean sure if we close our eyes and ears we can ignore the elephant in the room that is proprietary, closed source software.
https://security-tracker.debian.org/tracker/CVE-2026-123007
Debian still vulnerable in all versions.
12 years on, OpenBSD's port of nginx still maintains its default chroot(2) patch.
https://raw.githubusercontent.com/rnagy/nginx_chroot_patch/m...
I recommend reading Reyk Flöter's AsianBSDcon 2015 paper about nginx and the thing that replaced it in base.
https://www.openbsd.org/papers/httpd-asiabsdcon2015.pdf
HTTP/2 again. First Apache, now NGINX
I rarely use HTTP/2, I rarely use a popular web browser, I use a TLS forward proxy that has HTTP/2 support but I prefer HTTP/1.1
I find that 1.1 is faster for the way I use the www (text-only, no images, CSS, Javascript, cookies, advertising, tracking), plus the 1.1 protocol comes from academia not Google, Akamai, etc.
Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).
We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.
We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:
Step 1: import socket # import the socket library from glibc or glibc.
Step 2: socket.socket() # socket.socket()
Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.
Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.
Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1
Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.
Thank you for reading my tutorial, and best of lucks.
In fact in general there were waaay less CVEs for Caddy lately compared to nginx
I guess it was the right move.
See also: https://queue.acm.org/doi/10.1145/2732266.2716278
This webserver is not vulnerable to PHP2Shell nor this TLS issue, because it's completely userland PHP:
https://github.com/Qbix/webserver
It's now been battle-tested in production, and also supports websockets, etc.
(However, it doesn't support sendfile, because PHP doesn't have support for that, so it's not as fast as nginx for sending static files. This shouldn't make a difference, though, if you use a CDN in front.)
Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.
And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.
Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.
Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.
Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.
We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
Theh people laughing at Anthropic and co for being cautios of Mythos just didn't understand whats happening.
Instead of finding, hiring and then finding security issues, you can throw money and compute at it. This alone is a complete game changer. Combined with models which are very very capable.
This can be potentially the death of certain programming lanugages or requirement of a complete rewrite. It can mean that only big companies have the resources and capacity to continue playing this game.
If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
I'm not saying domain knowledge is not required. But a lot of people do have domain knowledge and are prevented from using these tools. Even to secure own code bases. This sucks. A lot.
Challenge accepted.
This is a monumentally dumb assertion that any Job Blow can just add a credit card and ram through these requests
So much cope on this site. Llms havent replaced the requirement of studying and becoming an expert.
But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.
Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.
People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..
Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.