Great post. I particularly like the 4 column pr view idea. I’ve built something similar to review speckit specs. The scaffolding to ensure that code actually works makes sense too.
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
>> A more serious irony is that the automatic control system has been put in place because it can do the job better than the operator, yet the operator is being asked to monitor that it is working effectively.
Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
Jumping into article, I assumed that the author was going to describe how he enjoys, by drawing it out as far as he can,
the process of surrendering to the machine god..
(Not the machina diabol, as it sadly turned out to be, though some BDSM zealots might disagree.. the things that bureaucracy does to us are mild in comparison? )
But.. we do monitor whether our prayers are effective .. right? Is that a smitable offense?
After that disappointment, I too was "curious": was the author going to talk about "curiosity", that magic virtue that we hackers believe, lifts design into the divine (but more often than not kills the pussy)
Except experience shows they aren't. Experience shows that when humans are just pressing "ok" 99% of the time they'll press it 100% of the time and completely miss the other cases.
"Yup, looks good to me!" said the human reviewing thousands of lines of generated code everyday.
This has become common now, and I've seen it ruin large projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
> ..the possibility of runaway dynamics: once a critical threshold is crossed, small increases in adoption can trigger rapid population-level shifts toward *persistent dependence, with abrupt losses in cognitive competence*.
We can all see it happening. As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
Even if the human's judgement is correct in every LGTM, there's a bias towards merging. Each merge that adds a feature but degrades understandability 1% will be merged. It won't be held back until it improves it by 5% instead. After 100 merges you have degraded understandability 100% even though nobody made an error.
You’re completely right that nobody is stopping a PR that just ever so slightly decreases understandability of something. It may not even be obvious to the reviewer that it’s slightly less understandable overall… Doubly so if they’re just skimming due to the volume, and tripley so if they’re having agents do the most of the review, which lets the human reviewer’s guard down.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
I hope there's still value to be found in a bunch of developers standing around a whiteboard and talking, and I think this scenario is where their ability and understanding really shows these days.
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
To be fair, models are more accurate than how they were earlier this year. In march I was still frustrated and was writing my own event loops because the model simply wasn't built for it. By January I was still hand checking statistical outputs to make sure it wasn't making any errors, and it was. After august it was pretty much over. There were no stats errors, code generated was shit but it could do what is asked. Instruction following is still a problem, and I sometimes catch silly things in the scrolling diffs, but it has been much easier since then.
The most difficult part of LLMs and agents seems to be - surprisingly - humans.
Specifically, humans and their dysfunctions. Their fragility. Their ego tied to stuff.
This is an excellent post/reflection on the topic of solo LLM use.
The only gap I see is that it assumes that one works on something alone and not in a team. Though, that might be a different essay in itself.
Edit: Turns out that footnote 3 actually mentions that already. Nice. Good post
i sometimes wonder how all carpenters or music producers in the world would react if tomorrow a magic box arrives where they type-in/dictate what they want and voila... out comes a table/composition.. Will the worlds music producers and carpenters just "accept" this new magic genie? I would hope they wont.
Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
> Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
It's worth remembering that, contrary to what HN acts like, software engineers are not uniformly accepting of LLMs. They are still quite controversial in the industry. Now management, on the other hand, is completely sold on the supposed benefits of LLMs. So many engineers feel forced to use them, but it's not enthusiasm on their part which drives them to that.
I mean more or less to an extent if you expand by what you mean by "out comes". With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours. Almost equally as magically as well as disruptive. People still use carpenters to make tables, but for the majority, the ease of same-day delivery and cheaper prices are worth it for a sacrifice in personalization and character.
Making a quick prototype is probably the best way to find out if an idea is worth keeping. Vibe-coding it and then throwing it away seems like a good way to go?
Yeah, definitely. Also at times I've vibecoded successive iterations with better designs after seeing where the first ones collapsed -- the agents do a good job with a template and need much less prompting for the successive versions.
I am often thinking recently of the littel loop diagrams from the "The Capability Trap" paper. We need to keep a loop reinvesting into our own Human capability in anything we're doing.
This is copium that is gone with the wind when the models perform up to the level of standard that the user requires. I'll give the author 6 months before they feel like its okay to just send it.
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
Writing is hard, but what I was trying to say is that it's okay to just send it as long as you're doing it consciously and intentionally (and I do that frequently enough); but if it's not an explicit choice that can be problematic.
Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved. You then advocate to these users, make them feel important, make them your cash cows, rinse and repeat.
Every team I have been on have been more successful when they gave up on the product ideals and just ensured that the customers were hooked and satisfied.
> Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved.
I care, because doing a sloppy job and seeing if people will tolerate it is not acceptable engineering practice. It's not engineering at all. More pragmatically, if you do the thing you are suggesting, you won't have users for much longer. They will abandon you and go to someone who actually makes good products.
I don't believe this in practice. I am at a FAANG-like firm, and we are just maxxing the engagement and people use the product more and more the more we serve stuff to keep them hooked using their data and behaviour.
The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.
I think there is a small subset of technical nerd/idealists who think that quality is all that matters but they are in their own little sandbox while 97% of users are in another one.
Why would Prime be in that list. He had an entire arc where he tried to make it work and then went back to the drawing board.
I am not saying he writes all code by hand, of course. Just that he is furthest in figuring out some sort of mid point between full comprehension and cognitive surrender.
I have missed the more recent content, so perhaps I have missed something, but this was my read last I watched.
I worked for one of the largest device manufacturers for a certain kind of sensor.
The requirements/standards for this sensor was made by this company 20 years ago. All the tests and requirements were made by this same company in a way that would minimize the costs for us to keep making new revision we could sell, and make it the hardest for any competitor to enter the market. Another competitor started with a better product abroad, so we copied it, but it didn't pass the tests, so then they installed an owner of the certification house that was a cousin of one of the VPs. No problems passing tests after that.
You think this is all controlled and nice and standardized but it's all just a market from software to hardware.
While quite believable, there is a diversity of industries and standard adherence / enforcement no?
Markets also range from under regulated to over regulated, and are usually in a transition from one end to another (while contending with other forces).
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
But.. we do monitor whether our prayers are effective .. right? Is that a smitable offense?
After that disappointment, I too was "curious": was the author going to talk about "curiosity", that magic virtue that we hackers believe, lifts design into the divine (but more often than not kills the pussy)
This has become common now, and I've seen it ruin large projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
> ..the possibility of runaway dynamics: once a critical threshold is crossed, small increases in adoption can trigger rapid population-level shifts toward *persistent dependence, with abrupt losses in cognitive competence*.
We can all see it happening. As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
Specifically, humans and their dysfunctions. Their fragility. Their ego tied to stuff.
This is an excellent post/reflection on the topic of solo LLM use. The only gap I see is that it assumes that one works on something alone and not in a team. Though, that might be a different essay in itself.
Edit: Turns out that footnote 3 actually mentions that already. Nice. Good post
It's worth remembering that, contrary to what HN acts like, software engineers are not uniformly accepting of LLMs. They are still quite controversial in the industry. Now management, on the other hand, is completely sold on the supposed benefits of LLMs. So many engineers feel forced to use them, but it's not enthusiasm on their part which drives them to that.
Sure but that doesnt require the person to be a carpenter though right? Anybody could do that.
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
However it takes more mental energy and people don't like that
I guess you could ask the model. "Yes well done, you're a visionary software engineer".
Every team I have been on have been more successful when they gave up on the product ideals and just ensured that the customers were hooked and satisfied.
I care, because doing a sloppy job and seeing if people will tolerate it is not acceptable engineering practice. It's not engineering at all. More pragmatically, if you do the thing you are suggesting, you won't have users for much longer. They will abandon you and go to someone who actually makes good products.
The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.
I think there is a small subset of technical nerd/idealists who think that quality is all that matters but they are in their own little sandbox while 97% of users are in another one.
I am not saying he writes all code by hand, of course. Just that he is furthest in figuring out some sort of mid point between full comprehension and cognitive surrender.
I have missed the more recent content, so perhaps I have missed something, but this was my read last I watched.
The lawyers and judges who trusted an ML system to be up to snuff are currently pounding sand.
The requirements/standards for this sensor was made by this company 20 years ago. All the tests and requirements were made by this same company in a way that would minimize the costs for us to keep making new revision we could sell, and make it the hardest for any competitor to enter the market. Another competitor started with a better product abroad, so we copied it, but it didn't pass the tests, so then they installed an owner of the certification house that was a cousin of one of the VPs. No problems passing tests after that.
You think this is all controlled and nice and standardized but it's all just a market from software to hardware.
Markets also range from under regulated to over regulated, and are usually in a transition from one end to another (while contending with other forces).