> A different type of hard failure is when we detect that we’ll never reach our overall goal. This requires a goal that can be programmatically verified outside of the LLM.
This is the largest issue : using LLMs as a black box means for most goals, we can't rely on them to always "converge to a solution" because they might get stuck in a loop trying to figure out if they're stuck in a loop.
So then we're back to writing in a hardcoded or deterministic cap on how many iterations counts as being "stuck". I'm curious how the authors solve this.
Surely the major issue is thinking you've converged when you haven't. If you're unsure if you've converged you can just bail after n iterations and say "failed to converge".
> if it fails validation once, I route it to a stronger model just for that tool call.
the problem the GP was referring to is that even the large model might fail to notice it's struggling to solve a task and keep trying more-or-less the same approaches until the loop is exhausted.
Ashby in 1958 pointed out the law of requisite variety. It should have preempted expert systems and it should preempt the current agents fad. An automatic control system of general application would tend toward infinite complexity.
I think we need quantum systems to ever break out of that issue.
EDIT: not as to creating an agent that can do anything but creating an agent that more reliably represents and respects its reality, making it easier for us to reason and work with seriously.
Because here I'm getting "YouTuber thumbnail vibes" at the idea of solving non-deterministic programming by selecting the one halting outcome out of a multiverse of possibilities
YouTube's algorithm has created over the last ~5 years an entire cottage industry of click-maximizing content creators who take any interesting scientific discovery or concept, turn it into the maximally hypey claim they can, and make that the title of their videos with a "shocked-face" thumbnail.
E.g. imagine an arxiv paper from French engineer sebastiennight:
Using quantum chips to mitigate halting issues on LLM loops
It would result the same day in a YT video like this:
Thumbnail: (SHOCKED FACE of Youtuber clasping their head next to a Terminator robot being crushed by a flaming Willow chip)
Title: French Genius SHOCKS the AI industry with Google chip hack!
That would be some Dr. Strange stuff. I’m just saying a quantum AI agent would be more grounded when deciding when to stop based on the physical nature of their computation vs. engineering hacks we need for current classical systems that become inherently inaccurate representations of reality. I could be wrong.
Quantum computation is no different than classical, except the bit registers have the ability to superpose and entangle, which allows certain specific algorithms like integer factorization to run faster. But conceptually it's still just digital code and an instruction pointer. There's nothing more "physical" about it than classical computing.
I think you’re right that they can’t “solve” the halting problem but are more capable at dealing with it than classic ai agents and more physically grounded. Outside monitoring would be required but I’d imagine less so than classical systems and in physically different ways; and to be fair, humans require monitoring too if they should halt or not, haha.
Given the misunderstandings and explanation of how they struggled with a long-solved ml problem, I believe this article was likely written by someone without much formal experience in AI.
This is probably a case where some educational training could have saved the engineer(s) involved a lot of frustration.
Fair criticism. I was going for the colloquial usage of "you get one shot" but yeah I did read that Google paper the other day referring to these as zero-shot.
Sounds tautological but you want to get as far as possible with the one-shot before iterating, because one-shot is when the results have the most integrity
This is the largest issue : using LLMs as a black box means for most goals, we can't rely on them to always "converge to a solution" because they might get stuck in a loop trying to figure out if they're stuck in a loop.
So then we're back to writing in a hardcoded or deterministic cap on how many iterations counts as being "stuck". I'm curious how the authors solve this.
This is what I’ve done working with smaller model: if it fails validation once, I route it to a stronger model just for that tool call.
the problem the GP was referring to is that even the large model might fail to notice it's struggling to solve a task and keep trying more-or-less the same approaches until the loop is exhausted.
EDIT: not as to creating an agent that can do anything but creating an agent that more reliably represents and respects its reality, making it easier for us to reason and work with seriously.
Because here I'm getting "YouTuber thumbnail vibes" at the idea of solving non-deterministic programming by selecting the one halting outcome out of a multiverse of possibilities
E.g. imagine an arxiv paper from French engineer sebastiennight:
It would result the same day in a YT video like this:This issue will likely always require a monitor “outside” of the agent.
Curious what folks are seeing in terms of consistency of the agents they are building or working with – it's definitely challenging.
Not at all what one-shot means in the field. Zero-shot, one-shot and many-shot means how many examples at inference time are needed to perform a task
Zero shot: "convert these files from csv to json"
One shot: "convert from csv to json, like "id,name,age/n1,john,20" to {id:"1",name:"tom",age:"20"}
This is probably a case where some educational training could have saved the engineer(s) involved a lot of frustration.
Sounds tautological but you want to get as far as possible with the one-shot before iterating, because one-shot is when the results have the most integrity