I want to preface that I've done LLM evals in a proper setting before. This is far sillier. It's derivative of the "pelican riding a bicycle" test, something simple that shows how capable an LLM is at perceiving intent, dealing with physics and general aesthetics. I run this for every new model now and it gives me a decent idea of what to expect.
The setup needs more description. Can it have pre-installed npm, npx, chrome, playright etc.? It turns out my agent did not have access to it and it is setting all of that itself. Will update when done.
3 comments