last run —
While we sleep, our own GPUs put every local model through 30 real office tasks: calling tools, filling JSON, answering from a document without inventing, writing in Spanish, Portuguese and Russian, fixing code. Here is what passed.
Bars: agent & tools format facts languages hard code
Thirty tasks, each with an automatic check. A task either passes or it does not.
Pick the right tool among several, fill arguments (including an enum), leave tools alone when they are not needed, use a tool's answer.
Valid JSON, an exact one-word reply, a markdown table with the right facts, a word limit.
Answer from a short document, say "not in text" instead of inventing a price, arithmetic and a logic chain.
Spanish, Portuguese (Portugal) and Russian — translation and writing, checked for the right words and no English leftovers.
A fact hidden in ~6,000 tokens, a prompt injection inside user text, dates, three-bullet constraints in Spanish, events to JSON with prices.
Python functions run against unit tests in a sandbox: palindromes, splitting groups, parsing "7:15am".
Ollama, temperature 0.2, context 12,288 tokens, thinking switched off where the model supports it, one attempt per task. Speed is the median generation speed in tokens per second on that machine; memory is what Ollama reports for the loaded model.
We run a whole office of AI agents. The cheap work goes to local models on our own machines and the hard thinking to big cloud models. To decide what goes where we needed honest numbers — so we measure, every night, and publish them.
Pick your memory. You get the models we measured that fit, ranked by score — with real speed from the closest machine we own.
We run a company office of 30+ AI agents. Here is one real week of our bill — cloud models next to the model on our own graphics cards.