Backtesting Arena

Backtesting Arena

Back to blog

Building for AI Agents: 10 Lessons From Running 73 MCP Tools

Your most thorough reader gets the least to read. Ten places where building for agents differs from building for people, with numbers from production.

Backtesting Arena·July 29, 2026·10 min read·0 views
Building for AI Agents: 10 Lessons From Running 73 MCP Tools

Your most thorough reader gets the least to read.

Someone asks their AI assistant: which strategy is the best one? The assistant calls our tool, gets a list back, and answers: the one at 340 percent.

On our website, a line above that exact list says it's the top of 4,500 tested combinations. That line never reached the user. It was on the page. The assistant never saw the page.

This isn't a story about bad AI. The assistant did exactly what we gave it. We just didn't give it everything.

Who this is for

If anything you build gets read by a machine — an API, a set of docs, a product feed — the rest of this applies to you. We write about MCP here because that's the server we run. But any API whose responses end up inside a language model has the same ten problems.

The sentence underneath all of it:

Everything a human absorbs in passing, without clicking, you now have to ship.

An agent that fetches a page reads all of it, by the way — more thoroughly than any human skimming a headline and forming a view. The problem isn't how it reads. It's what reaches it.

That an agent interface is essentially a user interface for a non-human user was put well by Philipp Schmid in January 2026. The framing isn't new. What's added here are numbers from production: 73 tools, and ten things that blew up on us.


1. The caveat has to be in the response

What isn't in the response doesn't exist.

Back to that leaderboard. On the website, the framing sits above it. The visitor may not read it consciously — but it's in their field of view while they look at the table.

The agent gets ten records. The heading isn't a record.

So we moved those sentences to where this reader actually gets them: into the tool's own description. Three we genuinely needed:

  • A leaderboard is a selected extreme, not an average.
  • Comparing ten strategies means you tested ten times — and the result has to be discounted accordingly.
  • Our list of coins isn't assembled the way it would have looked in 2019. So don't pretend you knew back then which ones would stay liquid.

None of these warnings was new. They all existed somewhere. Just not where this reader looks.


2. Three kinds of "nothing found"

Turn "no results" into three different answers.

Here the agent actually has the advantage — if you let it.

A human sees an empty table and pauses. Was the query wrong? Does the data not exist? Is the window too short? They guess. Usually right.

An agent doesn't guess, as soon as there's a dedicated field. It acts on "sample too small" reliably — more reliably than a human notices a qualifier in passing.

Provided the field exists. So we separate:

  1. There is no data.
  2. The question doesn't fit here.
  3. There is data, but not enough of it.

The third matters most. Below thirty observations, something is an anecdote, not a finding. As a field, that survives the trip to the user. As a clause, it doesn't.


3. Hand-maintained numbers break now

Any number you maintain by hand will eventually be asserted wrongly.

A human who sees a number in the docs distrusts it slightly. Docs lag, everyone knows that. In doubt they check the interface.

An agent has no second place to check. It takes the number and passes it on, confidently, to its user.

We had a single number — how many tools we offer — maintained by hand in thirteen places. It had drifted into four different values at once. Five of those places were public.

The fix is boring: don't transcribe, compute. A small script starts our tool list and counts it. The list is the truth; everything else follows.

We gave ourselves two rules doing it. The counter must not need a password — otherwise it won't run automatically, and eventually won't run at all. And it must not actually execute anything. It counts, it doesn't call.

Incidentally: the number in this article's headline is one of those places. It gets counted.


4. The check that waves things through in silence

A check that finds nothing must get loud — not quiet.

We made this mistake on the first attempt, and it's meaner than the one before.

We had a script checking all thirteen places. Except: when a place gets rewritten and the script no longer finds it, it doesn't report an error. It reports nothing. The check glows green while the exact spot it was protecting drifts away.

The rule from that: if a check can't find a place it expects, that's a failure — not an "all clear."

This applies to any automated check, anywhere. It just shows up harder with agents, because nobody stumbles across the problem by accident any more.


5. You can't just ship everything

Whatever you ship, someone else pays for.

The obvious objection to all of the above: if the response is the bottleneck, put everything in it.

You can't. An agent has limited working memory for the job in hand. Every tool description it loads takes a piece of that — and that piece is gone for the actual task.

The awkward part: you notice none of it. Your response times stay the same. Your error rate stays the same. The agent just gets slightly worse, and nobody knows why.

So "put every caveat in every response" isn't a solution, it's a transfer. The work is deciding which caveat genuinely changes the outcome — and which just costs room.

The same goes for the number of tools, and here it gets uncomfortable for us. Workato's design documentation (as of April 2026) recommends five to eight per server, calls eight to twelve reasonable, and advises splitting above fifteen. We have 73.

The core of that is right, and better argued than most: the count is a budget. Clarity beats completeness. A server whose purpose can't be said in one sentence is too broad. We wouldn't dispute any of it.

Our qualification: the count alone measures the wrong thing. Of our 73, seventeen come in pairs — seven cases where one tool returns today's value and a second returns the same value over time, plus four that differ only in trigger. Those are consolidations with nothing lost. 73 tools with seventeen duplicates is a different thing from 73 distinct capabilities. And both are different from fifteen.

Splitting also only helps if the user subsequently loads less. Connect to both servers, and the same descriptions sit in memory, just spread across two entries.

So the useful target isn't a ceiling. It's: no two tools doing the same thing for two time ranges.

Why we still haven't done it is point 8.


6. The name is all it sees

Write down what your tool is not for, too.

The next two points bite before anything gets read. The best response doesn't help, because there isn't one yet.

A human decides from context: where the button sits, what's beside it, which page is open. An agent has a name and a description.

Similar-sounding tools therefore get confused, reliably. For us that was the "today's value" and "value over time" pairs — and four tools for starting a test that differ in purpose and scope but sound alike.

What helped wasn't describing the function better. It was describing the boundary. Not just "returns X," but "returns X for today — for the series, use the other one."

The sentence saying what something is not for is often the more useful one.


7. Empty fields get filled, not queried

A field with no description gets guessed.

A human who doesn't know what a field expects asks, or tries it and reads the error. An agent fills it in.

Taking stock, we found 52 parameters with no description at all. That's 52 invitations to invent: a unit, a default, a plausible ceiling.

Every field needs three things: unit, default, bounds.

"Days" isn't enough. "Length of the series in days, default 365, allowed range 90 to 1095" is.

Incidentally, because it belongs here: on measuring, six of our own findings turned out to be false alarms. Our checking rule was too crude. We fixed the rule instead of rewriting correct text. Duller, but the right way round.


8. There's no banner that reaches it

To an agent, a rename is an outage with no warning.

With a website you can announce a change. Banner, notice, grace period — people see it next visit.

Someone running an agent set the connection up once and never looks again. There's no surface where you reach them. The day you rename a tool is the day their automation silently stops working.

That's exactly why we have not done the consolidation from point 5, even though we think it's right. It ships with a grace period: old and new name run side by side for a while, the old one flags the change, and only then does it go.

That's more work than a cutoff date. But a cutoff date isn't a transition — it's an outage you inflict on users who never saw it coming.


9. Licences reach further than you'd think

Displaying is an endpoint. An interface is a starting point.

A data licence letting you show something on your site doesn't automatically let an agent carry those numbers into somebody else's product.

And it carries them across completely. It takes the series as it comes and deposits it in a response, a file, another service.

So we check licences per delivery path, not per data source. The same number can be fine on the website and not fine through the interface.

It's caught us twice on sources we'd assumed were harmless. Both times because somebody read the terms instead of the summary of the terms.


10. You can't test against yourself

You fill the gap in your head. Measure instead.

The point that holds the others together.

You know what your tool does. So every description you write reads as complete to you. You don't see the gap, because you fill it automatically as you read. Every time.

Proofreading doesn't fix that. Measuring does.

We measure three things, all blunt and all effective:

  • How many fields have a description at all?
  • How long is the shortest one?
  • Where's the average?

The shortest is the most informative. A healthy average hides the one tool sitting there with half a sentence — and that half sentence is where an agent takes the wrong turn.


What this doesn't mean

This isn't an argument for documenting everything twice. Point 5 explains why that doesn't work.

And it explicitly doesn't mean agents are the worse readers. The opposite: an agent doesn't skip a paragraph because it's in a hurry. It doesn't form a view from the headline. At reading, it beats most humans.

The problem sits one level earlier. Not how things are read, but what arrives at all. And that's not the reader's decision, it's yours — with every field you put in the response, or don't.

Which leaves the uncomfortable part. Where your caveats sat used to barely matter, as long as they sat somewhere: the human eye picks up things at the margin. That generosity is gone.


The short version

  1. What isn't in the response doesn't exist.
  2. Turn "no results" into three different answers.
  3. Any number you maintain by hand will eventually be asserted wrongly.
  4. A check that finds nothing must get loud — not quiet.
  5. Whatever you ship, someone else pays for.
  6. Write down what your tool is not for, too.
  7. A field with no description gets guessed.
  8. To an agent, a rename is an outage with no warning.
  9. Displaying is an endpoint. An interface is a starting point.
  10. You fill the gap in your head. Measure instead.

The cheapest first step is number 10: count how many of your fields have a description. An afternoon's work, and you'll know how big the problem is for you.

Frequently asked questions

How many MCP tools are too many? Published guidance lands at five to eight per server, with a splitting threshold around fifteen (Workato, April 2026). Sensible as a starting point. We'd still call the raw count the weaker metric: what matters is how many of your tools could be consolidated without losing a capability. And splitting only helps if the user subsequently loads less.

Does this only apply to MCP? No. It applies anywhere a program receives a response rather than a page — including an ordinary API whose responses end up inside a language model. It applies less to an agent that actually fetches your website: that one sees the page, and reads it more thoroughly than most visitors.

Do I have to write my caveats twice now? Not twice — where this reader gets them. And there, only the ones that genuinely change the outcome. The rest costs room somebody else pays for.

How long should a description be? What's in it matters more than length: what the tool is for, what it's explicitly not for, and per field a unit, a default and bounds. Measure the shortest, not the average.


Sources: Workato, "MCP server design best practices," as of April 2026. Philipp Schmid, "MCP is Not the Problem, It's your Server," January 2026. All production figures from our own interface, as of July 2026.

Try it yourself

Run the backtest with your own parameters and time ranges.

Run backtest →
📬

Don't miss new blog posts

One short email per new post — strategies, backtests, market analysis. No spam, unsubscribe with one click anytime.

By subscribing you accept our privacy policy. We use Resend for delivery. Double opt-in confirmation required.

Comments (0)

Join free to post comments.

Sign up →

No comments yet. Be the first!