This article was co-authored with generative AI. Facts have been checked against public documentation where feasible, but errors may remain. Please verify primary sources before relying on this for important decisions.

I added a Router Agent — in which the LLM looks at the type of question and automatically picks the right "tool" — to a vector-search RAG (retrieval-augmented generation) over a corpus of classical documents in TEI (Text Encoding Initiative)/XML format (on the order of 5,000 items). This article is an end-to-end record of that design (an 8-tool setup) and the 5 traps I hit while getting it to run on a stack of Cloudflare Workers + Azure OpenAI + Vercel AI SDK + TiDB Cloud Serverless.

In particular, the conclusion most strongly confirmed by this implementation was that "vector RAG and structured data are not competitors but complements."

Why a Router Agent: the territory vector RAG alone cannot reach

I received feedback in advance on the existing vector RAG. Specifically, recall was low for exhaustive queries like "a list of all book titles that appear in the corpus," and for attribute-filtering queries like "references to books written by a particular author."

In fact, when I collated a little over 50 hand-annotated book titles against the RAG output, recall was 17–19%. Concrete failure patterns looked like this:

  • Attributing the author of a work A to a different writer B (a contemporary but a different person)
  • A work appears in the body text under 7 variant forms (abbreviations, alternate names, common names, etc.), but vector search picks up only 1 or 2 of them
  • Three works by a particular author are mentioned in the body text, but all of them are missed

This looks less like a quality problem with the individual RAG and more like a structural limitation of vector similarity search. Exhaustiveness, attribute matching, and reconciling variant name forms are inherently hard for cosine similarity. Meanwhile, the TEI side of the corpus already has about 20,000 person-name tags, about 10,000 place-name tags, and about 5,000 organization-name tags, plus a seed of a little over 50 book titles from manual annotation.

In other words, if there were a mechanism to "route queries that vector search is bad at over to structured-data tools," each side's strengths could shine. That is the starting point for the Router Agent.

Designing the 8-tool setup

Using an LLM (a reasoning-class model) as the router, I let it automatically pick from the following 8 tools according to the question. The implementation uses Vercel AI SDK v6's tool() + JSON Schema, with 1 tool = 1 file (src/lib/agent-tools/<name>.ts) to minimize conflicts during parallel editing.

#ToolOriginGood at
1search_diary_ragVector (text-embedding-3-large)"thoughts about ○○", "what things were like around year ○○"
2search_diary_structuredStructured (persons / places / dates)"interactions with △△", "records of visiting ○○"
3fulltext_diaryFull-text LIKE"references to a specific work"
4list_books_by_authorBook-title seed"records of reading a specific author's books"
5list_all_titlesBook-title seed"a list of books in the corpus"
6find_book_aliasesBook-title seed"all variant forms of the same work"
7suggest_follow_upsAI automaticadditional suggestions after answering (2–4)
8ask_clarificationAI automaticasking back on overly vague questions

Of the 8 tools, 5 originate from structured data or manual annotation. The overall design philosophy is to cover "answers that vector search alone cannot produce" with tools on the structured-data side.

The router itself uses streamText + toolChoice: "auto" + stepCountIs(8) to allow multi-step calls (retries, using multiple tools together). The system prompt explicitly states retry guards such as "if the result is thin, try a different tool" and "do not repeat the same tool with the same arguments."

Behavior examples

I verified the post-implementation behavior across 4 scenarios.

① "Please list the book titles that appear in the corpus" (exhaustive)

Previously, vector search retrieved 15 items, only 17–19% of the 50+ manual seed titles. This time the AI judged "this is exhaustive, so RAG won't do," called list_all_titles once, and listed all de-duplicated book titles at once.

② "What records are there of reading a specific author's books?" (attribute filtering)

Previously it missed all 3 books and answered "no clear records found" (or mistakenly attributed another writer's work). This time the AI retrieved titles and entry_ids with list_books_by_author({author: "..."}), ran fulltext_diary in parallel for each title, and integrated the results. It correctly listed all 3 books with their dates.

③ "I want to know all mentions of a certain work" (variant-name reconciliation)

Previously only 1 item was picked up under the normalized title. This time the AI judged that "variant forms may be scattered," used find_book_aliases to retrieve the entry_ids for all 7 variant forms at once, and integrated them.

④ "Tell me about ○○" (vague)

Previously it would force some sort of answer. This time the AI judged "this is too broad to search on" and now returns 1–3 questions to the user via ask_clarification.

The 5 traps hit during implementation

Here is the main topic. The "design" itself is as above, and writing it out takes only a few paragraphs, but the traps I hit before getting it running were more numerous than expected, so I'll record them. Consider it a minefield map for anyone touching the same stack.

Trap 1: Cloudflare Workers' 3 MiB limit

I got CI passing, and then deployment gave:

Your Worker exceeded the size limit of 3 MiB.
Please upgrade to a paid plan to deploy Workers up to 10 MiB.

Looking at the size of handler.mjs, it was 8.4 MB. Gzip-compressed it came to just under 3 MB, apparently exceeding the Free plan's limit.

Scanning node_modules, the openai package stood out at about 9 MB (v6.38's unpacked size). It was used only for embed() and getChatClient(). Meanwhile @ai-sdk/azure is only 184 KB. If both hit the same Azure OpenAI, it seemed I could get by with the ai-sdk side alone.

The refactor:

  • Removed the AzureOpenAI import from openai
  • Replaced embed() with @ai-sdk/azure + ai-sdk's embed() function
  • Replaced the planner's .chat.completions.create({ response_format: json_object }) with generateObject() + a zod schema
  • Removed openai from package.json

Result:

uncompressedgzip
Before8.4 MB~3 MB (exceeded)
After6.7 MB1.72 MB (54% of the limit)

Comfortably within the Free plan.

Trap 2: The Azure OpenAI URL pattern (v1 Responses API vs. legacy)

Having removed the openai SDK and unified on @ai-sdk/azure, I then hit a different error:

API version not supported
url: '<endpoint>/openai/v1/responses?api-version=...'

@ai-sdk/azure apparently defaults to the newer /openai/v1/... URL (the Responses API). Because this is the new API it requires a newer api-version, and the 2024-10-21 I was specifying is not supported.

At first I tried useDeploymentBasedUrls: true to revert to the legacy URL (/openai/deployments/{id}/...), but with this combination (legacy URL + old api-version), generateObject apparently doesn't work and returned 404.

Conclusion: the right answer was to not specify apiVersion at all (leave it to the SDK's internal default). Another file using the same @ai-sdk/azure had been working from the start without specifying apiVersion, and I just needed to match that behavior. Note that @ai-sdk/azure's type-definition comment says "Defaults to preview," but in practice the SDK seemed to resolve it internally to "v1" (this may change by version).

// Before (doesn't work)
createAzure({
  resourceName: ...,
  apiKey: ...,
  apiVersion: process.env.AZURE_OPENAI_API_VERSION ?? "2024-10-21",
});

// After (works)
createAzure({
  resourceName: ...,
  apiKey: ...,
  // no apiVersion → leave it to the SDK's internal default
});

As a lesson, I subsequently centralized Azure provider creation into a single file. Having the same settings scattered across multiple files means one of them gets left stale, which is exactly how this kind of inconsistency arises — something I learned firsthand.

Trap 3: Structured search mass-producing 0 results (JSON_CONTAINS blind spot)

Searching for person names with search_diary_structured returned 0 results across the board.

Investigating, the <persName corresp="..."> on the TEI side apparently used surname-only abbreviations predominantly.

correspoccurrences
Surname-only tag Amany
Full-name tag A'few

JSON_CONTAINS(persons, JSON_QUOTE("full name")) is an exact match, so it was completely missing the surname-only tags.

The fix was three-pronged:

  1. Changed the SQL to JSON_SEARCH(persons, 'one', CONCAT('%', ?, '%')). The %...% partial match picks up both surname and full name. I first tried bidirectional fuzzy matching with JSON_TABLE, but TiDB explicitly documents JSON_TABLE as an unsupported function in its official docs, so it was rejected, and I settled on simple partial (contains) matching only.
  2. Tell the LLM in the tool description to "prefer surname only." "Because the tags are mostly abbreviations, pass just the surname rather than the full name."
  3. Added an honorific/title stripper. Trailing honorifics like "○○男爵" (Baron ○○), "○○公" (Duke ○○), "○○翁" (Elder ○○) — 男爵 / 子爵 / 伯爵 / 侯爵 / 公爵 / 公 / 翁 / 侯 / 候 / 氏 / 君 / 殿 / 先生 / 博士 … — are removed with stripHonorific() before searching.
const HONORIFIC_RE = /(?:閣下|大臣|長官|大使|総裁|社長|頭取|主任|博士|先生|男爵|子爵|伯爵|侯爵|公爵|公|侯|候|翁|殿|氏|君|様|さん)$/;

export function stripHonorific(name: string): string {
  let cur = name.trim();
  for (let i = 0; i < 3; i++) {
    const stripped = cur.replace(HONORIFIC_RE, "");
    if (stripped === cur || stripped.length === 0) break;
    cur = stripped;
  }
  return cur || name;
}

These three steps improved searches for "○○男爵"-type names from 0 results to over 30.

Trap 4: Reasoning-model multi-turn (msg_/rs_ chain breakage)

Trying multi-turn conversation with the agent that had finally started working:

Error: Item 'msg_09e0e...' of type 'message' was provided without
its required 'reasoning' item: 'rs_09e0e...'

Reasoning-class models return assistant output as a (msg_*, rs_*) pair. The Azure Responses API apparently requires this pair to be sent together in multi-turn, but the front end's prepareSendMessagesRequest, which resends history, was passing only the text part, so only msg_ was sent with no rs_, and it was being rejected.

// Before
parts: m.parts.filter((p) => p.type === "text"),

// After
parts: m.parts.filter(
  (p) => p.type === "text" || p.type === "reasoning",
),

The tool-call parts (which are heavy) continue to be dropped, but reasoning is kept. This preserves chain consistency when a follow-up is clicked. Non-reasoning models don't produce a reasoning part, so there's no impact.

Trap 5: The TPM (Tokens Per Minute) bottleneck

Rate limits were frequent, so I checked Azure OpenAI's capacity and found the deployment used by the agent and planner was throttled to 10K TPM. The agent's multi-step tool loop uses 20–30K tokens per request, so it was hitting the rate limit immediately.

az cognitiveservices account deployment list \
  --name <resource-name> \
  --resource-group <rg>

Looking at the other deployments on the same account, I discovered a different reasoning model waiting unused on a different resource at 100K TPM. Its quality is also better for the agent's use case.

So I added agent-specific env vars (AZURE_OPENAI_AGENT_ENDPOINT / AZURE_OPENAI_AGENT_API_KEY / AZURE_OPENAI_AGENT_DEPLOYMENT) to the Azure provider helper, and pointed only the agent at the high-TPM resource.

export function getAgentModel() {
  return createAzure({
    resourceName: resourceName(agentEndpoint()),  // a different resource
    apiKey: agentApiKey(),
  })(
    process.env.AZURE_OPENAI_AGENT_DEPLOYMENT ??
      process.env.AZURE_OPENAI_CHAT_DEPLOYMENT!,
  );
}

An asymmetric setup: the agent on the high-TPM side, the planner on the low-TPM side. The planner's load is light so it holds up even at low TPM.

Azure OpenAI's TPM turned out to be a two-tier structure: (1) the quota cap for subscription × region × model (determined by Azure; increasing it requires a support request), and (2) each deployment's capacity allocation (you decide it freely, within quota). This time I touched (2), but (1) can also be checked with az cognitiveservices usage list --location <region>.

The evaluation dataset: turning failure patterns into assets

The traps hit during implementation become "regression tests to avoid hitting the same ones again." This time I built golden JSON for the agent and turned each failure pattern covered in this article into a case (14 in total).

{
  "id": "agent-honorific-strip",
  "description": "structured search with an honorific ('○○男爵') → normalized to the abbreviation and returns 30+ items",
  "discovered": "JSON_CONTAINS→JSON_SEARCH + added stripHonorific",
  "user_input": "調べて (look up other days ○○男爵 visited)",
  "expectations": {
    "tool_called_any": ["search_diary_structured"],
    "tool_result_count_min": 5,
    "answer_must_not_contain": ["見つかりませんでした"]
  }
}

The runner parses the agent endpoint's UIMessage stream from Python and asserts against tool calls and the final answer. It has retries built in to absorb transient rate limits, and currently 13/14 cases pass (the remaining 1 is a multi-turn test of the reasoning chain, which is hard to reproduce completely without going through the UI, and is awaiting a Playwright E2E test).

Getting into a state where you can run the loop of "trap hit during implementation → failing test case → fix → test passes" makes it harder to reintroduce the same hole in later work, and also makes it mechanically clear "if I change ○○, where does it have impact." LLM-based systems are non-deterministic in behavior and manual review can't keep up with detecting regressions, so this kind of asset-building felt like it paid off.

What this work confirmed

Vector RAG and structured data are complements

For specific questions, there were many situations where only one of the two functioned. Conceptual questions like "what is the thinking about ○○?" are vector search's exclusive domain, while attribute filtering like "records of visiting △△ in year ○○" is structured search's exclusive domain. LLM tool calling seemed to function naturally as a layer that routes between the two.

I got the sense that investment in data curation (structured tags, manual annotation, authority control) pays off even in the LLM era — indeed, especially in the LLM era.

The friction of getting it running is greater than the design

The design decision to "go with an 8-tool setup" is settled in minutes, but getting it running on the real stack of Cloudflare Workers + Azure OpenAI + Vercel AI SDK + TiDB Cloud meant hitting the 5 traps above.

In particular:

  • LLM-specific problems (the tag-naming variance vs. LLM-output mismatch in Trap 3, the reasoning chain in Trap 4) are hard to notice by reading only the SDK documentation
  • Infrastructure limits (Worker size in Trap 1, TPM in Trap 5) open up more options if noticed early
  • SDK URL/api-version compatibility (Trap 2) can break quietly on a version bump

Grow the evaluation dataset in parallel

By fixing the traps hit during implementation as golden JSON test cases on the spot, the peace of mind for later modifications and model swaps felt quite different. LLM-based systems behave probabilistically, and full coverage via manual review is hard. Writing up "the things I absolutely must keep passing" as JSON seems to pay off.