I set up a call center on top of my second brain, cause why not

I keep everything in one folder of plain Markdown. Meeting notes, half-formed project ideas, journal entries, the occasional 2am thought about my career. A few hundred files. I’d built RAG systems before, but I wanted to do it on Google Cloud this time, and on something I actually cared about getting right, so I indexed the folder, put a grounded agent in front of it, and gave it a real phone number. You can call it and a robot reads you my notes, which is exactly as strange as it sounds.

The spec is the one every company means by “an AI trained on our docs”: retrieve from approved text, answer only from what you retrieved, escalate to a human when unsure. I’d assembled these pieces before and figured the new cloud and the personal corpus would be the only twists. The twists were elsewhere: retrieval, thresholds, quotas, voice formatting, permissions. Gemini was rarely the thing that broke.

The shape of it

flowchart TB
  CHAT["Web chat"] --> AGENT
  VOICE["Phone call<br/>(a real, live number)"] --> STT["Speech-to-Text"]
  STT --> AGENT
  AGENT["The agent<br/>Gemini + retrieval"] --> RAG["Search index<br/>316 of my notes"]
  RAG --> KB[("My second brain<br/>in cloud storage")]
  AGENT --> GATE{"Confidence gate<br/>is the answer supported?"}
  GATE -->|"yes"| ANSWER["Answer<br/>(typed or spoken)"]
  GATE -->|"no / out of scope"| HUMAN["Hand off to a human<br/>with a summary + sources"]
  ANSWER --> METRICS["Metrics → BigQuery"]
  HUMAN --> METRICS
  METRICS --> LOOP["Evals + CI gate<br/>+ drift monitor"]

A question arrives by text or phone. The agent retrieves relevant passages, drafts an answer from those passages only, then runs a second pass that scores how well the draft is actually supported by what it retrieved. High score, it answers. Low score or out of scope, it escalates. Every turn gets logged so I can measure the thing.

That support score is my grounding check: does the answer stick to the retrieved notes? The pitch for grounding is that it stops hallucination by checking the draft against your documents. It’s a narrower guarantee than it sounds, and it broke in the first place I tested it.

Grounded is not correct

I asked about Leaguemate, a side project. Clean sourced summary, support score 0.91, a couple dozen citations into the exact notes it used. The demo that makes everyone nod.

Then I asked something out of scope: “What is my bank account balance?” It answered. Not with a real number, that isn’t in my notes, but with a confident non-answer padded out around a note that happened to mention “bank transactions.” That answer also scored 0.91. Same as the good one.

The reason is mechanical once you see it. The grounding check measures one property: is this answer entailed by the retrieved text? It says nothing about relevance, correctness, or scope. The bank-balance reply was faithful to a note that genuinely discussed bank transactions, and useless.

flowchart TD
  Q["What is my bank balance?"] --> R["Retrieve the most<br/>relevant notes"]
  R --> A["Draft an answer from<br/>only those notes"]
  A --> G["Grounding check:<br/>is the answer supported<br/>by the retrieved text?"]
  G -->|"score 0.91, supported"| SHIP["Looks confident. Ship it."]
  SHIP --> OOPS["Grounded. Confident.<br/>And completely useless."]
  OOPS --> FIX["The fix: also ask<br/>is it in scope? is it relevant?<br/>then pair with real user feedback"]

Groundedness and correctness are different properties. A gate that watches only the support score waves wrong answers through at full confidence. If the only dials you watch are groundedness and containment (the share of questions handled without a human), both stay green while the bank-balance answer sails right through.

One threshold on a noisy signal

It gets worse in a useful way. I built a 15-question eval set, some answerable and some deliberately out of scope, and ran it repeatedly. The grounding score was noisy right around my 0.7 cutoff: the same answerable question landed at 0.69 on one run (escalate) and 0.74 on the next (answer). Pure jitter across a hard threshold.

Tracking escalation rate hides this. The honest metric is escalation appropriateness: when it escalated, was that the right call? Across that 15-question run it came out to 0.00. Every escalation was an answerable question shoved over the line by a wobbly score.

A single cutoff on a noisy signal makes both mistakes: it answers some questions it should have escalated, and escalates answerable ones that landed just under the line. The fix is a thermostat’s: a hysteresis band instead of one line, plus a couple of extra signals (scope, relevance) instead of betting everything on one number. I wired this into a CI test that fails the build if the system ever contains something it should have escalated.

The bugs I hit were in the plumbing

Once the gate worked, I kept poking at the system and kept finding the next failure in the plumbing around the model, never the model itself.

The eval batch died on a quota. The interactive demo was snappy; running all my eval questions back-to-back tripped a limit. The free tier caps you at 10 model calls a minute and each question spends two (answer, then grounding check), so the loop hit RESOURCE_EXHAUSTED partway through. Fast in the demo, dead under a batch, same system. The quota math is the sort of thing the demo never makes you look at.

The voice agent read its own formatting. I called the number and asked what league means. The grounded answer was correct but contained `previous_league_id`, and the text-to-speech read it out character by character: “backtick previous underscore league underscore id backtick.” Right content, wrong delivery for the medium. The fix was a per-channel system prompt: plain spoken prose for voice, spell identifiers as words; formatting preserved for web.

Retrieval surfaced exactly the wrong document. Searching for one of my projects, the top hits were my detailed working notes on it rather than the overview, just because they were the longest thing I’d written. Scale that to a company and the risk is obvious: a normal question retrieves the most detailed matching document, which can be the one the asker wasn’t cleared to see. Grounding governs whether the answer is faithful; it says nothing about whether the user was allowed to see the source. In my own notes this was harmless. On a shared corpus, retrieval has to be permission-scoped before the model sees the text, or that’s the reason you can’t ship.

The handoff resumes the conversation

The part I’m happiest with is what happens on “I’m not sure.” Most bots treat escalation as a dead end: you land in a queue and the human opens cold with “how can I help you today?” while you re-explain everything.

Here the human resumes the same conversation. The desk opens with a generated summary, a suggested reply drawn from the same notes with its sources, and the full transcript already loaded, including the call transcript if it came in by phone. I built and verified the desk against the live API, but I never sat a real person in the seat, so I won’t call it solved. The handoff at least carries the context forward instead of dropping it.

Closing the loop

flowchart LR
  TRAFFIC["Live chat + voice"] --> LOG["Cloud Logging"]
  LOG --> BQ["BigQuery<br/>groundedness, no-match,<br/>containment, latency, thumbs up/down"]
  BQ --> EVAL["Golden-set evals"]
  EVAL --> GATE{"CI gate:<br/>did we contain anything<br/>we should have escalated?"}
  GATE -->|"pass"| PROMOTE["Promote: dev → prod"]
  GATE -->|"fail"| BLOCK["Block the release"]
  PROMOTE --> TRAFFIC

Every turn streams into BigQuery I can query: no-match rate, containment, grounding scores, latency, thumbs up/down. The golden set runs as an automated test wired to fail on the failure modes that matter (did we confidently contain something we should have escalated) instead of the vanity ones (look how few questions needed a human). I gated promotion on that test: dev to prod only after it passes. I also built a drift monitor that fires when the notes fall out of sync with the index, since an answer grounded in a stale copy of reality is its own quiet kind of wrong, though I’ve only run that on demand, not on a cron.

None of this is exotic: a fixed eval set, logged turns, a build that fails on bad containment, a check for a stale index.

What I learned

• Grounded isn’t correct. Faithful to the data and actually right are different properties. Don’t gate on one number.
• Retrieval is a permissions problem before it’s an AI problem. The model will happily ground an answer in a document the user was never allowed to see.
• Match the medium. The answer that reads perfectly on screen can be unusable read out loud.
• The quota, the stale index, and the handoff path changed the system more than model choice would have. That’s where most of my time went.
• Spend on the loop, not the model. The gate, the handoff, and the measurement loop are what made it trustworthy. Prove it on one moderate-risk workflow before funding the rest.

The first version worked in an afternoon. Getting to where I’d trust an answer it gave me took every fix above, and I’d built one of these before.