MemLedger

Oct 18, 2026 · @rudraksh_soni
Four technical posts drawn from building MemLedger for GraphHacks 2026, each one ready to publish on its own.
Agent memory that can take things back
When several AI agents share one memory, a bad fact doesn't stay where it landed. Triage reads an alert and writes down that a service is failing. The Investigator reads that and recommends a failover. Comms reads both and drafts a status update. If the alert was forged, three agents are now wrong, and most memory layers can't tell you which of their facts came from it.
MemLedger is what we built for GraphHacks 2026 to fix that: shared memory for a team of incident-response agents, stored as a graph in FalkorDB, where every belief keeps its receipts.
What a belief records
Each belief carries four things besides its value. It records which agent wrote it, which source it came from, when it was valid, and which other beliefs it was based on. That last field does most of the work. "Based on" turns memory into a chain you can walk backwards and forwards.
Why a graph
Revoking a source means finding everything downstream of it, across agents, to whatever depth the chain goes. In a relational database that's a recursive join. A vector store doesn't model the relationship at all. In a graph it's one traversal.
Here is a simplified sketch of the revoke. Labels are shortened; the real queries live in memledger/queries.py.
MATCH (src:Source {name: $source})<-[:FROM]-(b:Belief)
MATCH (d:Belief)-[:BASED_ON*0..]->(b)
WHERE d.revoked_at IS NULL
SET d.revoked_at = $now, d.revoked_reason = $reason
Preview runs the same traversal without the SET. An operator sees the blast radius before anything changes, which matters when the thing being revoked is a whole feed.
The demo story
Our acme scenario runs 20 steps across three sessions. At step 16 a message on the alerts feed claims checkout-api is failing too. Triage stores that. At step 17 the Investigator concludes the team should fail checkout-api over to the standby database, and cites Triage's fact.
Step 18 previews revoking the alerts feed. The blast radius is five facts: three from Triage, one from the Investigator, and one from Comms, whose earlier status update was built on alert facts from step 1. Step 19 revokes all five with one query. The runbook's root cause (a Redis outage), the rollback plan and the owner stay active, because none of them depended on the alerts feed. At step 20, Comms writes a fresh status from what survived.
Revoked beliefs aren't deleted. They keep their timestamps and the reason, and an as_of query shows what the agents believed at any earlier moment. For a post-incident review, that history is half the value. A second tenant, globex, runs alongside acme, and acme's revoke never touches it.
A FalkorDB gotcha
Our seed query was supposed to create five sources. On the real FalkorDB 6.0.2 server it created one. The query used UNWIND ... MERGE after an aggregating WITH, and only the first row made it through. Rewriting the loop with FOREACH fixed it. Trimmed down to its shape:
// before: one source created on the server
WITH count(*) AS _
UNWIND $sources AS s
MERGE (:Source {name: s})
// after: all five created
FOREACH (s IN $sources | MERGE (:Source {name: s}))
If your seed data looks thin on a real server and fine elsewhere, check for this pattern before you check anything else.
Making Gemini extraction cite its evidence, on a 20-request quota
[image: Cover: a raw alert goes through Gemini; two claims are kept with quoted evidence and one is dropped]
MemLedger's agents never write raw text into memory. Alerts, logs, webhooks and the agents' own notes go through Gemini first, which turns them into structured claims. The model is the part of the system most likely to make something up, so every claim it returns has to quote the input text that supports it. If the quoted evidence isn't in the input, the validator drops the claim.
Strict output
We use Gemini's response_schema so the model returns claims in a fixed shape: predicate, value, certainty and evidence. Our extraction test prints each result as a KEEP line with the quote after a |, or a DROP line when the validator rejects it. Seeing the drops is useful. It shows exactly where the model stretched.
We looked at moving to Groq late in the build. We stayed with Gemini because extraction leans on that strict schema, and swapping providers on deadline day was the bigger risk. A second provider can sit behind agents/llm.py later.
Prompt rules that mattered
Two rules in the second version of the prompt (ext-v2) fixed most of our misses. Text filed under an incident counts as being about that incident even when it doesn't name it. Recommended actions count as mitigations.
One step still separated the models. At step 8, a webhook, gemini-3.5-flash-lite left out a "flush cache" mitigation that gemini-3.5-flash returned. That one missing claim failed four later checks in the scenario, because those steps built on it.
Living on the free tier
The free tier gave us 20 requests a day on gemini-3.5-flash, resetting at midnight Pacific (12:30 PM in India). gemini-2.5-flash returned 404 for our key, so 3.5 Flash became the default and 3.5 Flash-Lite the fallback.
Our retry layer, memledger/llm_retry.py, handles 429 and 503 errors. It waits as long as the error asks, or backs off up to 60 seconds, retries up to GEMINI_MAX_RETRIES times (default 5), then tries GEMINI_FALLBACK_MODEL.
Two kinds of 429 needed different handling:
A per-minute limit clears in seconds, so waiting and retrying is right. Spacing calls about 12 seconds apart avoided most of these.
A daily-quota 429 won't clear until tomorrow, so retrying only burns time. On that error the code now switches to the fallback model immediately.
A 429 during an agent step used to crash the whole demo run. Now it fails that step's check, records the reason, and the run continues.
Don't cache the fallback
Code review caught a quieter bug. When Flash hit its quota and Flash-Lite answered, the cache stored Flash-Lite's answer under Flash's key. A weaker extraction could sit in the cache and replay forever as if Flash had written it. Now generate reports which model actually answered, and the extractor writes to the cache only when the main model did.
One more thing we learned the hard way: free-tier quota belongs to the Google Cloud project, not the API key. Two keys in the same project share the same 20 requests. A key in a new project starts fresh.
The cache that could never replay
[image: Cover: two runs share the step 3 cache key, then every key from step 4 on differs]
We wanted the public MemLedger demo to replay real Gemini extractions from a cache, so visitors would see real model output without spending any quota. We spent most of a morning on it and shipped without it. The reason turned out to be a random ID inside a sentence.
How the cache is keyed
The cache key covers the input text and the beliefs already known for the incident at that step. That's deliberate: the same alert, read in a different state, can mean something different, so it should be extracted again. It also means every key depends on everything the scenario wrote before it.
First miss: different agents, different keys
Our committed cache came from a run with live agents. The deployed site runs offline agents, which write different beliefs. So at steps 5, 7 and 8 the known values differed, the keys differed, and the lookups missed. The smoke test failed on those steps and on the later ones that depended on them.
Our fallback mode, golden, replays hand-written reference claims and passed every step. We deployed on golden and planned to fill the cache for the offline configuration once the daily quota reset.
A dry run that couldn't fail
To budget quota, we ran a dry run that answered each cache miss with the golden claims and counted the misses. It reported 11 calls, well under the 20 a day we had. Every step passed.
The live run made 8 successful calls, lost step 8 to a failed request, and hit the daily limit at step 20. We stopped, committed nothing and got a key from a fresh project. Before spending it, we replayed the partial cache in cache-only mode as a check. Step 5 missed, even though the live run had just written it.
The root cause
Fact IDs are random (uuid4). Comms' status text cites them inline, like payments-api [f-23e93a9b], and that text gets extracted like any other input. So:
Comms' text is different on every run, so its cache key never matches.
Step 4 misses, so no status gets stored.
The known values at step 5 change, so step 5's key changes, and every later key changes with it.
Even with matching keys it would have failed. Cached answers quote the old IDs as evidence, and the validator drops any claim whose evidence isn't in the new text.
The dry run hid all of this. Golden claims answered every miss, so the chain never broke and the IDs never mattered. A dry run that substitutes answers can count calls, but it can't tell you whether a replay will match.
What we'd do next
There are two fixes. Deterministic fact IDs in scenario runs would make text reproducible, but that changes the memory core and touches nearly every test. Stripping [f-...] citations from the text before computing the cache key and checking evidence is smaller, but it touches the validator, the one piece that stops the model from inventing facts.
On deadline day, neither was worth the risk. The live site runs on golden, its header says so, and the live Gemini runs are recorded in the project log. Deterministic replay is first on the list after submission, since any memory system that caches model output on top of its own state will need it.
Shipping the demo to Cloud Run without spending Gemini quota
[image: Cover: a single Cloud Run instance with golden extraction, offline agents and zero Gemini calls]
The MemLedger demo is public, so anyone can click through it as often as they like. We had 20 Gemini requests a day. The deploy had to make zero model calls, keep the API key out of everything, cost close to nothing, and share a Google Cloud project with another app without disturbing it.
No key at all
Two environment variables put the app in its public mode: MEMLEDGER_EXTRACT=golden replays fixed reference extractions, and MEMLEDGER_AGENTS=offline runs the agents without a model. In that mode the app never creates a Gemini client, so it doesn't need a key.
Our original deploy.sh pulled the key from Secret Manager and passed it to the service. We skipped the script and deployed with plain gcloud, with no secrets attached. Before pushing, we checked the image itself: no .env, no local demo state file, no GEMINI variables, and the extraction cache present.
Sharing a project safely
The only project with billing enabled already ran another of our services. Cloud Build wasn't enabled there, and turning it on meant IAM changes we didn't want to make in a shared project. So we built the image locally with Docker, pushed it to a new Artifact Registry repo, and deployed with --image.
The service runs as a new service account, memledger-run, with no project roles. The default compute account may be able to read the other service's secrets, and MemLedger needs none of them. Every command passed --project explicitly, so the local gcloud default never changed.
One instance on purpose
The deployed app uses an embedded graph database, so each Cloud Run instance holds its own graph. With two instances, consecutive requests could land on different graphs, and the demo would appear to jump between states. We set --max-instances=1. With --min-instances=0 the cost is the same, since an idle service scales to zero.
Scaling to zero has a side effect we liked. Each cold start seeds the demo tenants fresh, so most visitors get a clean run. With --cpu-boost, the instance passed its startup probe 4.3 seconds after starting, 3.1 of those spent seeding. The probe allows 240 seconds. We asked for 512 MiB, and the app uses about 81.
Checking it
scripts/smoke.py --full drives all 20 acme steps against the live URL over HTTP and fails on any broken check. It also resets both demo tenants, so it shouldn't run while someone is presenting.
One bug only showed up in a browser. After a redesign, the server had the new CSS but browsers kept the old file, because the static files sent no cache headers. Adding a version tag to the asset links (style.css?v=ui1) forced a fresh download.
Redeploying
With the image built locally, a redeploy is one line:
IMG=asia-south1-docker.pkg.dev/<project>/memledger/memledger:$(git rev-parse --short HEAD) && docker build -t $IMG . && docker push $IMG && gcloud run deploy memledger --project=<project> --region=asia-south1 --image=$IMG
The image tag is the commit hash, so every revision in Cloud Run maps back to the exact code it runs.
Comments
Join the discussionNo comments yet. Be the first to comment.