OPO Startups · July 2026

AI-Powered Hybrid Search
& Agentic Assistants

on Cloudflare's Edge

Build production search and durable AI agents that scale to zero — and back up.

Brett Bartrum Founder · Juiced Wheels & ShopDAWG Senior DevOps Engineer · Applied Systems

⚙

Why I Built This — It Starts at the Workbench

At Juiced Wheels a bench tech has a customer's e-bike on the stand and the clock running — and the answer is buried in a parts catalog and a stack of manuals no keyword box can navigate.

84K+

parts to search

The right innertube is one SKU among tens of thousands across QBP · JBI · OSCO.

2,845

PDF manuals

The fix is on page 47 of a Bafang PDF — not in a tidy, queryable database.

"30h?"

cryptic error codes

Bafang, Bosch & Shimano each speak a different dialect — the code is meaningless without the table.

≠

symptom vs. keyword

"Motor cuts out climbing" isn't a phrase in any manual — it's a trace through the system.

A bench tech doesn't want ten documents — they want the answer, cited, in seconds, and bench time is a shop's biggest cost. That's the bar generic search couldn't clear, so I built Shop Shepherd on Cloudflare for ~$80/mo — and the stack underneath isn't bike-specific.

Why I Built This — 1.5 min. The hook.

Ground the whole talk in the physical reality before any architecture. "I run Juiced Wheels and ShopDAWG. Picture the bench: a customer's e-bike on the stand, the clock running, and the tech needs an answer — now."

Walk the four cards:

84K+ parts — the catalog is enormous; the right part is one SKU among tens of thousands.
2,845 manuals — answers live deep inside PDFs, not in clean rows.
Cryptic error codes — every brand's codes mean something different; useless without the table.
Symptom ≠ keyword — how a rider describes a fault never matches the doc; this sets up the diagnostic graph later.

Land the punchline: the tech wants the answer, cited, fast — not a pile of documents, because bench time is the shop's biggest cost. That bar is why the rest of this talk exists.

The pivot to land: I built this for my own bench at Juiced Wheels — and it works, on a ~$80/mo Cloudflare stack. But the takeaway isn't the bike shop: the stack underneath isn't bike-specific. Same primitives — manuals, parts, diagnostics — point at any industry (healthcare is our next target). This is a knowledge-share, not a sales pitch. THEN pivot to the infra problem (next slide): even if you build it, traditional infra bills you to serve nobody.

The Problem with Traditional Search Infrastructure

⚠ EVEN WHILE YOU READ THIS SLIDE…

$0.0000

Burned by always-on infra ($1,950/mo baseline, $0.0007/sec)

Queries actually served

100%

Of spend on idle capacity

Elasticsearch / OpenSearch

Dedicated clusters running 24/7 even with zero traffic.

$200–800+/mo minimum, no queries needed

Pinecone / Weaviate

Vector DB subscription bills for all 30 days, used or not.

$70–250+/mo serverless tier

Regional Lock-in

Pinecone, ES clusters pin to one region. EU/Asia queries pay 200ms+ tax.

Multi-region · 2–3× the cost or self-managed replication

Stitched Stack

Vector DB + keyword search are separate systems. Four vendors, four bills, glue code to merge.

No native hybrid · sync drift · double the ops

The Problem — 3 min.

Hook: Let the burn ticker spin silently for 5 seconds before you talk. The dollar number goes up while you stand there saying nothing. That's the slide.

Then: "This is what Pinecone, Elasticsearch, OpenAI, and AWS bill you while nobody's even using your app."

Four cards, ~30s each:

Elasticsearch: 3-node cluster runs 24/7. You pay for idle.
Pinecone: Per-pod billing. Standard tier minimum $50/mo before you query anything.
Regional lock-in: Pinecone + ES pin to one region. EU/Asia users pay a 200ms+ trans-Atlantic tax.
Stitched stack: ES + Pinecone + OpenAI + Postgres = 4 vendors, 4 bills, 4 dashboards, 4 incident pages.

Anticipated questions:
Q: "ES has serverless now."
A: It does, but it still pins to a region. The edge model is fundamentally different.
Q: "Cloudflare also bills you."
A: Workers Paid is $5/mo flat. The bill doesn't grow when nobody shows up.

Why Cloudflare? The Edge Advantage

✓ SAME SLIDE, ON CLOUDFLARE'S EDGE…

$0.0000

Always-on cost (flatline — pay only for what runs)

Queries served while you read

$0.0000

Total spend (~$0.00002 / query)

Usage-Based Pricing

Pay per request, per query, per GB stored — not per hour.

Zero traffic = zero cost

Global Edge Network

Workers run within ms of the user, not in us-east-1.

330+ PoPs worldwide

Unified Platform

Workers + D1 + Vectorize + KV + R2 in one ecosystem.

One bill, no glue code

Zero Cold Starts

V8 isolates start in sub-ms — the shop lights flick on, no warming a cold engine.

~0ms cold start vs ~200ms containers

Why Cloudflare — 3 min.

Hook: The relief hero mirrors the burn ticker. Where the burn-ticker slide was money going out, this is queries going through. Let it run silently.

Then: "Now look at the same load profile on Cloudflare. I'm serving queries instead of burning capacity."

Four advantages — ~30s each:

Truly idle = $0. Workers cold-start in <5ms, so scale-to-zero doesn't punish users.
330+ cities. Your request hits the nearest PoP. No regional flag day.
One vendor, one bill. D1, Vectorize, R2, KV, Workers AI — all on the same dashboard.
Free tier is real. You can ship the search section of this deck on the free plan and stay there for a long time.

Anticipated questions:
Q: "Vendor lock-in?"
A: Yes, that's real. The Long Road slide near the end addresses how to keep the stack portable.
Q: "What about cold starts?"
A: V8 isolates. Sub-5ms. Three Generations slide is up next — that's the proof.

⚡

Three Generations of Compute

Each one cut cost and blast radius by an order of magnitude. V8 isolates are what makes scale-to-zero affordable at edge density.

DENSITY · TENANTS PER HOST log scale · the punchline

GEN 01 · VMs

10s1×

GEN 02 · Containers

100s10×

GEN 03 · V8 isolates

10,000s1,000×

GEN 01 · ~2001 VMware · KVM · EC2

Virtual machines

Full guest OS · hypervisor

FootprintGB

Cold start30–120 s

Density / host10s

Idle cost100%

Bare-metal era. Pay 24/7 for capacity you may never use.

GEN 02 · ~2013 Docker · Lambda · Fargate

Containers

Shared kernel · namespaced procs

Footprint100s MB

Cold start200–800 ms

Density / host100s

Idle costscale-to-zero*

*Then a 500 ms cold start the next time someone shows up.

GEN 03 · ~2017 Cloudflare Workers · Deno Deploy · Fastly

V8 isolates

One process · thousands of tenants

Footprint~3 MB

Cold start<5 ms

Density / host10,000s

Idle cost$0 · truly

Same V8 sandbox Chrome uses for tabs — battle-tested at billions of executions/day.

Three Generations of Compute — 3 min.

This is the conceptual foundation. Non-technical evaluators need this to understand why the Cost Comparison slide's numbers are real, not aspirational.

Walk left-to-right:

VMs (~2001): Full guest OS. GB footprint. 30–120s cold start. Tens per host. You pay 24/7.
Containers (~2013): Shared kernel, namespaces. Hundreds of MB. 200–800ms cold. Hundreds per host. Scale-to-zero, BUT the cold start is what your user feels.
V8 isolates (~2017): Same sandbox Chrome uses for tabs. ~3MB. <5ms cold. Tens of thousands per host. Truly $0 when idle.

The density delta is the punchline: 10s → 100s → 10,000s of tenants per box is three orders of magnitude. That's what makes "scale-to-zero at edge density" economically possible.

Anticipated questions:
Q: "Aren't isolates limited compared to containers?"
A: Yes — no arbitrary native code, no long-running processes, CPU-time caps. I hit zero of those limits for this workload.
Q: "What about Deno Deploy / Fastly?"
A: Same V8 model, same advantages. I picked Cloudflare for the bindings ecosystem.

Part One

Search Architecture Deep Dive

Three engines, one router — all on the edge

The Search Stack — Think of It Like Shopping

Three search engines, three shopping behaviors. The customer's voice picks the station: semantic for symptoms, faceted for browsing with constraints, keyword for exact parts.

FIND THE RIGHT AISLE

Vectorize semantic

"My bike has a flat tire"

768d cosine · topK=10 · ~15 ms^†

PICK OFF THE SHELF

D1 + Meilisearch faceted

"Innertube — 26" × 4", Schrader valve"

facets · typo-tolerant · ~28 ms^‡

SCAN THE BARCODE

D1 + FTS5 keyword

"I need part TU1200"

SQL · BM25 · ~3–11 ms

Measured engine times, warm queries. ^†Semantic embed: ~3ms cached / ~200ms cold. ^‡Meili is remote-hosted (NYC); warm round-trip adds ~400–700ms — FTS5 + Vectorize are co-located in the worker.

Search Stack — Shopping Analogy — 4 min. THE slide.

Open with the metaphor: "Imagine walking into a bike shop. You arrive with one of three things in your head — a symptom, a rough idea, or an exact part number — and each one routes you to a different station."

Three stations, ~75s each — walk them left to right, the way one repair actually flows:

Aisle directory (semantic) — start here, point at the aisle sign: "A customer comes in: 'My bike has a flat tire.' That's a symptom, not a part number. Vector search maps the plain-language problem onto a meaning map and sends them to the right aisle — Wheels & Tires — even with zero shared keywords. Vectorize, 768-dim embeddings, nearest-neighbor by cosine. ~15ms warm."
Shelf (faceted) — now narrow it down: "They're in the aisle facing a wall of tubes. Faceted search filters by the constraints that matter — 26×4 fat, Schrader valve — and ranks what's left. Meilisearch over the D1 catalog, typo-tolerant, built for facets. They grab the right innertube. ~28ms engine."
Register (keyword) — ring it up, point at the scanner + barcode: "By now they have the exact part — TU1200, the fat-bike innertube. The cashier doesn't browse; they scan the barcode and the register matches one SKU. D1 + FTS5 — SQLite's built-in full-text engine. I index every part number, SKU, and error code; with a strong identifier I go straight to BM25 and return the exact match in 3 to 11 milliseconds. No AI in this path — it's overkill once the user has typed the answer. This is the layer most teams skip because it's unglamorous; it's also where the most-precise queries land."

On the Register graphic specifically — point at the elements as you talk:

Barcode on the box: the part number is the index key. No interpretation needed.
Laser pulse: the lookup is deterministic — same query, same row, every time.
POS terminal screen: exact-match result, structured fields (description, fitment, price). No re-ranking, no LLM, just SQL.
EXACT MATCH badge: when keyword search works, it works perfectly. When it doesn't, you fall through to the next station.
The receipt: done. One transaction, no follow-up needed. Most customer queries to a parts catalog actually look like this.

The take-home: these aren't redundant — they're different mechanisms for different question shapes. The router decides which to call (or all three) per query.

Footnote honesty: point at the bottom caveat. "These are engine times. Cold semantic adds ~200ms for embedding. Meili adds ~500ms because I host it in NYC, not on Cloudflare."

Anticipated questions:
Q: "Why three engines instead of one good one?"
A: There's no one good one for all three query shapes. Keyword search on a vector index is bad. Facet aggregation on FTS5 is brutal. Each tool earns its place.
Q: "Who routes? An LLM?"
A: Mostly the agent (Sonnet 5) using tools. I also have a pre-agent classifier (next half of deck) that primes the routing.

Layer 1: Find the Right Aisle — Vectorize

Layer 1 of 3

Why Vector Search Matters

Keywords fail when users describe problems instead of naming parts. "My bike has a flat tire" should match "innertube replacement" — vector search makes this connection.

Workers AI maps each sentence to 768 numbers — coordinates on a "meaning map," where similar meanings land near each other even with zero shared words. embeddinggemma-300m generates them; Vectorize finds the nearest via cosine similarity.

Embedding Pipeline

1Auto-chunk content > 7,500 chars

2Workers AI → 768-dim vector

3Vectorize index → stored at edge

4KV cache → skip re-embedding

Semantic Routing

CUSTOMER ASKS

"My bike has a flat tire"

↓

MEANING MAP · NEAREST AISLE WINS

your query nearest aisle other aisles

VECTORIZE STAGES

1embed (Workers AI)

→

2cosine search

→

3topK=10

Layer 1 — Vectorize — 3 min.

Lead: "The aisle directory. You describe your problem in your own words. Someone points you to the right aisle."

The hard part to explain: embeddings. Use the meaning-map analogy on the slide. Workers AI takes a sentence, converts it to 768 numbers — coordinates on a high-dimensional map. Two sentences that mean similar things land close together. Cosine similarity measures that distance.

What makes Vectorize specifically work:

Co-located with the worker — same network as D1, same edge PoP
768-dim native — matches Workers AI's EmbeddingGemma output exactly
Vector lookup itself is ~15ms warm
Embed cost is the tax: ~3ms cached, ~200ms cold. I cache aggressively.

Anticipated questions:
Q: "Why 768 dimensions?"
A: The embedding model's output dimension. EmbeddingGemma is 768. Larger models go to 1024, 1536, 3072 — diminishing returns past 768 for my domain.
Q: "How does this beat OpenAI embeddings?"
A: It doesn't, on raw quality. It beats them on latency (co-located) and cost ($0–10/mo vs $5–25). For my domain, the quality delta is invisible.

▦

Layer 2: Pick Off the Shelf — D1 + Meilisearch

Layer 2 of 3

What Faceting Is

Filtering a big catalog by structured attributes — brand, wheel size, valve type, price, in-stock — with a live count on every option. It's how a tech narrows 84,000 parts down to the 8 that fit, without typing a perfect query.

Why D1 and Meilisearch?

D1 system of record

Normalizes 84K parts from 3 vendor feeds, scores e-bike relevance, and serves keyword search via FTS5 — the fallback if Meili is ever down.

Meilisearch the speed layer

A sub-30ms count per facet checkbox across all 84K rows — more than a D1 GROUP BY can handle — plus typo tolerance ("shrader" finds "Schrader"). It's a faceted-search engine with its own hybrid keyword + vector ranking.

Faceted Shelf in Action

CUSTOMER ASKS

"Innertube — 26" × 4", Schrader valve"

THE LONE THIRD-PARTY

Meilisearch ($30/mo) is the only non-Cloudflare piece of this whole stack — today's one platform gap for fast faceting + typo tolerance. If Cloudflare ships a native equivalent, this line drops to $0 and the architecture is 100% first-party.

MEILISEARCH STAGES

1parse facets

→

2filter (typo-tolerant)

→

3rank by relevance

Layer 2 — Meilisearch — 2 min.

Lead: "The shelf. You know exactly what you want — 26", 4", Schrader valve — and you need filters to navigate 84,000 products."

Why Meili, not D1 FTS5:

Typo tolerance: "shrader" → "Schrader" out of the box. FTS5 won't do that.
Facet aggregation: sub-30ms count-by-category over 84K products. D1 GROUP BY can't keep up at this shape.
Faceted UI is what users want: "valve type ▾" "wheel size ▾" — every checkbox needs a fast count.

Honest disclosure: this is the ONE paid third-party. $30/mo. I could drop it and the parts catalog would degrade — slower facets, no typo tolerance — but it would still work via D1 FTS5 fallback.

Note on latency: Meili lives in NYC, not on Cloudflare. The 28ms is engine time. Round-trip from edge is ~500ms. That's the tradeoff for the only third-party in the stack.

Anticipated questions:
Q: "Why not host Meili on Workers?"
A: Meili's a Rust binary, not edge-deployable. The Workers-native equivalent (D1 FTS5) covers two of the three things Meili gives me; the third (typo tolerance) is the part I pay for.

Layer 3: Scan the Barcode — D1 + FTS5

Layer 3 of 3

How It Works

D1 is Cloudflare's serverless SQLite; FTS5 is its built-in full-text search — co-located with your Worker, no extra service, no extra bill.

Finding · progressive relaxation one round trip

EXACT"TU1200" → 1 hit

PREFIX"TU120…" → 6 hits

FUZZY"TU~1200" → 18 hits

Ranking · how BM25 scores a hit 3 signals → one score

RARITY"TU1200" outweighs a common "tube"

FREQUENCYmore hits help — but less each time

BREVITYa title hit beats one deep in a manual

Progressive Lookup

CUSTOMER ASKS

"I need part TU1200"

TU1200

Fat Bike Tube · 26×4 Schrader

BM25 score 8.4 · 7 ms (measured)

FTS5 STAGES

1tokenize

→

2match (virtual table)

→

3rank (BM25)

Layer 3 — D1 + FTS5 — 2 min.

Lead: "The register. You walk up with an exact part number. The cashier scans it. Done."

What FTS5 is: SQLite's full-text search extension. BM25 ranking — think of it as modern TF-IDF. Free with D1.

The two halves of the left column: "Finding" is HOW I match — progressive relaxation, exact → prefix → fuzzy in one round trip. "Ranking" is HOW I order the hits — BM25, which is just three intuitions:

Rarity (IDF): a rare token like "TU1200" is worth far more than a common one like "tube."
Frequency (TF): more mentions score higher — but with diminishing returns, so keyword-stuffing doesn't win.
Brevity (length norm): the same match in a short title beats one buried in a 900-word manual.

Tie it to the card on the right: those three signals are what produce the "BM25 score 8.4."

What's interesting:

Lives inside D1, so no separate cluster, no separate bill
Co-located with the worker — engine time ≈ user-perceived time
Progressive relaxation: exact → prefix → fuzzy. Three queries, one round trip.
Measured 3–11ms in production. That's not a marketing number.

Anticipated questions:
Q: "Doesn't FTS5 fall over at scale?"
A: My parts catalog is 84K rows and the broadest prefix scan returns in 11ms. I'll worry at 10M.
Q: "What about typos?"
A: FTS5 doesn't do typo tolerance — that's why Meilisearch is on the team for the parts catalog.

⊕

Hybrid Search: Merging Keyword + Vector

When one engine isn't enough, fire all three in parallel and merge by weighted relevance score. The agent never sees three result lists — just one ranked output.

User Query

FTS5

Keyword

8.4

Vectorize

Semantic

0.84

Meilisearch

D1 Catalog

7.1

Weighted Score Merge + Dedup

Source Weights

Manuals 2.0× · Docs 1.5× · Parts 0.2×

OEM manuals win every tie — they're the authoritative source for technicians
Parts get demoted unless the query is clearly a SKU lookup
A result returned by multiple engines stacks the boosts

Short Query Optimization

1–2 words

70% keyword

30% vector

3+ words

50% keyword

50% vector

Short queries lean keyword — prevents false matches like "brake" ≠ "break"
Below-floor results get cut, with a top-3 fallback so the agent never starves

Hybrid Search — 3 min.

Lead: "What if your query crosses categories? 'TU1200 tube keeps going flat.' Part of that is keyword, part is semantic."

How it works:

Run FTS5 and Vectorize in parallel against the same query
Normalize each engine's scores (BM25 ranges 0–20; cosine ranges 0–1)
Weighted merge: typically 60% keyword + 40% semantic, tuned per query type
One ranked list goes to Sonnet for synthesis

The winning result strip: "TU1200 final 18.2" — that's the combined score after merging. Best result wins.

Why this matters: a pure keyword search misses "has a flat tire." A pure semantic search misses "TU1200." Hybrid catches both query shapes in one round trip.

Anticipated questions:
Q: "How do you tune the weights?"
A: I have a 30-case eval suite (mentioned on the Contextual Retrieval slide). When I change weights, I re-run the eval and check Top-1/Top-3 accuracy.
Q: "Why not let the LLM combine?"
A: I do — Sonnet re-ranks the top 30. The weighted merge is the prefilter.

📄

PDF Pipeline: OCR, Extract, Optimize

Why OCR Every PDF?

Most OEM service manuals are scanned images or flattened exports — there's no selectable text layer. OCR (Optical Character Recognition) turns those page images into searchable text. I then inject that text back into the PDF as a hidden layer, so the manual is fully searchable in any viewer while looking identical to the original. You can't chunk what you can't read.

Raw OEM PDF → Mobile-ready

Raw OEM

80 MB

Optimized

2 MB

40× smaller · same content · ready for a 4G phone at the bench

How I get there

✓ Subset embedded fonts (strip unused glyphs)
✓ Recompress images (JPEG2000 / lossy DCT)
✓ Inject OCR text layer (hidden, selectable, indexable)
✓ Optimize to PDF/A (archival standard — keeps the OCR text map)
✓ Web-linearize (first page renders before full download)

📄 manual.pdf

Page 12 loaded · 180 ms

Tech reads at the bench

PDF STAGES

1OCR

→

2chunk + embed

→

3D1 + Vectorize

→

4R2 (mobile + thumbs)

PDF Pipeline — 4 min.

Lead: "I have 2,845 PDF manuals — 593,000 chunks. But the input is brutal: OEM service manuals arrive as 80MB scanned-image PDFs, no selectable text. The phone mockup on this slide shows what I ship: a 2MB mobile-ready PDF that a tech can scroll through on 4G at the bench, AND a fully indexed retrieval layer for the AI."

This is actually two parallel pipelines from one OCR pass:

Retrieval pipeline (for the AI): OCR → chunk → embed → D1 + Vectorize
Distribution pipeline (for the human): OCR text → re-inject into PDF → font/image optimize → R2

The OCR detour worth calling out: "OCR doesn't just feed my search index — I also put the extracted text back into the PDF as a hidden layer. Visually the page looks identical to the original scan. But now if a tech opens the manual in any PDF viewer, search works. Copy-paste works. Accessibility tools work. That's a separate win from the AI side."

How I get 80MB → 2MB (point at the recipe list):

Subset embedded fonts: OEM manuals ship with the entire font family. I strip everything except the glyphs actually used in this document.
Recompress images: scanned pages are usually 600dpi PNG or uncompressed TIFF. JPEG2000 / lossy DCT at 150dpi keeps screen-readable quality at a fraction of the size.
Inject OCR text layer: the searchable layer adds ~50KB per manual — negligible compared to the image savings.
Adobe optimize to PDF/A-2u: the latest archival standard. Validates structure, strips redundant objects, normalizes color profiles.
Web-linearize: bytes reordered so the first page can render before the rest finishes downloading. Critical for mobile.

The mobile-ready brag: a 2MB linearized PDF means a tech in a shop on bad cellular can open it in under 2 seconds. The competitor experience (download an 80MB OEM PDF on 4G) is 90+ seconds and frequently times out.

Anticipated questions:
Q: "Why not just use ChatGPT and let it read the PDF?"
A: Two reasons. (1) Cost and latency — re-uploading 5MB on every query is wasteful. (2) The text-layer reinsertion is a separate product win — the human-readable optimized PDF only happens if I OCR myself.
Q: "Why Adobe and not Ghostscript / qpdf?"
A: Adobe's pipeline is closer to producing PDF/A-2u in one pass; open-source tooling needs more orchestration to hit the same standard reliably.
Q: "What's PDF/A-2u?"
A: ISO 19005-2 — long-term archival format. Embeds all fonts, normalizes structure, ensures the file renders the same in 10 years as it does today. The "u" means Unicode-mapped — important for the text layer.

⛓

Why Chunking Context Matters

The Naive Chunking Problem

Most tutorials split documents into fixed-size blocks and embed them as-is — like cutting a manual into unlabeled index cards. You get a match back, but no idea which card it's from.

An orphaned chunk like "torque the bolts to 40 Nm" is useless without knowing it's a Bosch CX mid-drive install vs. a Shimano hub motor teardown.

📄 Long manual PDF · 142 pages

↓ chunked + tagged

chunk_0042

"Torque bolts to 40 Nm. Check alignment..."

bosch-cx install

chunk_0043

"Speed sensor mounting: 5 Nm. Use anti-seize..."

bosch-cx sensor

chunk_0044

"Battery seat alignment: shim to 0.2 mm..."

bosch-cx battery

Naive Chunk (no context)

{
  text: "Torque bolts to 40 Nm.
        Check alignment before
        tightening...",
  vector: [0.23, -0.41, ...]
}

Metadata-Tagged Chunk (structural context)

{
  text: "Torque bolts to 40 Nm...",
  vector: [0.23, -0.41, ...],
  metadata: {
    title: "Bosch CX Installation",
    path:  "/guides/bosch-cx-install",
    tags:  ["mid-drive","torque"],
    category: "installation",
    source: "manual"
  }
}

NEXT Metadata fixes naming. But what about OCR garbage where even the chunk text is garbled? →

Why Chunking Context Matters — 2 min. Set-up for the Contextual Retrieval slide (next).

Lead: "Here's the problem with naïve chunking. The chunk that contains the answer doesn't always contain the context that makes it findable."

The example: a chunk from the middle of a Bosch manual. The text says "the speed sensor must be torqued to 5Nm." It doesn't say "Bosch." It doesn't say "DriveUnit." It doesn't say "speed sensor installation procedure." Those words are on page 3; the chunk is from page 47.

The fix on this slide (metadata): tag every chunk with structural metadata — manual brand, model, section heading. Now FTS5 can find the chunk even when the body doesn't have the query terms.

The bridge: "Metadata fixes naming. But what about OCR garbage where even the chunk text is garbled?" — Click forward to Contextual Retrieval.

Anticipated questions:
Q: "Couldn't you just use bigger chunks?"
A: Bigger chunks dilute embeddings (everything looks medium-related to everything). Smaller chunks lose context. Metadata + contextual retrieval is the win.

Contextual Retrieval: Quality Up, Cost Down

CHUNK TEXT

raw bytes

METADATA

structural · prev slide

context_llm

semantic · this slide

Metadata gives chunks a structural home. A 1–2 sentence LLM summary — Anthropic's Contextual Retrieval pattern — adds a sticky note that stays findable even when the page is OCR garbage.

Labor division: LLMs author chunk context; humans author the graph (slides 19–21) — enrich text, don't invent ontology.

BEFORE — RAW OCR CHUNK

Indexes can't find it

"340 % 8 1 240% TURBO 340%
3 4 0 SPEED SENSOR BDU
4xx INSTALL torque 5Nm..."

✗

FTS5 BM25: 0 useful matches — query terms don't appear in garbled OCR

✗

Vectorize: embedding of garbage = garbage similarity

Query: "Bosch Drive Unit speed sensor installation"

AFTER — CHUNK + context_llm

Both indexes hit at rank 1

context_llm
Bosch DriveUnit BDU4xx (MY24) installation guide — speed sensor mounting and torque spec.

"340 % 8 1 240% TURBO 340%..."

✓

FTS5 BM25: "Bosch", "DriveUnit", "speed sensor" all match the summary

✓

Vectorize: coherent summary embeds cleanly → high cosine sim

RETRIEVAL QUALITY · n=30 cases

Top-1 accuracy66.7%→93.3%+26.6 pp

Top-3 accuracy83.3%→100%+16.7 pp

COST PER TURN · fewer tokens reach Sonnet

Dollars per turn$0.201→$0.075−63%

Sonnet synthesis tokens49.6K→13K−73%

Part Two

From RAG to Agentic AI

Everything so far was stateless — every request started cold. Now I add what search can't have: durable per-user memory, crash recovery, and query-aware context compression. Watch one number fall the whole way down:

~36¢ → ~3¢ per tool turn

⌬

A Durable Object Per User (Project Think)

A Worker is a hotel room — fresh every request, your stuff is gone. A Durable Object is your apartment — same address, same key, your memory is still on the counter.

TOOL

PURPOSE

search_documentation

unified hybrid search

search_manuals

PDF manual sections

search_parts

faceted parts catalog

diagnose

symptom → ranked causes (graph)

read_manual

deep-read one manual

get_content_section

fetch a known section

search_internet

web fallback — recalls, news

PER-USER DURABLE OBJECT STATE

Soul block

System prompt from KV — live-refresh per turn

Memory block

Per-user facts via UserMemoryProvider — durable identity

SessionPool

Per-thread history, FTS5-searchable

chatRecovery

runFiber — mid-crash saves partial text

Project Think DO — 4 min.

Lead with the hotel/apartment analogy on the slide.

"A REST API is a hotel — every request walks into a fresh room. State is gone the moment you leave. A Durable Object is an apartment — your stuff is still there when you come back. Each user gets their own apartment."

What's in the apartment:

Conversation history (the back-and-forth with the agent)
User preferences (skill level, response mode, filters)
Recent tool results (so the agent doesn't re-fetch)
Working memory (the agent's notepad for the current session)

The agent loop diagram in the middle: walk through one turn. (1) User message arrives at the DO. (2) Pre-agent classifies intent. (3) Sonnet decides which tools to call. (4) Tools run in parallel. (5) Mini-agent formats output. (6) Sonnet synthesizes the final answer.

Why DOs matter for follow-up questions: "Tell me more about the second one" — only works if the agent remembers what "the second one" was. The DO keeps that state per-user.

Anticipated questions:
Q: "Aren't DOs expensive?"
A: Billed per request + per CPU-second. A typical chat session is pennies.
Q: "What if the DO crashes?"
A: State is persisted to durable storage. Restart picks up exactly where it left off.

⇆

Sub-Agent Offloading: Cheap Bulk Work, Premium Synthesis

High-token tool work goes to a near-free model; the premium model only sees the compressed result and writes the answer. Like a shop, the apprentice does the bulk grunt work and the master tech makes the call.

~$0.36 → ~$0.030 per turn — a 90% cut, same answer quality.

User

Asks question

Pre-agent · gpt-oss-120b

Cerebras · intent classifier · ~$0.001

Sonnet 5 — Premium

$3 in / $15 out per MTok

Cerebras gpt-oss-120b

~$0.10 in / $0.40 out per MTok

Throughput
tokens / second

Cerebras gpt-oss-120b (used here)

~3,000 t/s 20–60× faster

Groq · Together (specialized inference)

~500–1,000 t/s ~7×

Sonnet 5 · GPT-5 · Gemini 2.5 (frontier APIs)

~50–150 t/s 1× baseline

20–60× faster than frontier APIs.
The pre-agent adds ~250ms, not ~3 seconds — that's what makes the offload pattern viable.

Why Cerebras for the mini-agent: I trialed Claude Haiku and Cloudflare Workers AI models too — both viable, but Cerebras's speed at low cost was the best fit for work that runs on every turn.

Sub-Agent Offloading + Cerebras Speed — 4 min.

This is the cost-cutting trick. Hammer it.

Lead: "I use Sonnet 5 — premium frontier model — for synthesis and tool calling. But I don't use it to format tool output. Formatting 30 raw parts results into a nice markdown table is a $0.005 problem; I don't pay $0.05 for it."

The lane diagram:

User — sends query
Pre-agent (Cerebras gpt-oss-120b, ~3000 t/s) — classifies intent, picks tools, ~250ms
Sonnet 5 — decides tool calls, synthesizes final answer
Sub-agent (Cerebras gpt-oss-120b again) — formats each tool's raw output

The throughput strip on the right is the kill shot. Cerebras runs gpt-oss-120b at ~3000 tokens/sec. Frontier APIs run at 50–150 t/s. That's a 20–50× speed delta. For tool-output formatting (a high-token, low-judgment task), Cerebras eats Sonnet's lunch.

The economics: Sonnet stays where quality matters (synthesis). gpt-oss-120b takes the volume work (formatting). Token cost drops ~10×; user-perceived quality stays the same.

What I evaluated: Claude Haiku and Cloudflare Workers AI models both work for the mini-agent; I landed on Cerebras because its speed at low cost is what makes a per-turn sub-agent viable.

Anticipated questions:
Q: "Won't users notice the smaller model?"
A: Not on formatting tasks. I benchmarked it — 100% marker preservation on parts output, 100% on diagnostic steps. The bench is in the codebase.
Q: "What about quality regression?"
A: Mini-agent bench runs against fixed fixtures. If I swap models, I re-bench before shipping.

⤒

Query-Aware Compression: 90% Cheaper, 10× Sharper

UNIT ECONOMICS

✗ BEFORE · DUMB TRUNCATION

The model gets noise.

Like clipping the manual at page 16.

Tool returns 15 KB raw PDF text
Char-count clip at 16K characters
Sonnet sees 8 irrelevant pages of OEM filler
Burns 8 follow-up tool calls narrowing down
No page citations, no section anchors

Response: 230 chars of vague advice

✓ AFTER · QUERY-AWARE COMPRESSION

The model gets answers.

Like a service writer marks the right page.

Same 15 KB result — pre-Sonnet hop adds gpt-oss-120b
Cerebras sub-agent extracts query-relevant sections
Page citations + chunk IDs preserved
Sonnet receives 3 KB of focused content
Answers in 2 calls with diagnostic depth

Response: 2,239 chars of specific guidance

−80%

Input tokens

15K→3K

per tool result sent to Sonnet

−75%

Tool calls / answer

8→2

no follow-up narrowing

−90%

Cost / turn

$0.36→$0.03

≈ $330 saved / 1K turns

+10×

Useful response

230→2,239

chars of specific guidance

RAG retrieves.Prompting reasons.Compression curates. Always-compress tools: read_manual · get_content_section · others compress only above the 4K-token budget

Query-Aware Compression — 3 min.

Lead: "RAG gets you the right document — but the tool hands back 15 KB of raw PDF. Dump that straight on Sonnet and you've turned a retrieval win into a token bill. The fix isn't a bigger window or a cleverer prompt — it's a cheap, fast hop that curates the result before the premium model ever sees it."

The two columns — point at them:

Before (dumb truncation): the tool returns 15 KB of raw PDF, char-clipped at 16K. Sonnet sees 8 irrelevant pages of OEM filler, burns 8 follow-up tool calls narrowing down, gets no page citations — and still only manages 230 chars of vague advice. "Like clipping the manual at page 16."
After (query-aware compression): the same 15 KB goes to a Cerebras gpt-oss-120b sub-agent first. It extracts only the query-relevant sections and keeps the page citations + chunk IDs. Sonnet receives 3 KB of focused content and answers in 2 calls with diagnostic depth — 2,239 chars of specific guidance. "Like a service writer marking the right page."

The four tiles are the whole argument:

Input tokens 15K → 3K (−80%) per tool result that reaches Sonnet
Tool calls / answer 8 → 2 (−75%) — no follow-up narrowing
Cost / turn ~$0.36 → ~$0.03 (−90%) — about $330 saved per 1,000 turns
Useful response 230 → 2,239 chars (+10×) — cheaper AND better, not a trade-off

How it's wired: the high-token deep-read tools always compress — read_manual, get_content_section; everything else compresses only when a result blows past the 4K-token budget. The compressor is the same Cerebras sub-agent offload from the previous slide, now aimed at curation instead of formatting.

The take-home: point at the kicker. "RAG retrieves. Prompting reasons. Compression curates." Compression is the third lever — the one most teams skip, and the cheapest 10× I've found.

Anticipated questions:
Q: "Why not just use a bigger context window?"
A: Bigger windows cost more and degrade quality at the edges. Curating what reaches the model wins on both cost and quality.
Q: "Doesn't the cheap model drop something important?"
A: It preserves page citations and chunk IDs, so Sonnet can always pull the exact source if it needs more. Extraction is exactly the bulk, well-scoped task a fast cheap model is good at.

CASE STUDY Part I of III · The Diagnostic Graph

When retrieval isn't enough.

Search returns text. Diagnosis needs causality — what supplies what, what cuts what, what symptom traces to which subsystem.

✓

01WHAT I HAD

Hybrid search over 2,845 manuals, 2,334 articles, 84K parts.

FTS5 + Vectorize + Meilisearch fused via /api/search/unified
Weighted merge + Gemini structural rerank — ranked passages with page citations

Great at finding the page. Useless at finding the cause.

02WHAT BROKE

"Motor cuts out climbing" returned 12 pages.

Pages described the symptom (motor cutting out under load)
None named the cause (controller overheating) or the subsystem (thermal cutoff disabling motor power)

A symptom isn't a keyword. It's a trace through the system.

⌬

03WHAT I BUILT

A diagnostic graph built on physics, real-world conditions & rider behavior.

825 profiled nodes — each carries tools · symptoms · tiered $0-first steps
2,523 typed edges — causal verbs the agent calls per turn via query_ebike_graph

SUPPLIES_POWER_TO · CUTS_POWER_TO · SYMPTOM_OF · SENDS_SIGNAL_TO · TESTED_BY

One causal trace, walked per query

WHY HAND-CRAFTED

I tried LLM extraction. It hallucinated. I shipped hand-crafted instead — and the agent still queries the graph every diagnostic turn. Here's the trade-off, the anatomy, and the live trace.

Case Study Intro — 3 min. Trilogy starts here.

Frame the trilogy: "Over the next three slides I'll walk you through one specific problem the standard RAG stack couldn't solve, and how I solved it."

The three cards:

What I HAD (green): 2,845 manuals indexed, hybrid search, Contextual Retrieval. Strong on lookup.
What BROKE (red): "My motor cuts out on hills" — RAG returned 30 chunks about motors, batteries, electricals. None named the cause (controller overheating) or the subsystem (thermal cutoff disabling motor power).
What I BUILT (orange): A hand-crafted diagnostic graph that encodes the real physics (thermal, electrical, and hydraulic load paths), real-world operating conditions (sustained climbs, heat, battery sag), and rider behavior (how the bikes are actually ridden and how riders describe a fault) — not just text. 825 nodes, each with its own diagnostic profile, queryable per turn.

The progress dots at the bottom: this is slide 1 of 4. I'm at "the problem." Next slide is the trade-off. Then the anatomy. Then the live demo.

Anticipated questions:
Q: "Why couldn't RAG figure it out?"
A: RAG retrieves text by similarity. It can't reason about causality. Chunks that say "motor cutout" don't necessarily say WHY. The graph encodes the why.

PART II OF III The Trade-off

Why hand-crafted beat LLM extraction.

Dead-end

LLM EXTRACTION · TRIED IT

High cost. High risk.

Hallucinated edges (motor PART_OF tire?) → fabricated repair advice
963 docs → Anthropic Batch API → entities + relationships
31 duplicate "Battery" variants → canonicalization nightmare
No auto-sync — graph drifts every time a doc changes
~$40 per full re-extraction × every doc update
No audit trail — can't diff a node's history

Kept it OFF in production

Shipped

✓

HAND-CRAFTED · SHIPPED IT

$0 cost. 100% deterministic.

825 nodes in TypeScript — one file, one source of truth
Expert-authored with e-bike diagnostic ordering
Regenerate SQL + seed in seconds — zero LLM cost
Zero hallucinated edges → no fabricated repair advice
Every change is a PR with a reviewable diff
Agent still uses GraphRAG to traverse — humans author, LLMs query

LIVE in production, every diagnostic query

" Truth is human work. Reasoning over it is LLM work. "

INTEGRATED WITH EVERY DIAGNOSTIC QUERY

diagnose('motor cutout climbing') → sym_motor_cutout_under_load (12 ms · D1)

query_ebike_graph('controller', 'interrupts') → brake_cutoff_sensor, shift_sensor (safety overrides)

lookup_error_code('E021') → controller_thermal_cutoff (69 codes indexed)

Trade-off: Dead-end vs. Shipped — 3 min.

The story arc: I tried the LLM-extraction approach first. It failed. Then I hand-authored. It worked.

The DEAD END card (left):

Anthropic Batch API on 963 docs → spit out an entity graph
31 duplicate "Battery" variants, hallucinated edges, no way to auto-sync
Every doc change re-broke the graph

The SHIPPED card (right):

825 hand-crafted TypeScript nodes
One file, one PR for any change, full diff audit trail
The agent still uses GraphRAG to TRAVERSE it — I just don't let LLMs AUTHOR it

The lesson under the cards: "Truth is human work. Reasoning over it is LLM work."

Anticipated questions:
Q: "Doesn't 825 nodes feel like a lot of manual work?"
A: It's about a week's work for someone who knows e-bikes. The agent uses it for every diagnostic turn — ROI was clear by week two.
Q: "How do you maintain it?"
A: New parts → new nodes via PR. Pre-commit hook catches structural inconsistencies. Same workflow as any TypeScript file.

PART III OF III Anatomy & GraphRAG Retrieval

Not a search index — a diagnostic brain.

825 expert-authored diagnostic profiles · 2,523 weighted causal edges — error codes are just one way in.

ONE DIAGNOSTIC SLICE · D1 + KV

Think of it as the bike's wiring diagram the AI can walk: parts are the nodes, the labeled wires between them (typed, weighted edges — SYMPTOM_OF · CUTS_POWER_TO) are the links, and the agent traces the circuit from symptom to cause instead of reading a paragraph that happens to mention the part. A real slice of the 825-node graph; live conditions (a climb) re-weight the paths. Expert-authored, not LLM-scraped.

symptom cause / failure component edge = relation · p = probability

ALSO IN THE GRAPH

18 node types · 85 typed relations · environmental multipliers (rain · cold · cargo)

SYMPTOM_OF · CAUSES_FAULT_IN · FAILS_AS · TESTED_BY · INTERRUPTS · CUTS_POWER_TO

A PLAYBOOK PER NODE — thermal_cutoff

TOOLSmultimeter · IR thermometer · torque wrench

CONSUMABLESdielectric grease · thermal paste

SYMPTOMScutout under load · power returns after cooldown

TIER 0 · $0temp check after a climb · airflow + brake drag

TIER 1 · metertemp-sensor resistance · connector continuity

TIER 2 · swapreplace thermal sensor / controller

Every node carries this — cost-tiered, e-bike-specific, cheapest-first.

HOW GRAPHRAG ANSWERS — "motor cuts out climbing"

1MATCHdiagnose maps the phrase to a node — codes & the 768D vector are other ways in → sym_motor_cutout_under_load

2TRAVERSEquery_ebike_graph walks causal edges to pull the connected causal subgraph

3RANKscored by base p × live conditions (climb = load + heat) → thermal_cutoff · battery_cell_sag · controller_overcurrent

4ANSWERSonnet reasons over the causal subgraph and names the cause + the fix

Plain RAG retrieves passages.
GraphRAG retrieves the causal chain.

Graph Anatomy — 3 min.

Lead with the framing, not the stat dump. "Not a search index — a diagnostic brain." The differentiator isn't error codes (that's a lookup feature); it's that every one of the 825 nodes carries a hand-authored diagnostic profile. Numbers in the sub-line: 825 profiles, 2,523 weighted edges, 18 node types, 85 typed relations. Don't dwell on the digits — make the point that this is curated, not scraped.

The three panels (left → right):

Curated ontology (graph slice): typed nodes (symptom · cause · component) linked by typed, weighted edges. Live conditions (a sustained climb) re-weight probabilities before ranking. Expert-authored, not LLM-scraped.
A playbook per node (the hero): every node knows how to diagnose itself — tools, consumables, symptoms, and cost-tiered steps: Tier 0 ($0 visual/physical) → Tier 1 (multimeter) → Tier 2 (replacement). Cheapest-first. This is what no generic RAG bot or PDF search has.
Probability-ranked fault isolation: for "motor cuts out climbing" — diagnose maps to a node (its internal find-symptom step), query_ebike_graph walks the causal edges, suspects ranked by base_p × environmental multipliers, Sonnet reasons over the causal subgraph. Error codes (69) and the 768D symptom vector are just two entry points.

The one-liner to land: a curated causal ontology where each node knows how to diagnose itself, cheapest-first.

Anticipated questions:
Q: "Why not Neo4j?"
A: Overkill. Adjacency list in D1 (two tables: nodes, edges) is fast at 825 nodes, free, and joins with everything else.
Q: "Probability scores — where do they come from?"
A: Hand-assigned base probabilities per edge, with environmental multipliers (rain, cold, cargo load). Calibrated by domain expert.

LIVE DEMO The Agent in Motion

From symptom to diagram — live.

Shop Shepherd · shepherd.shopdawg.appclaude-sonnet-5

Customer says the motor cuts out when climbing hills. Bafang M620.

→ tool: diagnose

{ "query": "motor cutout climbing", "limit": 3 }

← sym_motor_cutout_under_load 12 ms · D1

→ tool: query_ebike_graph

{ "node": "controller", "verb": "interrupts" }

← 4 nodes, 5 edges · rendering trace… 8 ms · D1

Likely controller overheating or battery cell sag — under sustained climbing load, pack voltage drops or the controller overheats, and its cutoff trips. Diagram on the right.

GENERATED MERMAID · RENDERED INLINE graph TD

symptom cause component → live demo

If you don't work on bikes motor = the web server · controller overheating = CPU throttling under load · battery sag = DB / resource latency under load

Live Demo — 4 min. Show, don't tell.

The animation does the talking. Let it run.

What's happening on screen:

User types: "My e-bike motor cuts out on hills."
Agent calls diagnose → matches the symptom node in 12ms (D1 SQL lookup, not LLM; find_symptom is the graph step inside diagnose)
Agent calls query_ebike_graph(controller, interrupts) → returns 4 nodes, 5 edges in 8ms
Agent synthesizes the Mermaid diagram on the right — that's actually generated by the model from the graph results.
Answer: "Likely controller overheating or battery cell sag — under sustained climbing load, pack voltage drops or the controller overheats, and its cutoff trips."

The "12 ms · D1" and "8 ms · D1" badges: emphasize these are SQL round-trip times, not the full chat turn. The full turn including Sonnet's thinking is several seconds. The graph queries themselves are cheap.

One-line translation for the non-cyclists (maps the nodes on screen): motor = the web server, controller overheating = CPU throttling under load, battery sag = DB/resource latency under load. Say it once here — don't repeat it on other slides.

If asked to do live: have shepherd.shopdawg.app pre-warmed and a similar query staged. Don't risk a cold-start in front of the audience.

Anticipated questions:
Q: "The Mermaid is generated by the LLM?"
A: Yes — Sonnet writes the diagram in its response, mermaid.js renders it client-side. Pattern detailed in my internal docs.
Q: "Why not just show the answer in prose?"
A: For diagnostics, the picture is worth the tokens. Mechanics scan a flowchart in seconds.

◎

Observability: One Pane of Glass

I was already running PostHog for product analytics — funnels, retention, session replay. Turning on its LLM observability dropped every AI trace onto that same dashboard: cost, tokens, latency, and tool-routing, per turn.

ONE REAL TRACE · POSTHOG

"Best margin on 26×4 fat tires + tubes, grouped by distributor"

$ai_trace $ai_generation $ai_span

the whole turn

31.8s

pre-agent

0.6s

route tools

6.3s

search: tires

2.1s

search: tubes

3.3s

format ×2

3.1s

synthesis

24.9s

44.9K tokens · $0.072 · 8 spans. The Sonnet synthesis alone is 24.9s — 78% of the turn (streaming a long answer, not a stall); the 3 searches + 2 Cerebras formatters total ~3s and $0.009. You only see that once it's traced.

ONE PANE OF GLASS

AI traces land right next to the product analytics I already had.

Per-turn cost & tokens — cache-aware, so prompt-cache savings show up
Tool routing — which tools fired, in what order, how long each took
Latency per step — pre-agent · tools · synthesis
Beside funnels, retention & session replay — already running

Recommended, not required — and not part of the ~$80/mo search infra. PostHog was already in the stack; the LLM layer was icing on the cake.

posthog.com/docs/ai-observability

Observability — 1.5 min. (Optional add-on slide.)

Framing first, so nobody thinks it's a cost line: "This isn't part of the ~$80/mo. I was already running PostHog for product analytics — funnels, retention, session replay. The LLM observability is icing on the cake: it puts every AI trace on the same pane of glass."

The waterfall (left card) is a REAL trace — a live parts query: "best margin on 26×4 fat tires + tubes, grouped by distributor." Every turn emits one $ai_trace (the whole turn), nested $ai_generation per model call (model, tokens, cache-aware cost), and a $ai_span per tool/mini-agent (args, output, latency). The punchline to point at: the synthesis bar. The two searches finished in ~2–3s and the Cerebras formatters in ~3s — but Sonnet writing the final answer took 24.9 seconds, 78% of the whole turn. Total: 44.9K tokens, $0.072. Sonnet (route + synthesis) is ~$0.063 of that; the searches + 2 Cerebras formatters are $0.009 combined — that's the sub-agent offloading from earlier, now visible in the trace. "You'd never guess the bottleneck is the write-up, not the search — until you trace it."

Why it matters (right card): cost and tokens per turn, which tools fired and in what order, latency per step — and because it's the same PostHog, it sits next to the product funnels and session replay. One pane of glass.

The lesson it sets up: next slide, lesson 1 is "observability first." You can't tune what you can't see — I had this wired before I optimized anything.

Link: posthog.com/docs/ai-observability for their LLM analytics.

Anticipated questions:
Q: "Does PostHog add latency to the chat turn?"
A: No — events are captured and flushed out-of-band (batch size 10, 2s interval, explicit flush in the finally). The user never waits on it.

∞

Real-World Scale: What This Handles

PDF Manuals Indexed

Gathered with AI search — Brave · Tavily · Firecrawl
OCR'd · ~593K chunks · full-text + vector indexed

Repair Articles

Built from Google Deep Research, human-curated
full-text + vector indexed

Parts in Catalog

Nightly refresh from 3 vendor feeds
D1 faceted, Meili-accelerated

Entire production stack

Workers + D1 + Vectorize + Meili included

Real-World Scale — 2 min. The "I actually shipped this" slide.

The four cards:

2,845 manuals — sourced from 1,200+ e-bike SKUs. ~593K chunks indexed.
2,334 knowledge-base articles — my own repair documentation
84,000+ parts — QBP catalog with e-bike relevance scoring
$80/mo production stack cost — your actual bill. Not a marketing number.

The $80/mo card is the brag. Hammer this. "That's the entire stack — D1, Vectorize, Workers, R2, KV, Workers AI, plus the Meilisearch instance for the parts catalog. Every model call, every embedding, every byte stored."

Anticipated questions:
Q: "What about LLM API costs?"
A: Sonnet + Cerebras + embedding calls run another $50–200/mo depending on traffic. The $80 is the infrastructure floor — model costs scale with usage.
Q: "How big is the user base?"
A: I'm pre-launch. The system is sized for shop staff (10s) initially, scales to thousands without architecture changes.

Cost Comparison: Traditional vs. Cloudflare

UNIT ECONOMICS

Where the monthly spend goes · representative production scale · JUN 2026 rates

Traditional

Elastic Cloud $500–2,000

Pinecone

EC2 · RDS

$622–2,392

Cloudflare

$80

~$80

Cloudflare's $80, zoomed in 25× — where it actually goes

Meilisearch $30

Workers AI $0–20

R2 + KV

Vectorize

Workers

Orange = Meilisearch, the only paid third-party — skip it and D1 FTS5 covers parts (Workers Paid $5 · D1 free tier). Infra floor only — LLM inference (Sonnet + Cerebras) adds ~$50–200/mo at usage.

25× cheaper · same throughput · scale-to-zero baseline

25×

cheaper than the
traditional stack

$0.5–2.3K

saved every
month

$6.5–28K

saved every
year

when idle ·
scale-to-zero

Cost Comparison — 3 min.

The story: 25× cheaper at production scale. Every line item is verified at June 2026 rates.

Traditional side (left):

Elastic Cloud Standard, 3 nodes: $500–2,000/mo
Pinecone Standard (serverless): $50–300/mo
OpenAI embeddings: $5–25/mo (text-embedding-3-small at $0.02/M tokens)
EC2 t3.medium + RDS db.t3.small: $67/mo
Total: $622–2,392/mo

Cloudflare side (right):

Workers Paid: $5
D1: free tier covers 10M reads / 1M writes
Vectorize at 593K vectors: $0–8
Workers AI + R2 + KV: $0–35
Meilisearch Cloud: $30 (the only third-party)
Actual production bill: ~$80/mo

The bar visualization at the bottom: $2,000 vs $80 = 25× cheaper. Same throughput. Scale-to-zero baseline.

Anticipated questions:
Q: "Aren't those Elastic numbers low?"
A: They're the Standard tier hot-node range. Enterprise tier easily hits $5–10K/mo. I chose the conservative comparison.
Q: "What about vendor lock-in cost?"
A: Real. The Long Road slide later addresses how to keep architecture portable.

💡

Lessons from Production

Five hard-won lessons from running this in production — not the order I learned them, but what I'd hand my past self on day one.

1 Observability

Observability first

Instrument before you optimize with PostHog — you can't tune what you can't measure.

2 Cost

Offload heavy work

Cheap models do bulk context; a premium model synthesizes. ~10× cheaper per token.

3 Judgment

Hand-author the graph

Truth is human work; let the LLM traverse it. 825 profiled nodes beat batch-API extraction.

4 Metadata

Contextualize chunks

Tag every chunk: source, section, vehicle class, date. Context disambiguates. 66.7% → 93.3% Top-1.

5 Testing · CI

E2E over mocks

Live end-to-end tests on every deploy catch what unit mocks never will.

The one-line version Let cheap models do the volume, keep humans on the judgment calls, and measure everything in between.

Lessons from Production — 4 min.

Five lessons, ~45s each. Walk through. Land lesson 3 carefully — it's the spicy one.

1. Observability first. PostHog from day one. You can't tune what you don't measure. Latency, tokens, tool routing, drop-off — all instrumented before any tuning happened.
2. Offload heavy context to sub-agents. Cerebras gpt-oss-120b for tool formatting. Sonnet for synthesis. ~10× token cost drop, no quality regression.
3. Hand-author the graph; let the LLM traverse it. Anthropic Batch API on 963 docs hallucinated edges, 31 duplicate "Battery" variants. 825 hand-crafted TypeScript nodes works. Truth is human work. Reasoning over it is LLM work.
4. Metadata-tagged chunks beat raw chunks. Layer 2 metadata + Layer 3 LLM context (Contextual Retrieval pattern). 66.7% → 93.3% Top-1.
5. Test in CI like any other code. 30-case regression suite. Every PR runs it. Quality gates that ship at the speed of CI, not at the speed of "I'll eval later."

On lesson 3 — be ready for pushback: "But Brett, you DO use LLMs to author the chunk context_llm summaries — isn't that a contradiction?" The answer is on the Contextual Retrieval slide: LLMs enrich existing text, humans invent ontology. Different jobs.

Part Three

Build It Yourself

Two paths, same destination — managed AI Search to start, raw primitives when you want full control.

None of it is bike-specific — the architecture holds for any domain.

STEP 1 · THE SHORTCUT Start Here

    1 · AI SEARCH — SHIP THIS WEEKEND
    →
    2 · CUSTOM STACK — WHEN YOU SCALE
  

Cloudflare AI Search.

The retrieval primitives you just saw — provisioned for you, fully managed.

WHERE AI SEARCH FITS IN YOUR STACK

~80% of the retrieval pipeline without writing a Worker. It provisions the whole retrieval pipeline you just built:

✓ AI SEARCH

Hybrid RAG — vector + BM25 keyword, reranked

R2 · Workers AI · Vectorize · AI Gateway · query rewriting · cited answers

Faceted partsMeilisearch · typo-tolerant filters

Agent + MemoryDurable Objects · diagnostic graph

↑ bolt these on in Step 2, as you outgrow the shortcut

Open beta · free within limits · 4 MB max per file

IF YOU REMEMBER ONE THING

Same R2 + Workers AI + Vectorize + AI Gateway pipeline.

Click to provision. No wrangler glue.
Drop docs into a bucket. Auto-chunked, auto-reindexed.
Hit one endpoint. Ranked results or full RAG answer.
Same scale-to-zero economics. Same edge latency.

developers.cloudflare.com/ai-search

The Shortcut: Cloudflare AI Search — 2 min.

Lead: "If you want 80% of what I just showed you with 5% of the work, this is the answer."

What Cloudflare AI Search gives you out of the box (verified against the docs, Apr 2026):

Hybrid search — vector + BM25 keyword with configurable RRF/max fusion (NOT vector-only)
Reranking, query rewriting, and similarity caching — all managed
Managed embedding (Workers AI) + R2-backed storage + Vectorize index, auto-synced
Metadata filtering, cited sources, a built-in MCP endpoint
One service: search() for ranked chunks, chatCompletions() for a generated answer

The navy "remember one thing" card: "Most teams should start here. You can always graduate to the primitives later."

What you actually give up (rerank + query rewriting are now managed, so it's not those): hand-tuned hybrid scoring and your own rerankers, the faceted parts catalog (Meilisearch), the diagnostic graph, and the agent loop + per-user Durable Object memory. If you don't need those, you save weeks.

Status: open beta — free within limits, 4 MB max per file. Fine to "ship this weekend," just flag the beta if asked.

Anticipated questions:
Q: "Why didn't you use this?"
A: I started before it launched. I'd also lose the graph and the agent. For my use case, the full primitive control was worth the build time.

STEP 2 · THE LONG ROAD When You Outgrow the Shortcut

    1 · AI SEARCH — SHIP THIS WEEKEND
    →
    2 · CUSTOM STACK — WHEN YOU SCALE
  

Same primitives, full control.

Started on AI Search? Steps 1–4 (the whole hybrid retrieval pipeline) come provisioned — your real build is the agent + memory + domain graph.

Database · SQL

D1 + FTS5 for keyword ✓ via AI Search

wrangler d1 create my-search-db CREATE VIRTUAL TABLE ... USING fts5()

Vectors

Vectorize for semantic ✓ via AI Search

wrangler vectorize create my-index \ --dimensions=768 --metric=cosine

Ingest · Compute

Worker to ingest & embed ✓ via AI Search

Chunk with metadata, embed via Workers AI, write to D1 + Vectorize in one transaction.

Search · API

Hybrid endpoint ✓ via AI Search

Parallel FTS5 + Vectorize, weighted merge by source, threshold + dedup.

Agent · Memory ★ your real build

Agent layer for tools & memory

Install @cloudflare/think — streaming, tools, per-user Durable Object memory, crash recovery.

Graduate to Step 2 when ANY of these is true

Big docs — PDFs past the beta's ~4 MB file ceiling

Causal diagnosis — typed-edge GraphRAG, not semantic proximity

Per-user state — durable memory + crash recovery (Durable Objects)

Faceted catalog — typo-tolerant facet filters over 84K parts (Meilisearch)

Custom scoring — hand-tuned hybrid ranking / your own rerankers

Tool breadth — a long tail of bespoke agent tools

The Long Road: Same primitives, full control — 2 min.

This is Step 2 — the DIY build, for when you outgrow AI Search. Lead: "Same primitives Cloudflare AI Search uses internally — now you own them."

The continuity (the green chips): steps 1–4 — keyword, vector, ingest/embed, the hybrid endpoint — are exactly what AI Search already provisions. Your real build is step 5: the agent layer, per-user Durable Object memory, and the hand-authored diagnostic graph.

Walk the five steps:

1: D1 + FTS5 — keyword
2: Vectorize — semantic
3: Worker to ingest & embed
4: Hybrid endpoint — merge & score
5: Agent layer — tools & memory (the differentiator)

The trigger matrix at the bottom answers "WHEN do I graduate?" — point at it: big docs past the ~4 MB beta ceiling, causal diagnosis needing typed-edge GraphRAG, per-user state, faceted parts (Meilisearch), custom scoring, or a long tail of agent tools. Any one = graduate.

Anticipated questions:
Q: "How long is the build?"
A: Bare-minimum hybrid in a weekend. Production-grade with eval suite + monitoring in 2–4 weeks.

SCAN FOR
THESE SLIDES

Start Building Today

Everything shown here runs on Cloudflare's free or pay-as-you-go tiers

Start on AI Search → grow into the primitives as you scale. Same edge, same economics, no rip-and-replace.

developers.cloudflare.com/d1 developers.cloudflare.com/vectorize developers.cloudflare.com/workers-ai developers.cloudflare.com/durable-objects blog: AI Search primitive blog: Project Think

See it live → shepherd.shopdawg.app

Built for e-bikes. The primitives fit any industry.
Same engine — healthcare is next.

Closing — 1 min + Q&A.

Read the thesis line — this is the takeaway: "Built for e-bikes. The primitives fit any industry." None of this stack is bike-specific — it's the general recipe (manuals + parts + diagnostics → hybrid search + a durable agent + a hand-authored graph). We're taking the same engine to healthcare next. Tell them: point these primitives at your own domain.

The CTAs: three doc links. Glance at them, don't read aloud.

The live link: shepherd.shopdawg.app is up. People can poke it.

Open the floor: "Happy to take questions, and if anyone wants the source pointers or has war stories from their own search builds, find me after."

If asked about capital / funding: "Not what I came here to ask for, but if you've got a thesis on edge AI tooling for vertical use cases, I'd love to talk after."

Most likely Q&A topics to prep for:

Eval methodology (the 30-case thing)
Cost numbers (verified June 2026 rates)
Why hand-author vs LLM-extract for the graph
Cold start tax on the live demo
Hand-author vs LLM-context contradiction (Contextual Retrieval slide covers it)

AI-Powered Hybrid Search& Agentic Assistants

Why I Built This — It Starts at the Workbench

The Problem with Traditional Search Infrastructure

Elasticsearch / OpenSearch

Pinecone / Weaviate

Regional Lock-in

Stitched Stack

Why Cloudflare? The Edge Advantage

Usage-Based Pricing

Global Edge Network

Unified Platform

Zero Cold Starts

Three Generations of Compute

Virtual machines

Containers

V8 isolates

Search Architecture Deep Dive

The Search Stack — Think of It Like Shopping

Layer 1: Find the Right Aisle — Vectorize

Why Vector Search Matters

Embedding Pipeline

Semantic Routing

Layer 2: Pick Off the Shelf — D1 + Meilisearch

What Faceting Is

Why D1 and Meilisearch?

Faceted Shelf in Action

Layer 3: Scan the Barcode — D1 + FTS5

How It Works

Progressive Lookup

Hybrid Search: Merging Keyword + Vector

Source Weights

Short Query Optimization

PDF Pipeline: OCR, Extract, Optimize

Why OCR Every PDF?

Why Chunking Context Matters

The Naive Chunking Problem

Naive Chunk (no context)

Metadata-Tagged Chunk (structural context)

Contextual Retrieval: Quality Up, Cost Down

Indexes can't find it

Both indexes hit at rank 1

From RAG to Agentic AI

A Durable Object Per User (Project Think)

PER-USER DURABLE OBJECT STATE

Sub-Agent Offloading: Cheap Bulk Work, Premium Synthesis

Query-Aware Compression: 90% Cheaper, 10× Sharper

The model gets noise.

The model gets answers.

When retrieval isn't enough.

Hybrid search over 2,845 manuals, 2,334 articles, 84K parts.

"Motor cuts out climbing" returned 12 pages.

A diagnostic graph built on physics, real-world conditions & rider behavior.

Why hand-crafted beat LLM extraction.

High cost. High risk.

$0 cost. 100% deterministic.

From symptom to diagram — live.

Observability: One Pane of Glass

ONE REAL TRACE · POSTHOG

ONE PANE OF GLASS

Real-World Scale: What This Handles

PDF Manuals Indexed

Repair Articles

Parts in Catalog

Entire production stack

Cost Comparison: Traditional vs. Cloudflare

Lessons from Production

Observability first

Offload heavy work

Hand-author the graph

Contextualize chunks

E2E over mocks

Build It Yourself

Cloudflare AI Search.

WHERE AI SEARCH FITS IN YOUR STACK

IF YOU REMEMBER ONE THING

Same primitives, full control.

Start Building Today

AI-Powered Hybrid Search
& Agentic Assistants