A personalized learning platform that turns what you read into a queryable knowledge base

The problem

People read constantly — articles, papers, docs — and almost none of it gets captured in a form they can come back to. I wanted a system that turns unstructured reading into a structured knowledge base you can actually query: capture a page now, and later ask questions and get answers grounded in what you have read.

That meant three hard problems in one:

1. An ingestion surface the user barely has to think about (a browser extension, not a form).
2. An AI pipeline that reliably extracts, classifies, and organizes knowledge from messy web content.
3. A retrieval layer — semantic search plus a chatbot — that makes the whole library queryable and keeps its answers grounded in the user’s own content.

Architecture

The system is a Turborepo monorepo — six apps and eight shared packages. The shape that matters is the split between the write path (a Step Functions pipeline that extracts knowledge with LLMs and writes content plus embeddings to Postgres) and the read path (a tRPC Lambda and a retrieval-augmented chatbot that serve the dashboard from cached and semantically-searched data), with EventBridge connecting the two and a WebSocket API pushing real-time progress back to the dashboard while content is processed.

Key decisions

Step Functions for the ingestion pipeline

Processing a captured link is a chain of AI workflows: convert HTML to markdown → evaluate quality → select context → extract metadata → extract the knowledge base → extract a summary → classify a review category. A Choice on whether the content is substantive short-circuits thin pages past the expensive LLM steps. Manually-entered content runs a separate, much shorter state machine.

Step Functions was the right fit because:

• Each AI call gets its own isolated retry / timeout / error branch.
• Per-execution history is observable without extra tooling.
• The Choice gate skips costly extraction for content that isn’t worth it.

Each step emits an EventBridge event on completion, so downstream work hangs off the pipeline without the pipeline knowing about it. Running this as one long Lambda would have traded that clarity for nothing I needed.

One Postgres, not three datastores

The system originally ran three stores: Neo4j for a topic graph, DynamoDB for relational data, and Chroma for vectors. I collapsed all three into a single Postgres (Drizzle ORM, Neon serverless driver):

• Relational tables for content, sources, topics, contexts, and subscriptions.
• vector(1536) pgvector columns on content sources and topics for embeddings.
• A generated tsvector column with a GIN index for full-text search.

The notable tradeoff: I dropped the hierarchical topic graph in favor of flat topics. The graph’s traversal queries were elegant but weren’t earning a second database in an MVP — and folding everything into one store deleted an entire class of cross-store consistency bugs (the old DynamoDB-streams-into-graph-rebuild sync). One database, one transaction boundary, one place for the truth.

Hybrid semantic search and reactive topic organization

Search is the core of the product, so it lives in the domain layer, not bolted onto persistence. Content search fuses dense cosine similarity (pgvector embeddings) with Postgres full-text (tsvector + ts_rank) via Reciprocal Rank Fusion; topic search is dense-only. Embeddings come from openai/text-embedding-3-small (1536 dims) through the Vercel AI SDK over OpenRouter.

That same vector search is how topics organize themselves. Topic assignment is deliberately not a pipeline step — when knowledge extraction finishes, the pipeline emits an event and a separate handler reacts: it pulls the new content’s embedding, queries pgvector for the 20 most similar existing topics, and hands those candidates to an attach-or-create-topics LLM workflow that decides what to reuse and what to create. Decoupling it this way lets the event dispatcher run multiple handlers off one event without growing the state machine.

A transactional outbox for event durability

Domain-event durability is owned by a transactional outbox in Postgres (replacing the DynamoDB Streams approach from before the migration). A tRPC mutation that needs to emit events wraps its write in withOutbox(...): the domain change and the event rows commit in the same transaction, so an event can never escape without its write — or vice versa. A scheduled drain-outbox Lambda forwards committed rows to EventBridge, which fans out to consumers like the WebSocket progress push and the topic-attach handler.

Observability as code

There’s no managed AI-tracing vendor. AI workflows are instrumented with the AI SDK’s experimental_telemetry plus structured logging, and a correlation ID carried through AsyncLocalStorage lands in every log line across Lambda handlers, tRPC, event handlers, and the outbox. Logs forward to Grafana Cloud Loki via a loki-forwarder Lambda; a CDK Aspect auto-attaches a subscription filter to every Lambda log group at synth time, so new functions inherit log shipping for free. Grafana dashboards and alerts themselves are managed as code in Terraform.

Stack

Infrastructure: AWS CDK (Lambda, Step Functions, API Gateway HTTP + WebSocket, EventBridge, SQS), Terraform for Grafana Cloud dashboards + alerts, Stripe webhooks. Data: PostgreSQL (Drizzle ORM, Neon serverless) with pgvector embeddings and tsvector full-text, Redis (ioredis) cache. Backend: TypeScript, tRPC, Zod, event-driven handlers, transactional outbox. AI: Vercel AI SDK routed through OpenRouter, openai/text-embedding-3-small embeddings, hybrid dense + full-text retrieval fused via RRF, experimental_telemetry tracing, a FAST / SMART model split for cost control. Frontend: Next.js dashboard with a retrieval-augmented chatbot, Next.js marketing site, WXT browser extension, shared Radix + Tailwind component library. Auth & payments: Better Auth (EdDSA JWT verified against a static JWKS), Stripe.

Impact

Beyond the product, this is the proving ground for the patterns I now bring to every architecture I lead: Step-Functions-as-workflow over ad-hoc chaining, a single Postgres over polyglot persistence when the second datastore isn’t earning its keep, hybrid vector + full-text retrieval for AI features, a transactional outbox for events that can’t be lost, and observability wired as code. It’s the most complete end-to-end system I’ve shipped solo — infrastructure, backend, AI, frontend, and browser extension.