AI Service Assistant: Build a Production AI App with Bedrock, Salesforce & Foundry
Build one realistic AI application from an empty folder to production: a service assistant that turns a customer's chat and photos into a triaged service case and a booked technician, for a fictional commercial kitchen-equipment maker. Beginner to advanced across 12 modules: Next.js, the Vercel AI SDK and Claude on Amazon Bedrock, Postgres, Redis and S3, Salesforce, Palantir Foundry, event-driven integration on AWS, LLM evals and server-side guardrails, CDK and CI/CD, and production monitoring, plus the engineering lessons real teams learned the hard way. Bilingual slides and notes (English + 中文).
Premium content
Upgrade to premium to access all lessons, tutorials, and hands-on exercises.
Lesson Overview
What We Are Building — The Acme Service Assistant and Its Architecture
Meet the app you will build across this bootcamp: an AI service assistant that turns a café owner's chat and photo into a triaged case and a booked technician. Follow one request end to end, walk the architecture layer by layer (Next.js, Bedrock, Postgres, Redis, S3, Salesforce, Foundry, EventBridge, AWS), learn the five rules from production that run through the course, and set up your tools and accounts.
The App Shell — Next.js App Router, TypeScript, Tailwind and shadcn/ui
Start building the Acme Service Assistant: create a Next.js 16 app, learn file-system routing (pages, layouts, route handlers, not-found, dynamic segments with typed params), split Server and Client Components, define Tailwind v4 design tokens, add shadcn/ui components you own, and apply the mobile-first rules an embedded app needs from day one.
Typed Configuration — Zod, Environment Variables and Failing Fast
Make bad configuration impossible to ship: describe every setting once in a Zod schema, coerce environment strings into real types, keep server config behind server-only and validate it lazily so builds need no secrets, exit at start-up when it is invalid, and learn why NEXT_PUBLIC_ values are public and frozen at build time.
The Database — Postgres, Drizzle and docker-compose
Give the Acme assistant its own database: run Postgres 18 in docker-compose, model sessions and messages in Drizzle with enums, a foreign key and a fitting index, generate and review SQL migrations, use one lazy connection pool, wrap related writes in a transaction, run scripts outside Next.js, and separate liveness from readiness.
API Design — Route Handlers, Validation, Errors and Idempotency
Build the first API on top of the repository: resource-oriented route handlers with typed params, one RFC 9457 problem-details error shape, request bodies read defensively with a streamed size cap, errors translated at the right layer, idempotent message creation proved with parallel requests, and request IDs with log-safe errors for the failures you did not expect.
Testing With Vitest — Unit Tests, Real Requests and a Real Database
Make every fix stay fixed: set up Vitest 5 for a Next.js 16 app, write table-driven unit tests, test route code with Web Request objects including streaming bodies, use spies, run the real route handlers against a real Postgres test database, prove the tests by putting bugs back, and read coverage as a map.
Git Workflow — Branches, Pull Requests and Pre-Push Hooks
Short-lived branches, Conventional Commits enforced by commitlint, a PR template and review checklist, squash merges, branch protection, and husky 9 hooks (lint-staged on commit, typecheck + lint + unit tests on push) so the fast checks run before code leaves your machine.
Your First Claude Call — Amazon Bedrock Through the AI SDK
Call Claude on Amazon Bedrock with the Vercel AI SDK 7: account setup (model access, the Anthropic form, least-privilege IAM for inference profiles), the AWS credential chain, regions and cross-region inference profiles, a route with instructions, output cap, retries and timeout, a mock model for development, and failures mapped to 503/504 without logging customer text.
Structured Output That Drives the UI
Return decisions as data: a Zod union of turn kinds (question, troubleshoot, emergency, confirm) sent to Claude on Bedrock as a forced JSON tool, validated after the call, rendered as different UI per kind — with a rule-based safety check that answers emergencies before the model is even asked.
Prompts as Modules, per Persona
Treat the prompt like code: small named modules (identity, audience, safety, turns, style) composed per persona — customer, support agent, field technician — in a fixed order, with a version over everything the model is told, prompt files reviewed as snapshot diffs, staff personas gated until login, and tests that keep the safety rule intact.
Streaming the Turn — Server-Sent Events and Partial Structured Output
Stream the structured turn over Server-Sent Events from a Next.js route: partial text for display, a validated final turn for UI, model errors as in-band events (including Bedrock exceptions mid-stream); why the Bedrock Converse provider's JSON-tool mode arrives in one piece while native output streams; the AI SDK's leaky default onError; and aborts that stop the model call.
Tool Calling and the Agent Loop — Lookups, Step Caps and Tool Errors
Give the model tools instead of guesses: a fault-code lookup and a troubleshooting-guide tool with Zod input schemas, run in the AI SDK's multi-step loop with stopWhen: isStepCount(4); how the tools sit next to the JSON tool on Bedrock (toolChoice any, the final turn counts as a step), what a too-small cap or a runaway model does (a clean 503), why unknown tools and bad inputs go back to the model as results, and how to bill and log a multi-step turn.
Grounding in the Fault Catalog — Catalog Prompts, Prompt Caching and Exact Values
Ground the assistant in data you own: a catalog prompt module generated from the same data the tools read, what belongs in the prompt versus behind a tool, Bedrock prompt caching with a cache point after the instructions (tools → system → messages, TTLs, what invalidates it), measuring the prefix against the model's minimum (and why it does nothing on Claude Haiku 4.5 today), reading cache reads and writes from usage, and snapping catalog values at the boundary instead of trusting the model's spelling.
Conversation State — Server-Side History, Idempotent Turns and Safe Replay
Move the conversation to the server: store each turn as the customer saw it, linked to the message it answers (migration with a unique reply_to), a session turn route that answers retries from storage, lets a failed call be retried once, allows one turn at a time with a lease that has an owner and an expiry (taken before anything is stored), re-validates and re-grounds stored turns, and why replaying the model's own messages fails on every later turn (MissingToolResultsError, pruneMessages, responseMessages in AI SDK 7).
Vision — Reading Rating Plates and Error Screens from Photos
Accept photos safely and read them with Claude on Bedrock: a capped raw-body upload whose type is sniffed from its bytes and whose size is read from its header, what an image costs in visual tokens (and the size to resize to on the phone), sending it as an image file part before the question, and grounding the transcription: serials snapped only where the format makes a look-alike unambiguous, fault codes checked against the catalog.
The Wizard as a State Machine — One Union, Pure Transitions, Effects as Events
Rebuild the customer page as a wizard on the session API: UI state as a discriminated union, a pure reducer whose transition table finds holes no reader sees, effects that answer with events and resend the same clientMessageId on a 409, one effect per job (and what StrictMode really replays), a photo reading the customer must confirm, and component tests with Testing Library on jsdom — plus the browser bug jsdom couldn't see.
Inside the App — a Webview, a Typed Bridge and the Back Button
Embed the wizard in a phone app's webview: a small versioned bridge protocol validated on both sides and free of customer data, WKScriptMessageHandler and Android's addWebMessageListener behind one interface, the back button as a pure table announced before the press, the app owning the frame (title bars, safe areas, 44 px touch targets), links and the camera leaving the page — tested with stand-in windows, Playwright at 360 px and a real WKWebView in the iOS Simulator.
Photos From the Phone — Resize, Strip, Upload Direct
Get real phone photos to the model: resize in the browser to exactly what the model sees, apply the EXIF orientation and drop EXIF and GPS by re-encoding, decide with a pixels-only allowlist, upload straight to storage with an S3 presigned PUT that allows one type, one size, one upload and five minutes (signed locally, checked against AWS's documented example), then re-check, hash and delete what fails — with a development stand-in for S3, HEIC across browsers, and real bytes in a real browser.
Booking the Visit — Slots, Holds and a Confirmation Screen
Turn the summary into a booked visit: slots as wall-clock times in the service region turned into instants with Intl (safe across daylight saving), the visit's clock first and the customer's added with its date, a five-minute hold decided by one SQL statement on a primary key, confirmation made idempotent with the hold as its key, a review screen that tells the truth and never dead-ends — and races tested with parallel requests against Postgres, which found two deadlocks.
Signing In — Two Identity Providers, and Owning What You Create
Add sign-in for customers and staff with OpenID Connect and Auth.js: the authorization code flow with PKCE, state and nonce on a real round trip, a development identity provider that speaks the protocol and can't reach production, a stable user id and a persona only the right provider can grant, checking the ID token's signature the library skips (with key rotation), ownership on every route (404 for strangers, 401 for nobody), and redirects that can't be turned against you.
Hand-Off Links — Single-Use Tokens, and No Personal Data in URLs
Let another app open the assistant for a user it has already signed in: the host's server exchanges what it knows for a link with one opaque token; the token is random, stored as a hash in Redis for two minutes and spent with an atomic GETDEL (the GET-then-DEL race reproduced); opening a link spends nothing, continuing takes a click on our own page (login CSRF, clickjacking), destinations are chosen by the server (no open redirect), and the user's machine reaches the conversation without touching the browser.
Embedded in Another Site — Frames, Third-Party Cookies and Proxies
Put the assistant inside a CRM's case page safely: let exactly the right site frame exactly the right pages with CSP frame-ancestors from a runtime setting; keep the sign-in alive in a cross-site frame with SameSite=None; Secure; Partitioned cookies (measured in a cookie lab); stop sessions vanishing behind a TLS-terminating load balancer; talk to the host over postMessage with named origins; and see why no CORS is the right amount here.
Who May Do What — Authorization from the Signed Session, and Least Privilege
Let the CRM's support agent see one customer's conversations, and nothing more: authorization from the signed session instead of the request (no IDOR), one deny-by-default policy with three answers (allow, 403, 404) that every route asks with its action, a grant sealed into the session that is narrow (one customer, one case, view only), short (an hour), revocable (a live key the CRM can delete) and recorded (an access log), and the policy tested as a table and the routes as a matrix.
Words Are Not Instructions — Prompt Injection, and Guarding What the Model Says
Assume the model can be fooled and design around it: no write tools and nothing to steal; invisible-character cleaning that keeps what the customer saw; the customer's words in a block named for each call, and a canary on the instructions' first line; attempts logged, never refused; and a narrow output guard that removes links, emails, phone numbers and markup, replaces recitals, promises and credential requests, never touches an emergency, and guards streamed text word by word — with an indirect injection through a photo, and a mock model that falls for it on purpose.
The Salesforce Data Model — Accounts, Machines, Cases and Work Orders
Model a service business in Salesforce's standard objects — Account, Contact, Asset, Case and WorkOrder — with custom __c fields, picklists and lookups; take 15- and 18-character record ids apart, checksum and all; use the REST API (create, read with fields, update, delete, describe) and branch on Salesforce's error codes; seed a customer with a composite tree, all or nothing; and build against a development stand-in shaped like a real org, through the one client the app will keep — with the steps for a Developer Edition org later.
Reading and Writing Salesforce Safely — SOQL, Escaping, Validation Rules, Upserts and Caching
Turn a booked visit into a Salesforce Case without the mistakes integrations make: SOQL built only through a tag that escapes every value (and the injection it prevents); an admin's validation rule that capped a field, handled by capping at write time with an overflow field; upserts on an external id that can't duplicate, and finding before resending so support's edits survive; people found by a stable id and machines linked only with proof; cached lookups and the API allowance; and a booking that never waits on the CRM — a deadline, late work that still finishes, and a resend job.
Signing In as the App — Salesforce JWT Bearer Flow, Server to Server
Sign in to Salesforce with no person and no password: an external client app, a certificate and a pre-authorized integration user; a short RS256 JWT (iss, sub, aud, exp) exchanged for an access token and the org's address; that address checked before the token goes anywhere; one sign-in per process, a 401 answered by signing in again once, and a back-off when sign-in fails; and a private key that is validated, kept secret and rotated — all against a stand-in login server that refuses what Salesforce refuses, with a runbook for a real org.
Salesforce Calling Out — Named and External Credentials
Let Salesforce call the app with no secret in code: Apex names a Named Credential (where), an External Credential (how: a JWT signed per callout with an org certificate; AWS Signature V4 for AWS; never no-auth) and a permission set (who); the app checks the signature, issuer, audience and the token's own lifetime, answers only for Cases it sent, and logs every read — while the Apex checks the person may see the Case first. The Salesforce files are written but not deployed; the app side runs.
When Salesforce Changes — Change Data Capture and Platform Events
Let Salesforce tell the app what changed: Change Data Capture on Case and a Platform Event for visit updates, subscribed to by replay id. Delivery is at least once, so every effect is idempotent and saved with the position in one transaction; the position is compare-and-set against a second subscriber; data the database refuses is set aside, while a bug stops the line; a lost place is found again by asking for the records. The customer sees where the visit stands, by the org's own definition of closed — never support's status. The Salesforce files are written but not deployed; the app side runs.
Into Foundry — Incremental Ingest, Dedupe and an Object Type
Module F begins: Salesforce's Cases into Palantir Foundry, a change at a time. An incremental extract with a watermark and a deliberate overlap (duplicates, never gaps); one append transaction per run; dates parsed to timestamps at the edge; a Pipeline Builder window that keeps the newest row of each Case; and an object type with a unique key on the clean dataset. Built for real in a Foundry training environment, through a branch and a proposal, including what to do when the pipeline cannot publish its object type.
Files Into Foundry — S3 Sources, File Syncs and Egress
Machine telemetry and service manuals reach Foundry as files in an S3 bucket. A write-once, time-partitioned bucket layout; what a file-based sync sees (paths, sizes, times); Foundry's four sync modes and when each is right; the file edited in place that an append sync never sees; files to rows in Pipeline Builder (JSON Lines, a parsed timestamp, the columns Foundry adds); and getting through the firewall: the S3 source, an approved egress policy, a bucket policy and a two-action IAM policy. Built in a Foundry training environment; no egress policy was requested.
Foundry From the Server — Developer Console, a Service User and the OSDK
The assistant reads Foundry from its Next.js server with the Ontology SDK, as a Developer Console application: a backend service with application permissions, the client credentials grant and the service user Foundry makes for it. The three gates that decide what a token can do (the service user's permissions, the application's restrictions, the scopes asked for); the OSDK's requests and Foundry's property API names; a secret that stays on the server; and the incident: an object type the application was never given, which Foundry answers with 404 — the same as a name that doesn't exist — and code that turned it into an empty list. Object type names read in a Foundry training environment; the application itself was not created.
Changing Foundry — Action Types, Submission Criteria and Backoff
The assistant changes Foundry through an action type, Change Acme Service Case Priority: a modify-object rule, a multiple-choice parameter and a submission criterion with a message people can read. Allowing edits on a synced object type and Foundry's default conflict resolution; validating before applying, because a refused apply says nothing; reading the answer as Palantir's published types define it — the lesson's incident, code written from an example and tested against the same example; and 429/503: the backoff the OSDK already does, why the app adds none, and actions that are safe to repeat. Foundry's settings and wizard looked at in a training environment, without changes.
Writing Back — Webhooks, Writeback vs Side Effect, and Retries
Foundry's action keeps Salesforce in step through webhooks: a writeback that patches the Case in Salesforce before Foundry changes anything, and stops the action when it fails, and a side effect that notifies support afterwards, best effort. Salesforce's own validation rules refusing the writeback in their own words; Data Connection's "not changed" status codes against "may have changed"; retryable codes, none by default, and why only idempotent requests are retried; time limits; and telling the agent the truth — refused, uncertain, or done. The webhooks run in the app's stand-ins with Data Connection's settings; the Foundry configuration follows Palantir's documentation.
Triage in Foundry — AIP Logic, Automate and AIP Evals
New Cases get a first priority without a person, the Foundry way: an AIP Logic function suggests one — the fault catalog first, a model only for the gaps, the answer snapped to the three values the action accepts — and the app calls it as an Ontology query with the OSDK, pinned to a version because the latest may be a pre-release. An Automate automation applies it to new Cases as its owner, at least once; the lesson's incident is a filtered "added to set" condition that re-triaged a Case put back to New and overwrote an agent's priority. An AIP Evals suite — test cases written by people, an exact match, three runs each — compares versions and catches a bad pre-release. Built against the app's stand-in with the documented semantics; the Foundry steps follow Palantir's documentation.
Governance in Foundry — Markings, Property Security, Model Access and Audit
Customer contact details arrive in Foundry, and the lesson governs them: a PII marking that travels with the data; an object and property security policy, and why a hidden property is simply absent from the API's answer — the lesson's incident, "none on file" for every customer; an app that reads as one service user and so reads only what every agent may see, selecting properties by name, with Palantir's guidance on the authorization code grant for per-person data; AIP's data-access policy, which checks the token's markings rather than the prompt; and audit, where Foundry records the application, the app records the person, and a trace id lines them up. Built against the app's Foundry stand-in with the documented behaviour; the Foundry steps follow Palantir's documentation.
A Case Assistant on Foundry — Ontology MCP and the LLM Proxy
A support agent asks about the Case in front of them, and a model answers with the ontology's tools: Ontology MCP turns the Developer Console application's object type, function and action type into MCP tools behind the application's own OAuth; Foundry's LLM proxy takes Anthropic-shaped requests with a Foundry token that has the model scope only and a model from Model Catalog. The server's search would read any customer's Cases, so the app gives the model tools of its own — reads bound to the open Case, and a change only as a proposal the person applies; the lesson's incident is a customer's Case subject that, read through the machine's history, changed another Case's priority when the model held the server's action tool. And an honest look at why an interactive agent should read as the person, not as the course's shared service user. Built against the app's Foundry stand-in and a deterministic stand-in model; the Foundry steps follow Palantir's documentation.
Case Events on EventBridge — a Bus, Rules and Targets
The app stops being asked and starts telling: Case Created and Case Priority Changed are published to an Amazon EventBridge bus with PutEvents — signed over the AWS JSON protocol, every entry's result read — and routed by rule to SQS queues: event patterns tested with TestEventPattern, queue policies for exactly the rules that send, an input transformer, retries and a dead-letter queue, and consumers that must be safe to run twice. The events are thin: ids and values, never free text or personal data. The lesson's incident is a bus name with a typo — a 200 for every event and nothing delivered — and the check that catches it. AWS now recommends its new Custom Event Bus for new applications; the lesson builds on the Classic bus of rules and targets and shows what carries over. Built against the app's EventBridge and SQS stand-ins; no AWS account.
Salesforce to AWS — Event Relay, a Partner Event Source and SQS
Salesforce tells AWS itself: Event Relay sends the org's Case change events to Amazon EventBridge — a named credential with the Region in capitals, a channel and its member, a relay created stopped — through a partner event source that drops everything until a partner bus of its name exists, and a rule puts them in an SQS queue the app reads. A queue keeps no order, so each booking keeps the commit time of the change it shows and the newest wins; a relayed change in a shape the consumer can't read is kept, never deleted. The lesson's incident is a relay stopped instead of paused for maintenance — changes never sent, bookings gone stale — and the reconcile that puts them right. Amazon AppFlow compared. Built against the app's Salesforce and EventBridge stand-ins; no org, no AWS account.
AWS to Salesforce — API Destinations with OAuth
An event on the bus becomes a change in Salesforce: a customer rates a visit, the app publishes Visit Rated, and a rule sends it to Salesforce through an EventBridge API destination — a connection that signs in with the OAuth client credentials flow of an external client app (new connected apps are restricted since Spring '26), a POST of a platform event, and a flow that puts the rating on the Case. Tokens are refreshed only after a 401, by the retry; only 401, 407, 409, 429, 5xx and timeouts are retried. The lesson's incident is the org's daily API limit: 403 REQUEST_LIMIT_EXCEEDED, never retried, and ratings lost without a dead-letter queue — then the DLQ, the alarm and the redrive, and a rotated secret that de-authorizes the connection. Built against the app's EventBridge and Salesforce stand-ins; no AWS account, no org.
Foundry and AWS, Both Ways — a Stream, a Side Effect and Cognito
A two-way bridge between Foundry and the app's event bus. AWS to Foundry: a rule and an EventBridge API destination push every Case event to a Foundry stream through the streams API (publishRecords, all or nothing), signed in as a Developer Console application allowed one operation and an Editor on the stream — compared with a Kinesis streaming sync and an SNS listener. Foundry to AWS: the priority action gets a side-effect webhook that chains the OAuth client credentials grant with Amazon Cognito and calls an API Gateway HTTP API whose JWT authorizer accepts machine tokens without aud; a function puts Case Priority Changed on the bus. A rule keeps Foundry's own changes from echoing back. The lesson's incident: two publishers for one fact, and on-call paged twice — then one publisher. Built against the app's stand-ins; no Foundry stack, no AWS account.
Long-Running Work — a Step Functions Workflow, Not a Long Request
Suggest a priority waited for Foundry's model inside the request — and the load balancer closed it at 60 seconds, the agent clicked again, and the model ran twice. Now the request answers 202 with a Location and Retry-After, and an AWS Step Functions Standard workflow does the work: it waits for the Case to reach Foundry (Wait and Choice), asks Foundry's triage with Retry (backoff, jitter) and Catch, waits up to eight hours for the agent's decision with a task token, and applies it. The click's Idempotency-Key names the execution, so a retried click is the same run. Failed runs alert through Step Functions' status events; JSONPath and JSONata compared. Built against the app's Step Functions stand-in; no AWS account.
When Messages Fail — Dead Letters, Redrive and Consumers That Run Once
A relayed change the consumer couldn't read came back every minute for two weeks, until the queue's retention deleted it unseen. Now every queue has a redrive policy (five receives) and a dead-letter queue that keeps longer than its source, with an alarm on it, and DLQ redrive (StartMessageMoveTask) puts fixed messages back — as new messages with new MessageIds. Events are written in the change's own transaction (a transactional outbox, drained in order with backoff and a dead state), the on-call pager pages once per escalation (keyed on the app's eventKey, with a ledger, a claim and a dedup key), and a reconcile publishes what Foundry's side effect never delivered. EventBridge archive and replay compared. Built against the app's AWS and pager stand-ins; no AWS account.
Evaluations — Scenarios, a Mock, Cassettes and the Real Model
In the lesson's scenario, a shortened prompt ships with every unit test green — the tests mock the model — and for a week the model skips the reviewed troubleshooting steps and offers technician visits instead. This lesson builds the evaluation suite: 27 scenarios as data, with code-graded checks of what the customer actually gets after grounding and the guard; a harness that runs the app's own turn with only the model swapped; three modes — the scripted mock, saved answers (cassettes) replayed through the real tools and guard, and the real model with trials, thresholds, every-trial for critical cases and a 95% interval; and eval:ci in the pre-push hook, which knows when the last real-model run no longer matches the prompt version or the model, or didn't pass. Bedrock's managed evaluations compared. Built against the mock and hand-written cassettes; no real model was called.
A Judge, Simulated Customers and Replayed Sessions
What code can't grade needs a reader, and on every change that reader is another model: a judge. In the lesson's scenario, a judge asked "is the answer helpful and polite?" reported 96% for a month and had never been checked against people. This lesson builds a versioned rubric of specific yes/no criteria, a judge prompt in Bedrock's order (role, task, rubric, input) that reasons before each verdict and treats the graded answer as data, and calibration against human-labelled answers — false passes, false fails and Cohen's kappa per criterion, deciding what the judge may gate and refusing when that no longer holds. Then whole conversations: simulated customers talking to the app's own turn, checked as wholes in eval:ci, and sessions exported with redaction and replayed against today's model. Judges and customers are labelled stand-ins; no real model was called.
Server Rules That Override the Model
Some answers aren't the model's to give: who pays for a repair, which steps a customer may try, whether a second technician drives out for the same oven. In the lesson's scenario, the model told customers their repair should be covered by the warranty, including customers whose warranty had ended, and a line in the prompt only made it rarer. This lesson adds server rules that run on every turn after grounding and before the output guard: a warranty decided from policy data and the machine's install date, customer steps taken word for word from the reviewed guide, and one open visit per machine, refused at the booking under a Postgres advisory lock. Every decision is named, evaluated with scenario facts and hand-written cassettes, and compared with Amazon Bedrock Guardrails. No real model was called.
Prompt Caching and Cost Engineering
Every model call sends the same few thousand characters before the customer's words: tools, the turn's schema, the instructions. Prompt caching charges less for that part, if the cache point is placed right and the model caches it at all. In the lesson's scenario, a per-session line in the instructions gave every session its own prefix, written to the cache at a premium and never shared. This lesson lays the request out in Bedrock's caching order and measures each checkpoint's prefix against the model's documented minimum (the app's prefix never cached on Claude Haiku 4.5), puts the session's context after the cache point with a regression test, adds a second checkpoint for the tool loop, works out when caching pays, and compares four layouts with a labelled caching stand-in. No real cache was read.
Retrieval, and When Not to Use It
Retrieval-augmented generation is the default answer to "the model needs our documents". In the lesson's scenario, a search tool over the troubleshooting guides rarely helped and cost a model call on every turn. This lesson builds retrieval properly (chunks by document structure with an audience per chunk, embeddings, pgvector and Postgres full-text search in one table, hybrid search with Reciprocal Rank Fusion) and then measures it: recall on labelled questions next to the exact code lookup, questions the guides can't answer, and an A/B of four ways to give the model the guides by model calls, the step cap and cost. The guides go in the instructions; the experiment stays. A real pgvector index with a labelled stand-in embedder; no real model was called.
When the Model Fails
A model call can return HTTP 200 and still not produce an answer: a provider filter declined, the answer was cut off, nothing came back, the output broke the schema, or the tool loop ran out of steps. In the lesson's scenario, one refused message was reported as "try again" and stayed in the history, so every later turn was refused too. This lesson names each failure from the stop reason and the provider's raw one, retries only empty and malformed answers once, answers the rest with a fixed, reviewed reply on both the JSON and streaming routes, and keeps the failed exchange out of the history the model is sent. A model that fails on purpose rehearses every fault through the real AI SDK; no real model was called.
One AI Account for Every App
When several apps call Amazon Bedrock from one account, they share one quota per model and one line on the bill. In the lesson's scenario, a nightly batch job throttled the customer chat, and nobody could say which app had spent what. This lesson moves Bedrock into a central AI account: the app assumes a role there with STS AssumeRole and the external id the hub gave it, caches the session and renews it before it expires, calls the model through its own tagged application inference profile, and keeps its own ledger of turns — outcome, model calls, tokens and latency, never the words. STS is a development stand-in and the hub's policies are files checked by tests; no AWS account was used.
One Image, Every Environment
The app becomes one container image: Next.js standalone output in a multi-stage Dockerfile, an allowlist build context, settings read at run time instead of NEXT_PUBLIC_ values inlined at build, a non-root user and a health check. In the lesson's scenario, each environment got its own build, and deploys cut chats off because a shell-form command kept SIGTERM from reaching Node. The lesson finds what the build's file trace copied (the whole project, development keys included) and fixes it, then measures docker stop seven ways: exec form against a shell, an answer streaming to a client that closes its connection and to one that keeps it, Node's keep-alive after SIGTERM, no grace period and --init — and maps the results to ECS stopTimeout, initProcessEnabled and health checks. Built and run locally with Docker; no AWS account was used.
The Staging Stack
The app's AWS, as code: an AWS CDK app with CloudFront in front of an internal Application Load Balancer through a VPC origin, ECS Fargate tasks running Lesson 51's image, one settings list (plain values, SSM Parameter Store Strings and SecureStrings, generated Secrets Manager secrets) and a timeout chain sized from the inside out. In the lesson's scenario, the first staging deploy crash-looped because the task definition left out a setting the app requires; the stack's tests now hold the settings list to the app's own rules, the timeouts to each other, and the permissions to the calls the server makes. Synthesized and tested with CDK assertions; nothing was deployed and no AWS account was used.
From Commit to Production
A GitHub Actions delivery pipeline for the app with no stored AWS keys: every pull request verified on the production image's Node against real databases, main's container image built once and promoted by its commit tag, and AWS CDK deploys to staging and then, after a GitHub environment approval, to production. Each job trades GitHub's OIDC token for an IAM role through STS AssumeRoleWithWebIdentity, and each role trusts exactly one subject. In the lesson's scenario, a long-lived CI access key leaked through a build log and production was deployed from a laptop. The workflow is held to its own rules by tests, run locally with act, and its trust policies are exercised against stand-in services; nothing reached GitHub or AWS.
Old Tabs, New Servers
What a rolling deploy can't fix by itself, for the app: browser tabs left open across a release keep running the previous build's code, and pipeline runs can reach an environment out of order. The build's commit becomes Next.js's deploymentId, so assets, navigations and the page carry it and a mismatch reloads; the server puts x-app-version on every response so the chat's own API calls can tell another build answered, with a notice rather than a forced reload; and each GitHub Actions deploy job skips when the version already answering is newer, judged by git history. In the lesson's scenario, CRM tabs kept calling the server with the previous release's code. Images ran locally and the pipeline with act; nothing was deployed.
The Schema the Code Expects
Database migrations that can't silently go missing, for the app: how drizzle-orm's migrate() decides what to run (only journal entries later than the last one it recorded, no hash check, no lock), so a migration generated on a branch first and merged after another is skipped on every database with history and runs on every fresh one. pnpm db:check holds the migrations folder to its rules in CI (journal and files agree, times only go up, against main's migrations, contract steps say why they're safe); pnpm db:drift compares a database by drizzle's own rules (pending, skipped, edited, unknown); readiness reports the schema, and the deploy's smoke check fails the run when it's behind — a report, not a prevention. Expand, then contract. drizzle's migrate() ran on a real local Postgres, the image locally and the pipeline with act; nothing was deployed.
Who May Do What
A least-privilege review of every IAM policy the app owns, for the app: the central AI account's role (Lesson 50) trusted the whole app account (:root) with one external id, so any role there whose own policy allowed sts:AssumeRole could call Bedrock on the app's profile. pnpm iam:review synthesizes the CDK stacks for every environment and holds every policy to ten rules — the task role grants exactly what the server calls (read from the code's import graph), "*" only for actions with no resource types, ECS trusts scoped with aws:SourceAccount and aws:SourceArn, an execution role with only what ECS reads, and a hub trust naming each environment's task role by aws:PrincipalArn with its own external id; pnpm iam:drill evaluates both sides of each AssumeRole, before and after. In CI on every change; IAM Access Analyzer and cdk-nag compared. Nothing was deployed.
The Log You Can Ask
Structured logging for the app, written for a question: the server's log was sentences, some with a request id in brackets and some without, so finding one request's lines or counting 5xx answers meant grep and reading. src/lib/log.ts writes one JSON object per line (time, level, a stable event name, service, env, version, the request's id and trace id from AsyncLocalStorage, then the event's fields), redacts secrets by key and by shape (JWTs, bearer credentials, passwords in URLs — error messages too), caps strings and lines, and never throws; every API request ends with one http.request line with its route, a streamed answer's when its stream ends; the server's console calls become events, and ESLint's no-console keeps it so. The first questions an operator asks are saved as CloudWatch Logs Insights query definitions per environment with CDK, tested against the events the code logs, and run here over the container's lines with pnpm logs:query. Nothing was deployed.
The Alarm That Stayed Quiet
Metrics and alarms for the app, built around the way an alarm fails worst: by staying quiet. In the lesson's scenario a 5xx alarm said OK through an outage twice — its metric filter matched a field the shipped log line doesn't write, a default value of 0 hid it, and when the service went down no logs arrived at all, missing data treated as fine. The request line now carries its metrics as CloudWatch's embedded metric format (durationMs and serverError by environment and route, never a request id), each server writes a heartbeat for a deadman alarm, and the alarm set is declared once: each alarm says what missing data means, and a test holds each to a metric the code makes, dimensions and all. An evaluator built from the CloudWatch missing-data page, tested against its own tables, replays the incident minute by minute; CDK makes the alarms, the metric filters and an SNS topic only CloudWatch may publish to. Nothing was deployed.
The Channel Nobody Read
Alerts without the noise, for the alarms Lesson 58 built. In the lesson's scenario the alarm topic went straight into the team's chat: every alarm's first OK after a deploy, a flapping alarm, two alarms for one outage. The channel was muted by the end of the afternoon, and the next real alarm (a customer's booking with no Case) sat unread for a day. Each CloudWatch alarm state change now goes through EventBridge to a small Lambda function that tells the chat the difference between where an alarm is and what the chat was told. A mute window, flapping, an alarm a bigger one already covers and a rate cap only delay a message: a five-minute EventBridge Scheduler sweep tells it once nothing holds it back, with how long it waited and why. Delivery is at least once and can be out of order, so each change is recorded by the alarm's own time in DynamoDB; a place under the cap is one conditional add, and a message is posted under a lease that's given back if the post fails. CDK bundles the function on its own with no AWS SDK and a role the access review holds to what the function is configured with; a queue keeps what still fails, and alarms on it and on the function's errors tell the topic. A drill replays the afternoon before and after. Nothing was deployed and no chat was posted to.
The Nineteen Seconds
A trace for every turn. In the lesson's scenario a customer waited nineteen seconds, and the request log said only that: not which model call, which tool, whether the retry ran, or the rules. The same afternoon a hosted tracer's quick start answered the question, and sent every customer's words to a third party, because its defaults record content. Now every request is a trace on its own trace id: the safety check, the turn and each attempt (so a retry shows), each model call and tool run from the AI SDK's own telemetry interface, and the rules, named by the OpenTelemetry GenAI semantic conventions. Message content is opt-in, as the conventions say: off by default and never from production. Every model call passes its own integration, which replaces a global one, and while anything listens on the AI SDK's diagnostics channel the SDK's telemetry is switched off. Spans go out as OTLP over HTTP with a JSON body, in the background: batched, retried with back-off on 429/502/503/504, never a 400, every drop said, and sending starts at shutdown. A development stand-in collector reads every field of the wire format and never answers 500, and a drill shows where the nineteen seconds went. No OpenTelemetry package was added and nothing was sent anywhere.