Streaming responses to real users
Lesson 2 of 4 in FM APIs, Streaming, and Application Wiring.
A model that takes twenty seconds to finish can still feel instant — if the first words arrive in under one. That is the entire case for streaming, and the stack has two halves: Bedrock streaming APIs (ConverseStream / InvokeModelWithResponseStream) emit the response as an event stream of incremental chunks the moment generation starts, and a delivery transport — WebSockets or server-sent events, or API Gateway with chunked/streamed transfer — carries those chunks to the user as they are born. The metric that moves is time to first byte: total generation time barely changes; perceived latency collapses.
Delivery is the part architects get wrong: every hop between model and browser must forward chunks, not buffer them. Lambda does this with response streaming (invoked via InvokeWithResponseStream), and API Gateway can now stream REST responses progressively — including SSE — while its WebSocket APIs push frames over a persistent two-way connection. One buffering hop anywhere, and the user waits for the whole answer anyway.
WebSockets
A persistent, bidirectional connection through an API Gateway WebSocket API: $connect/$disconnect/custom routes, and the backend pushes frames to a stored connection id. The fit: chat UIs where the client also talks mid-stream — typing signals, cancel-generation, multi-turn sessions. The cost: connection state to manage (ids, timeouts, reconnects) on both sides.
Server-sent events
One-way streaming over plain HTTP: the server holds the response open with Content-Type: text/event-stream and emits data: frames; browsers consume via EventSource, which auto-reconnects. The fit: display-only token streams — most GenAI UIs — with no connection registry to run. Carried end to end by Lambda response streaming and API Gateway's streamed responses.
Chunked / response streaming
The plumbing underneath: HTTP responses delivered progressively instead of as one buffered body — API Gateway's response streaming (the AIP-C01 exam guide's "chunked transfer encoding") paired with Lambda's InvokeWithResponseStream. Improves time-to-first-byte for any large payload, SSE included. The rule stands at every hop: forward, don't buffer.
AppSync (GraphQL)
The third delivery option: AWS AppSync pushes GraphQL subscriptions over WebSockets that AWS manages — no $connect routes, connection registry, or reconnect logic to build. The backend publishes each chunk as a mutation; every subscribed client receives it in real time. The fit: applications already speaking GraphQL, or multi-client UIs that want WebSocket-style push with the connection plumbing managed.
Key terms: Server-sent events (SSE), WebSocket API (API Gateway), Converse API
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.