Streaming LLM Responses in FastAPI: SSE, Backpressure, and the Latency Traps Nobody Warns You About

Short answer: Use FastAPI's StreamingResponse with an async generator that yields tokens the instant the provider emits them, format each chunk as an SSE event, and strip out every buffering layer (gzip, proxy buffering) on that route. The latency you add is almost never FastAPI itself — it's a buffering proxy, a blocked event loop, or a client disconnect you never cancelled. Get those three right and your end-to-end TTFT tracks the provider's.

The working SSE pattern

Stream end-to-end so the first token reaches the user the moment it exists (Latitude). The generator should do nothing between receiving a provider chunk and yielding it — no accumulation, no post-processing that waits for the full response.

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
 
app = FastAPI()
 
@app.get("/chat")
async def chat(request: Request, q: str):
    async def gen():
        stream = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": q}],
            stream=True,
        )
        async for chunk in stream:
            if await request.is_disconnected():
                await stream.close()   # stop paying for tokens nobody sees
                break
            delta = chunk.choices[0].delta.content
            if delta:
                yield f"data: {delta}\n\n"   # SSE frame
        yield "data: [DONE]\n\n"
 
    return StreamingResponse(
        gen(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache",
                 "X-Accel-Buffering": "no"},  # disable nginx buffering
    )

The \n\n terminator per event is not optional — the browser's EventSource won't dispatch a message until it sees the blank line.

The gotchas from production

• Buffering proxies and gzip. This is the number one cause of "why does my stream arrive all at once?" A buffering proxy or gzip layer commonly breaks streaming and inflates perceived TTFT (practitioner-known). nginx buffers proxied responses by default; set X-Accel-Buffering: no or proxy_buffering off. Never gzip a text/event-stream route — compression buffers to fill a window before flushing.
• Blocking the event loop. FastAPI only stalls if you stall it. Any sync call inside the generator — a blocking SDK, time.sleep, a heavy tokenizer, a sync DB write — freezes the whole loop and every concurrent stream with it. Keep the generator fully async; push CPU-bound work to run_in_threadpool.
• Client disconnects. A user closing the tab does not automatically stop your upstream request. Poll request.is_disconnected() between chunks, or catch asyncio.CancelledError.

Propagating cancellation so you stop paying

This is the one that shows up on your bill. When a client disconnects mid-stream, cancelling the upstream request stops paying for tokens the user will never see. Without it, the model keeps generating to completion and you're charged for every output token into the void.

• Check await request.is_disconnected() inside the loop and await stream.close() on the provider handle.
• Wrap the provider call so CancelledError (raised when the ASGI server drops the connection) closes the upstream stream in a finally block.
• On long generations under bursty traffic, this is real money — a 2,000-token answer abandoned at token 50 is ~1,950 tokens you shouldn't pay for.

Measuring real TTFT end-to-end

The trap: teams measure TTFT at the provider SDK and call it done. That number is a lie if a proxy buffers 40 tokens before flushing. Measure where it matters — at the client.

• Client-side: timestamp the request start and the first onmessage event in the browser. That's the TTFT your users actually feel.
• Server-side: log the delta between generator start and first yield, and separately between first yield and the ASGI http.response.body.
• If provider TTFT is 300ms but browser TTFT is 1.2s, your latency is in the transport — go hunt the buffering layer, not the model.

Streaming latency is a transport problem, not a model problem — fix buffering, cancellation, and measurement before you touch the prompt.

Related: more writing.