Stream LLM Responses Without Breaking Server Logic

When you build with large language models, streaming responses to the UI is a must. Users expect to see text appear instantly, not wait for the entire response to generate. This perceived speed improves user experience significantly. However, simply piping raw LLM output to the client creates a challenge: your server often needs the complete LLM response to perform critical actions.

The Challenge: Server-Side Logic and Streaming

Many applications require the server to do more than just relay text. You might need to:

Save the full conversation to a database.
Trigger a subsequent API call based on the LLM's final output.
Update an internal state or analytics.
Perform moderation or validation on the complete text.
Execute a tool or function call that the LLM suggested.

If you stream tokens directly and discard them on the server, you lose the ability to perform these actions on the complete response. The naive approach is to wait for the LLM to finish, process the full response, and then send it to the client. This works for server logic, but it kills the streaming UX. Your users will stare at a blank screen until everything is done.

The Solution: Buffer on Server, Stream to Client

The effective pattern is to buffer the LLM's full response on the server while simultaneously streaming chunks to the client. This gives users the immediate feedback they expect, and your server retains the complete context for its own processing.

Here’s how it works:

Initiate LLM Call: Your server makes an API call to the LLM, requesting a streaming response.
Buffer and Stream: As each chunk of the LLM response arrives at your server, append it to an internal buffer. Immediately send that same chunk to the client using a streaming protocol like Server-Sent Events (SSE) or WebSockets.
Process on Completion: Once the LLM stream signals completion (e.g., an [DONE] token or the stream closes), your server's buffer holds the entire, final response. Now, you can execute all necessary server-side logic using this complete text.
Final Client Update: After your server-side logic completes, send a final signal or data payload to the client. This can confirm that background tasks are done or provide additional information.

This approach ensures that the user sees the response building character by character, while your server has the full context needed for reliable operations.

Implementing with Server-Sent Events (SSE)

SSE is a good fit for this pattern because it's a simple, unidirectional protocol for sending data from the server to the client. The client opens a persistent connection, and the server pushes events over it.

Server-Side Example (Python with FastAPI):

from fastapi import FastAPI, Response
from fastapi.responses import StreamingResponse
import asyncio
import json

app = FastAPI()

## Simulate an LLM streaming response
async def mock_llm_stream():
    chunks = [
        "Hello", " ", "developer", ".", " Here", " is", " your", " streamed", " response", "."
    ]
    for chunk in chunks:
        yield chunk
        await asyncio.sleep(0.05) # Simulate network delay for chunks

@app.get("/stream-llm")
async def stream_llm_response():
    full_response_buffer = []

    async def generate_events():
        async for chunk in mock_llm_stream():
            full_response_buffer.append(chunk)
            # Send the text chunk to the client
            yield f"data: {json.dumps({"type": "text", "content": chunk})}\n\n"

        # LLM stream has finished. Process the full response.
        final_text = "".join(full_response_buffer)
        print(f"Server received full LLM response: {final_text}")

        # Simulate server-side logic (e.g., database write, external API call)
        await asyncio.sleep(1) # Simulate a longer processing time
        server_processed_data = {
            "status": "success",
            "processed_text_length": len(final_text),
            "summary": "Response processed and logged."
        }
        print(f"Server-side logic completed: {server_processed_data}")

        # Send a final event to the client indicating server processing is done
        yield f"event: server_done\ndata: {json.dumps(server_processed_data)}\n\n"

    return StreamingResponse(generate_events(), media_type="text/event-stream")

Client-Side Example (JavaScript):

const eventSource = new EventSource('/stream-llm');
let fullResponseContent = '';
const outputDiv = document.getElementById('llm-output');
const statusDiv = document.getElementById('llm-status');

eventSource.onmessage = function(event) {
    const data = JSON.parse(event.data);
    if (data.type === 'text') {
        fullResponseContent += data.content;
        outputDiv.textContent = fullResponseContent; // Append to UI
    }
};

eventSource.addEventListener('server_done', function(event) {
    const data = JSON.parse(event.data);
    statusDiv.textContent = `Server processing complete: ${data.summary}`;
    console.log('Final server-side data:', data);
    eventSource.close(); // Close the connection
});

eventSource.onerror = function(error) {
    console.error('EventSource failed:', error);
    statusDiv.textContent = 'Streaming error occurred.';
    eventSource.close();
};

In this example, onmessage handles the streaming text, and addEventListener('server_done', ...) specifically listens for the final event indicating server-side tasks are complete. This cleanly separates UI updates from backend processing.

Handling More Complex Server Actions During a Stream

Sometimes, you don't need the entire LLM output to trigger a server action. If your LLM can produce structured output (like JSON) or uses function calling, you can trigger server-side logic mid-stream.

For example, if the LLM generates a tool call, you can parse that tool call from the incoming stream, execute it on the server, and then continue streaming the LLM's subsequent response. This is common in AI agent development.

This requires more sophisticated parsing of the incoming stream, often looking for specific delimiters or JSON structures. You'd still buffer the full response for a final log or validation, but certain actions could be dispatched earlier.

Error Handling in Streaming

Robust error handling is crucial. On the server, wrap your LLM call and streaming logic in try...except blocks. If the LLM API fails or the connection breaks, you need to:

Log the error.
Send an event: error message to the client, possibly with a helpful message.
Close the SSE connection gracefully.

On the client, the onerror event listener for EventSource will catch network issues or errors sent by the server. Use this to inform the user and potentially retry or display a fallback message.

Where to go from here

Implement this buffering pattern in your LLM-powered applications to improve user experience without sacrificing essential server-side functionality. If you need help architecting robust AI systems, consider AI consulting services.