Skip to content

06. AI Responses & Streaming

Jump to bottom
Maruf Bepary edited this page Jun 17, 2026 · 2 revisions

Overview

AI Client is a real-time AI chat application built on the Vercel AI SDK v6. The streaming pipeline is the core of the application — it handles sending user messages, streaming AI responses token-by-token, processing tool calls, and displaying reasoning tokens.

The architecture is composed of three key layers:

  • Multi-provider architecture — the system supports any OpenAI-compatible AI provider (OpenRouter, Ollama, Groq, Azure, DeepSeek, etc.) selected per-message by the user
  • Server-Sent Events (SSE) is the transport protocol, delivering incremental tokens and structured events from server to client
  • Vercel AI SDK is the orchestration layer, abstracting provider differences, managing multi-step tool-calling loops, and producing a typed stream

Architectural Components

The streaming pipeline is composed of several distinct components, each with a clear responsibility:

Component Role
Vercel AI SDK (streamText) Orchestrates model calls, tool-calling loops, and stream generation
User's AI Provider OpenAI-compatible API endpoint (OpenRouter, Ollama, Groq, Azure, etc.)
@ai-sdk/openai provider Connects the SDK to the user's selected provider via dynamic baseURL and API key
Custom SSE transport Manual ReadableStream encoding SSE events from the fullStream
Client-side SSE parser Async generator that consumes a ReadableStream and yields typed events
Zustand store Manages the message tree state with optimistic updates

The Streaming Protocol

The project uses Server-Sent Events (SSE) as the transport protocol between the API route and the browser. SSE is unidirectional (server → client), text-based, and ideal for token-by-token AI streaming.

Why Not WebSockets?

SSE is unidirectional (server → client), which is exactly what token streaming requires — the client sends a single request and the server streams back tokens. WebSockets provide full-duplex communication, which introduces unnecessary complexity:

  • Connection management — WebSockets require explicit connection lifecycle handling (open, close, reconnect), whereas SSE streams are tied to a single HTTP request
  • Infrastructure overhead — WebSockets require sticky sessions or dedicated infrastructure for load balancing; SSE works with standard HTTP load balancers
  • No duplex need — user messages are sent via a single POST request, not as a continuous stream; full-duplex communication adds no value here

Why Not Native EventSource?

The project does not use the browser's native EventSource API. Instead, it uses fetch() combined with ReadableStream for three reasons:

  • POST requests — EventSource only supports GET; the chat endpoint requires a POST with a JSON body
  • Custom headers — authentication tokens and content-type headers must be sent with each request
  • Fine-grained control — manual parsing enables precise cancellation, error handling, and typed event dispatch

SSE Frame Format

Each SSE frame follows the standard format: data: {JSON}\n\n. The response is served with the required headers:

  • Content-Type: text/event-stream
  • Cache-Control: no-cache
  • Connection: keep-alive

SSE Event Types

Event Type Purpose Payload
text Incremental text token { delta: string }
reasoning Chain-of-thought token { delta: string }
tool-call Tool invocation started { toolCallId, toolName, args, serverName? }
tool-result Tool execution completed { toolCallId, toolName, result }
transform-start Transform agent run started { runId: string }
transform-step-start Transform step began { stepIndex: number, label: string }
transform-complete Transform agent run finished { runId: string, status: string }
error Stream error { message, code? }
done Stream complete { id, metadata? }

The Vercel AI SDK Integration

The Vercel AI SDK v6 is the orchestration layer. It abstracts provider differences, handles tool-calling loops, and produces a typed stream of chunks that the API route encodes into SSE events.

The project uses streamText(), which returns a fullStream — an async iterable of typed chunks representing text deltas, reasoning tokens, tool calls, tool results, and completion signals.

Key streamText Configuration

  • model — uses provider.chat(modelId) to force the Chat Completions API, compatible with all OpenAI-compatible providers
  • messages — the assembled ModelMessage[] array containing system prompts and conversation history
  • tools — the registered tool set, sourced from MCP servers and built-in tools
  • stopWhen: stepCountIs(10) — conditionally set when the model supports tool-calling and tools are registered; caps multi-step tool-calling loops at 10 iterations to prevent runaway execution
  • abortSignal — tied to the HTTP request lifecycle, enabling client disconnect propagation

Multi-Step Tool Loop

The SDK handles the multi-step tool loop automatically. When the model emits a tool call, the SDK executes the tool, sends the result back to the model, and continues generation. This repeats until the model produces a final text response or the step limit is reached.

The following diagram illustrates the tool-calling loop managed by the SDK:

flowchart TD
    A[Model generates response] --> B{Tool call emitted?}
    B -- Yes --> C[Execute tool]
    C --> D[Send tool result back to model]
    D --> A
    B -- No --> E[Emit text tokens]
    E --> F[Stream complete]
Loading

Model Selection and Provider Architecture

Per-Message Model Selection

Models are selected per-message, not globally. Each message in a conversation can use a different model. The model ID is sent in the request body and stored in the message metadata after generation.

Dynamic Provider Resolution

The application supports any OpenAI-compatible provider (OpenRouter, Ollama, Groq, Azure, or proprietary). When a chat message is sent:

  1. Model selection — the request specifies a model ID (e.g. openai/gpt-4o or meta-llama/llama-3:8b)
  2. Provider lookup — resolveProviderForModel(userId, modelId) determines which provider owns this model
  3. Credential retrieval — the provider's encrypted API key is fetched from the database and decrypted (AES-256-GCM)
  4. SDK instantiation — a Vercel AI SDK provider is created dynamically with the provider's base URL and decrypted key
  5. API call — the model is requested from the resolved provider's endpoint

Fallback behaviour: If no model is explicitly selected, resolveDefaultChatProvider(userId) retrieves the user's configured default chat model and its associated provider. If neither is set, the request fails with a descriptive error prompting the user to configure a provider in settings.

Each user maintains a Provider Registry (see wiki/18.-Provider-Registry-&-Model-Management.md) containing multiple providers, each with a name, base URL, and encrypted credentials. This decoupling allows seamless provider switching without breaking existing model references.

Capability-Based Feature Gating

Models declare capabilities (vision, tool-calling, JSON output) which gate features at request time:

  • Non-vision models have image attachments stripped from the conversation history
  • Non-tool-calling models have the entire tool set omitted from the request

The following diagram illustrates provider resolution and SDK instantiation:

flowchart TD
    REQ["Chat Request<br/>(model: 'openai/gpt-4o')"] --> RES{"resolveProviderForModel<br/>(userId, modelId)"}
    RES --> LOOKUP["Query ai_model &<br/>ai_provider tables"]
    LOOKUP --> FETCH["Fetch encrypted<br/>credentials"]
    FETCH --> DEC["Decrypt key<br/>(AES-256-GCM)"]
    DEC --> SDK["createOpenAI({<br/>apiKey: decrypted,<br/>baseURL: dynamic<br/>})"]
    SDK --> CALL["Vercel AI SDK<br/>streamText()"]
    CALL --> UP["User's Provider<br/>(OpenRouter, Ollama, etc.)"]
    UP --> MODEL["Upstream Model"]
Loading

Server-Side Request Lifecycle

The POST /api/chat endpoint follows a strict pipeline from request to response. Each stage must complete successfully before the next begins.

  1. Authentication — the session is validated via Better Auth; unauthenticated requests receive a 401 response
  2. User settings fetch — the user's global system prompt is loaded from userSettings
  3. Validation — the request body is parsed and validated with Zod schemas
  4. AI provider initialisation — the selected model's provider and encrypted API key are fetched from the database and decrypted via resolveProviderForModel() (or resolveDefaultChatProvider() if no model is specified)
  5. Ownership verification — the chat record is fetched and confirmed to belong to the authenticated user
  6. Context resolution — the project prompt, assistant prompt, and knowledge base ID are resolved from the chat's associations
  7. Tool registration — MCP servers are connected and tools are filtered and registered based on the user's selection
  8. History preparation — vision/capability filtering is applied to strip images from non-vision model requests
  9. Message assembly — the conversation history is converted into the ModelMessage[] format expected by the SDK
  10. Spreadsheet URL generation — presigned S3 URLs are generated for any spreadsheet attachments in the latest user message
  11. System prompt construction — global, project, and assistant prompts are concatenated into system messages (with spreadsheet URLs and a default fallback if all prompts are empty)
  12. Stream initiation — streamText() is called with the fully assembled configuration
  13. SSE emission — the fullStream is iterated and each chunk is encoded as an SSE event and enqueued to the ReadableStream
  14. Post-stream persistence — the assistant message and its metadata are saved to the database, and currentLeafId is updated
  15. Cleanup — MCP connections are closed and the stream controller is finalised

The following sequence diagram illustrates the full server-side lifecycle:

sequenceDiagram
    participant Client
    participant API as API Route
    participant Auth as Better Auth
    participant DB as Database
    participant MCP as MCP Servers
    participant SDK as Vercel AI SDK
    participant Provider as User's AI Provider

    Client->>API: POST /api/chat
    API->>Auth: Validate session
    Auth-->>API: Session valid
    API->>API: Zod validation
    API->>DB: Verify chat ownership
    DB-->>API: Chat record
    API->>DB: Resolve project, assistant, KB
    DB-->>API: Context data
    API->>MCP: Connect & register tools
    MCP-->>API: Tool definitions
    API->>API: Filter history & assemble messages
    API->>SDK: streamText(config)
    SDK->>Provider: Chat Completions request
    Provider-->>SDK: Token stream
    SDK-->>API: fullStream chunks

    loop SSE emission
        API-->>Client: data: {event}\n\n
    end

    API->>DB: Persist assistant message
    API->>MCP: Close connections
Loading

Client-Side Streaming Flow

The client-side flow is managed by a dedicated streaming hook that orchestrates the entire lifecycle from user input to final message rendering.

  1. User sends a message — a UUID is generated for the user message
  2. Optimistic insert — the user message is immediately added to the Zustand store and rendered in the UI
  3. Server persistence — the user message is persisted to the database via a Server Action (awaited, but wrapped in try-catch so a failure does not block the stream)
  4. Attachment upload — files are uploaded to S3 in parallel
  5. Thread reconstruction — the active conversation branch is reconstructed from the message tree
  6. HTTP request — a POST is sent to /api/chat with the conversation history, model ID, and tool selections
  7. SSE consumption — the response body is consumed via an async generator that parses SSE frames
  8. Real-time rendering — text and reasoning deltas are accumulated and rendered progressively
  9. Tool call display — active tool calls are shown inline with status indicators
  10. Finalisation — on the done event, the assistant message is added to the store with its server-generated ID
  11. Cleanup — all streaming state is reset

The following sequence diagram shows the client-side flow:

sequenceDiagram
    participant User
    participant UI as Chat UI
    participant Hook as Streaming Hook
    participant Store as Zustand Store
    participant SA as Server Action
    participant API as POST /api/chat
    participant Parser as SSE Parser

    User->>UI: Send message
    UI->>Hook: Trigger send
    Hook->>Store: Optimistic insert (user message)
    Store-->>UI: Re-render with user message
    Hook->>SA: Persist user message (async)
    Hook->>API: POST request (history, model, tools)
    API-->>Hook: ReadableStream (SSE)
    Hook->>Parser: Consume stream

    loop Token-by-token
        Parser-->>Hook: Typed SSE event
        Hook->>Hook: Accumulate deltas
        Hook-->>UI: Progressive render
    end

    Parser-->>Hook: done event
    Hook->>Store: Insert assistant message (final)
    Store-->>UI: Replace streaming bubble
    Hook->>Hook: Reset streaming state
Loading

Message Assembly for the AI SDK

The conversation history must be converted from the internal message tree format into the Vercel AI SDK's ModelMessage[] format before each request. This assembly ensures the model sees the complete conversation context, including all previous tool interactions.

User Messages

User messages are constructed with:

  • Text content as the primary part
  • Document attachments (PDFs, spreadsheets) as text parts containing the extracted content
  • Image attachments as image parts (only included when the selected model supports vision)

Assistant Messages with Tool History

When an assistant message includes tool call history (stored in metadata), it is reconstructed as a multi-part message:

  • The text response as a text part
  • Each tool call as a tool-call part (with ID, name, and arguments)
  • Followed by a single role: "tool" message containing all tool-result parts (each with ID, name, and result)

This reconstruction is essential for models that require the full tool-calling context to continue coherent multi-turn conversations.

System Messages

System messages are constructed by concatenating multiple prompt sources:

  • The global user prompt (from user settings)
  • The project-level prompt (if the chat belongs to a project)
  • The assistant persona prompt (if an assistant is selected)
  • Optional knowledge base instructions (if a knowledge base is attached)
  • Spreadsheet attachment URLs (presigned S3 URLs for tool-based file access)
  • A default fallback ("You are a helpful AI assistant.") if all other prompt sources are empty

Attachment Handling in the Streaming Pipeline

File attachments are a first-class part of the streaming pipeline. The application supports three attachment types, each with a distinct processing strategy that determines how the content reaches the AI model.

Attachment Types

Type Accepted Formats Max Size Processing Strategy
Image PNG, JPEG, GIF, WebP 2 MB Base64 data URL → ImagePart sent directly to the model
Document PDF, plain text, Markdown 20 MB Text extracted client-side → TextPart prepended to the user message
Spreadsheet XLSX, XLSM, XLS, CSV 50 MB Uploaded to S3 → presigned URL injected into the system prompt for tool-based access

A maximum of 5 attachments per message is enforced (with images additionally capped at 3).

Upload Flow

Attachment processing follows a two-phase pattern: client-side preparation and server-side persistence.

  1. Client-side validation — MIME type, file size, and quota limits are checked immediately on selection
  2. Content extraction — images are read as base64 data URLs; documents have their text extracted (PDFs via a parsing library, text/Markdown read directly); spreadsheets retain the raw file blob
  3. Optimistic display — attachments appear as preview chips in the input area before any upload occurs
  4. S3 upload — after the user message is persisted to the database, each attachment is uploaded to S3 sequentially via a Server Action that re-validates ownership and stores a record in the attachment table
  5. Store sync — the Zustand store is updated with the S3 key for each uploaded attachment

How Each Type Reaches the Model

The three attachment types take fundamentally different paths into the AI prompt:

flowchart TD
    A[User attaches file] --> B{Attachment type?}
    B -- Image --> C[Read as base64 data URL]
    C --> D[Vision model check]
    D -- Supported --> E["Inject as ImagePart in user message"]
    D -- Not supported --> F[Strip from history]
    B -- Document --> G[Extract text content]
    G --> H["Inject as TextPart before user text"]
    B -- Spreadsheet --> I[Upload to S3]
    I --> J[Generate presigned URL]
    J --> K["Inject URL into system prompt"]
Loading

Images are encoded as base64 data URLs on the client and sent to the model as ImagePart objects within the user message content array. The full base64 payload is transmitted directly to the provider — no intermediate URL is generated. Images are placed after the user's text content in the parts array.

Documents have their text content extracted on the client before the message is sent. PDF text is parsed and truncated to 50,000 characters; plain text and Markdown are read directly. The extracted content is injected as a TextPart prefixed with a header (e.g. [Document: report.pdf]) and placed before the user's text in the parts array.

Spreadsheets follow a completely different path. They are not inlined into the message at all. Instead, after upload to S3, the server generates presigned URLs and injects them into the system prompt with instructions for the model to use a file-loading tool. This enables MCP-based spreadsheet manipulation tools to download and process the files.

Vision Gating

Image attachments are conditionally included based on the selected model's capabilities:

  • Server-side — for non-vision models, all image attachments and image content parts are stripped from every user message in the conversation history before message assembly
  • Client-side prevention — the UI proactively blocks image attachment when a non-vision model is selected, displaying an error toast and greying out any existing image attachments with an "Unsupported" label

This dual-layer approach ensures images never reach models that cannot process them, regardless of client behaviour.

Streaming State Management

Streaming state is deliberately kept in React local state (useState), not in the Zustand store. This is because streaming content is ephemeral — it only exists during active generation and changes at high frequency.

State Tracked During Streaming

State Type Purpose
isLoading boolean Whether a stream is currently in progress
streamingContent string | null Accumulated text deltas for progressive rendering
streamingReasoning string | null Accumulated reasoning/chain-of-thought tokens
isStreamingReasoning boolean Whether the stream is currently in the reasoning phase
activeToolCalls array Live tool call states transitioning from calling → complete

Store Interaction Pattern

The Zustand store only receives final, complete messages:

  • Once on optimistic insert — the user message is added immediately before the stream begins
  • Once on the done event — the assistant message is added with its server-generated ID and full metadata

During active generation, a temporary "streaming" message bubble is rendered using the local state values. This bubble is replaced by the real persisted message node when streaming completes.

Error Handling and Resilience

The streaming pipeline employs a three-tier error handling architecture to handle failures at every stage.

Pre-Stream Errors

Before the stream begins, standard HTTP error responses are returned:

  • 401 — session expired or unauthenticated
  • 400 — invalid request body (Zod validation failure)
  • 404 — chat not found or not owned by the user

In-Stream Errors

Once streaming has begun, errors are emitted as SSE error events:

  • Rate limits — detected via HTTP 429 status or "rate-limited" patterns in error messages; emitted with a specific RATE_LIMIT error code
  • Model failures — upstream provider errors are wrapped and forwarded
  • Empty response guard — if the model returns no text, no tool calls, and no reasoning, an error event is emitted

Client-Side Error Handling

  • HTTP status codes are checked before stream consumption (429 for rate limiting, 401 for session expiry)
  • SSE error events are processed and displayed via toast notifications
  • Abort handling preserves partial content — text accumulated before an abort is saved as a store-only message (not persisted to the database)

Resilience Guarantees

  • Cleanup guarantee — MCP connections are always closed in a finally block regardless of success or failure
  • Graceful degradation — individual failures in database persistence, attachment uploads, or SSE parsing are logged and surfaced via toasts but do not crash the stream

Message Metadata

Every message carries a metadata JSON object that captures the full context of the interaction. This metadata is essential for replaying conversations, displaying tool interactions, and auditing model usage.

User Message Metadata

Field Purpose
Model ID The model selected for this message
Selected MCP server IDs Which MCP servers were active
Selected tools Which specific tools were enabled
Assistant ID The assistant persona used (if any)
Knowledge base IDs Attached knowledge bases (if any)
Prompt shortcut ID The slash-command used (if any)

Assistant Message Metadata

Field Purpose
Tool calls Array of tool call objects (ID, name, arguments)
Tool results Array of tool result objects (ID, name, result)
Reasoning text Full chain-of-thought reasoning (if emitted)
Model ID The model that generated the response

Metadata Use Cases

  • Tool call history reconstruction — previous tool interactions are replayed as multi-part messages in subsequent requests
  • UI rendering — tool calls and reasoning tokens are displayed inline in message bubbles
  • Model tracking — each response is attributed to the specific model that generated it
  • Audit trail — a complete record of which tools, servers, and knowledge bases were active for each interaction

Optimistic Updates and Persistence Strategy

The application uses a dual-persistence strategy to balance responsiveness with durability.

Persistence Responsibilities

Message Type Persisted By Timing
User messages Client (via Server Action) Before the stream starts
Assistant messages Server (in API route handler) After the stream completes

Optimistic Update Pattern

The user message follows a standard optimistic update flow:

  1. Apply the optimistic change immediately via addMessage — the user sees their message instantly
  2. Run the Server Action (persistMessage) to persist to the database
  3. Surface errors — if the Server Action fails, a toast notification warns the user but the optimistic message remains in the store (no rollback)

This ensures the user never waits for a network round-trip before seeing their own message.

Assistant Message Lifecycle

The assistant message transitions through two phases:

  1. Streaming phase — a temporary bubble rendered from local React state, showing progressive text and tool calls
  2. Persisted phase — on the done event, the final message (with server-generated ID and full metadata) is inserted into the Zustand store, replacing the temporary bubble

Aborted Streams

When a user aborts a stream mid-generation, the partial content accumulated up to that point is preserved as a store-only message. This message exists in the client-side Zustand store for the duration of the session but is not persisted to the database, ensuring the message tree remains clean.

References

  1. Vercel AI SDK Documentation
  2. Vercel AI SDK streamText Reference
  3. Vercel AI SDK Tools and Tool Calling
  4. Server-Sent Events (MDN)
  5. WHATWG Server-Sent Events Specification
  6. OpenAI-Compatible API Documentation (applies to all compatible providers: OpenRouter, Ollama, Groq, Azure, etc.)
  7. OpenRouter Documentation (example multi-provider gateway)
  8. Next.js Route Handlers

Clone this wiki locally