Repository navigation
05. Storage
MinIO provides S3-compatible object storage for the application. In development, a containerised MinIO instance replaces AWS S3. The AWS SDK v3 connects to either environment transparently via a single configuration switch (forcePathStyle).
All binary assets — chat attachments, knowledge base documents, and transform pipeline files — are stored as S3 objects with metadata tracked in PostgreSQL.
MinIO runs as a Docker container alongside PostgreSQL within the project's Docker Compose stack. Data is persisted via a named Docker volume (minio_data), ensuring uploads survive container restarts.
- Port 9000 serves the S3-compatible API
- Port 9001 exposes the MinIO web management console
- Default credentials are shared between the application and MinIO via environment variables
All storage-related configuration is driven by environment variables, validated at startup through a Zod schema. The application fails fast if any required value is missing.
| Variable | Required | Default | Purpose |
|---|---|---|---|
S3_ENDPOINT |
Yes | — | MinIO or S3 API endpoint URL |
S3_REGION |
No | us-east-1 |
AWS region identifier |
S3_ACCESS_KEY |
Yes | — | Access key credential |
S3_SECRET_KEY |
Yes | — | Secret key credential |
S3_BUCKET |
No | ai-client-uploads |
Target bucket name |
A single S3 client instance is created at module level and shared across all server-side code. The client is strictly server-side — credentials are never exposed to the browser.
-
forcePathStyle: trueenables MinIO compatibility (path-style URLs instead of AWS virtual-hosted style) - Six utility functions abstract all common operations: bucket creation, upload, single delete, batch delete, presigned URL generation, and download
graph LR
subgraph Docker Compose
MINIO[MinIO<br/>Port 9000 API<br/>Port 9001 Console]
PG[(PostgreSQL<br/>Port 5432)]
VOL[(minio_data<br/>Docker Volume)]
MINIO --> VOL
end
subgraph Next.js Application
SA[Server Actions]
API[API Routes]
S3C[S3 Client Module]
SA --> S3C
API --> S3C
end
S3C -->|AWS SDK v3<br/>forcePathStyle| MINIO
SA --> PG
API --> PG
The application uses a single bucket for all storage. Logical separation is achieved through key prefixes that encode ownership and context. This simplifies IAM policies and bucket management whilst maintaining clear organisational boundaries.
| Prefix Pattern | Subsystem | Example |
|---|---|---|
attachments/{userId}/{id}-{filename} |
Chat & Transform Attachments | attachments/abc123/def456-report.pdf |
kb/{kbId}/{docId}/{filename} |
Knowledge Base Documents | kb/kb001/doc789/manual.pdf |
transform-inputs/{userId}/{id}-{filename} |
Transform Pipeline Inputs | transform-inputs/abc123/run01-data.xlsx |
transform-outputs/{userId}/{id}-{filename} |
Transform Pipeline Outputs | transform-outputs/abc123/run01-result.xlsx |
Key design principles:
- User-scoped prefixes prevent cross-user key collisions
- Filename sanitisation strips directory separators and special characters
- UUIDs in the key path prevent overwrites of files with identical names
All uploads are proxied through Next.js Server Actions — the client never communicates directly with S3. The bucket is lazily created on first upload if it does not yet exist.
sequenceDiagram
participant Client as Browser
participant SA as Server Action
participant DB as PostgreSQL
participant S3 as MinIO / S3
Client->>SA: Upload file (multipart form data)
SA->>SA: Authenticate (requireSession)
SA->>SA: Validate type & size
SA->>S3: ensureBucket()
SA->>S3: PutObject (file buffer)
S3-->>SA: Success
SA->>DB: Insert attachment row<br/>(key, name, mimeType, size)
DB-->>SA: Attachment ID
SA-->>Client: Attachment metadata
- Uploads are single-request operations (no multipart chunking) — suitable for the enforced file size limits
- Both client-side and server-side validation enforce file type allowlists and size constraints
- The attachment metadata record in PostgreSQL tracks the S3 key, enabling later retrieval and deletion
Files are never streamed through the application server for downloads. Presigned URLs enable direct client-to-S3 transfer, reducing server load and latency.
sequenceDiagram
participant Client as Browser
participant SA as Server Action
participant DB as PostgreSQL
participant S3 as MinIO / S3
Client->>SA: Request file access
SA->>SA: Authenticate (requireSession)
SA->>DB: Verify ownership (userId match)
DB-->>SA: Attachment record
SA->>S3: Generate presigned GET URL<br/>(1-hour expiry)
S3-->>SA: Signed URL
SA-->>Client: Presigned URL
Client->>S3: Direct HTTP GET
S3-->>Client: File stream
- Every URL generation is preceded by an ownership check ensuring the requesting user owns the resource
- URLs default to a one-hour expiry window
- Attachment galleries lazily resolve presigned URLs on demand (not pre-fetched)
Deletion behaviour varies by subsystem and represents a deliberate design trade-off:
| Subsystem | S3 Cleanup | Database Cleanup | Trade-Off |
|---|---|---|---|
| Knowledge Base Documents | Explicit — S3 object removed | Cascade delete | Full consistency |
| Chat Attachments | Not automatic | Cascade on chat/message delete | Potential orphaned S3 objects |
| Transform Files | Not automatic | Cascade on run delete | Potential orphaned S3 objects |
Chat and transform attachments do not trigger S3 deletion to avoid complex distributed transactions. This is a known design trade-off that prioritises simplicity over storage efficiency.
graph TB
S3[(MinIO / S3<br/>Object Storage)]
subgraph Chat System
CA[Chat Attachments]
AI[AI Streaming Pipeline]
end
subgraph Knowledge Base
KBU[Document Upload]
ING[Ingestion Pipeline]
end
subgraph Transform Pipeline
TI[Transform Inputs]
TO[Transform Outputs]
end
CA -->|Upload & Download| S3
AI -->|Presigned URL<br/>Generation| S3
KBU -->|Upload| S3
ING -->|Download for<br/>Text Extraction| S3
TI -->|Upload| S3
TO -->|Upload & Download| S3
| Subsystem | Upload | Download | Delete | Notes |
|---|---|---|---|---|
| Chat Attachments | Yes — images, PDFs, spreadsheets, text files | Yes — presigned GET URLs | No — orphaned on chat deletion | Files attached to individual messages; max 5 per message |
| AI Streaming Pipeline | No | Yes — presigned URLs injected into the system prompt | No | Spreadsheet attachments on the latest user message receive presigned URLs, which are embedded into the system prompt so MCP tools (e.g. Excel analysis tools) can access the files directly. Modified files produced by MCP tools are persisted back to S3 after the stream completes. |
| Knowledge Base | Yes — PDF, text, Markdown documents | Yes — for ingestion pipeline | Yes — explicit cleanup | Documents downloaded server-side for text extraction, chunking, and embedding |
| Transform Inputs | Yes — typically spreadsheets | Yes — presigned URLs for review | No — orphaned on run deletion | Uploaded when a transform run is initiated |
| Transform Outputs | Yes — generated XLSX results | Yes — presigned URLs for download | No — orphaned on run deletion | Created by the transform agent after processing |
The knowledge base ingestion pipeline is the most complex storage interaction, combining upload, download, text extraction, and AI embedding in a multi-stage process.
sequenceDiagram
participant User as User
participant App as Application
participant S3 as MinIO / S3
participant DB as PostgreSQL
participant AI as Embedding Model
User->>App: Upload document
App->>S3: Store document
App->>DB: Create document record (status: pending)
App->>App: Trigger ingestion
App->>S3: Download document
S3-->>App: Raw file bytes
App->>App: Extract text (up to 500k chars)
App->>App: Chunk text (~400 tokens, 200-char overlap)
App->>AI: Batch embed chunks (2048-dim vectors)
AI-->>App: Embeddings
App->>DB: Upsert chunks with embeddings
App->>DB: Update status → ready
The document transitions through four status states: pending → processing → ready (or failed). Text extraction supports PDF, plain text, and Markdown formats.
| Layer | Mechanism | Scope |
|---|---|---|
| Authentication | Every storage operation requires a valid session (requireSession()) |
All endpoints |
| Ownership Verification | Database queries filter by userId = session.user.id before any S3 operation |
Per-resource |
| Presigned URL Scoping | URLs are GET-only, single-object, time-limited (1 hour) | Per-download |
| Bucket Privacy | Default private ACL — no public access configured | Bucket-level |
| Credential Isolation | S3 credentials are server-side only, validated at startup, never exposed to the client | Application-level |
| Check | Client-Side | Server-Side |
|---|---|---|
| File type allowlist | Yes — HTML accept attribute | Yes — MIME type validation |
| File size limits | Yes — pre-upload check | Yes — buffer size validation |
| Filename sanitisation | No | Partial — chat uploads strip path separators only (basename); KB uploads apply full regex sanitisation |
| Files per message | Yes — max 5 enforced | No — count is enforced client-side only |
Size limits and allowed types by category:
| Category | Allowed Formats | Maximum Size |
|---|---|---|
| Images | PNG, JPEG, GIF, WebP | 2 MB |
| Documents | PDF, plain text, Markdown | 20 MB |
| Spreadsheets | XLSX, XLSM, XLS, CSV | 50 MB |
Per-message quotas: maximum 3 images and 5 total attachments per message.
Knowledge base documents have separate restrictions — only PDF, plain text, and Markdown are accepted, with a 50 MB size cap.
The application accepts user-configured MCP server URLs that could potentially target internal storage endpoints. An SSRF guard validates all user-supplied URLs against a blocklist of internal address ranges before any outbound connection is established.
IPv4 blocked ranges:
-
Localhost and loopback —
127.0.0.0/8 -
Private networks (RFC 1918) —
10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 -
Link-local addresses —
169.254.0.0/16 -
Cloud metadata endpoints —
169.254.169.254 -
Unspecified address —
0.0.0.0
IPv6 blocked ranges:
-
Loopback —
::1 -
Link-local —
fe80::/10 -
Unique local addresses —
fc00::/7 - IPv4-mapped IPv6 — addresses that embed a blocked IPv4 address
Unparseable URLs are also blocked by default.
This prevents attackers from using MCP server configuration to reach the internal MinIO instance.
Development bypass: A configuration flag (NEXT_PUBLIC_ALLOW_PRIVATE_NETWORK_MCP) can be set to true to bypass the SSRF guard entirely, allowing connections to localhost MCP servers during local development. This flag must remain false in production.
- MIME validation relies on client-reported
Content-Type— no server-side magic byte verification is performed - No upload rate limiting beyond per-message attachment quotas
- No virus or malware scanning on uploaded files
- Chat attachment S3 objects are not cleaned up on deletion — this causes storage accumulation but does not constitute a security risk
The forcePathStyle configuration enables transparent switching between MinIO (development) and AWS S3 (production) by changing only environment variables. No application code changes are required to migrate.
Production deployments should consider:
- IAM roles instead of static access key credentials
- VPC endpoints for private S3 access without traversing the public internet
- Bucket versioning for accidental deletion recovery
- Lifecycle policies for automated cleanup of orphaned objects
- CloudFront or CDN distribution for presigned URL delivery at edge locations
- Server-side encryption (SSE-S3 or SSE-KMS) for data at rest
- Access logging for audit trails on all object operations