langchain-xns

LangChain document loaders and a ByteStore for XNS — S3-compatible, distributed object storage with $0 egress. Reading your data back is free, at any volume, so the retrieval side of a RAG or agent pipeline costs nothing to run repeatedly.

pip install langchain-xns

from langchain_xns import XNSLoader

docs = XNSLoader("corporate-strategy", suffixes=(".md",)).load()

That is the whole setup. No endpoint, no keys — see Zero-config.

Why not the generic S3 loader

You can point a generic S3 loader at XNS by overriding endpoint_url. This package exists because three things then still fall to you:

• Credentials. XNS writes ~/.xns/credentials during setup. This package reads it, so nothing has to be threaded through your app config or environment.
• Addressing. A self-hosted gateway is often reached by host:port during bring-up, where virtual-hosted addressing cannot work. Path-style is the default here.
• Cost of a bad list. Filtering happens on the ListObjectsV2 response, so a key excluded by suffixes, glob, or max_bytes is never downloaded at all.

Zero-config credentials

XNS has no shared hostname and no vendor API key. Every customer runs their own Relayer (the S3 gateway) and reaches it at a domain they control, with their own TLS certificate. A connection is therefore always endpoint + access key + secret.

Configuration is resolved per field, highest wins:

1. Arguments you pass to the loader or store.
2. XNS_ENDPOINT, XNS_ACCESS_KEY_ID, XNS_SECRET_ACCESS_KEY, XNS_REGION, XNS_PROFILE.
3. ~/.xns/credentials — written by the XNS MCP server or by xns auth login. Override the path with XNS_CREDENTIALS_FILE, pick a profile with profile=.

Step 3 is why the example above takes no arguments. If you have not set XNS up yet, an agent can do the whole install for you:

claude mcp add relayer -- npx @xns-cloud/relayer-mcp@latest

Then ask it to "set up XNS storage". It checks prerequisites, registers the account, installs and starts the Relayer on your Docker host, provisions S3 credentials, and writes the file this package reads.

XNSLoader — text objects as Documents

from langchain_xns import XNSLoader

loader = XNSLoader(
    "corporate-strategy",
    prefix="2026/",              # pushed to the server, not filtered client-side
    suffixes=(".md", ".txt"),
    glob="*/notes/*",            # optional fnmatch over the full key
)

for doc in loader.lazy_load():   # streams; never lists the whole bucket into memory
    print(doc.metadata["source"], len(doc.page_content))

Bodies are decoded incrementally as they stream, so a multi-byte character split across a chunk boundary is handled correctly and the object is never held in memory twice.

Each Document carries id="s3://bucket/key" — stable across re-indexing, so vector stores that support upsert will replace rather than duplicate.

Metadata key	Value
source	s3://bucket/key
bucket, key	as given
size	bytes, from the listing
etag	quotes stripped
last_modified	ISO 8601
content_type	as stored

Pass metadata_fn=lambda meta, entry: {...} to add your own.

Large objects. max_bytes defaults to 64 MiB. Over that, XNSLoader raises XNSObjectTooLargeError before fetching the body; pass on_oversize="skip" to ignore them, or max_bytes=None to disable the check. A text document that trips this is usually a media file — use XNSBlobLoader instead.

XNSBlobLoader — binary objects for parsers

from langchain_xns import XNSBlobLoader
from langchain_community.document_loaders.parsers import PyPDFParser

for blob in XNSBlobLoader("research", suffixes=(".pdf",)).yield_blobs():
    docs = list(PyPDFParser().lazy_parse(blob))

Yields langchain_core Blob objects with mimetype set from the object's Content-Type, so bytes reach a parser without a decode step.

XNSByteStore — a ByteStore for caches and multi-vector retrieval

from langchain_classic.embeddings import CacheBackedEmbeddings
from langchain_xns import XNSByteStore

store = XNSByteStore("agent-cache", prefix="embeddings/")
embedder = CacheBackedEmbeddings.from_bytes_store(
    underlying_embeddings, store, namespace="text-embedding-3-small", key_encoder="sha256"
)

As of LangChain 1.x, CacheBackedEmbeddings ships in langchain-classic, not langchain. key_encoder defaults to SHA-1 and warns; pass "sha256" to silence it.

It also works as the document store behind a retriever:

from langchain_classic.retrievers.multi_vector import MultiVectorRetriever

retriever = MultiVectorRetriever(
    vectorstore=vectorstore,
    byte_store=XNSByteStore("parents", prefix="mv/"),
    id_key="doc_id",
)

Implements the full BaseStore[str, bytes] contract — mget, mset, mdelete, yield_keys. mget returns None for absent keys (it does not raise); other errors propagate. mdelete batches at S3's 1000-key limit. prefix namespaces a store so several can share one bucket.

Because reads are unmetered on XNS, there is no cost argument for stacking a local cache in front of this store — the usual reason to do that is egress billing.

Multimodal and checkpoint workloads

The pipelines this is built for re-read the same bytes many times: a video re-processed into transcription and then embedding models, a checkpoint pulled to a fresh GPU host on every run, an eval suite replayed against a growing corpus. On egress-billed storage, each pass is charged. On XNS it is not, which is why XNSByteStore deliberately has no cache layer and XNSLoader re-reads rather than mirroring to local disk.

Compatibility

• Python 3.10+
• langchain-core >= 0.3, < 2 — uses BaseLoader, BlobLoader, and BaseStore only
• boto3 >= 1.34
• XNS Relayer, or any S3-compatible endpoint. Both path-style (default) and virtual-hosted addressing are supported; pass addressing_style="virtual" for the latter.

XNS is validated against the ceph/s3-tests conformance suite on real hardware; per-capability results are published at https://xns.tech/s3-compatibility.

Links

• XNS — https://xns.tech
• Set up with an AI agent — https://xns.tech/setup-with-ai
• S3 compatibility matrix — https://xns.tech/s3-compatibility
• MCP server — @xns-cloud/relayer-mcp

License

Apache-2.0