Skip to content

[FEATURE] Support Amazon OpenSearch Service (managed, non-serverless) with SigV4/IAM auth #407

Description

@noel-improv

Package

lexical-graph

Problem statement

The opensearch:// vector store (added in #397) connects to OpenSearch endpoints that don't use AWS authentication — HTTP basic auth from OPENSEARCH_USERNAME/OPENSEARCH_PASSWORD, or no auth. It's been validated against a localhost Docker/Finch container. Two deployment shapes aren't covered yet:

  • A managed Amazon OpenSearch Service domain (not Serverless) that authenticates with IAM/SigV4.
  • Any auth beyond basic/none (e.g. mutual TLS).

Raised in review of #397 (@acarbonetto): OpenSearch can run well beyond localhost (EC2, on-prem, managed Service), and we should support more of those once the localhost path is in.

Proposed solution

Decouple the two concerns the current single flag ties together:

  • Auth mechanismis_sigv4_auth selects AWS SigV4 vs basic/no auth in create_os_client / create_os_async_client.
  • AOSS document handling — the same flag currently also sets the llama-index client's is_aoss (custom id field, no post-ingest refresh()), via DummyAuth(service='aoss').

A managed Amazon OpenSearch Service domain needs SigV4 auth with non-AOSS document handling (_id + refresh()), so these can't stay coupled. Scope:

  • Separate the SigV4-auth signal from the AOSS-behavior signal so a managed OpenSearch Service domain can use SigV4 without AOSS document semantics.
  • Add a connection path / config for managed Amazon OpenSearch Service (IAM/SigV4), distinct from both aoss:// and the basic-auth opensearch://.
  • Decide how the endpoint scheme is expressed (opensearch-py accepts an endpoint without an explicit http:///https://); document the default.
  • Tests for the managed-Service auth path and document handling.

Alternatives considered

Leave opensearch:// as basic-auth/no-auth only and document managed Amazon OpenSearch Service as unsupported (current state in #397). Works for self-hosted OpenSearch but excludes managed non-serverless domains.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions