[FEATURE] Add OpenSearch datasource and log query plugin with PPL support#641
[FEATURE] Add OpenSearch datasource and log query plugin with PPL support#641Oliver-ke wants to merge 20 commits into
Conversation
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
05cd7bf to
bc65cad
Compare
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
The bound was being appended at the end of the user query, which fails once the user pipeline drops @timestamp from the schema (stats, fields, top). Tests now describe the intended ordering. Implementation follows. Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
…cker file Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
0963d7f to
496c296
Compare
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
Signed-off-by: Azorji Kelechi Oliver <kelechioliver96@gmail.com>
…search-plugin chore(plugins): update with remote main branch
|
Hi @Oliver-ke, |
Sure, I'd share my testing.md file shortly |
|
TESTING.md Testing the OpenSearch pluginManual QA steps. Verifies datasource, log query, and trace → logs pivot. Prerequisites
1. Run unit testsInstall once at the workspace root so cd plugins
npm install
cd opensearch
npm test
npm run type-check
npm run lintValidate the plugin schemas with cd plugins/opensearch
percli plugin lint
percli plugin test-schemasGo SDK: cd plugins/opensearch
go test ./sdk/...Expect all green. No skipped suites. 2. Start a local OpenSearchUse the bundled compose file (also boots OpenSearch Dashboards on cd docs/examples
docker compose up -d
docker compose logs -f opensearch # watch boot, Ctrl+C once "started" appearsWait until ready: curl -s http://localhost:9200/_cluster/health | jq .statusExpect Teardown when done: docker compose down -v3. Seed sample log dataIndex two log documents with a shared curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_doc" \
-H 'Content-Type: application/json' -d '{
"@timestamp": "2026-04-29T10:00:00Z",
"body": "request received",
"traceId": "abc123",
"severity": "INFO"
}'
curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_doc" \
-H 'Content-Type: application/json' -d '{
"@timestamp": "2026-04-29T10:00:01Z",
"body": "request failed",
"traceId": "abc123",
"severity": "ERROR"
}'
curl -X POST "http://localhost:9200/otel-logs-2026.04.29/_refresh"Sanity-check PPL endpoint directly: curl -X POST "http://localhost:9200/_plugins/_ppl" \
-H 'Content-Type: application/json' \
-d '{"query":"source=otel-logs-* | where traceId='\''abc123'\''"}'Expect 2 datarows in response. 4. Build and serve the plugincd plugins/opensearch
npm run devDev server listens on 5. Register the datasource in PersesThe current Perses UI does not accept raw YAML for datasource creation, so create Option A — via the UI formNavigate to Admin → Global Datasources → Add Datasource. Fill in:
Save. Option B — via
|
| Case | Input | Expected |
|---|---|---|
| Empty query | `` | No request, no error toast |
| Bad PPL | source=foo | invalid |
Error banner with PPL error body (SyntaxCheckException) |
| Wrong index | source=does-not-exist |
Error banner with IndexNotFoundException from OpenSearch (the PPL endpoint returns 404; the plugin surfaces that as an OpenSearchPPLError) |
| Unreachable DS | stop docker, re-run query | Error banner, retry works after restart |
| Aggregation (stats) | source=otel-logs-* | stats count() by service |
Returns one row per service, no error |
| Field projection | source=otel-logs-* | fields service, body | head 10 |
Returns 10 rows with only those two columns |
7. Field overrides
Index a doc with non-standard field names:
curl -X POST "http://localhost:9200/legacy-logs/_doc" \
-H 'Content-Type: application/json' -d '{
"time": "2026-04-29T10:05:00Z",
"message": "legacy log line",
"trace_id": "abc123"
}'
curl -X POST "http://localhost:9200/legacy-logs/_refresh"In the query spec, set:
timestampField: time
messageField: message
query: source=legacy-logs | where trace_id='abc123'Expect row to render with time as timestamp and message as body. Without
the overrides, the row still renders without crashing: missing timestamp falls
back to epoch 0, and missing message falls back to a JSON dump of the row.
8. Trace → logs pivot
Load docs/examples/trace-to-logs.json as a dashboard. Requires a Tempo (or Jaeger) datasource also configured.
- Set
traceIdvariable toabc123. - Logs panel re-runs and shows the 2 seeded rows.
- Change variable to
xyz999. Logs panel goes empty without error. - Swap Tempo panel for Jaeger per
docs/examples/README.md. Logs panel unchanged.
9. Cleanup
cd plugins/opensearch/docs/examples
docker compose down -v
looks like a comprehensive test plan |
|
Hi @Oliver-ke, first of all great work! I have tested against a local OpenSearch with the opensearch_dashboards_sample_data_logs sample. I did not review the source-code. I have only tested the functionality and below are my findings. Bugs/Improvements
Screenshot 1
Screenshot 2Working after I add Timestamp field (optional) placeholder @timestamp implies the plugin will fall back gracefully if left blank. it doesn't
6.. PPL Query field is a plain textarea. It has no syntax highlighting, no autocomplete, no Query Examples disclosure (Please compare ClickHouse query editor in the same panel type).
http://localhost:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open otel-v1-apm-service-map-sample Nio8tHEHTWK3uNI4PHmyXA 1 0 49 0 17.7kb 17.7kb
yellow open otel-logs-2026.04.29 _3tNZKIVTU2eseJhMHb_cw 1 1 4 0 17kb 17kb
green open otel-v1-apm-span-sample 14q5nFcqQq-OwewznwFAZQ 1 0 13061 0 6.1mb 6.1mb
green open .ql-datasources KdnzV0EoT96Y1XpY9NfQ7A 1 0 0 0 208b 208b
green open top_queries-2026.05.22-84957 hi43N36cRxqNUGtFImwVSg 1 0 16 2 104.1kb 104.1kb
green open ss4o_metrics-otel-sample 2Z4CkH5JTGWQXW6NZgHc7A 1 0 39923 0 4mb 4mb
green open .kibana_1 uKgkLwTLThyWFoSS5ztn2Q 1 0 246 1 155.6kb 155.6kb
green open ss4o_logs-otel-sample EkQueqJiQJmMWszbrNRe-Q 1 0 16286 0 5.5mb 5.5mb
green open .plugins-ml-config QmvGWldgQ_-5TnMuA35KeQ 1 0 1 0 4kb 4kb
green open .opensearch-observability BqmQfOVnSNaeth8GcjBEkw 1 0 0 0 208b 208b
green open opensearch_dashboards_sample_data_logs OgFiObqIR4GAuT_GPovfFA 1 0 14074 0 7.4mb 7.4mb
green open opensearch_dashboards_sample_data_flights 1jp6tGfmSuqBK1YUjKYmNA 1 0 13059 0 5.5mb 5.5mb
green open opensearch_dashboards_sample_data_ecommerce dxMt1PfqSP-9-4G5Cd-WbQ 1 0 4675 0 3.9mb 3.9mb8 Error banner reads OpenSearch PPL request failed (400): Invalid Query — the actual details field from the response (e.g. can't resolve Symbol(name=@timestamp)) is dropped, removing the important info.
|
@ibakshay Thanks for the feedback. |
There was a problem hiding this comment.
Pull request overview
Adds a new OpenSearch plugin module to Perses plugins, including an OpenSearch datasource, a PPL-backed log query plugin, schemas, Go SDK helpers, tests, and trace-to-logs examples.
Changes:
- Adds TypeScript OpenSearch datasource/client and log query implementation with time-range PPL bounding, variable replacement, and row-to-log mapping.
- Adds CUE schemas, Go SDK builders/options, and unit/schema fixtures for datasource and log query specs.
- Adds plugin module packaging/build config plus docs and example dashboards/docker compose.
Reviewed changes
Copilot reviewed 50 out of 53 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
opensearch/package.json |
Defines npm package metadata, scripts, peer deps, and Perses plugin registrations. |
opensearch/README.md |
Documents installation, development, trace-to-logs usage, and field overrides. |
opensearch/rsbuild.config.ts |
Configures module federation exposes and shared dependencies. |
opensearch/jest.config.ts |
Adds Jest configuration for the plugin. |
opensearch/tsconfig.json |
Adds TypeScript config. |
opensearch/tsconfig.build.json |
Adds declaration-build TypeScript config. |
opensearch/go.mod |
Adds Go module for OpenSearch SDK code. |
opensearch/go.sum |
Adds Go dependency checksums. |
opensearch/cue.mod/module.cue |
Adds CUE module metadata. |
opensearch/src/index.ts |
Exports plugin module APIs. |
opensearch/src/index-federation.ts |
Adds federation bootstrap entry. |
opensearch/src/bootstrap.tsx |
Adds empty React bootstrap used by module federation. |
opensearch/src/env.d.ts |
Adds rsbuild type references. |
opensearch/src/getPluginModule.ts |
Builds plugin module metadata from package.json. |
opensearch/src/setup-tests.ts |
Sets up Testing Library and echarts mock. |
opensearch/src/model/index.ts |
Re-exports OpenSearch model utilities. |
opensearch/src/model/opensearch-client.ts |
Implements PPL POST client and typed HTTP error. |
opensearch/src/model/opensearch-client-types.ts |
Defines PPL response/error/header types. |
opensearch/src/model/opensearch-client.test.ts |
Tests PPL client URL, headers, and error behavior. |
opensearch/src/model/opensearch-selectors.ts |
Adds datasource selector helpers. |
opensearch/src/datasources/index.ts |
Re-exports datasource plugin. |
opensearch/src/datasources/opensearch-datasource/index.ts |
Re-exports datasource implementation/types/editor. |
opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx |
Adds datasource plugin and client factory. |
opensearch/src/datasources/opensearch-datasource/OpenSearchDatasourceEditor.tsx |
Adds HTTP settings editor defaults. |
opensearch/src/datasources/opensearch-datasource/opensearch-datasource-types.ts |
Defines datasource spec. |
opensearch/src/queries/index.ts |
Re-exports query plugin. |
opensearch/src/queries/constants.ts |
Adds datasource and default field constants. |
opensearch/src/queries/query-editor-model.ts |
Adds shared query editor state hook. |
opensearch/src/queries/opensearch-log-query/index.ts |
Re-exports log query implementation. |
opensearch/src/queries/opensearch-log-query/OpenSearchLogQuery.tsx |
Registers log query plugin and dependencies. |
opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.tsx |
Adds datasource/index/field/query editor UI. |
opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.test.tsx |
Tests editor rendering and field updates. |
opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.ts |
Adds PPL bounding, execution, and log conversion. |
opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.test.ts |
Tests query execution, variable dependencies, bounding, and row mapping. |
opensearch/src/queries/opensearch-log-query/log-query-plugin-interface.ts |
Adds local log query plugin interface. |
opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.ts |
Defines log query spec/response types. |
opensearch/schemas/datasources/opensearch.cue |
Adds datasource CUE schema. |
opensearch/schemas/datasources/tests/valid/opensearch.json |
Adds valid datasource fixture. |
opensearch/schemas/queries/opensearch-log-query/query.cue |
Adds log query CUE schema. |
opensearch/schemas/queries/opensearch-log-query/tests/valid/basic.json |
Adds valid log query fixture. |
opensearch/schemas/queries/opensearch-log-query/tests/invalid/empty-query.json |
Adds invalid empty query fixture. |
opensearch/schemas/queries/opensearch-log-query/tests/invalid/empty-index.json |
Adds invalid empty index fixture. |
opensearch/schemas/queries/opensearch-log-query/tests/invalid/empty-timestamp-field.json |
Adds invalid empty timestamp field fixture. |
opensearch/schemas/queries/opensearch-log-query/tests/invalid/empty-message-field.json |
Adds invalid empty message field fixture. |
opensearch/sdk/go/datasource/datasource.go |
Adds Go datasource builder/spec. |
opensearch/sdk/go/datasource/options.go |
Adds Go datasource options. |
opensearch/sdk/go/query/log/log.go |
Adds Go log query builder/spec. |
opensearch/sdk/go/query/log/options.go |
Adds Go log query options. |
opensearch/sdk/go/query/log/log_test.go |
Tests Go log query JSON shape and validation. |
opensearch/docs/examples/README.md |
Documents trace-to-logs example and field conventions. |
opensearch/docs/examples/trace-to-logs.json |
Adds example dashboard wiring Tempo traces to OpenSearch logs. |
opensearch/docs/examples/docker-compose.yml |
Adds local OpenSearch example stack. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| type OpenSearchQueryEditorProps = OptionsEditorProps<OpenSearchLogQuerySpec>; | ||
|
|
||
| export function OpenSearchLogQueryEditor(props: OpenSearchQueryEditorProps): ReactElement { |
| "@perses-dev/components": "^0.53.1", | ||
| "@perses-dev/core": "^0.53.0", | ||
| "@perses-dev/dashboards": "^0.53.1", | ||
| "@perses-dev/explore": "^0.53.1", | ||
| "@perses-dev/plugin-system": "^0.53.1", |
There was a problem hiding this comment.
we have indeed updated the various package and remove the usage of the package @perses-dev/core. Please update accordingly your package.json.
There was a problem hiding this comment.
before that I would advise to merge the main branch into yours, so you have the latest changes with the up to date package in the repo
Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>
…ins into feature/opensearch-plugin chore: update with remote branch
| "@emotion/react": "^11.7.1", | ||
| "@emotion/styled": "^11.6.0", | ||
| "@hookform/resolvers": "^3.2.0", | ||
| "@perses-dev/components": "^0.54.0-beta.1", |
There was a problem hiding this comment.
last version is 0.54.0-beta.3 and please remove the usage of @perses-dev/core as it is deprecated and replace mostly by @perses-dev/spec and @perses-dev/client
There was a problem hiding this comment.
@Nexucis Package deps updated.
Doing another round of e2e. Meanwhile, open for full review.
There was a problem hiding this comment.
Thank you, @Oliver-ke. Did you also address my feedback?
There was a problem hiding this comment.
Yes, @ibakshay, I did, I was able to cover the items in scope for this PR.
Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>
6c643e3 to
fbddd72
Compare
|
@shahrokni @Gladorme @jgbernalp can you please review the code ? Thank you ! |
|
Hi @Oliver-ke, seems like my suggestions in this comment are not worked on. All my comments are still valid. Can you please check and let me know. Thank you. |





Description
Adds an OpenSearch datasource + log query plugin
These changes includes:
remaining columns (e.g. traceId) as labels.
opensearch/docs/examples/trace-to-logs.json (Tempo + OpenSearch; Jaeger swap documented).
and 2 Go tests (builder JSON shape + omitempty for optional fields).
Screenshots
DataSource

Checklist
[<catalog_entry>] <commit message>naming convention using one of thefollowing
catalog_entryvalues:FEATURE,ENHANCEMENT,BUGFIX,BREAKINGCHANGE,DOC,IGNORE.UI Changes