sql: add CREATE METRIC SINK as a durable catalog object (SQL-554) by mtabebe · Pull Request #37958 · MaterializeInc/materialize

mtabebe · 2026-07-29T20:11:51Z

Problem:

To maintain cluster metrics we defined the MetricSink compute operator. However, there is no way to ask for one. We need a SQL surface that names a metric sink and a durable catalog representation that survives a restart.

Solution:

Add CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel> and DROP METRIC SINK, gated behind enable_metric_sink. Creating a sink writes a catalog item and nothing else: no dataflow is optimized or shipped, so a sink created today publishes no metrics.

Planning checks that the FROM relation exposes the five columns the operator reads (metric_name, metric_type, labels, value, help). Note: order is not enforced and extra columns are fine. Nullability is not checked.

Metric sinks need no new durable record. They persist as ordinary Items and item_type works the type out from create_sql.

However, the changes to audit and serialization does bump the catalog version.

There are some gaps in:

mz_comments has no MetricSink branch, but CommentObjectType has no variant either, so COMMENT ON METRIC SINK does not parse and no such record can exist
MZ_DEFAULT_PRIVILEGES has no CASE arm, but ON METRIC SINKS is rejected during planning

Note: enable_metric_sink is off by default

Testing:

New test/sqllogictest/metric_sink.slt: the column contract and each way of violating it, FROM targets with no rows to read, IF NOT EXISTS, the flag-off refusal, and the seven views above answering with
a metric sink present.

Note to reviewers: The most important files to look at are:

ddl.rs which has the actual plan for create/drop metric sink
create_metric_sink.rs which writes the catalog item
objects.rs which defines the item_type
mz_catalog.rs small tweak here
rbac.rs/acl.rs for privilege handling
lib.rs/jsonb.rs for audit logging
v90_to_v91.rs the no-op migration
metric_sink.slt for what is expected behaviour

The rest is threading this all through

Problem: To maintain cluster metrics we defined the MetricSink compute operator. However, there is no way to ask for one. We need a SQL surface that names a metric sink and a durable catalog representation that survives a restart. Solution: Add `CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>` and `DROP METRIC SINK`, gated behind `enable_metric_sink`. Creating a sink writes a catalog item and nothing else: no dataflow is optimized or shipped, so a sink created today publishes no metrics. Planning checks that the `FROM` relation exposes the five columns the operator reads (`metric_name`, `metric_type`, `labels`, `value`, `help`). Note: order is not enforced and extra columns are fine. Nullability is not checked. Metric sinks need no new durable record. They persist as ordinary `Item`s and `item_type` works the type out from `create_sql`. However, the changes to audit and serialization does bump the catalog version. There are some gaps in: - `mz_comments` has no `MetricSink` branch, but `CommentObjectType` has no variant either, so `COMMENT ON METRIC SINK` does not parse and no such record can exist - `MZ_DEFAULT_PRIVILEGES` has no CASE arm, but `ON METRIC SINKS` is rejected during planning Note: `enable_metric_sink` is off by default Testing: - New `test/sqllogictest/metric_sink.slt`: the column contract and each way of violating it, `FROM` targets with no rows to read, `IF NOT EXISTS`, the flag-off refusal, and the seven views above answering with a metric sink present. Co-Authored-By: Moritz Hoffmann <mh@materialize.com>

SangJunBak

Had a few other comments:

What's the FROM relation for these metric sinks going to be? Wasn't too clear from the design doc. Are we going to simply ship these "builtin metric sources" per replica like our mz_introspection relations? Wonder if it's possible to simply create builtin views over the mz_introspection relations to match the schema. Are we expecting one giant metric sink for each cluster or one metric sink per metric? How would spreading out across multiple sinks affect performance?
Are we expecting users to do the DDL, for us to create builtin metric sinks, or both?
Should we implement EXPLAIN for these metric sinks in the future?

SangJunBak · 2026-08-03T18:12:30Z

+/// `CREATE METRIC SINK`
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct CreateMetricSinkStatement<T: AstInfo> {
+    pub name: Option<UnresolvedItemName>,


I wonder if we can make this non-nullable. I know CreateSink has the same schema, but I think it's because EXPLAIN KEY SCHEMA FOR CREATE SINK FROM bar (where we don't care about the name) is valid syntax

SangJunBak · 2026-08-03T18:12:31Z

+    } = &mut stmt;
+
+    // The parser always fills the name in. `Option` is here only because the AST shape is
+    // shared with `CreateSinkStatement`, where a sink can be created unnamed.


Are we assuming metric sinks can be unnamed for parity with a sink? I wonder how valuable maintaining parity would be 🤔

If we decide to keep it nullable, I think we should extend ddl in testdata with this case

SangJunBak · 2026-08-03T18:12:31Z

+    {
+        use CatalogItemType::*;
+        // `Index` belongs with the rejects even though it lives on a cluster: it has no
+        // relation description, so letting it through here only buys a vaguer error below.
+        match from_item.item_type() {
+            Table | Source | View | MaterializedView => {}
+            Sink | MetricSink | Index | Type | Func | Secret | Connection => {
+                sql_bail!(
+                    "cannot create metric sink from {} because it is a {}",
+                    scx.catalog.minimal_qualification(from_item.name()),
+                    from_item.item_type(),
+                );
+            }
+        }
+    }


I wonder if this is redundant given relation_desc does the same filter, except it includes builtinlogs?

SangJunBak · 2026-08-03T18:12:32Z

+        # `mz_catalog_raw` needs a system connection.
+        return Testdrive(dedent("""
+                ! CREATE METRIC SINK metric_sink_one FROM metric_sink_view
+                contains:metric sink "materialize.public.metric_sink_one" already exists


Good test for now but would be worth to refactor this test to point to an actual builtin object afterwards, or the way we would actually run this in prod! Otherwise an slt for this test case would be preferred over a platform check

mtabebe · 2026-08-03T20:03:20Z

Had a few other comments:

What's the FROM relation for these metric sinks going to be? Wasn't too clear from the design doc. Are we going to simply ship these "builtin metric sources" per replica like our mz_introspection relations? Wonder if it's possible to simply create builtin views over the mz_introspection relations to match the schema. Are we expecting one giant metric sink for each cluster or one metric sink per metric? How would spreading out across multiple sinks affect performance?

See the next draft PR for what the from will be: #38018

It is any relation exposing the five-column contract (metric_name, metric_type, labels, value, help).

It is not quite one sink per cluster or one per metric. It is one sink per replica for metrics that share a source view (e..g, arrangement size metrics all have the same source view)

Are we expecting users to do the DDL, for us to create builtin metric sinks, or both?

The plan is that we (adapter) will do the DDL. We aren't exposing this to users, but could in the future.

Should we implement EXPLAIN for these metric sinks in the future?

Deferred to a future PR!

Make CreateMetricSinkStatement.name non-nullable, collapse the redundant item_type reject match into the relation_desc None branch, and add a TODO(SQL-572) on the platform-check survival probe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

mtabebe force-pushed the ma/prom-metrics/sql-554-catalog-objects branch from 0060131 to a469492 Compare July 30, 2026 00:11

mtabebe requested review from SangJunBak and antiguru July 30, 2026 15:27

mtabebe marked this pull request as ready for review July 30, 2026 15:28

mtabebe requested review from a team as code owners July 30, 2026 15:28

SangJunBak reviewed Aug 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

sql: add CREATE METRIC SINK as a durable catalog object (SQL-554) - #37958

sql: add CREATE METRIC SINK as a durable catalog object (SQL-554)#37958
mtabebe wants to merge 2 commits into
MaterializeInc:mainfrom
mtabebe:ma/prom-metrics/sql-554-catalog-objects

mtabebe commented Jul 29, 2026 •

edited

Loading

Uh oh!

SangJunBak left a comment

Uh oh!

SangJunBak Aug 3, 2026

Uh oh!

SangJunBak Aug 3, 2026

Uh oh!

SangJunBak Aug 3, 2026

Uh oh!

SangJunBak Aug 3, 2026

Uh oh!

SangJunBak Aug 3, 2026

Uh oh!

mtabebe commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mtabebe commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SangJunBak left a comment

Choose a reason for hiding this comment

Uh oh!

SangJunBak Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

SangJunBak Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

SangJunBak Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

SangJunBak Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

SangJunBak Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

mtabebe commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mtabebe commented Jul 29, 2026 •

edited

Loading