metric sinks: run, rehydrate, and track dataflow progress (SQL-571) - #38018
Draft
mtabebe wants to merge 2 commits into
Draft
metric sinks: run, rehydrate, and track dataflow progress (SQL-571)#38018mtabebe wants to merge 2 commits into
mtabebe wants to merge 2 commits into
Conversation
Problem: To maintain cluster metrics we defined the MetricSink compute operator. However, there is no way to ask for one. We need a SQL surface that names a metric sink and a durable catalog representation that survives a restart. Solution: Add `CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>` and `DROP METRIC SINK`, gated behind `enable_metric_sink`. Creating a sink writes a catalog item and nothing else: no dataflow is optimized or shipped, so a sink created today publishes no metrics. Planning checks that the `FROM` relation exposes the five columns the operator reads (`metric_name`, `metric_type`, `labels`, `value`, `help`). Note: order is not enforced and extra columns are fine. Nullability is not checked. Metric sinks need no new durable record. They persist as ordinary `Item`s and `item_type` works the type out from `create_sql`. However, the changes to audit and serialization does bump the catalog version. There are some gaps in: - `mz_comments` has no `MetricSink` branch, but `CommentObjectType` has no variant either, so `COMMENT ON METRIC SINK` does not parse and no such record can exist - `MZ_DEFAULT_PRIVILEGES` has no CASE arm, but `ON METRIC SINKS` is rejected during planning Note: `enable_metric_sink` is off by default Testing: - New `test/sqllogictest/metric_sink.slt`: the column contract and each way of violating it, `FROM` targets with no rows to read, `IF NOT EXISTS`, the flag-off refusal, and the seven views above answering with a metric sink present. Co-Authored-By: Moritz Hoffmann <mh@materialize.com>
Problem: `CREATE METRIC SINK` only wrote a catalog item. It never optimized or shipped a dataflow, so nothing actually ran. On top of that a metric sink export had neither a trace nor a `sink_write_frontier`, so every frontier report logged `collection without write frontier`, the controller never saw progress, and the since of the sink's input stayed pinned. Solution: `CREATE METRIC SINK` now optimizes and ships a dataflow, `DROP` tears it down, and bootstrap re-renders every metric sink so one survives a restart. The assembly optimizer starts from the `GlobalId` of the collection to export, like `CREATE INDEX`, pushes the row-wise shaping into MIR, and exports a single `MetricSink` sink. Sequencing is staged so optimization runs off the coordinator thread. Bootstrap reuses the global expression cache, like materialized views do. A cached plan embeds a transient id for the shaped view, and reusing that id across a boot is safe because build ids are dataflow-local on the worker and never registered in the controller's instance-global collections. For progress, the operator already folds the combined ok+err input frontier, so we hand that to the `ComputeState`. Every worker reports it, not just the one that owns the registry collector. The controller meets the per-worker frontiers, so a worker stuck at the minimum would otherwise hold the input back. Rendering create-item notices and shipping a dataflow under a read hold move into the shared `render_create_item_notices` and `ship_new_dataflow` helpers, with `CREATE INDEX` moved onto both. Testing: testdrive drives a sink's series into the replica's Prometheus registry and back out through `mz_cluster_prometheus_metrics`, then checks that `DROP METRIC SINK` and `DROP VIEW ... CASCADE` retract them. The restart platform check now probes for the re-rendered dataflow, not just the re-parsed catalog item. Introspection is per-replica and the checks default to two replicas, so it targets one. An optimizer unit test pins the assembled dataflow to exactly one `MetricSink` export over the shaped view. Co-Authored-By: Moritz Hoffmann <antiguru@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
CREATE METRIC SINKonly wrote a catalog item. It neveroptimized or shipped a dataflow, so nothing actually ran. On top of that
a metric sink export had neither a trace nor a
sink_write_frontier, soevery frontier report logged
collection without write frontier, thecontroller never saw progress, and the since of the sink's input stayed
pinned.
Solution:
CREATE METRIC SINKnow optimizes and ships a dataflow,DROPtears it down, and bootstrap re-renders every metric sink so one survives
a restart. The assembly optimizer starts from the
GlobalIdof thecollection to export, like
CREATE INDEX, pushes the row-wise shapinginto MIR, and exports a single
MetricSinksink. Sequencing is staged sooptimization runs off the coordinator thread. Bootstrap reuses the global
expression cache, like materialized views do. A cached plan embeds a
transient id for the shaped view, and reusing that id across a boot is
safe because build ids are dataflow-local on the worker and never
registered in the controller's instance-global collections.
For progress, the operator already folds the combined ok+err input
frontier, so we hand that to the
ComputeState. Every worker reports it,not just the one that owns the registry collector. The controller meets
the per-worker frontiers, so a worker stuck at the minimum would
otherwise hold the input back.
Rendering create-item notices and shipping a dataflow under a read hold
move into the shared
render_create_item_noticesandship_new_dataflowhelpers, with
CREATE INDEXmoved onto both.Testing:
testdrive drives a sink's series into the replica's Prometheus
registry and back out through
mz_cluster_prometheus_metrics, thenchecks that
DROP METRIC SINKandDROP VIEW ... CASCADEretract them.The restart platform check now probes for the re-rendered dataflow, not
just the re-parsed catalog item. Introspection is per-replica and the
checks default to two replicas, so it targets one.
An optimizer unit test pins the assembled dataflow to exactly one
MetricSinkexport over the shaped view.Co-Authored-By: Moritz Hoffmann antiguru@gmail.com