Effective MLComputePolicy exposure

@handellm

Opening this fresh issue per:

RESOLUTION: Close issue #836 and open a new issue for Effective MLComputePolicy exposure discussion.

Effective MLComputePolicy is the policy actually used by the implementation. To that end, this effective policy can be exposed only after compilation, likely on MLGraph.

Based on today's discussion, the group believes the policy is a better abstraction than devices, thus we resolved that this to-be-defined policy-centric mechanism will supersede the earlier device-centric graph.devices proposal that was discussed in issue #836 and PR #854 that was closed.

The group's next step is to make sure the effective policy is exposed in a way that allows developers to understand it.

@handellm owns the use case and a product that would benefit from this feature, and can enlighten us on the preferred design.

For your reference, here's the latest MLComputePolicy that's passed in at context creation time (from #923):

enum MLComputePolicy {
  "default",
  "high-performance",
  "low-power",
  "fallback"
};

@anssiko

Thanks for filing the issue, @anssiko!

Here are the use-cases that I think effective compute policy would help solve:

Real-Time Media Constraints Audio models need to be executed with low and reasonably predictable latency and can typically not run well on GPUs and NPUs. Video models have unacceptable power or latency characteristics when executed on CPU. A reported mismatching effective compute policy enables an execution router to consider alternatives before ruining user experience by simply trying.
Dynamic execution routing The system must account for real-time device conditions when considering how to run a new workload (locally or in the cloud). If effective compute policies indicate potential hardware contention, the router can rearrange workloads to avoid degrading the user experience.
QoS monitoring There could be usage of ops and data types in a model that force unintended effective quality of service causing poor user experience. Trying to figure out all incompatibilities at development time is impossible due to the cartesian zoo of implementations (hardware, drivers, browsers, OSes). The effective compute policy enables aggregation of outcomes.

The concrete proposal (which is similar to already discussed alternatives) is to expose a set of MLComputePolicies that match the effective compiled graph's compute characteristics. A set of policies seems to better express the situation for the implementation than a single policy, for example to be able to express partial fallback.

interface MLGraph {

  // Set-like
  readonly attribute FrozenArray<MLComputePolicy> effectiveComputePolicies;

}

Interpretations

["high-performance", "fallback"] could indicate partial hardware acceleration
["low-power", "high-performance"] could indicate inference is using a device which is both high performance and low power
["fallback"] there won't be hardware acceleration when dispatching this onw

There are also non-sensical combinations, not sure how to deal with them.

Examples of use:

Use case	Requested policy	Criteria for using WebNN
Audio denoising	`"fallback"`	`"fallback"` in `effectiveComputePolicies` AND `effectiveComputePolicies` does not contain `"low-power"` or `"high-performance"`
Video processing + dynamic routing example	`"low-power"`	`"fallback"` not in `effectiveComputePolicies` AND (no significant NPU workload OR `"low-power"` not in `effectiveComputePolicies`) AND (no significant GPU workload OR `"high-performance"` not in `effectiveComputePolicies`)

Wdyt?

This proposal works well with the current opaque design for MLGraph. Needs to be checked against others' use cases, though.

There are also non-sensical combinations, not sure how to deal with them.

Implementations will own the sense of them, in this proposal.

I assume the "Criteria for using WebNN" are checked inside the application. Would there be something WebNN could do to make these checks simpler?

Would there be something WebNN could do to make these checks simpler?

The end result of our use cases will result in us either using or not using the compiled graph, so an alternative solution could be to pass some kind of "acceptable" representation when compiling the graph and fail it if it doesn't go the right way. That could impact the rejection time positively.

@jonathanding

RESOLUTION: Review Dynamic AI Offload Protocol in context of the effective MLComputePolicy proposal.

@jonathanding, the WebML WG discussed dynamic execution routing today and wants to invite you to share an update on the Dynamic AI Offloading Protocol (DAOP) incubation to understand synergies.

The group would like to hear your learnings from the DAOP incubation, both web and native side, to understand how to shape the effective MLComputePolicy proposal.

@handellm represents the Google Meet product that would benefit from dynamic execution routing, a use case that is increasingly more important given the global compute shortage.

@mwyrzykowski

@mwyrzykowski what do you think of the proposal?

@handellm

Examples of use:

Use case	Requested policy	Criteria for using WebNN
Audio denoising	"fallback"	"fallback" in effectiveComputePolicies AND effectiveComputePolicies does not contain "low-power" or "high-performance"
Video processing + dynamic routing example	"low-power"	"fallback" not in effectiveComputePolicies AND (no significant NPU workload OR "low-power" not in effectiveComputePolicies) AND (no significant GPU workload OR "high-performance" not in effectiveComputePolicies)

@handellm it seems like sites should not be trying to rely on this

E.g., site requests fallback, but UA implements no such fallback, similar to WebGPU's fallback which is optional https://www.w3.org/TR/webgpu/#fallback-adapter but rather a UA reports high-performance, site chooses not to run the model which would have otherwise ran great.

Similarly, high-performance and low-power are not necessarily mutually exclusive on some hardware.

I will try to make the next meeting the the WG time slot is lately a conflict after day light savings pushed the meeting back an hour.

It would seem some type of high performance timer could achieve the same with far greater reliability than exposing an effective compute policy.

E.g., site requests fallback, but UA implements no such fallback

Why would it not implement fallback? Is there an implementation that doesn't implement it?

but rather a UA reports high-performance, site chooses not to run the model which would have otherwise ran great.
It would seem some type of high performance timer could achieve the same with far greater reliability than exposing an effective compute policy.

"high-performance" would typically not be an acceptable result in the case of Audio models. Although high performance timers are not possible on the web platform due to security, you might still be able to use normal performance.now() for typical Audio dispatch speeds. But regardless - the problem with "trying anyway" is still that on a device with unsuitable implementation the absence of an API will cause bad experience for the user before detected and falling back to working solutions. Why would that be desireable?

E.g., site requests fallback, but UA implements no such fallback

Why would it not implement fallback? Is there an implementation that doesn't implement it?

WebGPU has the concept of fallback adapters which are optional. Their target use case is often dealing with non-conformant GPU drivers whose bugs are too challenging or not practical to work around.

There is no requirement to provide such an adapter. A site requests it and the UA is free to ignore it, for instance on GPU hardware with few bugs to work around.

WebGL and WebGPU both have power preference enums, WebGLPowerPreference and powerPreference, again a UA is free to ignore these as they are hints.

"high-performance" would typically not be an acceptable result in the case of Audio models.

Currently the WebNN specification defines high-performance as:

"high-performance"
Prioritizes execution speed over power consumption.

One would think prioritizing execution speed is precisely what is desired for audio processing. A UA reports high-performance and then the site decides not to run the model due to anticipated latency => that does not make sense since the UA never said there was going to be latency.

Although high performance timers are not possible on the web platform due to security, you might still be able to use normal performance.now() for typical Audio dispatch speeds. But regardless - the problem with "trying anyway" is still that on a device with unsuitable implementation the absence of an API will cause bad experience for the user before detected and falling back to working solutions. Why would that be desireable?

It is a reliable method handling system load from other actively running sites / applications / system operations and performance.now() should be sufficient. We could also consider exposing some graph compilation metrics regarding rough latency and runtime estimates to avoid requiring such timings.

I don't think we should build some matrix of hints to latency / runtime / chip-the-model-runs-on expectations, that is not the purpose of hints and would be objectionable.

One would think prioritizing execution speed is precisely what is desired for audio processing.

Note low-latency differs from throughput - a horse runs faster than a rabbit, but for short distances, a rabbit beats a horse because it takes longer for the horse to gain speed. We saw that multiple times with DirectML (RIP) where customers would complain that their tiny audio models took too long to upload the audio data to the GPU, process it, and then download data back. So we recommended to just keep the audio model on the CPU instead (or these days, perhaps NPU's if UMA and beefier than they used to be). Now if they passed a whole mp3 to the GPU, yes, that would certainly beat the CPU, but audio filters like live noise cancellation during meetings have really tiny time slices and need immediate results to avoid audio lag.

@RafaelCintron

One would think prioritizing execution speed is precisely what is desired for audio processing.

The reality is that on many platforms, hardware execution is associated with queueing delays. For Audio this is a killer reason to avoid it.

This is one example where the UAs view of "it should fit" might not match actual reality for app developers. There are others, like hybrid graphs which are likely to not match realtime performance expectations for video.

When the UA makes the wrong choice, the user has to endure poor performance before the system notices. This is what we want to avoid.

Also interested in perspectives from @RafaelCintron and @huningxin on the matter.

@handellm

We again hit a resource optimization problem seen and discussed before, also in this group: "high-performance" is a compound term and depends also on the application (model): whether audio, video, generative, etc. I'd assume that for audio a "low-latency" description would be more generic and more relevant. Or, we could create more hint classes (compute policies) describing the intended inputs and outputs.

These (counter-)examples don't necessarily invalidate the proposed design (that returns an array of compute policy descriptors), but we need to de-confuse the combinations by choosing a higher resolution "sampling" of the use case descriptors, which are as much orthogonal to each other as possible, in order to reduce overlaps. If we introduce more descriptors that are overlapping then we'll face exponentially bigger confusion space when these overlaps happen.

Fortunately we don't need to solve the whole world, then the use cases, but first focus on the use cases.

Still the question remains, @handellm - for your use cases, could you come up with a single descriptor that would quantify your needs, or if it must be a combination, is there an orthogonal set of descriptors that would solve your use case, and could be also used to decompose other use cases?

@zolkis

@zolkis re: low-latency. For the case of realtimeish Audio processing, use of hardware is a no-go so the present "fallback" perfectly serves the purpose. Not saying that there might be other use cases of low-latency or in what sense, just I don't know what those would be. Maybe low-latency can be separately discussed?

It looks like the primary question is whether exposing scheduled compute devices to the app is necessary. So as I mentioned above, since the anticipated end result would be about not using the compiled graph, one alternative proposal could be that the UA fails MLGraphBuilder build if the results turns out to be incompatible with a specified MLContext computePolicy. That pattern kind of aligns with the existing pattern of WebRTC getUserMedia constraints whose promise is rejected if constraints are not met.

Examples needed to solve use cases 1-3 (drafty):

MLComputePolicy	Description
low-power, high-performance	Prefer [low-power, high-performance] device but accept ops falling back
strict-low-power	Fail build if some op is not scheduled to a low-power equivalent device
strict-high-performance	Fail build if some op is not scheduled to a high-performance equivalent device
strict-fallback	Fail build if ops are scheduled on HW
strict-accelerated	Fail build if some op falls back to CPU
strict-accelerated-prefer-low-power	Fail build if some op falls back to CPU, prefer scheduling to low-power devices

Wdyt?

In the table above, I assume we'd also use these MLComputePolicy values as context creation hints? So context creation would not fail, but the implementation will decide what config is closest to the hints, and then the next time the app could check the details (and make a decision) was when a compute graph was available?

one alternative proposal could be that the UA fails MLGraphBuilder build if the results turns out to be incompatible with a specified MLContext computePolicy.

Instead of failing the build, I'd prefer to just expose the internal state, and based on this the app could determine the next steps (using also other factors).
Besides, the requirement to trigger failure based on a notion of "incompatibility" with a given compute policy will always be underspecified, i.e. the impl has no way to know for sure whether a given state actually counts as a failure for the app (under the current circumstances).

If we can spec an exact set of descriptors and a consistent compatibility algorithm, then we could consider failing graph building if compatibility is broken.

In specific use cases such as this, which is still generic, I wonder why not dedicate a compute policy like "for audio"?
The primary requirements seems to be latency, rather than throughput, power or memory -- but we need a latency specific to audio processing. Of course, we know that CPU (fallback) was going to be fine, but what if the given HW can fulfill the requirements also with other HW configuration than assumed? We'd hardcode assumptions that HW evolution might render obsolete.

What I meant with a "low-latency" compute policy is that it could be CPU (fallback) or the closest accelerator that can fulfill that. But even for audio, we might have multiple classes (pro audio 3-5 ms, VoIP 10-20 ms, AI models with a small lookahead buffer < 30 ms. We could have a "low-latency--audio" hint meaning e.g. < 30 ms. That would be a clear criterion.
Then we could even fail the context creation if the impl knew this was not possible.
Also, the build (or running the graph) could fail if that condition was not met.

(I am fine with either proposal, but we should avoid assumptions made at current levels of HW design, and should try to find the descriptors relevant to the use case in question. Since we opted for unified/collated compute policies, we might need to add a new compute policy for many such use cases).

@mwyrzykowski

Instead of failing the build, I'd prefer to just expose the internal state

That would be acceptable to me as well, but considering @mwyrzykowski's concerns with exposing the right info, failing build for "strict" versions could be an implementable option whose form already has precedent in shipping Web APIs (eg getUserMedia, MediaCapabilities...)

Regarding the audio options, some classes that come to mind are: audio-render (must complete 10-ish ms deadline or else glitch), voip-capture (say 20 ms deadline). I'm not aware of any HW API that would be able to soft-guarantee these limits today though so I'm unsure what value it would bring in practice. It's more like there could be some risk that over-eager UA implementors assume things about my app and do the wrong thing :) but sure that would open the door from accelerated Audio processing when the day comes.

It would seem preferable to expose approximate characteristics regarding graph latency and execution time, as opposed to repurposing MLComputePolicy to mean something which the site should interpret.

We don't want sites interpreting these values to mean 'my model is running with low latency on the CPU'. There is no guarantee that is consistent over the application's lifetime.

@mwyrzykowski

@mwyrzykowski do you have a concrete counter proposal that solves the use cases?

I'd be interested in hearing @RafaelCintron and @huningxin 's views as well.

The system characteristics may change at any time, applications open and close throughout a website's / web app's lifetime.

Periodic timer measurements would solve this reliably and can be in the 10ms resolution time or less. Repurposing MLComputePolicy seems fragile and the concern is that websites will make wrong decisions based on values UAs report, they already do this frequently with just the user agent.

We could also consider something similar to WebGPU's timestamp query: https://www.w3.org/TR/webgpu/#timestamp though it sounds like we need the time the model takes to execute along with any latency

We had a good discussion on this topic today (minutes), covered the following:

… - 1) compilation metrics & runtime estimates by MikeW
… - 2) low latency v high throughput tradeoff implications by Dwayne
… - 3) strict hints to fail at build by MarkusH, Reilly to speak to this?
… - 4) "low-latency" and "precision" hints by Dwayne

As for the next steps, the path forward is to continue discussion here to see if any of these 4 proposals have strong support from the group, and if so, which one(s) we should prioritize for further exploration and potential implementation?

@mwyrzykowski

@mwyrzykowski your point about runtime drift is valid — system load, thermal state, and accelerator contention do change at any time, and performance.now() / timestamp queries are the right tool for that.

But there's a compile-time signal the API can expose: the intended compute policy — the best result the context can deliver against the user's hint, decided at MLGraphBuilder.build() and stable for the MLGraph's lifetime. This is decision-useful for @handellm's gating: "if fallback is in the set, route to cloud." It does not claim runtime device scheduling.

interface MLGraph {
  // Compile-time assignment, not a runtime guarantee.
  readonly attribute FrozenArray<MLComputePolicy> intendedComputePolicies;
}

compile-time signal the API can expose: the intended compute policy — the best result the context can deliver against the user's hint

The objection to reusing the hint for this purpose is the hint should remain strictly a hint and nothing that the website will make decisions based on.

A conformant UA should be free to ignore the hint. Associating fallback with CPU processing is fragile at best. A UA should never be required to implement a fallback if the 'accelerated', which could be CPU SIMD acceleration, can fully implement the specification.

@handellm

Today we discussed Dynamic AI Offloading Protocol (DAOP) incubation learnings (estimateQoS() API, Performance Tiers, Boolean Requirement API) that inform this issue. We also had an insightful discussion on the compile-time vs runtime performance characteristics. We agreed to validate the discussed solutions with real-world use cases, shared by @handellm, inviting additional use case contributions as well as concrete solutions.

See minutes for more.

I wanted to chime in with my experience report as I'm poking around in this area: TinyStories/CoreML on A12/iOS 18.7.10 delivered 104 correct tokens/s with CPU-only vs 24 with CPU+GPU+ANE allowed, consistent repeated 63-token continuations after the first token. On M4 Max/macOS 27, optimization hints gave about 565 tokens/sec on repeats but 14.8 on the first observed continuation but was not a cold-cache measurement. I eyeballed every generated token and full logit vector. That made me think, could performance reporting distinguish preparation, first continuation and repeated execution, rather than treating a device/policy/etc label as a latency promise?

@anssiko

@anssiko Can this #963 be discussed alongwith. I can share some slides/videos for same.

Here is very initial explainer draft: https://github.com/shubhamg13/webnn-instrumentation/blob/main/webnn-ml-compute-timing-explainer.md

@handellm

We'll consider the MLComputeTiming proposal along with other options.

We need to evaluate the proposals along multiple axes: cross-platform implementability, product-market fit (cc @handellm), privacy implications.

Prototyping input is especially valuable to help inform this.

Navigation Menu

Repository navigation

Effective MLComputePolicy exposure #934

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Effective MLComputePolicy exposure #934

Description

Activity

handellm commented on May 8, 2026

zolkis commented on May 8, 2026

handellm commented on May 21, 2026

anssiko commented on May 21, 2026

handellm commented on May 21, 2026

mwyrzykowski commented on May 22, 2026

handellm commented on May 22, 2026

mwyrzykowski commented on May 22, 2026

fdwr commented on May 22, 2026

handellm commented on May 22, 2026

zolkis commented on May 22, 2026

handellm commented on May 25, 2026

zolkis commented on May 25, 2026

handellm commented on May 26, 2026

mwyrzykowski commented on May 26, 2026

handellm commented on May 27, 2026

mwyrzykowski commented on May 27, 2026

anssiko commented on Jun 4, 2026

huningxin commented on Jun 4, 2026

mwyrzykowski commented on Jun 4, 2026

anssiko commented on Jun 18, 2026

matthargett commented on Sep 15, 2026

shubhamg13 commented on Oct 22, 2026

anssiko commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions