Incremental UI layout updates

[ickshonpe]

Objective

Perform UI layout incrementally for better performance.

Other sub-objectives:

• bevy_ui stores a TaffyTree in the UiSurface resource that has to be kept synchronised with the bevy ecs UI node hierarchy. This has been a long time source of bugs and isn't very efficient. Instead, an adapter implementing Taffy's tree traits (from taffy::tree::traits) can give Taffy direct access to the UI node component data.

• Taffy has had calc support for a while, but it's not usuable through TaffyTree. The adapter will allow us to add a Val::Calc variant.

• ui_layout_system has too many responsibilities:

  • Synchronising the Node component data with Taffy's Style data.
  • Synchronising the ecs hierarchy with the TaffyTree. This includes GhostNodes and FixedNodes which generate all sorts of edge cases that need to be carefully handled.
  • Injection of measure funcs into the TaffyTree.
  • Managing the implicit root viewport nodes.
  • Removal detection and subsequent cleanup.
  • Updating ComputedNode's from the Taffy layout.
  • Resolving non-taffy generated geometry like outlines and border radius.
  • Invalidation of unreachable or diasabled UI nodes.

  It should be split up into multiple simpler systems that can be understood, tested and profiled in isolation.

• Interactions between certain components, in particular GhostNode, FixedNode, OverrideClip and UiTransform, haven't always been well defined or documented. Incrementality forces us to come up with a precise answer for every case.

Fixes #25150

Solution

Apologies for the massive size of this PR. The number of line changes is inflated by the large number of trait impls, tests, and other boilerplate, and also the deletion of the ui_surface and experimental modules. But even so, it's still huge.

I chose to focus more on correctness over super aggressive optimisation for this PR. Any hierarchy changes to a Node entity trigger a full walk over every UI node from each UI root. Geometric updates cascade through all descendants, which is overkill for changes that can only affect the node itself. The UI roots list has no change detection, and is rebuilt every frame by update_ui_roots. The queries for ui_layout_system and mark_dirty_ui_trees overlap somewhat, they use the results differently, but there is some redundancy and room for consolidation there.

There are four main dirty flags used to track changes:

• A UiTreeDirty marker component that uses change detection to indicate that a layout input changed for a Node entity or one of its descendants.
• Three fields on the new ComputedLayout component:
  • self_dirty: A change to a local input, either for Taffy (such as Node or ContentSize) or geometry only (such as UiTransform or OutLine).
  • subtree_dirty: The entire subtree needs a geometry update.
  • layout_dirty: The output layout returned from Taffy changed for this node.

I'd recommend for any reviewers to concentrate on these four functions, in order:

1. bevy_ui::layout::mark_dirty_ui_trees
2. bevy_ui::layout::ui_layout_system
3. bevy_ui::layout::layout_tree::sync_runtime_layout_tree
4. bevy_ui::layout::update_uinode_geometry_recursive

Otherwise It'd be good to look at existing projects built on top of bevy_ui, especially bevy_reactor and bevy_immediate, and make sure they still work correctly. Also look at the more complex UI examples, testbed_ui, feathers_gallery, mines and testbed_full_ui are probably the most likely to reveal any regressions.

Also be aware that some of the notes below might be out of date. This PR has gone through a lot of revisions, and I've started to lose track of everything that was changed myself.

UiSurface and ui_surface have been removed.

The UiSurface resource and the ui_surface module have been removed.

Instead of maintaining a separate TaffyTree in the UiSurface resource, bevy_ui now has a new UiLayoutTree adapter that allows Taffy to access the ECS component data directly.

The Taffy NodeId <-> Entity bimaps are gone

Since NodeIds are just u64s, we can generate them from each Node's entity id using Entity::to_bits as needed. Entity::from_bits is used on the NodeId to map back.

entity.to_bits() can't be zero, so a NodeId of zero is used to represent viewport nodes.

UiRoots resource

All the different types of root UI nodes are collected into a resource UiRoots by the update_ui_roots in UiSystems::Prepare. UI root discovery is quite complicated now, repeating the logic in each system is too fragile. SystemParams could have been used instead, but that makes it harder to enforce a stable ordering.

UiStack and propagate_ui_target_cameras don't use UiRoots as they don't need any special rules for dealing with GhostNodes and FixedNodes.

New layout_tree module

The bevy_ui::layout module has a new submodule layout_tree containing the majority of the new incremental layout implementation and taffy communication layer.

Measure func changes

Previously, NodeMeasures were moved out of the ContentSize components during layout. Now they are accessed directly by querying for Ref<ContentSize>. As a result, the receiver for the Measure trait and its implementation for NodeMeasure no longer needs to be mutable, instead &self is sufficient.

GhostNode reimplementation

GhostNodes now require Node and behave more like regular UI nodes. During layout a GhostNode is replaced by its children recursively, so that its nearest non-ghost descendants become children of its nearest non-ghost ancestor.

Each GhostNode has a corresponding taffy node now, but it's zero-sized and disconnected singleton node. When updated their ComputedNode is set to zero size (I had planned to give a GhostNode bounds that encompasses its children, but left this out for now, it seems useful but it would require a new mechanism to collect the children's bounds).

UiTransform is propagated through GhostNodes normally except that for percentage translations, the closest non-ghost ancestor's base size is used otherwise, since GhostNodes always have zero-size, percentage translations would always resolve to zero.

GhostNodes requiring Node makes traversal much simpler, we only need to consider ghost nodes during ComputedNode updates and in layout when they are replaced. The UiChildren and UiRootNodes system params are no longer needed. The bevy_ui::layout::experimental module and the ghost_hierarchy submodule have been removed.

The "ghost_nodes" feature gate has been removed. GhostNodes are always enabled now. Profiling indicated that even with the previous system params implementation, it would be cheaper to have the GhostNode feature enabled all the time, even when they aren't used.

FixedNode changes

A FixedNode creates a new layout context, its ancestors layout is not affected by the FixedNode or its descendants. So in mark_dirty_ui_trees the upwards walk to set the dirty subtree flags stops at any FixedNode.

A GhostNode cannot also be a FixedNode. If a node has both FixedNode and GhostNode components, FixedNode is ignored.

There were some OverrideClip changes which were split off into a separate PR and have already been merged: #25613

Taffy

Taffy's TaffyTree is no longer used, instead implemented all the taffy::tree::traits on a new struct UiLayoutTree that acts as an adaptor allowing Taffy direct access to the necessary component data.

Each UI entity has new components TaffyStyle and ComputedLayout.

TaffyStyle contains the input layout data, and is updated from Node on changes by the sync_taffy_styles_with_nodes system.

ComputedLayout holds the cached calculations, resolved children, dirty flags and data and output layout geometry for each node.

These changes also allow us to add a Calc variant to Val. I implemented a basic version already, just waiting for this to get merged.

ui_layout_system changes

Calls compute_layout for each UI root to update its Taffy layout data.

No longer responsible for component change dectection or updating ComputedNodes.

After layout updated, it clears any UI nodes that became unreachable since the previous update.

New update_computed_nodes system

New system that takes over responsibility for updating ComputedNodes from ui_layout_system.

Previously the entire UI hierarchy was walked to update every ComputedNode, now only those nodes marked dirty are updated.

GhostNodes are also updated here now, their size is set to zero.

New update_border_radius system

ResolvedBorderRadius is updated after layout in a separate system. Profiling lead change.

mark_dirty_ui_trees and UiTreeDirty

UiTreeDirty is a new marker component that is set change detection to indicate that some input for layout changed for a node or one of its descendants.

mark_dirty_ui_trees watches for all UI component changes, insertions and removals that will require a layout update. For each dirty Node entity and its direct ancestors it sets the UiTreeDirty component changed.

update_clipping

The clipping bounds are updated incrementally.

Since GhostNodes are Nodes, I had to add some special casing for them. Clipping is propagated downwards through ghosts, but the overflow field on the ghost's own Node component is ignored.

accessibility

The changes here shouldn't introduce any new problems, unless someone does something weird with GhostNodes, maybe.

UiStack

UiStack updates are still immediate. The only change is that it no longer uses the UiChildren traversal params. GhostNodes are full UI nodes now and have a ComputedStackIndex so they are visted during the UI stack walk and this does this does affect the visual ordering of nodes. Consider:

fn setup(mut commands: Commands) {
    commands.spawn(Camera2d);
    commands.spawn_scene(bsn! {
        Node
        Children [
            GhostNode
            Children [
                Node {
                    position_type: PositionType::Absolute,
                    width: px(10),
                    height: px(10),
                    top: px(5),
                }
                BackgroundColor(RED)
                ZIndex(1)
                --
                Node {
                    position_type: PositionType::Absolute,
                    width: px(10),
                    height: px(10),
                    left: px(5),
                }
                BackgroundColor(BLUE)

            ]
            --
            Node {
                position_type: PositionType::Absolute,
                width: px(10),
                height: px(10),
            }
            BackgroundColor(GREEN)
        ]
    });
}

On main, the GhostNode is skipped and its children hoisted, so the render order would blue, green, red.
But with this PR the GhostNode is a node, so its children are z sorted only relative to each other. The render order ends up blue, red, green instead. This isn't ideal, the blue, green, red ordering is the correct one. I left it to be fixed in a follow up as it would need a rewrite of the system and possibly changes to the ComputedStackIndex component as well.

---

Future work

• Change detection is still too broad. Maybe UiTransform should be updated separately.
  -There is still full tree walk on structual changes, this could be done incrementally.
• Maybe ComputedLayout + ComputedNode could be consolidated.
• The implicit taffy viewport nodes aren't retained. I don't think it's worth it, but not certain, could investigate whether there's any advantage in retaining them.
• ui_layout_system walks up through all the ancestors of each FixedNode every frame checking that each is a valid UI Node. Should only have to do this on changes.
• Since this implements all of Taffy's tree traversal traits, we can add Val::Calc support once this is merged.
• Reimplement the accessibility module.
• Clipping updates are delayed a frame when positions are updated after layout, like with popover. Needs some mechanism to set nodes dirty again after layout. Or maybe the popover implementation should be part of layout.
• UI stack updates are still immediate.
• Ghost nodes now have a ComputedStackIndex. This affects the stack ordering of nodes, and needs to be fixed in a follow up.
• Clipping is updated incrementally now, but otherwise the implementation is still quite inefficient.
• UiSystems should be split up further with a separate Sync set between Content and Layout, where the pre-layout synchronisation is handled and the dirty flags are set.

AI use disclosure

A review was done by Claude, which picked up the tree_changed_query mistake, and an inconsistancy with root ghost nodes and transforms.

Testing

Includes a lot of new regression tests now. layout/mod.rs was getting too large, so moved them into a layout/tests.rs file.

The more complex UI examples such as testbed_ui, feathers_gallery, mines and testbed_full_ui are most likely to show up any regressions.

The examples are unchanged, so screenshot CI should be passing.

Benchmarks

There are UI layout benchmarks now in benches, you can run them with:

cargo bench -p benches --bench ui

You can perform a comparison by first running the UI benchmarks on main and saving a baseline:

cargo bench -p benches --bench ui -- --save-baseline main

Then switch to this PR and run:

cargo bench -p benches --bench ui -- --baseline main

I saw a ~75% improvement with the static layouts and a ~10% regression on full updates.

---

Showcase

These were run about a hundred commits ago, but nothing should have changed substantially 🤞 :

cargo run --example many_buttons --release --features="trace_tracy"

FPS comparison doesn't show much because pipelined rendering:

But just PostUpdate:

cargo run --example many_buttons --release --features="trace_tracy" -- --text

Just PostUpdate again:

cargo run --example many_buttons --release --features="trace_tracy" -- --respawn

It's even a little faster for complete rebuilds.