Derive the command environment from .dsv descriptors

[mjcarroll]

The .dsv descriptors are colcon's own representation of an environment change. They arrived in 2019 (cbee991) together with a pure-Python evaluator generated into every install prefix (bd55f3f); sh and bat were moved onto that evaluator for their prefix scripts within days (bccec1d, 8fc78e3), and #244 made the descriptor authoritative even where no shell-specific script exists. The per-shell hooks are renderings of the same data.

generate_command_environment is the one consumer that never moved. It still writes a script which dot-sources each dependency's per-shell rendering plus all of its hooks and spawns a shell to dump the result. This implements it on DsvShell instead, at a priority above the shell extensions, and falls back to them whenever a descriptor references a script that has no descriptor of its own, or a dependency provides no package.dsv.

Not spawning a shell is also considerably cheaper. On a 363 package ROS 2 workspace on Windows the shell path costs roughly 438 + 60.5 * n_deps ms per package and the median package has 95 dependencies; evaluating the descriptors instead costs about 390 ms per package, measured over all 32664 dependency pairs of that workspace.

package	dependencies	shell	descriptors
rcutils	53	11.2 s	4.8 s
rviz_default_plugins	182	22.7 s	5.5 s

[codecov]

Codecov Report

❌ Patch coverage is 85.59322% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.31%. Comparing base (6bfb100) to head (537b783).

Files with missing lines	Patch %	Lines
colcon_core/shell/dsv.py	85.59%	14 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #748      +/-   ##
==========================================
+ Coverage   87.25%   87.31%   +0.06%     
==========================================
  Files          75       75              
  Lines        4692     4809     +117     
  Branches      817      845      +28     
==========================================
+ Hits         4094     4199     +105     
- Misses        468      477       +9     
- Partials      130      133       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cottsay]

Thanks for the work here.

In addition to promoting DsvShell to be a fully-featured and higher priority shell than the "native" primary shells, this change introduces some aggressive caching that I'm a bit concerned about (namely _flattened_files). While this mechanism should properly detect when the .dsv itself changes, DSV effectively supports chaining that wouldn't be caught by this invalidation strategy.

Given the impact of spawning subprocesses in Windows, I'm skeptical that this aggressive caching is even worth the risk of invalid caches causing cryptic and difficult to reproduce bugs. Please try simplifying the change to focus only on the high-impact DSV capabilities and priority changes so we have the numbers behind that specific part of the change. If we decide that there is an opportunity for some package-level or hook-level caching, I'd rather see that explored in a way that would provide the benefit to all shells, not just .dsv.

It's probably worth noting here that the type of workspace plays a huge role in describing the impact of BatShell's poor performance. In many of our CI builds, we intentionally use isolated workspaces to help us uncover dependency issues. Merged workspaces, like the ones used to distribute binaries like the comprehensive .zip archive, would not see even close to the same penalty.

[mjcarroll]

Thanks for the early look, wasn't quite ready for another review yet, but this is good feedback, I will incorporate.

[mjcarroll]

Okay, I dropped the caching bits. The bugs are too subtle to be worth it.

Also, agreed on workspace type. My profiling is done with merge-install, which means all the scripts should be effectively no-op, right? An isolated workspace would have the same number of scripts, but they would actually be doing things in each script.