dotnet test fails on Linux under proot when the current process is not detected as the dotnet muxer (ARM64 'dotnet host' not found)

[Starsky16]

Describe the bug

Running dotnet test inside an aarch64 Debian rootfs under proot (Android Termux) fails with:

找不到“ARM64”体系结构的“dotnet”主机。
(Could not find a 'dotnet' host for the 'ARM64' architecture.)

uname -m = aarch64, RID linux-arm64. Reproduced with .NET SDK 8.0.424 and 10.0.400.

Root cause

In DotnetHostHelper.TryGetDotnetPathByArchitecture (src/Microsoft.TestPlatform.CoreUtilities/Helpers/DotnetHostHelper.cs) the fast path only applies when the current process is the dotnet muxer:

if (_processHelper.GetCurrentProcessArchitecture() == targetArchitecture)
{
    string currentProcessFileName = _processHelper.GetCurrentProcessFileName()!;
    if (Path.GetFileName(currentProcessFileName) == _muxerName)
    {
        muxerPath = currentProcessFileName;
        ...
        return true;
    }
}

On Unix GetCurrentProcessFileName() comes from Process.MainModule.FileName, which .NET derives from the first mapping in /proc/self/maps. Under proot an empty placeholder loader file is injected at the lowest address of every process, so MainModule.FileName returns

/data/data/com.termux/files/usr/libexec/proot/loader

even though the real executable is /usr/share/dotnet/dotnet (probe from inside the same process):

Environment.ProcessPath      = /usr/share/dotnet/dotnet
/proc/self/exe               = /usr/share/dotnet/dotnet
Process.MainModule.FileName  = /data/data/com.termux/files/usr/libexec/proot/loader
maps[0]                      = .../usr/libexec/proot/loader  (injected placeholder)

Because the basename is not dotnet, resolution falls through to the search branch (DOTNET_ROOT_ARM64 / DOTNET_ROOT, global registration, default installation location). Every candidate is then rejected because IsValidArchitectureMuxer only inspects PE headers on Windows and Mach-O on macOS — on Linux it never determines an architecture (actual ''; see #4042, closed as By Design). Even a valid, explicitly configured muxer is refused:

DotnetHostHelper.IsValidArchitectureMuxer: Incompatible architecture muxer, target architecture 'ARM64', actual ''
DotnetHostHelper: Invalid muxer resolved using env var key 'DOTNET_ROOT' in '/usr/share/dotnet'

#4042 was closed as By Design because on regular Linux the current-process fast path always succeeds. This environment breaks that assumption: the process is the muxer, but the /proc/self/maps-based image-name check reports a different name, so vstest lands in a search branch that can never validate a muxer on Linux.

Expected behavior / suggested fixes

1. Do not rely on the first /proc/self/maps mapping to decide "current process is the muxer" — prefer Environment.ProcessPath / /proc/self/exe for GetCurrentProcessFileName(). (These return the correct /usr/share/dotnet/dotnet in the same environment.)
2. Implement ELF e_machine inspection in IsValidArchitectureMuxer on Linux, mirroring the PE (Windows) and Mach-O (macOS) paths, so the search branch can actually validate real muxers.
3. At minimum, when an arch-specific DOTNET_ROOT_ARM64 (or DOTNET_ROOT) explicitly points to an existing muxer, honour it instead of failing the check on a platform where the validator always reports "incompatible".

Steps to reproduce

1. On an aarch64 Linux rootfs running inside Termux proot, install the .NET SDK (linux-arm64).
2. Create a minimal xUnit project and run dotnet test -c Release.
3. Observe the failure and the trace above with --diag.

Environment: OS Debian 13 (trixie) aarch64 inside proot; runtime RID linux-arm64; .NET SDK 8.0.424 and 10.0.400.

Workaround

Run tests with the xunit.runner.console runner directly (bypasses vstest) — this confirms the failure is isolated to vstest's dotnet-host resolution.

Related: #4042, #4194, #4349, #1749

[azat-msft]

Thank you for your feedback, Starsky16,

On the first glance it looks related to #16151 , I'll take a closer look at it next week and reach out if I was able to reproduce the issue or find a way to fix it.