fix(mtp): analyse projects without a design-time build

[slang25]

Problem

Running Stryker with --test-runner mtp against a strong-named project fails: every mutant prints E (runtime error), Stryker logs "Stryker was unable to calculate a mutation score", and then exits successfully — masking that nothing was actually tested. Reported by @martincostello while trying MTP on Polly (#3094 comment, App-vNext/Polly#3131).

Root cause

E = MutantStatus.RuntimeError, which Stryker assigns when the MTP test host crashes during a run. The host dies with an unhandled exception on every run:

Could not load file or assembly 'Polly.Core, Version=8.0.0.0, ... PublicKeyToken=c8a3ffc3f8f825cc'
   at System.Reflection.RuntimeAssembly.GetExportedTypes()
   at Xunit.v3.XunitTestFrameworkDiscoverer.GetExportedTypes()

Comparing assembly identities:

Assembly	Version	PublicKeyToken
Original Polly.Core.dll	8.0.0.0	c8a3ffc3f8f825cc
Stryker's mutant	1.0.0.0	c8a3ffc3f8f825cc

The assembly version is normally injected by MSBuild — or by tools such as MinVer / Nerdbank.GitVersioning (Polly uses MinVer) — during a real build. But Stryker analyses projects with a Buildalyzer design-time build, which sets DesignTimeBuild=true. MinVer (among others) deliberately skips version generation on a design-time build, so the source Stryker captures — and recompiles the mutant from — carries the default 1.0.0.0 instead of the real version.

For a strong-named assembly that downgrade is fatal: the unchanged test assembly still references Polly.Core, Version=8.0.0.0, the loader cannot bind a lower-versioned assembly, the test host throws on a background thread and exits before any test runs. Non-strong-named assemblies ignore the version, which is why this only surfaces on signed projects.

Fix

Stop using a design-time build for analysis. Stryker now runs a regular build (so MSBuild computes the version exactly as a normal build would), while keeping SkipCompilerExecution=true — plus the output-copy / project-reference suppressions — so the C# compiler never runs and analysis stays fast and side-effect free.

Concretely, the Buildalyzer EnvironmentOptions is configured with DesignTime = false and a curated set of global properties that reproduce the cheap parts of a design-time build but omit the two flags that cause the problem: DesignTimeBuild (suppresses version generation) and BuildingProject (short-circuits the Build target). This is general — it fixes any strong-named project whose version comes from MinVer / Nerdbank.GitVersioning / GitVersion / etc., with no per-repo configuration.

Verification

Validated through Buildalyzer's own API (replicating Stryker's exact analysis call) against a strong-named MinVer project tagged 8.0.0:

Analysis build	Source files	References	Captured AssemblyVersion	csc ran?
Design-time (before)	3	163	1.0.0.0 ❌	no
Regular build, skip csc (after)	3	163	8.0.0.0 ✅	no (skipped)

Identical captured source files and references, the correct version, and the compiler is still skipped.

That the corrected version is what unblocks Polly was confirmed independently: forcing the analysed version to match the real one (via MinVerSkip on an otherwise-unfixed Stryker) takes the real Polly.Core (xunit v3 + MTP) from 0 mutations covered / all E to 1324 mutations covered, real results (.....SS.S.S...S.S), final score 81.82%, zero Es. This change achieves the same by making the analysis build produce the real version in the first place.

Added a unit test asserting analysis uses a non-design-time build that still skips compilation. All Initialisation + Compiling unit tests pass.

Reported in #3094

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes a failure mode when running Stryker with --test-runner mtp against strong-named projects by ensuring the emitted mutant assembly keeps the same AssemblyVersion as the originally built assembly, preserving assembly binding for the unchanged test assembly.

Changes:

• Read the original built assembly’s version (metadata-only) and inject a matching [assembly: AssemblyVersion(...)] into the mutated compilation when needed.
• Remove any existing assembly-level AssemblyVersion attributes from syntax trees to avoid duplicate-attribute compilation errors before injecting the replacement.
• Add unit tests that verify the mutant version is forced to match the original, and that behavior is a no-op when the original assembly version cannot be read.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/Stryker.Core/Stryker.Core/Compiling/CsharpCompilingProcess.cs	Forces mutated compilation’s AssemblyVersion to match the original assembly to prevent strong-name binding failures under MTP.
src/Stryker.Core/Stryker.Core.UnitTest/Compiling/CSharpCompilingProcessTests.cs	Adds coverage ensuring version alignment occurs when possible and is skipped when the original cannot be read.

[dupdob]

Disclaimer, thanks for the proposal, but it looks like a bolt on patch for a very specific issue that affect the overall behavior and I feel it would be good to have an understanding on the issue at hand to identify the right solution.

Assuming analysis is correct (*), it would be simpler, and faster, to use a regular build instead of a design time one.

Also, I wonder why this problem surfaces just now, as I see not recent change that would explain it.
Finally, I had a cursory look at MinVer and GitVersioning, it appears that only MinVer explicitly excludes DesignTime build. Granted, there is DesignTime build specification besides providing whatever your IDE of choice needs to run; but one could argue this situation warrants opening an issue on MinVer to evaluate changing this logic.

So, we have, at least, 3 possibilities:

1. Stop using designtime builds for analysis or offer an option to do so
2. Open an issue on MinVer to ask support for designtime build. Or an option to do so, but I don't think it makes sense
3. Use this solution: retrieve version numbers from the present assembly to reinject them. Note that we should probaby still mutate the description to track the fact this is a mutated assembly

(*) I consider the analysis is an assumption since fixing this by retrieving version number from the generated assembly would work disregarding the reason of version generation failing.

[slang25]

@dupdob I agree that this isn't the right approach, it's "addressing the symptom". From what you've said, I think the most appropriate approach is no not do a design-time build for the analysis and workspace initialization.

This updated PR now does this, and also addresses the issue. I was keen to keep the MSBuild properties in a way that prevent Buildalyzer from emitting assemblies, but to not do a design-time build, as we are kind of creating a full build pipeline.

Let me know what you think.

[slang25]

I'm also of the opinion that Buildalyzer should be dropped in favour of a stryker out-of-process build host, which I've prototyped here:
master...slang25:stryker-net:slang25/msbuild-workspace-support

[dupdob]

I'm also of the opinion that Buildalyzer should be dropped in favour of a stryker out-of-process build host, which I've prototyped here: master...slang25:stryker-net:slang25/msbuild-workspace-support

But why? what problem will it solve ? we have accumulated years of experience working with Buildalyzer, learning and dealing the the intricacies of various SDKs, why would we want to start over?

[dupdob]

Thinking about it, this clearly appears to be a nice issue: it happens when one uses MinVer AND assembly signing.
We could narrow this down to two approaches:

1. Add an option and log a relevant warning when Stryker detects MinVer + signing combo
2. Add an automatic no DesignTime retry when detecting said combo

Or maybe a combination of both, that would bring out the best:

1. Add an option
2. When we detect a MinVer + signing combo (w/o the option) raise a warning message suggesting to add the option AND retry the build

[slang25]

I'm also of the opinion that Buildalyzer should be dropped in favour of a stryker out-of-process build host, which I've prototyped here: master...slang25:stryker-net:slang25/msbuild-workspace-support

But why? what problem will it solve ? we have accumulated years of experience working with Buildalyzer, learning and dealing the the intricacies of various SDKs, why would we want to start over?

I've faced issues in the past that I've attributed to Buildalyzer (which I've not raised issues for, so not the best citizen here), and at work I process hundreds of projects daily using Buildalyzer, it had a success rate of about 80-something percent, with MSBuildWorkspace that's in the high 90s. Buildalyzer solved a problem when it was first introduced, however since mid 2024 the new rework in MSBuildWorkspace has made it not just fit-for-purpose, but a better option generally. I say that as someone who (and will continue to) contributes to and sponsors Buildalyzer.

I'm not being pushy with this, just saying that there is an alternative implementation (behind an opt-in flag) that we could look at if we ever face odd bugs, rather than having to back-and-forth between Stryker and Buildalyzer.

[slang25]

Closing as the approach here needs considering and discussing.

[rouke-broersma]

I also agree that it's time we look at msbuild workspace again, it's come up multiple times in the past but it's not an easy choice to make. Even adding it as opt in creates a maintenance burden we need to consider. I am willing to make that consideration at this point, we will discuss it soon and decide the path forward.

[dupdob]

The main benefit of switching to msbuild workspace is to get rid of Buildalayzer dependency. Which would probably reduce reaction times as there are far more contributors to Stryker.Net than to Buildalyzer.
OTOH it means increasing the responsibilities for Stryker.

An alternative would be for Buildalyzer to switch to said workspaces 😄 .