SE2 DXC build - #7
Merged
Merged
Conversation
Space Engineers 2 compiles every shader at runtime through Vortice.Dxc, which P/Invokes DxcCreateInstance from dxcompiler.dll. Until now the Linux port used hand-built libraries pinned inside the recompiled-game workspace. Move them here so they ship in se2-dependencies.tar.gz like every other SE2 native dependency. Two files are staged, not three: * libdxcompiler.so — DirectX Shader Compiler at tag v1.9.2607, commit 0d3ee6b5, built from unpatched upstream sources. * libSE2DxcCompiler.so — the ABI shim, built in the same step from Sources/dxc-bridge/DxcCompilerBridge.cpp against the headers of the DXC tree just compiled. libdxil.so is deliberately dropped. Microsoft open-sourced the DXIL validator hash, so ComputeHashRetail is compiled into libdxcompiler.so directly; compiling a shader with libdxil.so beside it and again with it absent produces byte-identical containers with the same non-zero hash, and LD_DEBUG=libs shows it is never dlopened either way. It also has no buildable source — it exists only as a prebuilt release artefact — so leaving it out is what makes a from-source payload possible. Because the compiler is now compiled rather than unpacked from Microsoft's release download, no binary distribution agreement applies to the shipped payload any more. The LICENSE-MS.txt the recompiled-game workspace carries for exactly that reason is no longer needed. The archive gains Licenses/se2/DXC-LICENSE.txt (the NCSA/LLVM Release License) and DXC-README.txt (pin and provenance) instead. The shim stays. DXC's Linux WCHAR is the 4-byte platform wchar_t while Vortice marshals the Windows 2-byte wire type, and building DXC with -fshort-wchar to close the gap is not an option: libdxcompiler.so imports eight wide-char functions from glibc (wcslen, wcscmp, wcsncmp, wcsncpy, wmemcmp, wmemcpy, mbstowcs, wcstombs) that a 2-byte wchar_t build would silently mismatch. Its vtable layout tracks the DXC version, which is why it is compiled next to the compiler rather than left in a consuming repo where the drift would go unnoticed. One behavioural fix while moving it: the backend fallback was a hardcoded /usr/lib/libdxcompiler.so, which picked up whatever the host had installed — in practice a v1.10 Shader Model 6.10 preview, which aborts inside LLVM while compiling Render12 shaders. It now dlopens the bare SONAME and resolves it through DT_RUNPATH=$ORIGIN to the sibling staged beside it. SE2_DXCOMPILER_BACKEND still overrides the path. Notes on the pin and the build: * v1.9.2607 is the newest stable release. Every v1.10.x release is flagged prerelease upstream. Its only breaking HLSL change over the previously shipped v1.9.2602.24 is that `volatile` is no longer accepted, and none of SE2's 660 shader sources use it. * The SPIR-V backend cannot be switched off on Linux: DXC's root CMakeLists.txt does an unconditional `if(NOT WIN32) set(ENABLE_SPIRV_CODEGEN ON) endif()` that shadows both the cache and the command line. external/SPIRV-Headers and external/SPIRV-Tools are therefore initialised alongside external/DirectX-Headers; only external/googletest stays uncloned. SPIRV-Tools links in statically, so DXC-BUNDLED-LICENSES.txt carries its Apache-2.0 text plus the SPIRV-Headers and DirectX-Headers ones. * Flag order in the cmake invocation is load-bearing: the -C cache file's set()s are non-FORCE, so a -D wins only when it precedes the -C. * The build tree (~270 MB) is deleted after staging, keeping the runner footprint around 550 MB. DXC_KEEP_BUILD_TREE=1 keeps it locally. Verified locally: both staged libraries resolve only glibc, libstdc++, libgcc_s and libm and carry DT_RUNPATH=$ORIGIN; a trivial compute shader compiled in a directory containing no libdxil.so yields a signed container with DXIL hash 19a5d3029c0cd4a5e1bf3d242c351712; the same shader compiled through libSE2DxcCompiler.so from a 2-byte-WCHAR caller produces a byte-identical container, with LD_DEBUG confirming the backend resolved out of $ORIGIN; ./build.sh --only=dxc stages cleanly and the expected-artefact assertion passes. CI wall-clock roughly doubles — DXC alone is 30-60 minutes.
A cold run is about 31 minutes now, roughly 19 of them the DirectX Shader Compiler, and every push paid it in full even when nothing that feeds a given library had changed. Cache each dependency's staged output and restore it before build.sh runs. The mechanism is the stamp files the scripts already keep. Restoring build/<dep>.stamp next to the staged .so files makes the script's existing "outputs present and stamp matches" check fire, so the step exits without a source tree at all. A fully warm run builds nothing and packages all three archives in under five seconds. The cache key IS the stamp: every Scripts/build_*.sh gains --print-stamp, which prints its stamp and exits without touching the network or the build tree, and the workflow keys on a hash of that plus the image label and the runner's gcc/glibc versions. Keying off the scripts' own value rather than hashFiles() means the two cannot drift apart, and that editing a comment does not cost a rebuild. The toolchain is in the key because the staged libraries link against the runner's gcc and glibc and ubuntu-24.04 is patched in place. The scripts re-derive and compare their stamp on every run regardless, so the CI key is only a performance hint: too loose costs a rebuild, it cannot ship a stale binary. Two supporting changes: * FFmpeg had no stamp and no early exit — it relied on an incremental make inside a 203 MB build tree, which is not something a cache of shipped files can stand in for. It now has both. Its stamp folds in a hash of the configure flags, which meant hoisting the flag array (minus --prefix, a build-tree path that must not affect the key) above the rationale comment so --print-stamp can hash it without downloading anything. The resulting CONFIGURE_FLAGS is byte-for-byte what it was. * SDL3 caches its build prefix rather than the shipped file, because build_sdl3.sh deliberately re-stages libSDL3.so into both game trees on every run. Restore and save are separate workflow steps rather than the combined cache action: the combined one only writes on a fully successful job, so a failure in a late dependency — or in the release step after a 30-minute build — would discard everything the run had already built. Note for future dependency bumps: GitHub scopes caches by branch. A run reads its own branch, the default branch, and for a PR its base branch, but a cache written by a PR lives on that PR's merge ref and cannot be read by main or by any other PR. So a bump is built twice, once on the PR and once on main after the merge; everything the PR did not touch restores from main's caches either way. Also corrects the DXC build-time figures throughout: 30-60 minutes came from the original task description, but the first CI run measured the whole cold pipeline at 30m45s, which puts DXC at roughly 19 minutes on a 4-vCPU runner. Verified by replaying the workflow's CACHE_PATHS_* lists into an empty build directory and running the pipeline against it: all seven steps report a cache hit, the 45 staged files are byte-identical to a locally built tree, and the steam archive is byte-identical to the last published release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Space Engineers 2 compiles every shader at runtime through Vortice.Dxc, which P/Invokes DxcCreateInstance from dxcompiler.dll. Until now the Linux port used hand-built libraries pinned inside the recompiled-game workspace. Move them here so they ship in se2-dependencies.tar.gz like every other SE2 native dependency.
Two files are staged, not three:
libdxil.so is deliberately dropped. Microsoft open-sourced the DXIL validator hash, so ComputeHashRetail is compiled into libdxcompiler.so directly; compiling a shader with libdxil.so beside it and again with it absent produces byte-identical containers with the same non-zero hash, and LD_DEBUG=libs shows it is never dlopened either way. It also has no buildable source — it exists only as a prebuilt release artefact — so leaving it out is what makes a from-source payload possible.
Because the compiler is now compiled rather than unpacked from Microsoft's release download, no binary distribution agreement applies to the shipped payload any more. The LICENSE-MS.txt the recompiled-game workspace carries for exactly that reason is no longer needed. The archive gains Licenses/se2/DXC-LICENSE.txt (the NCSA/LLVM Release License) and DXC-README.txt (pin and provenance) instead.
The shim stays. DXC's Linux WCHAR is the 4-byte platform wchar_t while Vortice marshals the Windows 2-byte wire type, and building DXC with -fshort-wchar to close the gap is not an option: libdxcompiler.so imports eight wide-char functions from glibc (wcslen, wcscmp, wcsncmp, wcsncpy, wmemcmp, wmemcpy, mbstowcs, wcstombs) that a 2-byte wchar_t build would silently mismatch. Its vtable layout tracks the DXC version, which is why it is compiled next to the compiler rather than left in a consuming repo where the drift would go unnoticed.
One behavioural fix while moving it: the backend fallback was a hardcoded /usr/lib/libdxcompiler.so, which picked up whatever the host had installed — in practice a v1.10 Shader Model 6.10 preview, which aborts inside LLVM while compiling Render12 shaders. It now dlopens the bare SONAME and resolves it through DT_RUNPATH=$ORIGIN to the sibling staged beside it. SE2_DXCOMPILER_BACKEND still overrides the path.
Notes on the pin and the build:
volatileis no longer accepted, and none of SE2's 660 shader sources use it.if(NOT WIN32) set(ENABLE_SPIRV_CODEGEN ON) endif()that shadows both the cache and the command line. external/SPIRV-Headers and external/SPIRV-Tools are therefore initialised alongside external/DirectX-Headers; only external/googletest stays uncloned. SPIRV-Tools links in statically, so DXC-BUNDLED-LICENSES.txt carries its Apache-2.0 text plus the SPIRV-Headers and DirectX-Headers ones.Verified locally: both staged libraries resolve only glibc, libstdc++, libgcc_s and libm and carry DT_RUNPATH=$ORIGIN; a trivial compute shader compiled in a directory containing no libdxil.so yields a signed container with DXIL hash 19a5d3029c0cd4a5e1bf3d242c351712; the same shader compiled through libSE2DxcCompiler.so from a 2-byte-WCHAR caller produces a byte-identical container, with LD_DEBUG confirming the backend resolved out of $ORIGIN; ./build.sh --only=dxc stages cleanly and the expected-artefact assertion passes.
CI wall-clock roughly doubles — DXC alone is 30-60 minutes.