Cache vertex orientations in a signed char, and test Linux on arm64

[danielepanozzo]

Linux on arm64 silently produces a BSP complex with the input surface never embedded. One-line cause, one-line fix, plus the CI job that would have caught it.

The bug

BSP.h cached orient3D results — -1, 0, +1 — in a std::vector<char>, and count_vrt_orBin() reads them back with:

else if (vrts_orBin[inds[i]] == -1) (*neg)++;

char is signed on x86-64 and under Apple's arm64 ABI, but unsigned under AAPCS64 (Linux/arm64). There -1 stores as 255, the == -1 test never matches, and every vertex below a constraint plane is counted as being above it.

splitCell() then sees vrtsUNDER == 0 for every constraint, takes the NO SPLIT branch each time, and returns having split nothing.

Measured

Native aarch64 Ubuntu 24.04, GCC 13, on a 9-vertex / 8-triangle input:

	initial cells	final cells	tets
macOS/arm64	12162	13884	20753
Linux/arm64 (before)	12162	12162	12162
Linux/arm64 (after)	12162	13884	20753

Instrumenting the orientation counts made it unambiguous — under was 0 on every constraint, never once nonzero:

before:  [ORIENT] over=1 under=0 on=1     (x12, always under=0)
macOS:   [ORIENT] over=1 under=2 on=1     (healthy)

The constraint→tet map was fine throughout (tets_with_constraints=1118, total_entries=1837), so cells did receive their constraints — they were all rejected one step later.

Test results, linux/arm64

	integration suite
without fix	22 / 28
with fix	28 / 28

The six failures include model outputs are byte-stable across platforms — precisely the invariant this breaks. The suite already detects the bug; nothing had ever run it on arm64 Linux.

Why it survived

The matrix tested Linux only on x86-64 and arm64 only on macOS, so no job ever compiled the combination. This adds ubuntu-24.04-arm (Debug + Release).

Scope

The other std::vector<char> members (tri_is_flat, used, undecided_sign) only ever hold 0/1 and are tested for truthiness — unaffected, left alone.

Found while investigating a wildmeshing-toolkit tetwild test failing only on linux/arm64; that repo gets the same CI addition and a pin bump once this lands.

🤖 Generated with Claude Code