Summary
FairShip track fitting and shipVertex extrapolation both use GenFit's
TGeoMaterialInterface. That interface operates on gGeoManager's current
TGeoNavigator, so fitting and vertexing share mutable boundary-navigation
caches and point-stack state.
With a tracking configuration that produces difficult low-momentum fits, an
exception or nonconvergent vertex extrapolation can be followed by a chain of
otherwise unrelated GenFit failures in later events. Starting a fresh process
at those later events succeeds, which shows that the event records themselves
are not sufficient to trigger the failure.
Full log: https://cernbox.cern.ch/files/spaces/eos/user/v/vkholoim/SHIP_smallest_example/shipreco_udate.log
Environment
- GenFit:
2.3.0, build ha28ff26_10
- ROOT:
6.40.02
- Input:
samples/sim_signal_Scalar_mixing_2.5_GeV_10_m_2mu.root
- Geometry:
samples/geo_signal_Scalar_mixing_2.5_GeV_10_m_2mu.root
- Pattern recognition:
AR
The geometry file predates the target.length_fixed metadata field. The
provided reproduction harness adds the legacy value, 158.64 cm, to a temporary
copy so current main can load it. It does not alter the detector geometry or
events.
Reproduction
VERY STRANGE BEHAVIOR. I did not manage to produce a fully reproducible result.
From the FairShip repository root:
Samples: https://cernbox.cern.ch/files/spaces/eos/user/v/vkholoim/SHIP_smallest_example
The deterministic early reproduction used no straw smearing:
pixi run python -u macro/ShipReco.py \
-f samples/sim_signal_Scalar_mixing_2.5_GeV_10_m_2mu.root \
-g samples/geo_signal_Scalar_mixing_2.5_GeV_10_m_2mu.root \
-n 5000 --noStrawSmearing 2>&1 | tee shipreco.log
Typical diagnostics are:
RKTrackRep::Extrap ==> maximum number of iterations exceeded
RKTrackRep::RKutta ==> momentum too low
MaterialEffects::getMomGammaBeta - Energy <= mass
The GenFit messages say fatal flag 0, but this text is stale: the installed
binary executes setFatal(), while Exception::what() retains text formatted
before the flag was changed. KalmanFitterRefTrack catches the exception
internally, marks the fit unsuccessful, and returns, so Python does not receive
an exception from processTrack().
Control tests
The following controls isolate the cross-event state carrier:
| Configuration |
Result |
| Start a fresh process at original event 2424 |
Events reconstruct normally |
| Cut the problematic entries from the original ROOT file and reconstruct only that subset |
The failure chain does not appear |
| Continuous fitting, veto and vertexing |
Failure chain begins |
| Continuous fitting only |
2500 events complete |
| Continuous fitting plus veto, no vertexing |
2500 events complete |
| Continuous fitting plus vertexing, no veto |
Failure chain reproduces at 2425--2427 |
| Separate TGeo navigator for vertexing |
2700 events complete without issues |
This rules out intrinsically corrupt input events, the DAF instance, and veto
linking as the cross-event carrier. Random straw smearing changes the onset but
is not required.
The cut-file result is particularly suspicious. The subset contains the same
MC event records and detector hits as the entries which fail during the long
continuous run, but reconstruction succeeds when those entries are placed in a
new ROOT file and processed by a fresh ShipReco process. Individual low-momentum
or nonconvergent hypotheses may still be reproducible, but they do not trigger
the subsequent chain. Thus the failure depends on reconstruction history before
the problematic range, rather than only on the contents of those events.
For example, original entries 6070--6210 were copied without modifying their
event data and reconstructed separately. The 141-event job completed normally.
Original event 6088 still produced its two invalid low-momentum hypotheses,
showing that those individual bad fits belong to the event, but another
hypothesis fitted successfully and later events did not collapse. Events which
had veto or extrapolation failures in the long run also reconstructed normally
in the cut sample.
Expected behavior
A failed or nonconvergent vertex extrapolation should affect only that vertex
candidate. Track fitting in later events should be independent.
Actual behavior
Vertexing and fitting use the same process-global current TGeo navigator. Under
the triggering tracking configuration, failures become correlated across later
events, while restarting reconstruction at those events—or reconstructing a
ROOT file cut to contain only that range—succeeds. This dependence on preceding
events is unexpected because each event's tracks and fit objects are intended
to be independent.
Proposed fix
Create a dedicated navigator for vertexing and always restore the fitting
navigator:
navigators = ROOT.gGeoManager.GetListOfNavigators()
fit_navigator = ROOT.gGeoManager.GetCurrentNavigator()
fit_navigator_index = navigators.IndexOf(fit_navigator)
vertex_navigator = ROOT.gGeoManager.AddNavigator()
vertex_navigator_index = navigators.IndexOf(vertex_navigator)
try:
ROOT.gGeoManager.SetCurrentNavigator(vertex_navigator_index)
Vertexing.execute()
finally:
ROOT.gGeoManager.SetCurrentNavigator(fit_navigator_index)This keeps the geometry and field shared while isolating mutable navigation
state. Calling TGeoNavigator::ResetAll() was tested and is not a safe
substitute; it caused a ROOT segmentation fault.
Important
I'm not sure that my solution is the best one, but it fixes the problem which was encountered.
Summary
FairShip track fitting and
shipVertexextrapolation both use GenFit'sTGeoMaterialInterface. That interface operates ongGeoManager's currentTGeoNavigator, so fitting and vertexing share mutable boundary-navigationcaches and point-stack state.
With a tracking configuration that produces difficult low-momentum fits, an
exception or nonconvergent vertex extrapolation can be followed by a chain of
otherwise unrelated GenFit failures in later events. Starting a fresh process
at those later events succeeds, which shows that the event records themselves
are not sufficient to trigger the failure.
Full log: https://cernbox.cern.ch/files/spaces/eos/user/v/vkholoim/SHIP_smallest_example/shipreco_udate.log
Environment
2.3.0, buildha28ff26_106.40.02samples/sim_signal_Scalar_mixing_2.5_GeV_10_m_2mu.rootsamples/geo_signal_Scalar_mixing_2.5_GeV_10_m_2mu.rootARThe geometry file predates the
target.length_fixedmetadata field. Theprovided reproduction harness adds the legacy value, 158.64 cm, to a temporary
copy so current
maincan load it. It does not alter the detector geometry orevents.
Reproduction
VERY STRANGE BEHAVIOR. I did not manage to produce a fully reproducible result.
From the FairShip repository root:
Samples: https://cernbox.cern.ch/files/spaces/eos/user/v/vkholoim/SHIP_smallest_example
The deterministic early reproduction used no straw smearing:
Typical diagnostics are:
The GenFit messages say
fatal flag 0, but this text is stale: the installedbinary executes
setFatal(), whileException::what()retains text formattedbefore the flag was changed.
KalmanFitterRefTrackcatches the exceptioninternally, marks the fit unsuccessful, and returns, so Python does not receive
an exception from
processTrack().Control tests
The following controls isolate the cross-event state carrier:
This rules out intrinsically corrupt input events, the DAF instance, and veto
linking as the cross-event carrier. Random straw smearing changes the onset but
is not required.
The cut-file result is particularly suspicious. The subset contains the same
MC event records and detector hits as the entries which fail during the long
continuous run, but reconstruction succeeds when those entries are placed in a
new ROOT file and processed by a fresh ShipReco process. Individual low-momentum
or nonconvergent hypotheses may still be reproducible, but they do not trigger
the subsequent chain. Thus the failure depends on reconstruction history before
the problematic range, rather than only on the contents of those events.
For example, original entries 6070--6210 were copied without modifying their
event data and reconstructed separately. The 141-event job completed normally.
Original event 6088 still produced its two invalid low-momentum hypotheses,
showing that those individual bad fits belong to the event, but another
hypothesis fitted successfully and later events did not collapse. Events which
had veto or extrapolation failures in the long run also reconstructed normally
in the cut sample.
Expected behavior
A failed or nonconvergent vertex extrapolation should affect only that vertex
candidate. Track fitting in later events should be independent.
Actual behavior
Vertexing and fitting use the same process-global current TGeo navigator. Under
the triggering tracking configuration, failures become correlated across later
events, while restarting reconstruction at those events—or reconstructing a
ROOT file cut to contain only that range—succeeds. This dependence on preceding
events is unexpected because each event's tracks and fit objects are intended
to be independent.
Proposed fix
Create a dedicated navigator for vertexing and always restore the fitting
navigator:
This keeps the geometry and field shared while isolating mutable navigation
state. Calling
TGeoNavigator::ResetAll()was tested and is not a safesubstitute; it caused a ROOT segmentation fault.
Important
I'm not sure that my solution is the best one, but it fixes the problem which was encountered.