Skip to content

Numba backend - #45

Open
ppp-one wants to merge 7 commits into
lgrcia:mainfrom
ppp-one:numba-backend
Open

ppp-one wants to merge 7 commits into
lgrcia:mainfrom
ppp-one:numba-backend

Conversation

@ppp-one

@ppp-one ppp-one commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Add optional numba backend to find_transform

Summary

This PR makes plate solving much faster when many asterism pairs must be tested. A new backend argument on find_transform and compute_wcs selects a numba-compiled loop. numba is optional. Without it, twirl uses the current NumPy loop and gives the same results.

Why

Testing one candidate asterism pair costs about 25 µs. Almost all of that is NumPy call overhead on tiny arrays (lstsq on 3–4 points, a 20×40 distance array). The number of pairs grows as about n⁸ for quads and n⁶ for triangles. When min_match is not reached, find_transform tests every pair. With 20 detections and 40 catalog stars, one solve then takes 10–20 s.

On 357 real SPECULOOS images, run through the astra pointing pipeline, 114 images (32%) took more than 1 s with the current code, and the slowest took 19 s.

Changes

  • find_transform(..., backend="auto") and compute_wcs(..., backend="auto"): "numba", "numpy", or "auto" (numba when it is installed). The default is "auto", so existing calls need no change.
  • twirl/_numba.py: the compiled pair loop (least-squares affine fit, projection, cross-match count, early stop). numba is imported only when this backend is used, so import twirl does not get slower.
  • Candidate pairs are built as NumPy arrays, not as a Python list. With quads the list could hold millions of pairs.
  • quads.reorder uses np.take_along_axis, not a Python loop. The output is identical, and this is about 2× faster on 40 stars.
  • pyproject.toml: new optional extra twirl[numba], and numba is added to the dev group so CI tests both backends. uv.lock is updated (no existing pins changed).
  • README: a short note on pip install "twirl[numba]".
  • benchmarks/: plate_solve.py times compute_wcs on synthetic fields and checks each solution against the true WCS. complexity.py plots how the cost grows with the number of stars. Both have a --backend option.

Results

Synthetic benchmark (benchmarks/plate_solve.py, median per case):

case before after speed-up
crowded, 20 stars, quads 627 ms 30 ms 21×
crowded, 20 stars, triangles 391 ms 12 ms 32×
30 stars, quads 282 ms 160 ms 1.8×
10–15 stars 6–15 ms 6–9 ms 1.0–1.7×

When early stop happens, the fixed cost of compute_wcs (about 5 ms) dominates, so the gain is small there.

357 SPECULOOS images, 20 detections and 40 Gaia stars each:

median slowest
quads (default), before → after 558 ms → 218 ms 19.3 s → 0.61 s
triangles (asterism=3, min_match=0.5), before → after 95 ms → 23 ms 4.0 s → 0.26 s

The solutions are identical to the current code on all 357 images, for both asterisms.

Measured on an Apple M2 Max with 8 parallel workers.

Compatibility

  • No breaking change. The new argument has a default, and results are the same with or without numba.
  • numba stays optional, for these reasons: it adds about 150 MB (llvmlite), it caps NumPy (numba 0.68 requires numpy<2.6), and it usually supports a new Python version some months after its release.
  • The first numba call in a process compiles the loop (about 0.6 s). numba then caches it on disk if the folder is writable.

Testing

  • tests/test_backends.py (50 cases):
    • reorder matches the old loop exactly
    • both backends return the same transform on simulated fields (n = 6, 12, 25; triangles and quads; with and without noise; with and without early stop)
    • compute_wcs gives the same WCS with both backends
    • an unknown backend raises ValueError
  • The full suite passes: 83 tests.
  • Checked end to end on the 357 SPECULOOS images against the unchanged code, as above.

Notes

  • I did not touch the ruff warnings that were already on main.

ppp-one and others added 3 commits October 11, 2026 18:13
benchmarks/plate_solve.py times compute_wcs on synthetic star fields made
from a known TAN WCS (missing and spurious detections, position noise) and
checks each solution against the true WCS. Results can be saved and compared
between runs.

benchmarks/complexity.py measures how hashes, candidate pairs, solve time and
cost per tested pair grow with the number of stars, and plots them on log-log
axes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Testing candidate asterism pairs cost ~25 us per pair, almost all NumPy call
overhead on tiny arrays. The number of pairs grows as ~n^8 (quads) and ~n^6
(triangles), so when min_match is not reached every pair is tested and
solves can take 10-20 s with 20 detections and 40 catalog stars.

- find_transform and compute_wcs get a `backend` argument: "numba", "numpy"
  or "auto" (default: numba when installed). The numba loop costs ~0.5 us
  per pair. numba is an optional dependency (`twirl[numba]`) and is only
  imported when used.
- Candidate pairs are built as NumPy arrays instead of a Python list.
- quads.reorder uses take_along_axis instead of a Python loop (same output).

Both backends return the same transform: tests compare them on simulated
fields, and on 357 SPECULOOS images run through astra the solutions are
identical to the previous code, while the slowest solve drops from 19.3 s to
0.6 s (quads) and from 4.0 s to 0.26 s (triangles).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ppp-one
ppp-one requested a review from dgegen October 11, 2026 18:33
dgegen and others added 4 commits October 25, 2026 15:00
backend="auto" checked only that numba was installed, so a numba that
cannot be imported (e.g. with an unsupported NumPy version) made every
call fail. "auto" now tries the import and falls back to the NumPy loop;
an explicit backend="numba" still raises.

Also run the backend tests that do not need numba without it, add a test
for the fallback, mention the first-call compile in the docstring, and
make the README speed-up claim match the benchmarks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add a Python 3.13 / astropy 7.0 job that uninstalls numba, so the NumPy
backend most users get is tested.

Run pytest with `uv run --no-sync`: `uv run` re-synced the environment
to uv.lock, which undid the astropy version pin (CI tested astropy 7.2.2
in every job) and would reinstall numba.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
astropy 6.1 and 7.0 use np.in1d, which NumPy 2.4 removed, so they fail
to import with the NumPy locked for Python >= 3.11. This only shows now
that the astropy pin is no longer undone by `uv run`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants