perf: cached model layers and tzfpy shrink image and update downloads; delete every media variant a detection owns

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M46kR13yG1Gi4zRDUZv6SY

Author: Suncuss

CHANGELOG.md

@@ -2,9 +2,12 @@
 
 ## [Unreleased]
 
+- Fixed detection deletion removing only one copy of a recording's media when duplicates existed under different naming eras (source-ID transition, legacy colon pattern); the surviving copy was orphaned on disk forever. Deletion now removes every variant it finds
 - Fixed detection deletion responses reporting the generated filename instead of the legacy media file actually removed
 - Fixed automatic storage cleanup skipping audio and spectrogram files created during the source-ID filename transition, allowing upgraded stations to reclaim those legacy recordings
 - Improved native updates on Raspberry Pis by stabilizing Docker base layers between releases and removing the backend compiler toolchain from the runtime image, reducing both routine downloads and slow SD-card extraction
+- Improved update download size again: bundled model files (V2.4 model, species table, eBird codes) now live in their own cached image layers instead of re-downloading with every release, shrinking a routine update's backend download from ~55MB to a few megabytes
+- Changed the offline timezone lookup (used when saving a station location) to a much smaller library, trimming ~55MB from the backend image and removing the last need for a compiler during image builds
 - Fixed failed update checks exposing the exact installed commit to anonymous visitors through detailed GitHub error URLs; diagnostics remain available to signed-in owners and in logs
 - Fixed authentication and settings temporary files being readable by other local users while sensitive content was still being written; private files now start owner-only
 - Fixed a password-change race that could leave previously issued audio links valid until the API service restarted

backend/Dockerfile

@@ -3,14 +3,10 @@
 # both the AMD64 and ARM64 images built by CI.
 FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7 AS python-deps
 
-# Build native wheels in a disposable stage. timezonefinder currently builds
-# from source on ARM64, so a compiler is still required during image creation,
-# but none of this toolchain needs to ship to stations.
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
-    --mount=type=cache,target=/var/lib/apt,sharing=locked \
-    apt-get update && apt-get install -y --no-install-recommends \
-    build-essential
-
+# Assemble the venv in a disposable stage. Every dependency ships a prebuilt
+# ARM64 wheel, so no compiler toolchain is installed; if a future dependency
+# starts building from source, pip fails loudly here instead of silently
+# requiring one.
 RUN python -m venv /opt/venv
 ENV PATH="/opt/venv/bin:$PATH"
 
@@ -24,14 +20,23 @@ RUN --mount=type=cache,target=/root/.cache/pip \
     pip install --extra-index-url https://www.piwheels.org/simple \
     -r requirements.txt
 
+# Prune bundled models out of the per-release source layer. They ship in the
+# dedicated COPY layers below instead, so a routine source change pushes a few
+# megabytes, not the ~55MB of model blobs. The prune must be explicit: BuildKit
+# only dedupes files identical to lower layers on some builders (CI yes,
+# local dockerd no), so leaving them in COPY . . re-ships them from local builds.
+FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7 AS source
+COPY . /src
+RUN rm -rf /src/model_service/models
+
 # Use the same pinned multi-platform base for the shipped runtime image.
 FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7
 
 # Build arguments for user/group IDs (defaults for fallback)
 ARG UID=1000
 ARG GID=1000
 
-# Install runtime libraries only; the compiler toolchain stays in python-deps.
+# Install runtime libraries only.
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     --mount=type=cache,target=/var/lib/apt,sharing=locked \
     apt-get update && apt-get install -y --no-install-recommends \
@@ -43,8 +48,8 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
 
 WORKDIR /app
 
-# Copy only the finished environment; build-essential and intermediate wheel
-# caches remain in the discarded python-deps stage.
+# Copy only the finished environment; intermediate wheel caches remain in the
+# discarded python-deps stage.
 COPY --from=python-deps /opt/venv /opt/venv
 ENV PATH="/opt/venv/bin:$PATH"
 
@@ -70,7 +75,15 @@ RUN python -m model_service.birdnet_v3_assets
 # Compose bind-mounts the checkout over /app, so runtime loaders repeat the
 # authoritative checks against the bytes each service actually reads.
 
-COPY --chown=${UID}:${GID} . .
+# Remaining bundled model data in its own rarely-invalidated layers, after the
+# gates so a species-table refresh does not re-run the memory-intensive checks.
+COPY --chown=${UID}:${GID} model_service/models/v2.4 ./model_service/models/v2.4
+COPY --chown=${UID}:${GID} model_service/models/species_table.csv \
+     model_service/models/ebird_codes.json \
+     ./model_service/models/
+
+# Source code last: the only layer that changes on a routine release.
+COPY --from=source --chown=${UID}:${GID} /src ./
 
 # Copy entrypoint script
 COPY docker-entrypoint.sh /usr/local/bin/

backend/Dockerfile.test

@@ -2,7 +2,6 @@
 FROM python:3.11-slim
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
-    build-essential \
     ffmpeg \
     && rm -rf /var/lib/apt/lists/*
 

backend/core/routes/settings.py

@@ -6,7 +6,6 @@
 core.settings_store. Registered on the shared ``api`` blueprint at import.
 """
 import re
-import threading
 
 from flask import jsonify, request
 
@@ -42,32 +41,21 @@
 logger = get_logger(__name__)
 
 
-# Singleton TimezoneFinder (loads ~40MB shape data on first use). The import
-# itself is deferred into _get_timezone_finder(): it drags numpy/cffi/h3 into
-# the worker, and the only caller is the settings handler resolving a newly
-# saved location — a station that never edits its location never pays for it.
-_timezone_finder = None
-_tz_finder_lock = threading.Lock()  # hub-only: timezone lookup runs in settings-route greenlets, never the DB lane
-
-
-def _get_timezone_finder():
-    """Lazy-import and lazy-load TimezoneFinder (loads ~40MB shape data)."""
-    global _timezone_finder
-    with _tz_finder_lock:
-        if _timezone_finder is None:
-            from timezonefinder import TimezoneFinder
-            _timezone_finder = TimezoneFinder()
-        return _timezone_finder
-
-
 def get_timezone_for_location(lat: float, lon: float) -> str | None:
-    """Offline timezone lookup. Returns IANA timezone or None on failure."""
+    """Offline timezone lookup. Returns IANA timezone or None on failure.
+
+    tzfpy's simplified polygons leave rare hairline gaps at zone borders
+    (and at exactly ±180° longitude) where lookup returns nothing; retrying
+    a few km to each side recovers a point sitting in such a gap. The import
+    stays deferred so only the location-save path ever pays for it.
+    """
     try:
-        tf = _get_timezone_finder()
-        timezone = tf.timezone_at(lat=lat, lng=lon)
-        if timezone:
-            logger.info(f"Resolved timezone: {timezone}")
-            return timezone
+        from tzfpy import get_tz
+        for dlat, dlon in ((0, 0), (0.1, 0), (-0.1, 0), (0, 0.1), (0, -0.1)):
+            timezone = get_tz(lon + dlon, lat + dlat)  # tzfpy takes lng first
+            if timezone:
+                logger.info(f"Resolved timezone: {timezone}")
+                return timezone
         logger.warning(f"No timezone found for ({lat}, {lon})")
         return None
     except Exception as e:

backend/core/storage_manager.py

@@ -75,29 +75,6 @@ def get_disk_usage(path=None):
     }
 
 
-def _resolve_path_with_legacy_fallback(filename, directory):
-    """Resolve file path, falling back to legacy colon-pattern if needed.
-
-    Args:
-        filename: Filename (dash-pattern)
-        directory: Directory containing the file
-
-    Returns:
-        Full path to the file (dash or legacy pattern, whichever exists)
-    """
-    path = os.path.join(directory, filename)
-    if os.path.exists(path):
-        return path
-
-    legacy_filename = get_legacy_filename(filename)
-    if legacy_filename:
-        legacy_path = os.path.join(directory, legacy_filename)
-        if os.path.exists(legacy_path):
-            return legacy_path
-
-    return path  # Return original path even if it doesn't exist
-
-
 def _detection_filename_candidates(detection):
     """Build ordered dash-pattern filename candidates for a detection.
 
@@ -137,38 +114,26 @@ def _detection_filename_candidates(detection):
     )
 
 
-def _resolve_detection_path(filename_candidates, key, directory):
-    """Return the first existing path for one detection media type."""
-    first_path = None
-    for filenames in filename_candidates:
-        path = _resolve_path_with_legacy_fallback(filenames[key], directory)
-        if first_path is None:
-            first_path = path
-        if os.path.exists(path):
-            return path
-    return first_path
-
-
-def get_detection_files(detection):
-    """Get full file paths for a detection record.
+def _existing_detection_paths(detection):
+    """Every on-disk path holding the detection's media, audio paths first.
 
-    Supports lazy migration: if new dash-pattern files don't exist,
-    falls back to checking for old colon-pattern files.
-
-    Args:
-        detection: dict with common_name, confidence, timestamp
-
-    Returns:
-        dict with audio_path and spectrogram_path
+    Checks each filename candidate in both dash and legacy colon patterns
+    and keeps all hits, not just the first: transition-era rows can own
+    duplicate copies under different naming eras, and a copy that survived
+    deletion would be orphaned forever (cleanup only ever revisits DB rows).
     """
-    filename_candidates = _detection_filename_candidates(detection)
-
-    return {
-        'audio_path': _resolve_detection_path(
-            filename_candidates, 'audio_filename', EXTRACTED_AUDIO_DIR),
-        'spectrogram_path': _resolve_detection_path(
-            filename_candidates, 'spectrogram_filename', SPECTROGRAM_DIR),
-    }
+    candidates = _detection_filename_candidates(detection)
+    paths = []
+    for key, directory in (('audio_filename', EXTRACTED_AUDIO_DIR),
+                           ('spectrogram_filename', SPECTROGRAM_DIR)):
+        for filenames in candidates:
+            for name in (filenames[key], get_legacy_filename(filenames[key])):
+                if name is None:
+                    continue
+                path = os.path.join(directory, name)
+                if os.path.exists(path):
+                    paths.append(path)
+    return paths
 
 
 def _disk_filename_sets():
@@ -205,51 +170,26 @@ def _has_files_on_disk(detection, audio_names, spectrogram_names):
 
 
 def delete_detection_files(detection):
-    """Delete audio and spectrogram files for a detection.
+    """Delete every audio and spectrogram file a detection owns.
 
     Args:
         detection: dict with common_name, confidence, timestamp
 
     Returns:
-        dict with deleted_audio, deleted_spectrogram, deleted_filenames,
-        and bytes_freed
+        dict with deleted_filenames (the names actually removed, audio
+        first) and bytes_freed
     """
-    paths = get_detection_files(detection)
-    result = {
-        'deleted_audio': False,
-        'deleted_spectrogram': False,
-        'deleted_filenames': [],
-        'bytes_freed': 0
-    }
-
-    # Delete audio file
-    audio_path = paths['audio_path']
-    if audio_path and os.path.exists(audio_path):
-        try:
-            size = os.path.getsize(audio_path)
-            os.remove(audio_path)
-            result['deleted_audio'] = True
-            result['deleted_filenames'].append(os.path.basename(audio_path))
-            result['bytes_freed'] += size
-        except OSError as e:
-            logger.warning("Failed to delete audio file", extra={
-                'path': audio_path,
-                'error': str(e)
-            })
+    result = {'deleted_filenames': [], 'bytes_freed': 0}
 
-    # Delete spectrogram file
-    spectrogram_path = paths['spectrogram_path']
-    if spectrogram_path and os.path.exists(spectrogram_path):
+    for path in _existing_detection_paths(detection):
         try:
-            size = os.path.getsize(spectrogram_path)
-            os.remove(spectrogram_path)
-            result['deleted_spectrogram'] = True
-            result['deleted_filenames'].append(
-                os.path.basename(spectrogram_path))
+            size = os.path.getsize(path)
+            os.remove(path)
+            result['deleted_filenames'].append(os.path.basename(path))
             result['bytes_freed'] += size
         except OSError as e:
-            logger.warning("Failed to delete spectrogram file", extra={
-                'path': spectrogram_path,
+            logger.warning("Failed to delete detection file", extra={
+                'path': path,
                 'error': str(e)
             })
 
@@ -422,7 +362,7 @@ def delete_pass(start_cursor, stop_cursor=None):
                 continue
 
             delete_result = delete_detection_files(detection)
-            if delete_result['deleted_audio'] or delete_result['deleted_spectrogram']:
+            if delete_result['deleted_filenames']:
                 result['files_deleted'] += 1
                 result['bytes_freed'] += delete_result['bytes_freed']
         return cursor

backend/requirements.txt

@@ -28,8 +28,8 @@ requests>=2.32.4
 apprise>=1.8.0
 paho-mqtt>=2.0.0
 
-# Timezone utilities
-timezonefinder>=6.2.0
+# Timezone utilities (Rust wheel, no compiler needed; ~50MB smaller than timezonefinder)
+tzfpy>=1.3.2
 
 # Visualization (spectrogram rendering)
 Pillow>=10.4.0

backend/tests/api/conftest.py

@@ -207,6 +207,23 @@ def media_dirs():
             yield audio_dir, spectrogram_dir
 
 
+@pytest.fixture
+def storage_media_dirs():
+    """Patch storage_manager's audio/spectrogram directories to empty temp dirs.
+
+    Detection deletion resolves and removes media files through
+    core.storage_manager; tests create files here to control what gets
+    deleted and reported."""
+    with tempfile.TemporaryDirectory() as tmp:
+        audio_dir = os.path.join(tmp, 'extracted_songs')
+        spectrogram_dir = os.path.join(tmp, 'spectrograms')
+        os.makedirs(audio_dir)
+        os.makedirs(spectrogram_dir)
+        with patch('core.storage_manager.EXTRACTED_AUDIO_DIR', audio_dir), \
+             patch('core.storage_manager.SPECTROGRAM_DIR', spectrogram_dir):
+            yield audio_dir, spectrogram_dir
+
+
 @pytest.fixture
 def create_recording_files(media_dirs):
     """Factory that creates on-disk audio+spectrogram files for a species'