Stabilization: reconciled runs notify, pip tasks hardened, QA-sweep fixes

Reliability (roadmap 1.1 + 1.2, meant for 1.16 but never shipped):
- Run.reconcile_stale now sends the script's normal failure notification for
  every run it flips to FAILED (running-past-deadline and lost-from-queue),
  best-effort per run so a broken channel can't stall the heartbeat.
- Package operations: pip runs through EnvironmentService._run_pip (own
  process group + kill_process_tree on timeout, stdin=DEVNULL, PIP_NO_INPUT);
  each django-q task gets pip's cap + 60s so pip always dies first; the
  worker retry floor accounts for the longest pip task; the heartbeat
  reconciles stale PackageOperations; STALE_AFTER 15 -> 30 min.

July QA sweep (docs/TEST_PLAN_e2e.md section 100):
- F2: a plugins/<slug> folder with no Plugin row is moved aside
  (<slug>.orphaned-<ts>) instead of blocking every upload of that slug; a
  dev-mode folder is refused with a clear message.
- F3: partial venvs are removed when `python -m venv` fails, and an
  unreferenced leftover folder is reclaimed on the next create.
- F4: the interpreter picker probes `import ensurepip, venv` and hides
  interpreters that can't build a venv.
- F5: pyrunner_db.connect() no longer claims psycopg ships with the runtime;
  it points at Environments -> Packages.
- F6: plugin.json min_pyrunner is enforced at upload and activation.
- F8: saving a changed admin URL slug warns that it applies after a restart.
- F1: run-local-postgres.ps1 -Fresh removes the three named volumes
  explicitly after `down -v` and fails loudly if any survive.
- F9: not reproducible - ScheduleFirstTickTests drives django-q's real
  scheduler loop and locks "interval fires once, clock modes don't fire on
  creation".

Tests: core/test_stabilization_v1_17.py (27 tests); full core suite 1644 OK.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Author: hassancs91

CHANGELOG.md

@@ -41,6 +41,50 @@ begins tracking at the current release; earlier history is in the git log.
   files under the test runner's `DEBUG=False`), and an unpinned Ruff picked
   up 0.16's much larger default rule set. Both pinned down; no product code
   changed.
+- **A worker dying mid-run now triggers the run's failure notification** —
+  runs the stale-run reconciler flipped to *failed* never went through the
+  normal finalize path, so the one failure the platform most needs to report
+  (your worker was killed or restarted while your script ran) was the one it
+  stayed silent about. Reconciled runs, including runs lost from the queue,
+  now send the script's configured notifications.
+- **Bulk requirements install no longer gets stuck** — the pip subprocess and
+  the worker task that hosts it both had a 600 s limit, so on a slow install
+  the worker killed the task first, pip kept running orphaned in the venv,
+  and the queue re-delivered the same install into it. Every package task
+  now gets its own timeout above pip's cap, pip runs in its own process group
+  (a timeout kills its build helpers too), with stdin closed and
+  `PIP_NO_INPUT` set so it can never wait on a prompt; the worker retry
+  window accounts for the longest pip task, and stuck package operations
+  are reconciled by the worker heartbeat instead of only when someone opens
+  the packages page.
+- **A failed environment create no longer poisons its name** — a `python -m
+  venv` that failed halfway (typically Debian's `/usr/bin/python3` without
+  `python3-venv`) left a folder behind that blocked the name forever with
+  "Path already exists" and no way to clean up. Partial venvs are removed on
+  failure, an existing folder no environment references is reclaimed, and the
+  interpreter picker only lists Pythons that can actually bootstrap pip.
+- **A plugin's `min_pyrunner` is now enforced** — the manifest field was
+  documented but never checked, so a plugin built for a newer PyRunner
+  uploaded and activated on an older one and failed later in confusing ways.
+  Upload and activation now refuse it with the required version.
+- **A stray plugin folder no longer blocks uploads of that slug** — a folder
+  under the plugins directory with no plugin record (a delete that crashed
+  halfway, or files shipped inside an image) made every upload of the same
+  slug fail with "already exists" and nothing in the UI could remove it. It
+  is now moved aside (never deleted) and the upload proceeds; a folder loaded
+  in dev mode is left alone with a clear message.
+- **`pyrunner_db.connect()` tells the truth about `psycopg`** — the error
+  claimed the driver "ships with PyRunner's runtime"; it doesn't ship inside
+  script environments. The message now points at Environments → Packages.
+- **Changing the Django admin URL slug says it needs a restart** — routes are
+  built at startup, so the old slug stayed live and the new one 404'd until
+  the next restart, with nothing telling you so. Saving a changed slug now
+  shows exactly that.
+- **`run-local-postgres.ps1 -Fresh` really wipes** — `docker compose down -v`
+  only removes volumes it can attribute to its own project, so a stack that
+  had been brought up under another project name kept its month-old Postgres
+  data and the "fresh" boot silently reused it. The launcher now removes the
+  three named volumes explicitly and refuses to continue if any survive.
 
 ## [1.16.0] — July 29, 2026
 

core/models/package.py

@@ -82,7 +82,10 @@ class Status(models.TextChoices):
     # An operation should never legitimately stay pending/running this long.
     # Past this, it almost certainly belongs to a worker that crashed/restarted
     # mid-task or a task stuck in a django-q2 re-queue loop.
-    STALE_AFTER = datetime.timedelta(minutes=15)
+    # Must outlast the longest package task (bulk install: 600s pip cap + 60s
+    # worker grace) plus realistic queue wait, or a slow-but-alive install is
+    # mislabelled failed while pip is still working.
+    STALE_AFTER = datetime.timedelta(minutes=30)
 
     class Meta:
         db_table = "package_operations"

core/models/run.py

@@ -3,6 +3,7 @@
 """
 
 import datetime
+import logging
 import uuid
 
 from django.conf import settings
@@ -12,6 +13,9 @@
 from .workspace import WorkspaceScopedManager
 
 
+logger = logging.getLogger(__name__)
+
+
 class Run(models.Model):
     """
     Represents a single execution of a script.
@@ -192,6 +196,7 @@ def reconcile_stale(cls) -> int:
 
         now = timezone.now()
         reconciled = 0
+        notify_pks: list = []
 
         # RUNNING: dead once well past started_at + the script's own timeout +
         # grace. The deadline is per-run, and the candidate set is tiny by
@@ -220,7 +225,7 @@ def reconcile_stale(cls) -> int:
                 f"{run.script.timeout_seconds}s timeout. A detached script "
                 "process may have kept running after the worker died.]"
             )
-            reconciled += cls.objects.filter(
+            flipped = cls.objects.filter(
                 pk=run.pk, status=cls.Status.RUNNING
             ).update(
                 status=cls.Status.FAILED,
@@ -229,14 +234,22 @@ def reconcile_stale(cls) -> int:
                 ended_at=now,
                 pid=None,
             )
+            reconciled += flipped
+            if flipped:
+                notify_pks.append(run.pk)
 
         # PENDING: only reconciled while workers are alive — with the cluster
         # down, "queued" is still the truth and these runs will execute when it
         # returns.
         if GlobalSettings.get_settings().worker_is_alive():
-            reconciled += cls.objects.filter(
-                status=cls.Status.PENDING,
-                created_at__lt=now - cls.RECONCILE_PENDING_AFTER,
+            stale_pending = list(
+                cls.objects.filter(
+                    status=cls.Status.PENDING,
+                    created_at__lt=now - cls.RECONCILE_PENDING_AFTER,
+                ).values_list("pk", flat=True)
+            )
+            flipped = cls.objects.filter(
+                pk__in=stale_pending, status=cls.Status.PENDING
             ).update(
                 status=cls.Status.FAILED,
                 exit_code=-1,
@@ -248,9 +261,37 @@ def reconcile_stale(cls) -> int:
                 ),
                 ended_at=now,
             )
-
+            reconciled += flipped
+            if flipped:
+                notify_pks.extend(
+                    cls.objects.filter(
+                        pk__in=stale_pending, status=cls.Status.FAILED
+                    ).values_list("pk", flat=True)
+                )
+
+        cls._notify_reconciled(notify_pks)
         return reconciled
 
+    @classmethod
+    def _notify_reconciled(cls, pks) -> None:
+        """Send the normal failure notifications for reconciled runs.
+
+        A run the reconciler flips to FAILED never went through
+        ``execute_run_task``'s finalize path, so without this the one failure
+        class the platform most needs to report - "your worker died mid-run" -
+        was the one it stayed silent about. Best-effort per run: a broken
+        channel must not stop the reconciler (which is the heartbeat).
+        """
+        if not pks:
+            return
+        from core.services.notification_service import NotificationService
+
+        for run in cls.objects.filter(pk__in=pks).select_related("script"):
+            try:
+                NotificationService.send_notification(run)
+            except Exception:
+                logger.exception(f"Notification for reconciled run {run.pk} failed")
+
     @property
     def duration(self) -> float | None:
         """Return the duration in seconds, or None if not completed."""

core/script_helpers/pyrunner_db.py

@@ -139,9 +139,10 @@ def connect(name: str, **kwargs):
         import psycopg
     except ImportError as e:
         raise PyRunnerDbError(
-            "The 'psycopg' package is required for pyrunner_db.connect(). It "
-            "ships with PyRunner's runtime; in a custom environment install it "
-            "with: pip install psycopg[binary]"
+            "The 'psycopg' package is not installed in this script's environment. "
+            "Add 'psycopg[binary]' under Environments -> (this environment) -> "
+            "Packages, then re-run. (pyrunner_db.dsn() / sqlalchemy_url() work "
+            "without it if you bring your own driver.)"
         ) from e
 
     return psycopg.connect(dsn(name), **kwargs)

core/services/environment_service.py

@@ -94,7 +94,24 @@ def discover_python_versions(cls) -> list[dict]:
                 pythons.append(p)
                 seen_paths.add(p["path"])
 
-        return pythons
+        # Debian/Ubuntu ship /usr/bin/python3 without ensurepip unless
+        # python3-venv is installed; offering it produces a create that fails
+        # with "ensurepip is not available" and an orphan folder. Probe first.
+        return [p for p in pythons if cls._supports_venv(p["path"])]
+
+    @staticmethod
+    def _supports_venv(python_path: str) -> bool:
+        """True when ``python -m venv`` can bootstrap pip with this interpreter."""
+        try:
+            result = subprocess.run(
+                [python_path, "-c", "import ensurepip, venv"],
+                capture_output=True,
+                timeout=15,
+                creationflags=subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0,
+            )
+            return result.returncode == 0
+        except Exception:
+            return False
 
     @classmethod
     def _discover_via_py_launcher(cls) -> list[dict]:
@@ -235,9 +252,21 @@ def create_environment(
         except ValueError as e:
             return False, str(e)
 
-        # Check if path already exists
+        # Check if path already exists. A folder no Environment row references is
+        # the leftover of an earlier create that failed after ``python -m venv``
+        # started (e.g. an interpreter without ensurepip): reclaim it instead of
+        # blocking the name forever with no UI way out.
         if os.path.exists(full_path):
-            return False, f"Path already exists: {full_path}"
+            from core.models import Environment
+
+            if Environment.objects.filter(path=env_path).exists():
+                return False, f"Path already exists: {full_path}"
+            logger.warning(
+                f"Reclaiming orphan environment folder {full_path} (no Environment row)"
+            )
+            shutil.rmtree(full_path, ignore_errors=True)
+            if os.path.exists(full_path):
+                return False, f"Path already exists and could not be removed: {full_path}"
 
         # Ensure parent directory exists
         os.makedirs(os.path.dirname(full_path), exist_ok=True)
@@ -256,6 +285,7 @@ def create_environment(
 
             if result.returncode != 0:
                 error_msg = result.stderr or result.stdout or "Unknown error"
+                cls._discard_partial_venv(full_path)
                 return False, f"Failed to create venv: {error_msg}"
 
             # Verify the environment was created
@@ -266,10 +296,18 @@ def create_environment(
             return True, "Environment created successfully"
 
         except subprocess.TimeoutExpired:
+            cls._discard_partial_venv(full_path)
             return False, "Timeout creating environment"
         except Exception as e:
+            cls._discard_partial_venv(full_path)
             return False, f"Error creating environment: {str(e)}"
 
+    @staticmethod
+    def _discard_partial_venv(full_path: str) -> None:
+        """Remove whatever a failed ``python -m venv`` left behind."""
+        if os.path.exists(full_path):
+            shutil.rmtree(full_path, ignore_errors=True)
+
     @classmethod
     def delete_environment(cls, environment) -> tuple[bool, str]:
         """
@@ -371,6 +409,66 @@ def pip_freeze(cls, environment) -> str:
             logger.error(f"pip freeze failed: {e}")
             return ""
 
+    # Wall-clock caps for the pip subprocess per operation. The django-q task
+    # that hosts each one is given this + TASK_TIMEOUT_GRACE (see
+    # ``task_timeout``) so pip's own timeout always fires first and kills the
+    # tree; the worker-level timeout is only a backstop. Before this the two
+    # were equal (600s), so the worker killed the task mid-install, the pip
+    # process kept running orphaned, and the broker re-delivered the task
+    # into the same venv - the "bulk install stuck" report.
+    PIP_TIMEOUTS = {"install": 300, "uninstall": 120, "bulk_install": 600}
+    TASK_TIMEOUT_GRACE = 60
+
+    @classmethod
+    def task_timeout(cls, operation: str) -> int:
+        """django-q per-task timeout for a package operation (pip cap + grace)."""
+        key = {"install": "install", "uninstall": "uninstall", "bulk_install": "bulk_install"}[
+            str(operation)
+        ]
+        return cls.PIP_TIMEOUTS[key] + cls.TASK_TIMEOUT_GRACE
+
+    @classmethod
+    def max_task_timeout(cls) -> int:
+        """The longest package-operation task the cluster can host."""
+        return max(cls.PIP_TIMEOUTS.values()) + cls.TASK_TIMEOUT_GRACE
+
+    @classmethod
+    def _run_pip(cls, cmd: list, timeout: int) -> subprocess.CompletedProcess:
+        """Run a pip command with the same hardening as script runs.
+
+        - Own process group / session, so a timeout kills pip *and* the build
+          backends it spawned (``kill_process_tree``), not just the leader.
+        - ``stdin`` closed + ``PIP_NO_INPUT=1``: pip can never block on a
+          prompt (e.g. a private index asking for credentials) inside a worker.
+        Raises ``subprocess.TimeoutExpired`` after killing the tree.
+        """
+        from core.executor_backends.local import kill_process_tree
+
+        popen_kwargs: dict = {
+            "stdin": subprocess.DEVNULL,
+            "stdout": subprocess.PIPE,
+            "stderr": subprocess.PIPE,
+            "text": True,
+            "env": {**os.environ, "PIP_NO_INPUT": "1"},
+        }
+        if os.name == "nt":
+            popen_kwargs["creationflags"] = (
+                subprocess.CREATE_NO_WINDOW | subprocess.CREATE_NEW_PROCESS_GROUP
+            )
+        else:
+            popen_kwargs["start_new_session"] = True
+        proc = subprocess.Popen(cmd, **popen_kwargs)
+        try:
+            stdout, stderr = proc.communicate(timeout=timeout)
+        except subprocess.TimeoutExpired:
+            kill_process_tree(proc.pid)
+            try:
+                proc.communicate(timeout=5)
+            except Exception:
+                pass
+            raise
+        return subprocess.CompletedProcess(cmd, proc.returncode, stdout, stderr)
+
     @classmethod
     def install_package(
         cls, environment, package_spec: str
@@ -395,15 +493,7 @@ def install_package(
 
         try:
             cmd = [pip_path, "install", package_spec]
-            creationflags = subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0
-
-            result = subprocess.run(
-                cmd,
-                capture_output=True,
-                text=True,
-                timeout=300,  # 5 minutes timeout
-                creationflags=creationflags,
-            )
+            result = cls._run_pip(cmd, timeout=cls.PIP_TIMEOUTS["install"])
 
             success = result.returncode == 0
             if success:
@@ -442,15 +532,7 @@ def uninstall_package(
 
         try:
             cmd = [pip_path, "uninstall", "-y", package_name]
-            creationflags = subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0
-
-            result = subprocess.run(
-                cmd,
-                capture_output=True,
-                text=True,
-                timeout=120,
-                creationflags=creationflags,
-            )
+            result = cls._run_pip(cmd, timeout=cls.PIP_TIMEOUTS["uninstall"])
 
             success = result.returncode == 0
             if success:
@@ -512,15 +594,7 @@ def install_requirements(
                 temp_path = f.name
 
             cmd = [pip_path, "install", "-r", temp_path]
-            creationflags = subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0
-
-            result = subprocess.run(
-                cmd,
-                capture_output=True,
-                text=True,
-                timeout=600,  # 10 minutes for bulk install
-                creationflags=creationflags,
-            )
+            result = cls._run_pip(cmd, timeout=cls.PIP_TIMEOUTS["bulk_install"])
 
             success = result.returncode == 0
             if success: