Permanent file deletion ignores request cancellation and can wait on a locked file-sync item

[stevelr]

Summary

DELETE /v1/spaces/{space_id}/files/{file_id}?skip_bin=true accepts the HTTP
request context, then drops it before permanent deletion begins. The deletion
path substitutes component or background contexts for file-sync queue access,
file offload, tree deletion, and cache removal. A client disconnect or timeout
therefore bounds only the caller. Server work can continue after the request is
gone.

The file-sync queue makes the failure operationally significant. Permanent
deletion calls a context-free GetById operation. If another worker holds that
file object's queue item, the delete has no request-level bound. The wait ends
only when the owner releases the item or the queue shuts down. Request
cancellation cannot remove the waiter.

This report makes the source-backed cancellation and queue-wait claim. Runtime
observations on Heart v0.50.10 suggest that an abandoned delete can delay later
deletes, but they do not establish a persistent or space-wide wedge.

Affected and current versions

• Runtime observation: anytype-cli v0.3.6, built 2026-06-17, CLI commit
  f9d819532b7440d18957b5c470fac2ab171ac0d8, embedding
  github.com/anyproto/anytype-heart v0.50.10.
• Latest stable Heart release inspected: v0.50.20, released 2026-08-09,
  peeled tag commit af6b987420ba8624e1874ac91641c07186fbb4e0.
• Latest Heart prerelease inspected: v0.51.0-rc2, released 2026-08-17,
  peeled tag commit 2642778c42a370140e5e4a631b8b258d37f77d0d.
• Current supported CLI asset: anytype-cli v0.3.6. No newer supported CLI
  asset is available for a v0.50.20 or v0.51.0-rc2 runtime check.

anytype version and go version -m establish the runtime coordinates. The
v0.50.20 and v0.51.0-rc2 sources establish that the cancellation boundary
remains unchanged. Source inspection is not runtime reproduction on either
Heart release.

Reproducible source evidence

Check out Heart v0.50.20 or v0.51.0-rc2 and inspect these functions in order:

1. core/api/handler/file.go, DeleteFileHandler, passes
   c.Request.Context() to Service.DeleteFile.
2. core/api/service/file.go, Service.DeleteFile, passes that context to
   Middleware.ObjectListDelete when skipBin is true.
3. core/block.go, Middleware.ObjectListDelete, does not read cctx. It
   calls Block.Service.DeleteArchivedObjects without a context.
4. core/block/service.go, DeleteArchivedObjects and
   DeleteArchivedObject, provide no context parameter. DeleteArchivedObject
   obtains the space with context.Background() and enters the permanent
   object-deletion path.
5. core/block/delete.go, DeleteObjectByFullID, uses background or component
   contexts for the delete stages, including file data deletion, tree deletion,
   and cache removal.
6. core/files/fileobject/service.go, DeleteFileData, calls file-sync delete
   without a request context and calls FileOffloadFullId with
   context.Background().
7. core/files/filesync/queue.go, fileSync.process, obtains the item through
   Queue.GetById(id).
8. core/files/filesync/filequeue/filequeue.go, Queue.GetById, has no caller
   context. A locked item adds the response channel to getByIdWaiters; the
   wait ends when the item is released or the queue is closed.
9. core/files/fileoffloader/offloader.go, FileOffloadRaw, removes each block
   with context.Background() even though the function accepts a context.

The same cancellation loss and context-free GetById path is present in the
locally inspected v0.50.10 source. Heart v0.50.20 changes the pending-deletion
queue update to reset an existing item's ScheduledAt value. That change can
shorten a retry backoff. It does not make the queue wait cancellable or retain
the HTTP request context. The v0.51.0-rc2 source check at commit
2642778c42a370140e5e4a631b8b258d37f77d0d finds the same request-context loss
and context-free GetById chain.

The source path can be enumerated without running a server:

rg -n \
  'DeleteFileHandler|func \(s \*Service\) DeleteFile|ObjectListDelete|DeleteArchivedObjects|DeleteArchivedObject|DeleteFileData|FileOffloadRaw|func \(q \*Queue.*GetById' \
  core/api/handler/file.go \
  core/api/service/file.go \
  core/block.go \
  core/block/service.go \
  core/block/delete.go \
  core/files/fileobject/service.go \
  core/files/filesync/queue.go \
  core/files/filesync/filequeue/filequeue.go \
  core/files/fileoffloader/offloader.go

Runtime observations on v0.50.10

Two permanent-delete requests were abandoned in one disposable probe space.
The first request ran inside a test process that was killed after 45 seconds.
A follow-up request in that probe space was killed after 120 seconds. A later
measurement observed a permanent delete return after approximately 154
seconds.

The adverse result did not reproduce consistently. Three sequential permanent
deletes in one fresh space and three deletes in separate fresh spaces all
returned within one second. The Rust client's cleanup-owned live test also
completed within its 180-second guard. These results show that the endpoint
works under ordinary conditions.

The retained evidence does not identify whether the two abandoned requests
targeted the same file object or different file objects, and it does not retain
a complete server log for those calls. It cannot distinguish a queue-item wait,
a shared space lock, or another delayed cleanup stage. The runtime evidence is
therefore corroboration for the cancellation defect, not proof of a space-wide
failure mode.

Bounded runtime confirmation

A runtime confirmation should use a disposable server data directory and a
dedicated test account. The test harness must start and own the complete
Heart/server process tree. Run that tree in an outbound-blocked network
namespace, or an equivalent verified sync-isolation boundary, so the fault
injection cannot reach the Anytype network. Before creating any object, record
the namespace and cgroup identities and prove outbound traffic is blocked with
a negative egress probe. Capture the server log from startup.

Use these outer bounds in addition to the client library's own deadline:

• 180 seconds for every individual API or CLI call;
• 8 minutes for each fresh-server branch, including startup and teardown;
• 30 minutes for the complete matrix.

Create a unique transient cgroup, capture its exact unit name, and install an
unconditional trap or finally handler before starting Heart. The handler must
stop that captured unit, wait until the unit is inactive with no remaining
processes, and only then remove the disposable server data.

Run one abandoned delete per fresh server instance:

1. Create a uniquely named private space and upload one small file.
2. Establish a normal permanent-delete baseline in another fresh space.
3. Start permanent deletion of the test file. Confirm that the request reached
   the handler, then terminate only the client. Bound dispatch confirmation to
   30 seconds even though the call-level guard is 180 seconds.
4. Snapshot the complete file inventories, direct object state, request and
   server timestamps, and a server-log boundary after client termination. Do
   not issue a same-object follow-up unless this snapshot completes within its
   180-second call guard.
5. Issue exactly one second delete in each isolated branch: the same object, a
   different file in the same space, or a file in another space. This second
   delete is deliberate disposable fault injection. It sends one physical
   attempt under the 180-second guard; it is not general retry guidance. Use a
   new server instance for every branch so one abandoned operation cannot
   contaminate the controls.
6. Snapshot the same inventories, direct state, timestamps, and fixed log
   categories after the branch completes or reaches its guard.
7. Restart the server on the disposable data. Before sending another delete,
   snapshot inventories and direct state, then run or await cleanup
   reconciliation within the branch guard. If deletion is already terminal,
   record that result and send no further delete. If state remains nonterminal
   and the report needs a recovery probe, send exactly one guarded deliberate
   delete with one physical attempt.
8. Compare the pre-restart and post-reconciliation snapshots to distinguish
   transient waiting from persisted cleanup state.
9. Invoke the unconditional teardown and verify that the exact captured cgroup
   is inactive and empty before deleting the disposable data directory.

Terminating only the HTTP client or test process does not stop Heart. The test
is complete only after its owned server cgroup has stopped and no process from
that tree remains.

Do not run this probe against user data. A hard delete is irreversible, and
the server can continue it after the client-side guard expires.

Retain only a bounded evidence derivative containing timestamps and fixed error
categories. Scan that derivative for credentials before attachment. Do not
attach raw logs, object or space IDs, local paths, request headers, tokens, or
other authentication material.

Expected result

Request cancellation should reach every cancellable wait in permanent file
deletion. Cancellation before an irreversible mutation starts should stop the
operation and remove any queue waiter. Permanent deletion needs one explicit
commit point immediately before its first irreversible mutation. At that point,
ownership transfers from the request to one server-owned lifecycle context
with one finite absolute cleanup deadline.

After the commit point, request cancellation must not cancel invariant-restoring
cleanup. If the request disappears, the handler returns no success and stops
waiting while the server-owned cleanup continues. Cleanup must release queue
and object locks and converge file data, object tree, index, and cache state
before its deadline.

Durable cleanup intent and its initial phase must be committed before, or
atomically with, crossing the commit point. A marker-write failure leaves the
operation pre-commit and authorizes no irreversible mutation. Each cleanup
phase advances durable state transactionally with its mutation, or uses an
idempotent mutation plus a durable phase transition that is safe to replay.
The marker is cleared only after file data, object tree, index, cache, and queue
ownership have converged. A failure, process kill, or deadline expiry leaves a
phase that restart reconciliation can resume. Later operations must not depend
on queue shutdown or server restart merely to release request-owned waiters.

The caller cannot infer whether a dispatched delete crossed the commit point
when its connection disappears. The server-side fix does not need to provide
rollback. It needs a defined ownership transfer, finite cleanup, and convergent
recovery that cannot retain request-owned waiters indefinitely.

Current client containment

The Rust anytype-api client applies a 600-second logical deadline to file
operations. A permanent delete is dispatched once and is never replayed by the
HTTP helper. Expiration returns a typed long_operation timeout with
mutation_indeterminate outcome and one physical attempt. A post-dispatch
transport failure is also classified as mutation-indeterminate.

The live happy-path test adds an independent 180-second guard in a fresh,
cleanup-owned space. This protects the test runner, but cancellation at that
guard does not stop Heart's background work. Callers must observe current file
state and reconcile an indeterminate outcome. Production code must not
automatically repeat the permanent delete. The deliberate second call described
above belongs only to the disposable fault-injection matrix.

Client containment limits caller retention and unsafe retries. It cannot
release a Heart queue waiter after the request context has been discarded.

Suggested implementation boundary

Use two contexts separated by the commit point:

• Before commit, the request context governs Middleware.ObjectListDelete,
  DeleteArchivedObjects, DeleteArchivedObject, object and file-data
  validation, and file-sync item acquisition. Cancellation removes any waiter
  and authorizes no irreversible mutation.
• After commit, one server-owned lifecycle context with one finite absolute
  deadline governs file deletion, offload, tree and index mutation, cache
  cleanup, and lock release. Request cancellation cannot cancel this context or
  reset its deadline.

Add a context-aware queue acquisition, such as GetById(ctx, id). Cancellation
must remove the response channel from getByIdWaiters without transferring the
item lock to the abandoned waiter. Releasing the current owner must still wake
the next live waiter.

Define a commit point before the first irreversible mutation. Pre-commit stages
use the request context. Crossing the commit point transfers ownership exactly
once to a server cleanup supervisor with one finite absolute deadline. Every
post-commit deletion, offload, tree, index, cache, and lock-release stage uses
that same lifecycle context; individual stages cannot reset its budget. If the
request is canceled after transfer, the handler detaches without a success
response and must not cancel cleanup.

Commit durable cleanup intent and the initial phase before, or in the same
transaction as, ownership transfer and the first irreversible mutation. If the
intent write fails, remain pre-commit and mutate nothing. Advance the phase
transactionally with each durable mutation where the storage boundary permits
it. Otherwise, make the mutation idempotent and order marker transitions so a
crash at either boundary replays safely. Startup or a bounded background
reconciler snapshots current state before acting, resumes the recorded phase,
and clears the marker only after file data, object tree, index, cache, and queue
ownership agree on the terminal state.

Suggested tests

1. Lock a filequeue item, enqueue a second GetById with a cancelable context,
   cancel it, and assert prompt context.Canceled plus waiter removal.
2. Release the locked item after canceling one waiter and prove that the next
   live waiter acquires it exactly once.
3. Cancel ObjectListDelete before permanent deletion dispatch and assert that
   no file, tree, or cache mutation begins.
4. Cancel while permanent deletion waits for a file-sync item, after item
   acquisition, and immediately before the commit point. Each pre-commit case
   must abort without mutation or a retained waiter.
5. Cancel immediately after the commit point and during every post-commit
   phase: file-sync scheduling, local offload, tree deletion, index removal,
   cache removal, and lock release. The handler returns no success, while the
   single server-owned cleanup context restores every invariant within its
   original deadline.
6. Fail the durable intent or initial phase write and prove that no irreversible
   mutation occurs. Kill the process after the marker commit and between the
   marker transition and mutation at every cleanup phase. Restart, snapshot
   state before reconciliation, and prove idempotent convergence to one
   terminal outcome without duplicate deletion, leaked locks, or live queue
   waiters.
7. Inject cancellation, failure, and deadline expiry before and after every
   phase mutation. Prove that the phase marker advances transactionally or
   replays the idempotent mutation, and is cleared only after all invariants
   converge.
8. Run an HTTP integration test with same-object, same-space different-object,
   and different-space follow-ups. Snapshot state before the deliberate second
   call. Every call sends one physical attempt under its own finite guard, and
   every branch and whole run obey their outer bounds.
9. After restart, snapshot and reconcile before deciding whether a recovery
   probe is warranted. If deletion is terminal, send no delete. Otherwise send
   at most one deliberate guarded probe and snapshot the resulting state.
10. Retain the existing successful skip_bin=true handler and permanent-delete
    coverage so cancellation support does not remove the documented API.

Impact

• Client timeouts and disconnects can leave server work running without an
  owner-visible completion result.
• A hard delete waiting for a locked file-sync item cannot respond to request
  cancellation.
• A deliberate repeated delete during isolated fault injection can expose
  another waiter or overlapping irreversible work. Production callers should
  reconcile indeterminate state instead of automatically retrying.
• Tests and automation must own and tear down the entire Heart/server process
  tree inside verified outbound-blocked sync isolation. Terminating only the
  client or test process leaves Heart free to continue abandoned delete work.

No server crash, cross-space data loss, or persistent space-wide wedge has been
established.

Evidence limitations

• Runtime evidence is from Heart v0.50.10 embedded in anytype-cli v0.3.6.
• Heart v0.50.20 is the latest stable source release. Heart v0.51.0-rc2 is the
  latest inspected prerelease. No newer supported CLI asset is available for a
  matching runtime test of either source version.
• The two adverse v0.50.10 calls lack complete server logs and exact object-ID
  relationship evidence.
• Later fresh-space calls succeeded quickly, so this report does not claim that
  permanent deletion is generally slow or unusable.
• Source inspection proves loss of cancellation and an unbounded queue wait. It
  does not prove which internal wait caused the observed 45-, 120-, or
  approximately 154-second durations.
• External evidence must be a credential-scanned bounded derivative containing
  only necessary timestamps and fixed error categories. Raw logs, IDs, local
  paths, headers, tokens, and authentication material must not be attached.