`POST /soc` silently discards an overwrite and returns 201 Created

[chrishobcroft]

Bee version: 2.8.1-7cf53193 (API 8.1.0), single node, default config.

Summary

Uploading a single-owner chunk to an address that already holds a SOC returns
201 Created, but the stored chunk is unchanged. The new payload is discarded
with no error, no warning, and a success status code. A client has no way to
tell a write that landed from a write that was dropped.

This happens on a mutable batch (immutableFlag: false), which is the case
where a caller is most likely to expect the write to take effect.

Reproduction

import requests
from eth_hash.auto import keccak
from eth_keys import keys

BEE   = "http://localhost:1633"
BATCH = "<a usable batch with immutableFlag: false>"

pk    = keys.PrivateKey(bytes.fromhex("11" * 32))
owner = pk.public_key.to_canonical_address()
ident = keccak(b"overwrite-demo")                 # fixed id -> fixed address
addr  = keccak(ident + owner)

def bmt_root(payload):                            # BMT over 32-byte segments
    lvl = [payload[i:i+32].ljust(32, b"\0") for i in range(0, 4096, 32)]
    while len(lvl) > 1:
        lvl = [keccak(lvl[i] + lvl[i+1]) for i in range(0, len(lvl), 2)]
    return lvl[0]

def put(payload):
    cac    = len(payload).to_bytes(8, "little") + payload
    digest = keccak(ident + keccak(cac[:8] + bmt_root(cac[8:])))
    sig    = pk.sign_msg_hash(keccak(b"\x19Ethereum Signed Message:\n32" + digest))
    sig_b  = sig.r.to_bytes(32,"big") + sig.s.to_bytes(32,"big") + bytes([sig.v + 27])
    return requests.post(f"{BEE}/soc/{owner.hex()}/{ident.hex()}",
                         params={"sig": sig_b.hex()}, data=cac,
                         headers={"swarm-postage-batch-id": BATCH,
                                  "content-type": "application/octet-stream"})

for v in (b"VERSION-ONE", b"VERSION-TWO", b"VERSION-THREE"):
    st  = put(v).status_code
    got = requests.get(f"{BEE}/chunks/{addr.hex()}").content[105:105+len(v)]
    print(f"wrote {v.decode():<15} POST={st}  GET -> {got!r}")

Actual

wrote VERSION-ONE      POST=201  GET -> b'VERSION-ONE'
wrote VERSION-TWO      POST=201  GET -> b'VERSION-ONE'
wrote VERSION-THREE    POST=201  GET -> b'VERSION-ONE'

All three uploads are signed correctly by the owner and all three are accepted
with 201. Only the first is stored.

Expected

Either of these would be fine; the current behaviour is neither:

1. Reject explicitly — 409 Conflict (or 403), with a body explaining that
   a SOC already exists at this address and replacement is not supported. The
   caller learns the write did not happen.
2. Honour the overwrite — replace the stored chunk, since the signature
   proves owner intent. 200 OK rather than 201 Created.

The problem is not which of these Bee chooses. It is that 201 Created asserts
a write that did not occur.

Why it matters

Any application treating SOC as a keyed store — a fixed address per logical
slot, updated over time — will appear to work in testing (the first write
lands) and silently serve stale data forever after. The failure is invisible at
the point of the bug and only surfaces much later as data corruption.

Feeds sidestep this by deriving a fresh identifier per update, so the mainline
Bee use case never hits it. That is exactly why it survives unnoticed.

Open questions

• Is the discard happening in the localstore dedup path, or in SOC validation?
• Would a node in the target neighbourhood accept the replacement, making the
  outcome depend on whether the uploader happens to be near the address? If so
  the behaviour is not just unreported but nondeterministic.
• Is "SOC is write-once per address" an intended invariant? If so it should be
  documented alongside the /soc endpoint, which currently does not say it.

[chrishobcroft]

Traced the root cause in the source (master @ ffea29f, same as 2.8.1).

The discard happens in uploadPutter.Put, before any stamp handling:

https://github.com/ethersphere/bee/blob/master/pkg/storer/internal/upload/uploadstore.go#L414-L423

// Check if upload store has already seen this chunk
ui := &uploadItem{Address: chunk.Address(), BatchID: chunk.Stamp().BatchID()}
switch exists, err := st.IndexStore().Has(ui); {
...
case exists:
    u.seen++
    u.split++
    return nil          // <-- new payload never reaches the chunkstore
}

The "seen" key is (Address, BatchID) only — no stamp hash, no timestamp. For a
SOC re-upload at a fixed identifier both are unchanged by construction, so the
second write is classified as a duplicate and returns nil. api/soc.go then
returns jsonhttp.Created unconditionally (L255), which is where the 201
comes from.

Two things make this look like an oversight rather than an intended invariant:

1. The collision handling already exists in this file, but is unreachable
here. errOverwriteOfImmutableBatch (L368) and errOverwriteOfNewerBatch
(L372) are defined precisely for stamp-index collisions — but the address-level
seen check short-circuits first, so a same-address SOC re-upload never gets
that far. Those errors can only fire for a different address colliding on a
stamp index.

2. The reserve implements the opposite semantics for the same situation.
reserve.Put compares stamp timestamps on collision and, for single-owner
chunks specifically, replaces the stored chunk:

https://github.com/ethersphere/bee/blob/master/pkg/storer/internal/reserve/reserve.go#L211-L213

if chunkType == swarm.ChunkTypeSingleOwner {
    r.logger.Debug("replacing soc in chunkstore", "address", chunk.Address())
    return s.ChunkStore().Replace(ctx, chunk, false)
}

So the reserve knows a SOC address can legitimately hold new content over time,
and the upload store does not. The two stores disagree, and the uploading node
serves its own stale copy from the upload store indefinitely — which is why this
is invisible from the node that made the write.

Fix direction, either of:

• include the stamp hash (or timestamp) in the uploadItem seen-check key, so a
  genuinely new stamp falls through to the existing collision path; or
• special-case swarm.ChunkTypeSingleOwner to skip the seen-check and defer to
  the stamp-index logic, mirroring reserve.Put.

If write-once-per-address is the intended invariant for the upload store, then
the minimal fix is just the status code — 409 instead of 201 — plus a note on
the /soc endpoint docs. The current behaviour is the one option that gives the
caller no signal at all.

[acud]

@chrishobcroft how would achieve this sort of behavior without having to ask every node in the network whether the chunk exists already yes/no (and even then you won't get good enough guarantees - how do you know it is not being uploaded/retried/etc)?