Impact
A destination stream key is a credential: whoever holds one can broadcast to the owner's channel.
Five separate defects let one escape. They were found and fixed independently, and they are collected here because they share a cause worth stating once: redaction was substring-based, and the key kept arriving in a spelling the redactor had never been given.
1. Cleartext at rest, in a world-readable file
polyemesis.db was created at SQLite's default 0644, inside a data directory both shipped deployments created with a plain mkdir -p under umask 022 — the systemd unit's install notes and the Dockerfile alike. Neither set UMask=, so nothing narrowed it at runtime.
That file held every destination stream key in cleartext, plus the admin bcrypt hash and session secrets. The -wal sidecar matters as much: SQLite gives it the mode of the main file, and it holds committed pages.
What upgrading does and does not clear. Destination stream keys are sealed at rest from 0.7.0. The other credentials in that database are not, by design: sources.token, the ingest JSON (SRT passphrases, legacy RTMP stream keys, and pull URLs with credentials embedded) and the settings blob remain plaintext after upgrading. So the database stops being the place your DESTINATION keys live, and does not stop being credential-bearing. Treat it as sensitive after upgrading, not only before.
Confirmed on a live deployment before the fix:
-rw-r--r-- polyemesis.db
-rw-r--r-- polyemesis.db-wal
$ sudo -u nobody head -c 32 /var/lib/polyemesis/polyemesis.db-wal
(succeeds)
2. A refused destination wrote its key to server.log on every retry
The supervisor logs the child's error on two lines, and both carried it unscrubbed: process exited, once per retry attempt, and giving up on process, once when the retry budget runs out. A destination that a platform refused therefore logged its own key repeatedly, so the least healthy destination produced the most copies.
The second line was missed when the first was fixed, and it is the worse of the two. It fires only after MaxRestarts consecutive failures — which is to say after a destination has been refused over and over, exactly the state an operator is in when they stop watching and start copying server.log into a bug report. The first line got the attention because it fires often; the second fires when somebody is about to hand the file to a stranger.
3. The stored spelling and the wire spelling were allowed to differ
The one worth reading closely, because nothing was unwired — the scrub ran and found nothing.
A configured key was 65 bytes, because a terminal had appended a bracketed-paste artefact: the real key followed by ESC [ 2 7 ; 2 ; 1 3. FFmpeg stopped reading the publish URL at the ESC, opened the 56-byte prefix, failed to connect, and printed that prefix on stderr.
SecretSet.Scrub is substring replacement — it finds the secret inside the text. Here the text held a prefix of the secret rather than the secret. A 65-byte needle does not occur inside a 56-byte haystack, so nothing matched.
Measured against the real process.log:
raw 65-byte value in log : False
percent-encoded in log : False
value-before-ESC in log : True (len 56)
The escape sequence is incidental. The defect is that what was stored and what reached the wire were permitted to differ at all.
4. A third-party model key reached the logs and the API
Not a destination stream key — the operator's own credential for whichever inference endpoint automod was pointed at. net/http puts the request URL verbatim into *url.Error, and that endpoint is free text an operator pastes, most often carrying ?api_key=…. A failed call therefore wrote the key into ModelStats.LastError and into server.log, once per message and with no backoff, because failing open means trying again on the next one.
The redaction reasoning already existed for exactly this and had been applied to the settings blob and nowhere else.
5. A minted Twitch key was only partially masked — not in any released version
Recorded here because it belongs with the other four and is the clearest illustration of the shared cause, but it never shipped: it was introduced and fixed inside the 0.7.0 development window (c7db212, four days after v0.6.0; git tag --contains is empty). No released version carries it, and it is excluded from the affected range above.
Twitch Enhanced Broadcasting mints a new 312-character key on a successful negotiation, carrying the operator's original key as its final segment. Only the original was registered for scrubbing — and because Scrub is strings.ReplaceAll, registering a substring masks that substring and leaves the rest of the containing string intact.
The output was a 312-character credential with its tail blanked. That reads as protection, which is worse than no masking at all: a redacted-looking log invites you to paste it into an issue.
Patches
All are fixed in v0.7.0. v0.6.0 and earlier carry defects 1 through 4;
defect 5 was introduced and fixed between v0.6.0 and v0.7.0 and has never
been in a release.
| # |
Fix |
Landed in |
| 1 |
The state database is private and neither deployment makes its directory world-traversable |
88ee8f6 (#299) |
| 1 |
The -wal and -shm sidecars are secured too, which an existing install does not inherit |
1fe0c82 (#300) |
| 1 |
Keys are sealed at rest; a key that will not open disables its destination rather than failing open |
abcc11f (#301) |
| 2 |
The supervisor scrubs the process exited line |
d6ea664 (#310, merged via #311) |
| 2 |
…and the giving up on process line, which the above missed |
#359 |
| 3 |
A key is stored in exactly the spelling that reaches the wire, so a prefix cannot escape the scrub |
be20540 (#306, #307, merged via #309) |
| 4 |
The model endpoint's own key is redacted out of errors and logs |
281821c (#324) |
| 5 |
The minted Twitch key is registered in full, not by its suffix (never released) |
dd6108a (#326, merged via #333) |
Until a release carrying them is tagged, the workarounds below are the only
remedy available to an installed deployment.
Workarounds
Before upgrading: chmod 600 the database and its -wal/-shm sidecars, and restrict the data directory. Restricting it matters more after upgrading, not less, because 0.7.0 puts the key that opens the sealed stream keys there.
Treat any existing server.log or process.log as containing live credentials.
On rotation, and the two defects scope it differently. Defect 2 exposed the keys of destinations that were refused, so those must be rotated. Defect 1 exposed every destination key stored on the host, to any local user, for as long as the file was readable — so if that file was ever reachable by someone you would not hand a stream key to, rotate all of them rather than only the ones that failed. If automod was configured against a hosted inference endpoint, rotate that provider key too (defect 4).
Credit
Found during an internal review of the redaction contract.
Impact
A destination stream key is a credential: whoever holds one can broadcast to the owner's channel.
Five separate defects let one escape. They were found and fixed independently, and they are collected here because they share a cause worth stating once: redaction was substring-based, and the key kept arriving in a spelling the redactor had never been given.
1. Cleartext at rest, in a world-readable file
polyemesis.dbwas created at SQLite's default 0644, inside a data directory both shipped deployments created with a plainmkdir -punder umask 022 — the systemd unit's install notes and the Dockerfile alike. Neither setUMask=, so nothing narrowed it at runtime.That file held every destination stream key in cleartext, plus the admin bcrypt hash and session secrets. The
-walsidecar matters as much: SQLite gives it the mode of the main file, and it holds committed pages.What upgrading does and does not clear. Destination stream keys are sealed at rest from 0.7.0. The other credentials in that database are not, by design:
sources.token, the ingest JSON (SRT passphrases, legacy RTMP stream keys, and pull URLs with credentials embedded) and the settings blob remain plaintext after upgrading. So the database stops being the place your DESTINATION keys live, and does not stop being credential-bearing. Treat it as sensitive after upgrading, not only before.Confirmed on a live deployment before the fix:
2. A refused destination wrote its key to
server.logon every retryThe supervisor logs the child's error on two lines, and both carried it unscrubbed:
process exited, once per retry attempt, andgiving up on process, once when the retry budget runs out. A destination that a platform refused therefore logged its own key repeatedly, so the least healthy destination produced the most copies.The second line was missed when the first was fixed, and it is the worse of the two. It fires only after
MaxRestartsconsecutive failures — which is to say after a destination has been refused over and over, exactly the state an operator is in when they stop watching and start copyingserver.loginto a bug report. The first line got the attention because it fires often; the second fires when somebody is about to hand the file to a stranger.3. The stored spelling and the wire spelling were allowed to differ
The one worth reading closely, because nothing was unwired — the scrub ran and found nothing.
A configured key was 65 bytes, because a terminal had appended a bracketed-paste artefact: the real key followed by
ESC [ 2 7 ; 2 ; 1 3. FFmpeg stopped reading the publish URL at the ESC, opened the 56-byte prefix, failed to connect, and printed that prefix on stderr.SecretSet.Scrubis substring replacement — it finds the secret inside the text. Here the text held a prefix of the secret rather than the secret. A 65-byte needle does not occur inside a 56-byte haystack, so nothing matched.Measured against the real
process.log:The escape sequence is incidental. The defect is that what was stored and what reached the wire were permitted to differ at all.
4. A third-party model key reached the logs and the API
Not a destination stream key — the operator's own credential for whichever inference endpoint automod was pointed at.
net/httpputs the request URL verbatim into*url.Error, and that endpoint is free text an operator pastes, most often carrying?api_key=…. A failed call therefore wrote the key intoModelStats.LastErrorand intoserver.log, once per message and with no backoff, because failing open means trying again on the next one.The redaction reasoning already existed for exactly this and had been applied to the settings blob and nowhere else.
5. A minted Twitch key was only partially masked — not in any released version
Recorded here because it belongs with the other four and is the clearest illustration of the shared cause, but it never shipped: it was introduced and fixed inside the 0.7.0 development window (
c7db212, four days afterv0.6.0;git tag --containsis empty). No released version carries it, and it is excluded from the affected range above.Twitch Enhanced Broadcasting mints a new 312-character key on a successful negotiation, carrying the operator's original key as its final segment. Only the original was registered for scrubbing — and because
Scrubisstrings.ReplaceAll, registering a substring masks that substring and leaves the rest of the containing string intact.The output was a 312-character credential with its tail blanked. That reads as protection, which is worse than no masking at all: a redacted-looking log invites you to paste it into an issue.
Patches
All are fixed in
v0.7.0.v0.6.0and earlier carry defects 1 through 4;defect 5 was introduced and fixed between
v0.6.0andv0.7.0and has neverbeen in a release.
88ee8f6(#299)-waland-shmsidecars are secured too, which an existing install does not inherit1fe0c82(#300)abcc11f(#301)process exitedlined6ea664(#310, merged via #311)giving up on processline, which the above missedbe20540(#306, #307, merged via #309)281821c(#324)dd6108a(#326, merged via #333)Until a release carrying them is tagged, the workarounds below are the only
remedy available to an installed deployment.
Workarounds
Before upgrading:
chmod 600the database and its-wal/-shmsidecars, and restrict the data directory. Restricting it matters more after upgrading, not less, because 0.7.0 puts the key that opens the sealed stream keys there.Treat any existing
server.logorprocess.logas containing live credentials.On rotation, and the two defects scope it differently. Defect 2 exposed the keys of destinations that were refused, so those must be rotated. Defect 1 exposed every destination key stored on the host, to any local user, for as long as the file was readable — so if that file was ever reachable by someone you would not hand a stream key to, rotate all of them rather than only the ones that failed. If automod was configured against a hosted inference endpoint, rotate that provider key too (defect 4).
Credit
Found during an internal review of the redaction contract.