You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 78cb8e0
Browse filesBrowse the repository at this point in the historyBrowse files
optical-codec: the QR path, a synthetic screen to read it off, and a measured size floor
Adds packages/optical-codec and the recognize.scan verb, so Screen Angel can grab a frame
and answer with what is written on it rather than handing back a picture. QR only for now;
the eggs field is present and empty so no call site changes when the egg reader lands.
The eggs are drawn rather than photographed, because The Sims does not have eggs yet: the
package renders synthetic game frames with eggs and QR codes in them and reports the
ground-truth boxes, which gives the reader targets today and keeps working afterwards as
the controlled half of the test set.
Sweeping pixels-per-module against scaling, blur, noise, colour cast and a hand-held tilt
puts the floor at four pixels per module. That is the number the About dialog has to draw
to, and it replaces the guess the specs were carrying.
Lighting is now cosmetic everywhere, per Don: the reader looks at caps, bands and segments
and never at illumination, the way a Factorio lamp is for the player. The glow moved behind
the egg body so the cosmetic layer cannot touch the pixels the reader samples, and there is
a test that the bands are pixel-identical lit and dark. The blink carrier in
OPTICAL-CHANNEL.yml is marked speculative rather than load-bearing.
Two corrections while in here: the hand-rolled downscale retry ladder went away because
zxing already does it better, and the Hope Chest is credited to SimProv — unreleased, built
with Donna (SimBabes), Steve Alvey (SimSlice) and Heather Castillo (SimFreaks) — rather
than to a shipped Sims 1 object.
Copy file name to clipboardExpand all lines: apps/screen-angel/EGGS.yml
+6-6Lines changed: 6 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -167,7 +167,7 @@ the_blip_editor:
167
167
it_is_existing_machinery: "Games already float things over objects — thought balloons, status icons, the diamond over a selected Sim. A badge is that overlay slot pointed at egg state (SCREEN-ANGEL.yml)."
168
168
a_standard_modular_set_todo:
169
169
todo: "Not specced. For looks first: the player needs to know the mailbox has a new message, and nothing more than that is required of a badge today."
170
-
the_precedent: "Donna's SimBabes modular wedding cake toppers— bride and groom in any combination of gender and skin tone, assembled from parts instead of drawn once per pairing. A handful of pieces covering a space that would otherwise need dozens of drawings (Don's recollection; find the creator's page before citing this publicly)."
170
+
the_precedent: "Donna's modular wedding cake toppers, from SimBabes — bride and groom in any combination of gender and skin tone, assembled from parts instead of drawn once per pairing. A handful of pieces covering a space that would otherwise need dozens of drawings. Same collaboration as the SimProv wedding playset (cracking_is_the_constructor)."
171
171
what_that_suggests: "One standard symbol set with combinable slots — subject, action, urgency — rather than a flat list of finished glyphs. Same combinatorial move as the bands, in a channel meant for people."
172
172
deliberately_deferred: "How large the set is, what the slots are, and whether the recogniser ever reads it. Decorative and honest beats specified and wrong."
173
173
custom_badges_are_the_safe_kind_of_content: "A badge is a picture with no authority: it labels, it cannot act. So players and shows can ship their own without anyone auditing anything, which makes it the natural place for personality in a system that is otherwise deliberately uniform (the_blip_editor)."
@@ -195,14 +195,14 @@ the_blip_editor:
195
195
steady: "Actionable NOW. The menu has live items and picking one will do something."
196
196
breathing: "Working. The call is out, the job is running, nobody has to do anything yet. A slow rise and fall is unmistakably different from steady at any distance, and reads as patience rather than as an alarm."
197
197
it_solves_finding_before_reading:
198
-
why_this_is_the_useful_part: "Emitted light is a local brightness maximum, so an egg segments itself out of a scene. Threshold for candidates, frame-difference for the breathing ones, and you have every egg on screen without matching a single template."
199
-
the_pipeline_order: "Light answers WHERE, the badge answers WHAT KIND, the bands answer WHICH ONE, the dialog answers everything else. Detection is cheap and constant; decoding is expensive and only happens where the light already pointed (MOBILE-CAMERA.yml, the_five_stages)."
200
-
humans_do_the_same_thing: "Attention goes to the bright thing before it goes to the detailed thing. Same signal, same ordering, two very different readers — which is the property worth designing for rather than a coincidence to note."
198
+
it_is_cosmetic_and_stays_cosmetic: "For the player only, the way a lamp wired to a circuit condition in Factorio is for the player. The reader never consults it: detection and decoding use the caps, the bands and the segments, and nothing else (RECOGNIZER.yml, lighting_is_never_a_signal)."
199
+
why_that_boundary_is_worth_keeping: "A decoder that depended on a glow would break when an artist dimmed it, and bloom already damages the colour fidelity the bands need. Keeping light out of the read means the art stays free and the reader stays testable on a still frame."
200
+
humans_do_the_same_thing: "Attention goes to the bright thing before it goes to the detailed thing, which is why this is worth drawing at all. It is a courtesy to the player, not a channel."
201
201
height_is_the_other_state_channel:
202
202
levitation: "An active egg rises off the ground. Elevation is a large, slow, unmistakable signal that survives being small on screen, and it reads at a glance in a way a colour never will — a floating thing is doing something."
203
203
it_leaves_its_base_behind: "The egg lifts off a base that stays on the tile. So the grid keeps its geometry when half the eggs are up, the empty socket says an egg belongs here, and an elevated egg matrix still reads as a matrix instead of as scattered debris (matrix_eggs)."
204
204
it_costs_nothing: "A z offset in the draw group and a slow animation. The same offsets that stack the bands lift the body (offsets_are_what_make_the_set_small)."
205
-
it_stacks_with_light: "Light says whether it can be acted on; height says whether it is working. Two independent low-frequency channels, both legible at the range where every band has collapsed."
205
+
it_stacks_with_light: "Light says whether it can be acted on; height says whether it is working. Two glanceable hints for the player at the range where every band has collapsed — and, like light, invisible to the reader."
206
206
207
207
one_vocabulary_for_every_object: "This is a house rule, not an egg feature. Anything with pending state uses the same three states — the machine chewing on a job, the mailbox with something in it, the appliance mid-cycle. Learned once, it makes the whole screen a status board, and a recogniser written for eggs already works on all of it."
208
208
excitement_as_a_dimmer: "Brightness and pulse rate are a continuous channel over an existing effect: brighter and faster for the errand about to resolve, dim and slow for the one with a week to run. Enthusiasm is a value here, not decoration."
@@ -295,7 +295,7 @@ the_egg_is_its_own_file:
295
295
cracking_is_the_constructor:
296
296
the_verb: "Eggs do not get constructed, they get CRACKED. Instantiation is an interaction a character performs in the world, in a place, at a time, in front of whoever is standing there — not an event in a log."
297
297
personalise_at_the_crack: "Cracking is where an egg stops being generic: name it, pick its badge, set what it is for, bond it to the character who opened it. The moment of instantiation is the moment of authorship, which is the right place for it and a good one to animate."
298
-
bonding: "Don's reference is the hope chest — an object you accumulate a relationship with rather than merely own (Sims 1 object, unverified as to which release). A cracked egg carries who cracked it and what they did with it, so a well-travelled egg is an heirloom, and its custody chain IS its sentimental value (moollm skills/soul-city/SOUVENIRS.md)."
298
+
bonding: "The reference is the Hope Chest from SimProv — Don's unreleased playset collection, built with Donna (SimBabes), Steve Alvey (SimSlice) and Heather Castillo (SimFreaks) — an object you accumulate a relationship with rather than merely own. Never shipped, so there is nothing to go and look at; the design is in WWSFF catalogs/simprov/. A cracked egg carries who cracked it and what they did with it, so a well-travelled egg is an heirloom, and its custody chain IS its sentimental value (moollm skills/soul-city/SOUVENIRS.md)."
299
299
cracking_it_like_a_safe:
300
300
the_gesture: "The crack menu carries a Dial submenu of eight compass points. Each pick is three bits, and you dial repeatedly to enter a code of any length — the menu stays eight items no matter how long the number (OPTICAL-CHANNEL.yml, dialing_in)."
301
301
what_you_are_dialling_into: "An identification number for a web object outside the game. Cracking with a combination binds this egg to that external thing, and the same combination dialled into eggs in other lots or other games binds them all to it — every one of them able to send messages up to the same place."
Copy file name to clipboardExpand all lines: apps/screen-angel/MOBILE-CAMERA.yml
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -35,7 +35,7 @@ what_it_unlocks:
35
35
# same package, imported by both.
36
36
37
37
stages:
38
-
find_the_bright_things_first: "Before any recognition, threshold for local brightness maxima and frame-difference for slow pulses. Eggs emit light when they are worth reading, so a two-instruction GPU pass produces the candidate list and everything expensive runs only inside those boxes (EGGS.yml, it_solves_finding_before_reading). On a phone this is the difference between a live viewfinder and a slideshow."
38
+
find_the_caps_first: "Before any decoding, look for the fiducial: a high-contrast white-over-black pair in the plausible size range, on a downscaled frame. Everything expensive runs only inside those boxes, which on a phone is the difference between a live viewfinder and a slideshow. Illumination is never part of the test — glow and pulse are cosmetic (RECOGNIZER.yml, lighting_is_never_a_signal)."
39
39
40
40
acquire:
41
41
desktop: "Screen capture into the ring buffer (RECORDER.yml). Pixel-exact, no optics, known geometry."
status: "SPECULATIVE, and nothing depends on it. Light is cosmetic by decision, so the reader takes its values from caps, bands and segments alone (RECOGNIZER.yml, lighting_is_never_a_signal). A blink carrier stays written down because the idea is sound, and it may not become load-bearing without a deliberate reversal of that rule."
140
141
what: "A known number of frames of brightness change, in a known rhythm. The value is in the schedule, not in the picture."
141
142
what_it_is_for: "Objects that must remain in the world and cannot pop a dialog, at zoom levels where no spatial detail survives. Chiefly: an egg saying which of its fixed signals just became true (EGGS.yml, the_wink)."
142
143
the_trade: "A band costs pixels side by side and is read in one frame. A wink costs one pixel and many frames. Same few bits, opposite budgets, and the reader spends whichever it has — which is why zooming out does not end the conversation, it only slows it down."
where: "packages/optical-codec (codec, renderer, fixtures, tests) · src/main/recognize/service.ts (the verb)"
13
15
14
16
# THE DECISION THAT SHAPES EVERYTHING ELSE
15
17
#
@@ -43,7 +45,8 @@ egg_reading:
43
45
self_delimiting: "The caps bound the message, so there is no length field to parse and no terminator to find (EGGS.yml, the_stack_is_self_delimiting)."
44
46
no_error_correction_needed: "A per-group check band turns a misread into a discarded read. No Reed-Solomon, no bit interleaving, no Galois field."
45
47
the_stages:
46
-
zero_find_the_lit_things: "Threshold for local brightness maxima, frame-difference for slow pulses. Eggs light up when they are worth reading, so the candidate list is a cheap GPU pass and everything expensive runs only inside those boxes (EGGS.yml, it_solves_finding_before_reading)."
48
+
zero_find_the_caps: "Search for the fiducial: a white region directly above a black one, at high contrast, in the size range the current zoom allows. That pair is the only thing the reader looks for, and finding it is a cheap local test over a downscaled frame."
49
+
lighting_is_never_a_signal: "Glow, pulse and levitation are cosmetic — for the player, the way a wired lamp in Factorio is for the player. The reader must work on a still frame of an unlit egg, so no stage may consult illumination. Coupling the decoder to a light effect would mean an artist dimming a glow breaks reads, and bloom already wrecks colour fidelity."
47
50
one_confirm_the_silhouette: "Match the egg template at the expected size. Where the game's zoom is known, the size is known, and the scale search disappears entirely."
48
51
two_lock_the_caps: "Find the white-over-black adjacency. It gives the axis, the scale, the extent of the stack, and a black and a white sample for normalisation, all from the same feature."
49
52
three_sample_the_bands: "Walk the axis, sample each band's centre, normalise against the cap samples so a sunset or a fire does not turn orange into brown."
@@ -62,7 +65,7 @@ the_stack:
62
65
what: "One interface over screen capture, a video element, a camera, and a synthetic test source, so the same reader runs against a fixture and a live game."
63
66
how: "getDisplayMedia into WebCodecs VideoFrame, driven by requestVideoFrameCallback so every frame carries the presentation timestamp the recorder's clock already wants (RECORDER.yml)."
what: "Greyscale, mip-chain downsample, separable box blur for an adaptive threshold, and a brightness-maximum reduction to produce candidate boxes."
68
+
what: "Greyscale, mip-chain downsample, separable box blur for an adaptive threshold, and a high-contrast vertical-adjacency test to produce candidate boxes. Geometry and contrast only — never illumination."
66
69
where: "WebGPU compute, with a WebGL2 fragment-shader pyramid as the fallback. Both are present in Chromium, so Electron gets this for free."
67
70
the_actual_bottleneck: "Readback, not arithmetic. Copy only the candidate tiles, double-buffer so the CPU never waits on the GPU, and never read a full frame back per frame."
68
71
honest_ordering: "CPU first. A 640-wide greyscale threshold on a modern machine is cheap, and a GPU stage added before it is proven necessary is a stage that has to be debugged for no reason."
@@ -76,7 +79,47 @@ the_stack:
76
79
# space heater.
77
80
78
81
when_to_look:
79
-
survey: "A few times a second at low resolution, to answer whether anything lit changed."
82
+
survey: "A few times a second at low resolution, to answer whether anything changed at all."
80
83
burst: "Full-rate reads on the candidate boxes for a moment after a change, then back to survey."
81
84
on_demand: "A dialog opening, a player selecting an egg, or the Angel deciding to act are all reasons to look immediately at one place."
82
-
idle_is_free: "A screen where nothing lit has changed costs one downscaled diff. The dance-floor case is the interesting stress test, and it is bounded by the candidate count rather than by area (EGGS.yml, matrix_eggs)."
85
+
idle_is_free: "A still screen costs one downscaled frame difference. The dance-floor case is the interesting stress test, and it is bounded by the candidate count rather than by area (EGGS.yml, matrix_eggs)."
86
+
87
+
# WHAT IS ACTUALLY BUILT, AND WHAT IT MEASURED
88
+
#
89
+
# The specification above was written before any of it ran. This section is what happened when it
90
+
# did, and it is allowed to contradict the section above — the numbers win.
91
+
92
+
the_verb:
93
+
recognize_scan: "One capture plus one decode, in the main process, returning payloads and boxes in screen points. One verb rather than two because handing the caller an image would put a decoder in every client and move pixels across a socket to be looked at once (src/common/protocol.ts, src/main/recognize/service.ts)."
94
+
png_always: "The capture service prefers JPEG for screens and is right to, but a small QR code is what JPEG damages worst — hard edges, tiny features, artifacts landing on the modules. Recognition overrides the default."
95
+
the_bytes_go_in_encoded: "zxing decodes PNG and JPEG itself, so the captured file goes straight to the wasm module. Decoding it first would mean carrying an image decoder to hand the decoder something it can already read."
96
+
eggs_field_is_empty_not_absent: "Callers get an empty array today and populated arrays later, so nobody rewrites a call site when the egg reader lands."
97
+
98
+
what_the_library_already_does:
99
+
the_correction: "The stage list above assumed we would do our own geometry. For QR we do none: zxing-cpp finds the finder patterns, solves the perspective transform from them, samples the module grid through it, and retries rotated, inverted and downscaled. A phone at an angle to a monitor is its ordinary case."
100
+
so_the_screen_and_camera_split_is_smaller_than_written: "It still holds for the egg reader, which has no such machinery and is why the caps exist. For QR, one call covers both paths and a hand-rolled retry ladder was deleted after the library turned out to do it better."
101
+
what_no_decoder_fixes: "Moire between the pixel grid and the sensor grid, specular glare, PWM banding, and too few pixels per module. Those are optical, and they are what sets the size floor below."
102
+
103
+
measured_legibility:
104
+
how: "packages/optical-codec/scripts/measure-legibility.ts — a 37-byte payload at ecLevel M in an 800x600 frame, swept across pixels-per-module and degradation."
105
+
the_number: "FOUR PIXELS PER MODULE survives everything in the suite. One pixel per module reads only on a clean grab."
106
+
the_grid:
107
+
clean_screen_grab: "1px and up."
108
+
window_scaled_to_50_percent: "2px and up."
109
+
window_scaled_to_33_percent: "4px and up."
110
+
soft_focus: "3px and up."
111
+
phone_noise_and_warm_cast: "1px and up — a colour cast and grain barely matter, which is the least intuitive result here."
112
+
phone_tilted: "2px and up."
113
+
phone_tilted_blurred_and_noisy: "4px and up."
114
+
the_caveat: "Synthetic degradations, not a camera. This is a floor on what will work, not a substitute for pointing a phone at a monitor. The perspective warp is a real homography, the noise is additive rather than sensor-shaped, and lossy encoding is not in the suite at all."
115
+
what_it_decides: "How big the About dialog draws its code, and what to say to anyone who wants it smaller."
116
+
117
+
the_eggs_are_drawn_before_the_game_draws_them:
118
+
what: "packages/optical-codec renders eggs into synthetic frames — caps, calibration swatches, bands, segments, levitation, glow — and reports the ground-truth box and digits for each one."
119
+
why_not_wait_for_screenshots: "There are no eggs in The Sims yet, so there is nothing to photograph. Generating the frame gives the reader targets today and stays useful afterwards: a real capture tells you whether it works, a synthetic frame tells you which pixel broke it."
120
+
the_test_that_matters: "The bands are pixel-identical lit and dark. The glow draws behind the body, so the cosmetic layer cannot reach the pixels the reader samples — which is the rule above, enforced rather than asserted."
121
+
122
+
open_items:
123
+
packaging_the_wasm: "The Node path finds the wasm inside node_modules by walking up from an exported entry point. A packaged app needs those two files unpacked from the asar and resolved from resources instead."
124
+
jpeg_in_the_abuse_suite: "Needs a real DCT and quantisation table. Standing in for it with blur plus noise would let us claim a pass we had not earned."
125
+
the_palette_is_provisional: "Ten separated RGB values, chosen against theory. The spec requires choosing them against the game's rendered output, and that has not happened."
0 commit comments