Repository navigation
MU1367 cluster artifacts: root cause in the MOST queue flush, and a twelve-byte fix #22
diasdio
started this conversation in
Show and tell
Replies: 1 comment
|
Hi, I'm also running omonob's files and having the same problems with lag and pixels breaking up. i have 12.3 inch MU1367 cluster and running version MHI2_ER_VWG13_P4521 retrofitted onto my 2014 MK7 GTI. wondering if the fix you have posted here would fix my same setup and is it possible that you could upload and adjusted file for me to install. (I'm not sure how to adjust hashes as you've described in your post) also interested in the next up in media display you mentioned. im happy to test things out with you and try to find a smooth fix. thanks from Australia |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi harman-f,
I hope you are keeping well. My name is Diogo, I am based in Nairobi and I run a Golf Alltrack MK7 with an ex-Japan MIB2 High that I converted from G11 to G13. I have been running omonob's CarPlay Maps VC mod on it, and following your AltScreen work for some time — the compatibility matrix and the firmware profile map have saved me a great deal of guesswork, so thank you for putting those out.
I have been chasing the cluster artifacts on the CarPlay map stream on MU1367, and I believe I have found the root cause. It is in the MOST queue flush in omonob's
altscreen-most-bridge: the flush zeroespacket_posat the same moment it setshead = tail, so a part-written access unit is abandoned mid-PES and the cluster decodes a truncated frame. Making those stores conditional onpacket_posbeing zero is twelve bytes, and the artifacts are gone.There is a second finding on the IDR deferral gate, and a forced-IDR cadence change that lines up with VW's own GOP sizing in
displaymanager.json.The full write-up follows, with the exact offsets, the bytes and the hashes. It opens with what it builds on, because very little of this is mine — your transport knowledge base is where the stream path came from, and the binaries are omonob's. It also lists three builds I got wrong earlier, which should be deleted if anyone picked them up. That is two drives on one car now, both of them over road that used to break it without fail, so I am confident about this car — but I would be glad to hear whether it holds or breaks on other firmware.
Separately, I am also working on two other cluster things — getting the CarPlay track text up there in place of the "Apple CarPlay" label, and the cover art. Both are still work in progress so I am not writing them up yet, but if either is of interest to you I am happy to share what I have so far, and I can test on this car.
Happy to send any logs or binaries that would help.
Kind regards,
Diogo
The artifacts are gone. Two drives now, over the multi-storey car park and the roundabout that between them used to tear the map on every single pass, and not one artifact has appeared. Two changes did it: twelve bytes in
altscreen-most-bridgeso the MOST queue flush can no longer abandon a part-written access unit, and a forced-IDR cadence incore.soaligned to VW's own GOP sizing.Firmware
MHI2_ER_VWG13_K4525_1, MU1367, HMI H29.319.75. The car is a Golf Alltrack MK7 RHD, ex-Japan, with a MIB2 High converted from G11 to G13.Worth stating plainly, because it bears on everything below: the cluster and the amplifier are both retrofits on this particular car, but only one of them is unusual for the chassis. The MK7 did offer Dynaudio from the factory and that amplifier runs on MOST, so the ring is standard equipment — on a car with the standard speakers VW simply caps the MOST connector with a plastic cover and codes it out. What the MK7 was never offered with at all is the digital cockpit; that only arrived with the 7.5. So the Dynaudio J525 went into a ring that was already there, the 5G1920791B 791 AID 12.3 is the node no factory MK7 ever carried, and the AID is being fed by a head unit that was never built to talk to one in this chassis.
What this builds on
None of this starts from scratch, and it would be wrong to let it read that way.
The binaries patched here are omonob's.
altscreen-most-bridgeandcore.soare part of the CarPlay Maps VC mod, and the low-latency recovery design around the flush is his work. The fault described below is one narrow edge case inside it, and it was only findable because the rest of it is good enough to drive behind every day.The stream path, and most of what I understand about how AltScreen frames reach the cluster over MOST, comes from your work, harman-f — your MIB2 cluster display transport knowledge base in particular, with your compatibility matrix and firmware profile map for the surrounding context, and the AltScreen discussions here.
Several other projects were read along the way and shaped how I came at this: luka-dev's mib2q-carplay-rgi, jamespan's mib2-carplay-rgi-altscreen, yuedizhibo's MHI2Q-CarPlay-AltScreen, and Lanye-z's MHI2Q-CarPlay-MMI-Mirror.
What is mine in here is the queue-flush diagnosis, the twelve bytes, and a car to test them on.
Symptom and reproducer
The cluster map tore into rectangular blocks of stale or scrambled picture, in bands across the frame. Not a freeze and not a stutter: the frame kept updating, but parts of it were wrong, and the wrong parts held until the next clean keyframe.
It was worst at roundabouts and junctions, where the map pans and rotates hardest, but it was not only a motion problem. It happened on straight runs too, with a destination set, which ruled out the obvious "too much change between frames" reading early on.
The reliable reproducer was a multi-storey car park on the way out of work. Tight helical ramps, constant heading change, poor GPS, and it triggered every single time without exception. One episode on the way back from the airport was bad enough that the phone had to be unplugged and reconnected before the stream settled.
That consistency is what made the diagnosis possible. A fault that fires every time on a known stretch of road can be tested against, which is why the result in the road test section below means something.
Root cause: the queue flush abandons a part-written access unit
The bridge binary is a help here, because it ships unstripped:
altscreen-most-bridge, 67,835 bytes, ARM ELF with symbols intact.The queue flush helper sits at
0x101ae0(T.193in the symbol table). Reduced to what it does:The three stores in the middle are the defect, and it is the first of them that matters.
packet_posat structure offset0x1cis the write cursor into the access unit currently being assembled. Zeroing it while simultaneously settinghead = tailthrows away a part-written access unit along with the queue it was being written into.What reaches the cluster is then a PES packet whose payload simply stops short. The decoder in the 791 AID gets a truncated H.264 access unit, and a truncated access unit decodes to exactly the symptom above: regions of wrong picture that persist until the next clean keyframe replaces them. It is not a transport error and not a bandwidth problem. The stream is malformed at the point it is handed to MediaLB.
The flush itself is reasonable and the recovery design around it is sound. The narrow problem is that it is unconditional: it runs identically whether or not an access unit is mid-write. On a quiet ring it almost never catches one. Under congestion, which is when the flush fires, it catches one often.
At a glance
flowchart LR A["Phone, CarPlay<br>stream type 111<br>H.264 AltScreen"] --> B["Annex-B to PES<br>access units"] B --> C["MPEG-TS<br>188-byte packets"] C --> D["Queue<br>1024 slots<br>head, tail, packet_pos"] D -->|truncated PES| E["Blocks<br>64 x 188 bytes<br>12,032 per write"] E --> F["isoTX2<br>MediaLB"] F --> G["MOST150 ring<br>12.288 Mbit/s"] G --> H["791 AID decoder<br>cluster display"] style D fill:#dbeafe,stroke:#2563eb,stroke-width:3pxEverything up to the queue is correct. The fault is created at the queue and travels onward as valid transport carrying an invalid access unit, which is why it looks like a decoder problem at the cluster end rather than a bug four stages upstream.
The fix: twelve bytes, same length, no relocation
Make the flush conditional on the access unit being aligned. Sixteen bytes at file offset
0x1b04, four instructions in, four out, so nothing moves and no offsets need fixing up.0x1b04mov r3, #0ldr r6, [r4, #0x1c]0x1b08str r3, [r4, #0x1c]cmp r6, #00x1b0cstr r2, [r4, #0x0c]streq r2, [r4, #0x0c]0x1b10str r3, [r4, #0x14]streq r6, [r4, #0x14]As bytes, which is what you would verify against:
Twelve bytes differ in total, across five runs, and the file size is unchanged at 67,835 bytes.
Two things worth pointing out in the replacement. ARM conditional execution does the whole job without a branch, so the control flow of the function is untouched. And
streq r6works as the zero store because on the equal pathr6is already known to be zero, which is what keeps the window at exactly four instructions rather than needing a fifth and a shuffle.The
packet_posstore is gone entirely. It is now only read. So when an access unit is mid-write the flush does nothing at all and the queue is left to drain normally; the next flush, after the access unit closes, behaves exactly as before.One honest caveat. The function still returns
r5, the packet count read before the conditional. On a skipped call that is now the count that would have been dropped rather than the count actually dropped, so any log line or counter fed from the return value will over-report on those calls. Nothing downstream appears to act on it, but it is worth knowing before anyone trusts that number.Second finding: the fresh-IDR request defers exactly when it is needed
This one explains why the artifacts lasted as long as they did, rather than clearing in a frame or two.
When
enter_low_latency_recoveryruns at0x101b20it setsg_waiting_for_keyframe = 1, setsg_discontinuity_pending = 1, clearsg_recovery_idr_requested, removes/ramdisk/carplay-alt-force-idr.request, and logsidr_request_deferred=1. So recovery starts by waiting for a keyframe, and the request for that keyframe is gated separately.The gate is at
main+0x1860:Because
cmphionly executes when the first compare left HI, both conditions must hold: more than 7 blocks completed since the last recovery, and a fast-write streak above 7.That second condition is the trap.
g_writer_fast_streakis reset to zero by any MOST write that takes longer than 30,000 microseconds — themovw r3, #0x7530at0x102354. Under ring congestion, writes routinely exceed 30 ms. So the streak keeps getting knocked back to zero, never reaches 8, andrequest_fresh_idris never called. Recovery sits waiting for a keyframe that nobody has asked for, until the periodic forced IDR comes round on its own schedule.The deferral is sensible in intent. You do not want to spam IDR requests at a struggling encoder. But the specific threshold is defeated by the very condition that triggers recovery, so in practice the request is deferred whenever it is most needed and granted when it is not.
Lowering the streak requirement from 7 to 0 — the single byte at
0x4174— keeps the "some progress since last recovery" condition intact while dropping the demand for a long clean streak that congestion will not allow. Under congestion the streak still reaches 1 or 2 between slow writes, which is now enough.Forced-IDR cadence against VW's own GOP sizing
The second change came from reading what VW had already decided, in
/etc/eso/production/displaymanager.json. The comments are theirs, not added:queue_size18: 210 kBytes => VW ca 200kB | max BPF 40KBnv_frames_per_gopSet FramesPerGOP to 1 to encode each frame as I-frame, else I+P BP encodenv_bytes_per_frameBytesPerGOP = BytesPerFrame * FramesPerGOPnv_channel_data_ratesocket bandwith * MOST rate * byte = 32*48000*8nv_encoder_data_rateshould be less then channel * data rate - (min. <10-20%)nv_maintain_mux_rateTo enable NULL packet stuffing, set to 1The arithmetic in those first three lines is the useful part. Five frames at 40,960 bytes is 204,800 bytes per GOP, and the queue is 210 kB. VW sized the MOST queue to hold almost exactly one GOP. That is a deliberate choice, and it sets the natural granularity for everything downstream of it.
The periodic forced-IDR divisor in
core.sosits at0x4370and0x452c, and stock is0x14, which is 20. With 20 there, the forced keyframe cadence spans roughly four GOPs' worth of frames while the queue only ever holds one. A flush can therefore cost up to a GOP, and the next guaranteed recovery point is up to four GOPs away. That mismatch is what stretched each artifact into something you had time to look at.Setting the divisor to
0x05puts the recovery granularity on the same footing as the queue depth VW designed: one GOP in the queue, one GOP between forced recovery points. It is not a tuning guess, it is the number already in VW's own config.Two footnotes for anyone reading the hashes later. First,
0x2a70in the same file is maxFPS, stock0x28(40), and this car was already running0x1e(30) before any of this work — so the only change attributable to the artifact fix is the divisor. Second, the two data rates confuse easily and both are real at different layers: 12,288,000 is the MediaLB channel bandwidth and 9,600,000 is the encoder's budget inside it, about 22 per cent below the channel, which is exactly the 10–20 per cent headroom VW's comment asks for.Road test
Both changes installed, full power cycle, then driven.
First drive. 5 km including the multi-storey car park that previously triggered artifacts on every single pass. Not one artifact. Nothing on the ramps, nothing at the junctions, nothing on the straight sections, and no unplugging and reconnecting the phone.
Second drive. The same journey again, deliberately, to test it properly rather than take one clean run as an answer. That route includes the roundabout that used to tear the map badly every time, and it was taken twice in the same run. Still nothing.
So the reproducer has now been hit several times over, across two separate drives, on the two stretches of road that previously had a perfect record of breaking it. On this car, the artifacts are fixed.
The map is not perfectly fluid. There is some lag, it does not last long, and it now behaves much as the factory maps do on this unit. For something being encoded, packetised and streamed over MOST to a retrofitted cluster, that is about as good a comparison as you could ask for, and anyone watching the road rather than the screen would not be troubled by it.
Earlier, before either drive, random destinations were set and cleared repeatedly on the driveway with no artifacts, and the forced-IDR change on its own made the map feel noticeably more responsive after a pan.
What makes this convincing rather than lucky is the prior. Both the car park and that roundabout produced artifacts every time, without exception, for as long as the stream has been running to the cluster. Several clean passes through both is not a coincidence.
Builds and hashes
All of these are byte patches of omonob's own binaries at the same size, so a hash is the only reliable way to tell them apart on a card.
altscreen-most-bridgec846e505da704f101815a4658f1b815930d0b6f779d435349b864ef96056c8f3altscreen-most-bridge.AUSAFEfed7dce45f923384fec47861c448b753913f0561ed7e2a279860a5d13f8a44f4altscreen-most-bridge.AUSAFE-Q51225e13b01b4593087f062ce2258f3cd5135905a7fd81055512a78d8ccd119287dcore.so00b48a8fe876e6ff405124b87ed9eace1bcb079220ff7f02b866f42676477a4acore.so.IDR5-30fps3d0238409c7542ee34122ed32b1112926ca519e9e0d175a7cf926fef08991a84The Q512 build is AUSAFE plus the MOST queue halved from 1024 to 512, built as a fallback in case the conditional flush left the queue sitting too full. It was not needed and the queue stays at 1024 in what is running. Keeping it listed in case anyone on a tighter ring wants a starting point.
Three builds to discard
These went out earlier in the investigation and were wrong. If they are on anyone's card they should be deleted, because each one makes this fault worse, not better.
core.so.IDR60-30fps5e060dfa2d22e6cb570f1a1d8b47822911427f942b3cd88b0d1ddd5d29656c06core.so.IDR120-30fps9701dd34445497b8fce38d2ba8f939e1f3bf91c8542066a46918be3e7a2ad048core.so.IDR60-20fpsfd92464d1d5aaccc2b9f9fb4335ca90e9b090662fadef721244da01cc9528fafThe reasoning behind them was that fewer forced keyframes would mean fewer bitrate spikes and therefore less ring congestion. That holds as a bandwidth argument and it is the wrong argument for this fault.
The forced IDR is not just a cost, it is the recovery point. It is the moment a corrupted region gets replaced with correct picture. Raising the divisor lengthens the window a corrupted region survives in, so these builds stretch each artifact from a fraction of a second to somewhere around two to four seconds. They make the thing you are trying to fix considerably more visible.
There is also
core.so.IDR10-30fps(divisor 20 → 10, sha256bbb9d8713906d895…), which is untested here. It sits between stock and the installed build and would be the obvious thing to try on a platform where a 5 divisor turns out to cost too much bitrate. Not a retraction, just not evidence.What this is tested on
Everything above was measured on one car and then driven twice: Golf Alltrack MK7 RHD, MU1367, HMI H29.319.75, ex-Japan MIB2 High converted G11 to G13, with a retrofitted 791 AID 12.3 cluster and a retrofitted Dynaudio J525 amplifier both on the MOST ring. On that car the result is not in doubt. What is untested is everything outside it, and the divisor of 5 is not a general recommendation — it is
nv_frames_per_gopexactly as VW set it in this platform's owndisplaymanager.json, chosen for that reason and then driven.The rest, honestly listed:
displaymanager.jsoncarries a comment line for the other platform:8: 94 kBytes => Audi ca 100kB | max BPF 20.000B. Whether the queue-to-GOP relationship that holds here also holds there is not something this car can answer, and I have not read an Audi config. Anyone porting this should read their ownnv_frames_per_gopandqueue_sizeand work from those, rather than copying a 5 across.0x1b04and0x4174are specific to this build of the bridge — 67,835 bytes, sha256c846e505…. On any other build the flush has to be found by pattern rather than offset. The pattern to look for is the four-store sequencemov r3,#0/str r3,[r4,#0x1c]/str r2,[r4,#0x0c]/str r3,[r4,#0x14]inside the function that takes the queue lock and returns the queued count. The binary ships unstripped, which makes that straightforward.AUSAFE-Q512was built for. No evidence either way yet.IDR10-30fpsexists as the obvious middle step if it turns out to.If any of the above turns out differently on another car, that is more useful than agreement.
All reactions