Skip to content

Commit ebfaed3

Browse files
committed
Step 9: Fix what the code review found
- Keep 4 significant digits in the relative value, so a report millions of times slower than the claimed one no longer rounds to 0 and breaks the log axis (26 values on run 37571671870); the axis also skips anything at or below 0, and labels tiny values as 1/50M×. - A claim that ties on some Rubies and loses on the rest says it never wins (✗), not "wins on 0 of N" (~). - "the one Ruby" instead of "all 1 Rubies". - A Ruby where the claim ran but could not be compared says "Not compared", not "Did not run". - "How to read a card" lists every icon. - Jump to keeps the chosen section, so arrow keys can move through the list. - Card titles are measured in one pass and again once the web font loads. - Drop an unused helper and a leftover from the find box.
1 parent 60badc3 commit ebfaed3

2 files changed

Lines changed: 27 additions & 17 deletions

File tree

‎script/build_results_site.rb‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,7 @@ def entry(report, name)
178178
def cross_run(report, names, ref_ips)
179179
entries = names.each_with_index.map do |name, i|
180180
e = entry(report, name) or next nil
181-
rel = ref_ips[0].empty? || !e["ips"].positive? || e["ips"] > SUSPECT_IPS ? nil : (e["ips"] / median(ref_ips[0])).round(4)
181+
rel = ref_ips[0].empty? || !e["ips"].positive? || e["ips"] > SUSPECT_IPS ? nil : sig(e["ips"] / median(ref_ips[0]))
182182
[e["ips"].round(3), e["error"].to_f.round(1), rel]
183183
end
184184
{ entries: entries, pass: report["pass"] }
@@ -193,7 +193,7 @@ def reference_run(passes, names, ref_ips)
193193
usable = es.select { |e| e["ips"].positive? && e["ips"] <= SUSPECT_IPS }
194194
pick = usable.empty? ? es : usable
195195
ips = median(pick.map { |e| e["ips"] })
196-
rel = ref_ips[0].empty? || ref_ips[i].empty? ? nil : (median(ref_ips[i]) / median(ref_ips[0])).round(4)
196+
rel = ref_ips[0].empty? || ref_ips[i].empty? ? nil : sig(median(ref_ips[i]) / median(ref_ips[0]))
197197
[ips.round(3), median(pick.map { |e| e["error"].to_f }).round(1), rel]
198198
end
199199
{ entries: entries, passes: passes.map { |r| r["pass"] }.sort }
@@ -208,6 +208,11 @@ def variation(ref_ips)
208208
cvs.empty? ? nil : median(cvs)
209209
end
210210

211+
# Four significant digits: a report 27M times slower than the claimed one keeps its value instead of rounding to 0.
212+
def sig(value)
213+
Float(format("%.4g", value))
214+
end
215+
211216
def median(values)
212217
s = values.sort
213218
s.size.odd? ? s[s.size / 2] : (s[s.size / 2 - 1] + s[s.size / 2]) / 2.0

‎script/results_site.html‎

Lines changed: 20 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -313,7 +313,7 @@ <h1>How fast is each idiom on each Ruby?</h1>
313313
<div id="rout"></div>
314314

315315
<div class="rhow" id="rhow">
316-
<p><b>How to read a card.</b> The heading says what happened on the build you picked: ✓ the claimed idiom (the one the benchmark file says is fastest) wins or ties on every Ruby, ~ it wins on some and another idiom wins on others, ✗ another idiom wins everywhere. It compares the claimed idiom with the fastest of the others, and a gap within the measured error is a tie.</p>
316+
<p><b>How to read a card.</b> The heading says what happened on the build you picked: ✓ the claimed idiom (the one the benchmark file says is fastest) wins on every Ruby, or wins on some and ties on the rest; = it ties on every Ruby; ~ it wins on some and another idiom wins on others; ✗ it never wins; – no Ruby could compare them. It compares the claimed idiom with the fastest of the others, and a gap within the measured error is a tie.</p>
317317
<p><b>How to read a chart.</b> The y axis is logarithmic: each gridline step is the same ratio, so a 100× gap between two idioms and a 10% change between two Rubies both show. Under "Relative", every line is divided by the claimed idiom on the reference Ruby, so 1× is that idiom there and the higher line is still the faster one.</p>
318318
<p><b>How this is measured.</b> Each shard of benchmark files ran on one GitHub runner: every build, in a random order, with benchmark-ips' default timing. The numbers are i/s as measured.</p>
319319
<p><b>How steady the machine was.</b> The reference Ruby ran first, again after every few builds, and last. It is the same code every time, so how much its i/s varies is the machine, not Ruby. "Numbers per build" shows that variation (standard deviation over mean). Over 5% the benchmark is marked noisy under its chart, and when the typical benchmark of a run is over 5%, the page warns at the top. Two Rubies whose i/s differ by less than that variation are shown as same-ish.</p>
@@ -653,7 +653,6 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
653653
function esc(s) { return String(s).replace(/[&<>"]/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" }[c])); }
654654
function md(s) { return esc(s).replace(/`([^`]+)`/g, "<code>$1</code>"); }
655655
function nice(l) { const [b, m] = l.split("+"); return b.replace(/^ruby_/, "Ruby ").replace(/^jruby_/, "JRuby ").replace(/^truffleruby_/, "TruffleRuby ") + (m ? " + " + m.toUpperCase() : ""); }
656-
function short(l) { const [b, m] = l.split("+"); return b.replace(/^ruby_/, "").replace(/^jruby_/, "jruby ").replace(/^truffleruby_/, "truffle ") + (m ? " +" + m : ""); }
657656
function fmt(v) { return v >= 1e9 ? (v / 1e9).toFixed(1) + "B" : v >= 1e6 ? (v / 1e6).toFixed(v >= 1e7 ? 1 : 2) + "M" : v >= 1e3 ? (v / 1e3).toFixed(v >= 1e5 ? 0 : 1) + "k" : v.toFixed(1); }
658657
function pct(a, b) { const p = (a / b - 1) * 100; return (p >= 0 ? "+" : "") + p.toFixed(1) + "%"; }
659658
function cpu(c) { return (c || "unknown CPU").replace(/^AMD /, "").replace(/ \d+-Core Processor$/, "").replace(/^Intel\(R\) Xeon\(R\) Platinum (\S+) CPU.*$/i, "Xeon $1"); }
@@ -663,7 +662,7 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
663662
// A result that can be drawn and compared: it ran and is not suspect.
664663
const ok = (e) => !!e && !sus(e) && e[0] > 0;
665664
// Relative needs the reference to have the claimed report.
666-
const val = (e) => ok(e) ? (showing === "rel" ? e[2] : e[0]) : null;
665+
const val = (e) => { const v = ok(e) ? (showing === "rel" ? e[2] : e[0]) : null; return v > 0 ? v : null; };
667666
// Two reports of one build, measured seconds apart: same-ish when the error bars overlap, as benchmark-ips says.
668667
const same = (a, b) => a[0] + a[1] > b[0] - b[1] && a[0] - a[1] < b[0] + b[1];
669668
// Two Rubies, measured minutes apart: also same-ish when the gap is within how much the reference varied on that machine, which is larger than benchmark-ips' error bars.
@@ -736,7 +735,7 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
736735
const res = cols.map((l) => [l, versus(b.runs[l])]).filter(([, v]) => v);
737736
const n = res.length, of = (s) => res.filter(([, v]) => v.s === s);
738737
const wins = of("ok"), ties = of("eq"), losses = of("no");
739-
const all = `all ${n} Rubies${where}`;
738+
const all = `${n === 1 ? "the one Ruby" : `all ${n} Rubies`}${where}`;
740739
let icon = "ok", head;
741740
if (!n) { icon = "na"; head = `Did not run on any Ruby${where}`; }
742741
else if (!losses.length && !ties.length) head = b.names.length === 2 ? `${claim} beats ${nm(1)} on <b>${all}</b>` : `${claim} is the fastest on <b>${all}</b>`;
@@ -752,6 +751,7 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
752751
if (rest) named.push(`others on ${rest} more`);
753752
if (losses.length === n && top.length === 1) { icon = "no"; head = `${nm(+top[0])} beats ${claim} on <b>${all}</b>`; }
754753
else if (losses.length === n) { icon = "no"; head = `${claim} loses on <b>${all}</b>: ${named.join("; ")}`; }
754+
else if (!wins.length) { icon = "no"; head = `${claim} never wins: it ties on <b>${ties.length} of ${n} Rubies${where}</b>; ${named.join("; ")}`; }
755755
else { icon = "mix"; head = `${claim} wins on <b>${wins.length} of ${n} Rubies${where}</b>${ties.length ? ` (${ties.length} tie${ties.length > 1 ? "s" : ""})` : ""}; ${named.join("; ")}`; }
756756
}
757757

@@ -787,8 +787,9 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
787787
}
788788
// Rubies on the chart that are not in the count.
789789
const left = cols.filter((l) => !res.some(([k]) => k === l)), notes = [];
790-
const susp = left.filter((l) => b.runs[l] && b.runs[l].entries.some(sus)), gone = left.filter((l) => !susp.includes(l));
790+
const susp = left.filter((l) => b.runs[l] && b.runs[l].entries.some(sus)), gone = left.filter((l) => !b.runs[l]), odd = left.filter((l) => b.runs[l] && !susp.includes(l));
791791
if (susp.length) notes.push(`Not counted: ${esc(runsOf(susp, cols))}, suspect results (over 1B i/s).`);
792+
if (odd.length) notes.push(`Not compared: ${esc(runsOf(odd, cols))}, an idiom has no result there.`);
792793
if (gone.length) notes.push(`Did not run: ${esc(runsOf(gone, cols))}.`);
793794
return { icon, head, tiles, foot: notes.join(" ") };
794795
}
@@ -817,8 +818,10 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
817818
// Tick labels without trailing zeros: 2M, 500k, 0.05×.
818819
function tickLabel(t) {
819820
const n = (v) => String(+v.toPrecision(3));
820-
if (showing === "rel") return n(t) + "×";
821-
return t >= 1e9 ? n(t / 1e9) + "B" : t >= 1e6 ? n(t / 1e6) + "M" : t >= 1e3 ? n(t / 1e3) + "k" : n(t);
821+
const big = (v) => v >= 1e9 ? n(v / 1e9) + "B" : v >= 1e6 ? n(v / 1e6) + "M" : v >= 1e3 ? n(v / 1e3) + "k" : n(v);
822+
// Far below the claimed idiom, as a fraction: 1/10M× rather than 1e-7×.
823+
if (showing === "rel") return (t < 1e-3 ? "1/" + big(1 / t) : n(t)) + "×";
824+
return big(t);
822825
}
823826

824827
function path(pts, c, extra, w) {
@@ -964,13 +967,12 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
964967
if (showing === "rel") hints.push(`1× is the claimed idiom on ${nice(REF)}.`);
965968
if (mode) hints.push("Dashed: the same Rubies without the JIT, on the same machine.");
966969
$("rhint").textContent = hints.join(" ");
967-
const shown = B.map((b, n) => [b, n]);
968970
const count = {};
969-
shown.forEach(([b]) => { count[b.section] = (count[b.section] || 0) + 1; });
971+
B.forEach((b) => { count[b.section] = (count[b.section] || 0) + 1; });
970972
const secs = Object.keys(count);
971973
$("rjump").innerHTML = `<option value="">A section</option>` + secs.map((x) => `<option value="${esc(slug(x))}">${esc(x)} (${count[x]})</option>`).join("");
972974
let h = "", sec = null;
973-
shown.forEach(([b, n]) => {
975+
B.forEach((b, n) => {
974976
if (b.section !== sec) { sec = b.section; h += `<h2 class="rsec" id="${esc(slug(sec))}">${esc(sec)}<small>${count[sec]} benchmark${count[sec] > 1 ? "s" : ""}</small><span class="on">${esc(MODES.find(([m]) => m === mode)[1])}</span></h2>`; }
975977
const legend = b.names.map((nm, i) => `<span><i style="background:${color(i)}"></i><code>${esc(nm)}</code>${i ? "" : " (claimed)"}</span>`).join("") + (mode ? '<span class="ghost"><i></i>without the JIT</span>' : "") +
976978
(hasSuspect(b) ? '<span class="sus"><svg viewBox="0 0 10 8" aria-hidden="true"><path d="M1,7 L5,1 L9,7Z"/></svg>suspect, over 1B i/s: kept off the chart</span>' : "");
@@ -988,17 +990,21 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
988990
pinned();
989991
}
990992
// A title of many report names stays on one line by getting smaller (20px down to 14px); one too long even at 14px wraps at 14px.
993+
// All writes, then all reads, then all writes: one layout for every card instead of one per card.
991994
function fitTitles() {
992-
$("rout").querySelectorAll(".rcard h3").forEach((h) => {
993-
h.classList.add("one");
994-
h.style.fontSize = "";
995-
const room = h.clientWidth, need = h.scrollWidth;
995+
const hs = [...$("rout").querySelectorAll(".rcard h3")];
996+
hs.forEach((h) => { h.classList.add("one"); h.style.fontSize = ""; });
997+
const fit = hs.map((h) => [h.clientWidth, h.scrollWidth]);
998+
hs.forEach((h, i) => {
999+
const [room, need] = fit[i];
9961000
if (need <= room) return;
9971001
const size = Math.floor(20 * room / need * 10) / 10;
9981002
if (size >= 14) h.style.fontSize = size + "px";
9991003
else { h.style.fontSize = "14px"; h.classList.remove("one"); }
10001004
});
10011005
}
1006+
// A title measured before the web font arrived is measured again once it has.
1007+
if (document.fonts) document.fonts.ready.then(() => { if (!$("view-across-rubies").hidden) fitTitles(); });
10021008
// A section heading reached from "Jump to" lands under the pinned picker, not behind it.
10031009
function pinned() {
10041010
const p = $("rpicker");
@@ -1010,7 +1016,6 @@ <h2 class="bh">FastRuby<span>.io</span> keeps fast-ruby running</h2>
10101016
$("rshow").addEventListener("click", (e) => { const c = e.target.closest(".chip"); if (c) { showing = c.dataset.s; window.SiteViews.setParams({ show: showing === "rel" ? "rel" : null }, true); render(); } });
10111017
$("rjump").addEventListener("change", (e) => {
10121018
const target = e.target.value && document.getElementById(e.target.value);
1013-
e.target.value = "";
10141019
if (target) { pinned(); target.scrollIntoView(); }
10151020
});
10161021
$("rout").addEventListener("toggle", (e) => { const d = e.target; if (d.matches("details")) d.open ? open.add(+d.dataset.n) : open.delete(+d.dataset.n); }, true);

0 commit comments

Comments
 (0)