Skip to content

feat: redesign crawler metrics figures with per-crawl bars on a shared time axis - #52

Open
parismic wants to merge 1 commit into
masterfrom
micha/crawler-metrics-figures
Open

parismic wants to merge 1 commit into
masterfrom
micha/crawler-metrics-figures

Conversation

@parismic

Copy link
Copy Markdown
Contributor

Supersedes #50, which was merged prematurely and has been reverted from master - all review discussion, Sebastian's change requests, and the approvals live there.

  • CrawlDb status and fetch status figures: vertical stacked bars over a datetime x-axis (derived from the crawl label), one fixed-width bar (21 days, the shortest crawl interval) per crawl, single crawls are distinguishable and the irregular intervals between crawls show up as gaps
  • figures keep a fixed landscape size instead of growing with every crawl; both time-axis figures share the same x-range and axes geometry
  • combined crawler metrics figure: unchanged standard dot/line plot, consistent with the other plots on the page (the mixed log-linear scale from the first iteration is dropped)
  • stronger CrawlDb status colors; vertical grid lines removed
  • rewrite plots/crawlermetrics.md as a funnel narrative, document the relation between fetch list, fetch total and the status counts, and explain the success-rate history (outdated seed donations until 2016; robots.txt/HTTP-403 denials and the 2022 exponential backoff recently

Address review feedback on #50:
- draw fixed-width bars (21 days, the shortest crawl interval) per
  crawl over a datetime x-axis, so single crawls are distinguishable
  and the irregular intervals between crawls show up as gaps
- keep the standard dot/line plot for the combined metrics figure,
  consistent with the other plots (no mixed log-linear scale)
- strengthen the CrawlDb status colors and drop vertical grid lines
- rewrite plots/crawlermetrics.md as a fetch funnel narrative and
  explain the success rate history (outdated seed donations until
  2016, robots.txt/HTTP 403 denials and exponential backoff since
  2022)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@sebastian-nagel sebastian-nagel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @parismic. Looks good.

There is a merge conflict related to #51. Likely this requires some additional work to make the new methods write the CSV files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants