Feat/Download domains-top-1000.csv from s3

[handecelikkanat]

As part of the task to prepare and publish domains-top-N.csv files as part of the crawl pipeline, some processing that was previously done here is moved to main pipeline.

In particular, main pipeline now takes over:

• Preparing domains-top-1000-extended.csv files, which include the features domain, pages, urls, hosts.
• Shares these files to here, together with other files that include crawl statistics.

cc-crawl-statistics then:

• Read also the new file in get_stats.sh
• Pipes size excerpt to plot/domain.py in plot.sh
• In domain.py combined the features from the read file + calculates and combines %pages and %urls features (using the size information from the argument excerpt)
• Publishes the resulting file in domains.html

Decisions, please check:

• Intentionally does not change stats/{crawl}.gz files (ie. does not change crawlstats.py and run_stats_hadoop.py, to ensure compatibility with previous files.
• Intentionally does not remove building of stats/excerpt/domain.json.gz, in case another downstream user needs it.

Validation:

I tested this setup using the Docker container and a test csv file that I upload beforehand to the test bucket.

@sebastian-nagel I left this file there so you too can test if you like using this.

Some settings are needed:

0. If CC-MAIN-2026-39 is the latest crawl in crawlstats.py, comment it out. Use August 2026 crawl (CC-MAIN-2026-34) as latest instead.

1. I had to add a || true at the end of line that reads from AWS each top-domains file, because for test purposes I have only one. This I will not commit to the eventual version because Id need the script to fail on prod.

We can make this a command line argument, but didnt look worthy to bog down the script to me.

Thoughts?

ie.:

     if [ -e ${TOP_DOMAINS_TARGET} ]; then
         echo "  ... top-domains exist"
     elif $ON_AWS; then
-        aws s3 cp ${TOP_DOMAINS_S3}/$crawl/stats/${TOP_DOMAINS_FILE} ${TOP_DOMAINS_TARGET}
+        aws s3 cp ${TOP_DOMAINS_S3}/$crawl/stats/${TOP_DOMAINS_FILE} ${TOP_DOMAINS_TARGET} || true
     else
         curl --silent --fail ${TOP_DOMAINS_URL}/$crawl/stats/${TOP_DOMAINS_FILE} -o ${TOP_DOMAINS_TARGET} || { rm -f ${TOP_DOMAINS_TARGET}; false; }
     fi

2. Then run Docker, TOP_DOMAINS_S3 must be overriden to the test location that crawl-tools test uploaded the temp test files. eg.:

docker run --rm -it \
    -e TOP_DOMAINS_S3=s3://<test-bucket>/<test-prefix> \
    -v ~/.aws:/root/.aws:ro \
    -v $(pwd -P):/app \
    ghcr.io/commoncrawl/cc-crawl-statistics/stats:latest bash

3. Then in Docker shell:

$ echo $TOP_DOMAINS_S3    ### Sanity check
<...>

$ aws s3 ls $TOP_DOMAINS_S3/CC-MAIN-2026-34/stats/domains-top-1000-extended.csv  ### Sanity check
<...> domains-top-1000-extended.csv

$ ./get_stats.sh
# This will skip many failed files, check only for CC-MAIN-2026-34

$ ls stats/top-domains/   
CC-MAIN-2026-34.domains-top-1000-extended.csv

$ ./plot.sh
# Check and compare values with current crawl values at `https://commoncrawl.github.io/cc-crawl-statistics/plots/domains.html`
#    - plots/domains.md
#    - plots/domains-top-1000.csv
#    - plots/domains-top-1000.html

4. In a local shell, can view the generated files using the site.Dockerfile, which will redirect output to http://localhost:4000/

docker build -f site.Dockerfile -t ccsite .
docker run --rm -p 4000:4000 ccsite bundle exec jekyll serve --host 0.0.0.0

[handecelikkanat]

@sebastian-nagel I intend to not touch crawlstats.py and run_stats_hadoop.py for now. They will continue calculating the statistics for top 500 domains (which can be updated to 1000?)

I will update plot/domain.py to use data from my own files rather than from crawl.gz files. Feels like less amount of change in one go.

Let me know if you prefer to update everything all at once. I thought if current system fails, its good to have the old one also running in parallel for another month. Plus I think some other stats calculations use these domain numbers, which I dont want to mess with.

Opinions?

[sebastian-nagel]

Thanks, @handecelikkanat. Looks good.

Don't forget: the top-500 domain files (CSV and HTML) need to be removed from Git and replaced by the top-1000 files. This should be part of the PR.