A small utility to scrape the European Commission's 'Have your Say' plattform (https://ec.europa.eu/info/law/better-regulation/have-your-say). Can scrape an initiative's feedback submissions, attachments of these submissions, and the by country and by category statistics.
pip3 install hys_scraperTested to work with Python 3.9 on a Linux machine and Google Colab notebooks.
To get started, you will need the publication id of the initiative you want to scrape. To get this, simply navigate to the initiative on 'Have your Say' and look at the URL - the number at the end is the publication id you will use in the next step. For example, for the AIAct commission adoption initiative, the publication id would be 24212003.
To scrape an initiative the following is sufficient (replace 24212003 with the publication id of the initiative you want to scrape):
python3 -m hys_scraper 24212003This will create a new folder in your current working directory with the following layout:
24212003_requirements_for_artificial_intelligence/
├── attachments
│ ├── 2488672.pdf
│ ├── 2596917.pdf
│ └── ...
├── attachments.csv
├── categories.csv
├── countries.csv
└── feedbacks.csv
1 directory, 263 files
The command line interface has a few more arguments. For example instead of having hys_scraper create a folder in the local working directory to save results into, you can also manually specify a target directory.
$ python3 -m hys_scraper -h
Scrape feedback and statistics from the European Commission's 'Have your Say' plattform.
positional arguments:
PID The publication id - what comes after 'p_id=' in the initiative's URL.
optional arguments:
-h, --help show this help message and exit
--dir target_dir, --target_dir target_dir
Directory to save the feedback and statistics dataframes to. Defaults to creating a new
folder in the current working directory.
--no_attachments Whether to skip the download of attachments.
--sleep_time t Minimum time between consecutive HTTP requests (in seconds).
Alternatively, you can also access hys_scraper from Python:
from hys_scraper import HYS_Scraper
feedbacks, countries, categories = HYS_Scraper("24212003").scrape()Similar options are available as for the command line interface, check out help(HYS_Scraper) for details.