A collection of web scraping scripts for collecting researcher and institutional data from different college/university websites, as well as structured data sources like Wikidata and DBLP.
Scripts using Selenium to scrape researcher and departmental data from different college/university websites.
Scripts that use the Wikidata API to query and extract structured researcher and institutional information (affiliations, fields of study, biographical data, etc.).
Scripts that use the DBLP API to fetch publication metadata for researchers.
- Python 3.x
- Selenium (with an appropriate WebDriver, e.g. ChromeDriver/GeckoDriver)
- Internet access for Wikidata and DBLP API calls
Clone the repository and install dependencies:
git clone <repo-url>
cd Web-ScrapperExplore each module — see the comments within each script for usage details specific to that college/data source.
Additional details for each component are documented within the corresponding scripts.