A production-oriented Python automation toolkit for enriching journal metadata from publisher websites into Excel workbooks. The current implementation supports two sources:
- TandF Online for journal metadata, publication history, and institutional pricing
- IndianJournals for journal metadata and pricing from their public journal pages
This project reads an Excel workbook containing journal titles and/or ISSNs, searches the relevant publisher websites, extracts metadata, and writes a new workbook with updated journal fields. It is intended for batch processing of large journal lists with minimal manual intervention.
For each row in the input workbook, the scraper can populate the following fields:
- Month
- Start Year
- Last issue year
- Selected Year
- Volume
- Number of issues in year
- Volume Per Anumn
- Currency
- Price
- Remarks
- Remarks 2
- Publication History
- Reference Link
The data is written to a new Excel file named <input>_updated.xlsx.
| File | Description |
|---|---|
| TAndFrancies.py | Selenium-based scraper for TandF Online |
| TAndFrancies_gui.py | Tkinter GUI wrapper for the TandF scraper |
| update_journal_excel.py | Requests/BeautifulSoup scraper for IndianJournals |
| build_exe.spec | PyInstaller spec for packaging the GUI app into an EXE |
| build_exe.bat | One-click Windows build script |
| README.md | Project documentation |
- Python 3.10+
- pandas for Excel I/O
- openpyxl for workbook support
- selenium + undetected_chromedriver for browser-based scraping
- beautifulsoup4 for HTML parsing
- requests for HTTP-based scraping
- tkinter for the desktop GUI
- pyinstaller for executable packaging
- Windows 10/11 recommended
- Google Chrome installed
- Python 3.10 or newer
The TandF scraper is configured with:
CHROME_VERSION = 150in TAndFrancies.py
If your installed Chrome version changes, update this value accordingly.
Install the required packages with:
pip install undetected-chromedriver selenium beautifulsoup4 pandas openpyxl requests pyinstallerpython -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txtIf a requirements file is not present, install the packages manually as shown above.
python TAndFrancies.py journal.xlsxThe workbook should contain at least the following identifiers:
| Column | Required | Description |
|---|---|---|
| publication_title | Yes | Journal title used for matching |
| ISSN | Recommended | ISSN used as a secondary lookup key |
The script will also ensure the following output columns exist if they are missing:
- Month
- Start Year
- Last issue year
- Selected Year
- Volume
- Number of issues in year
- Volume Per Anumn
- Currency
- Price
- Remarks
- Remarks 2
- Publication History
- Reference Link
If the scraper cannot find new data for a row, existing values are preserved.
Run the command-line version:
python TAndFrancies.py journal.xlsxOptional output filename:
python TAndFrancies.py journal.xlsx output_file.xlsxpython update_journal_excel.py journal.xlsxpython TAndFrancies_gui.pyThe GUI provides:
- Excel file selection
- Start / Stop processing
- Live log output
- Partial save support when processing is interrupted
The TandF scraper uses a multi-step search strategy:
- Title search through the public journal listing
- ISSN-based search via the search endpoint
- About-page ISSN verification for unlisted or difficult matches
- Journal information lookup for discontinued journals
After a journal is matched, the scraper extracts:
- ISSNs from the about page
- Publication history from the LOI page
- Volume and year data from volume buttons
- Issues per year from issue links
- Institutional USD pricing from the pricing page
Important behaviors:
- Combined issues such as
Issue 3-4are counted as 2 - Journal code redirects are handled automatically
- Cloudflare challenges are managed using undetected-chromedriver
- ISSN values with an
Xsuffix are matched correctly
The IndianJournals scraper:
- Searches by title and ISSN
- Verifies the matched journal using the page content
- Extracts pricing and metadata from the journal page
- Writes the enriched data back to Excel
The output workbook is saved as:
<input>_updated.xlsx
Example:
python TAndFrancies.py journal.xlsxProduces:
journal_updated.xlsxbuild_exe.batpip install pyinstaller
python -m PyInstaller build_exe.specThis produces:
dist\TandF_Journal_Updater.exe
This project is functional for automation, but it is still dependent on the structure of publisher websites. Some limitations to keep in mind:
- Publisher pages may change layout and break selectors
- Cloudflare or anti-bot protection may require browser-based retries
- Matching quality depends on the quality of the input title and ISSN data
- Some journals may require manual review when no strong match is found
For production use, it is recommended to:
- run the scraper in a controlled environment
- keep Chrome updated to the version expected by the code
- validate the output workbook before bulk processing
- log failures and unmatched rows for review
If Selenium fails or browser launch breaks, verify the Chrome version and update CHROME_VERSION in TAndFrancies.py.
If Python reports import errors, reinstall the dependencies:
pip install undetected-chromedriver selenium beautifulsoup4 pandas openpyxl requests pyinstallerIf a journal is not found:
- check whether the journal title is spelled correctly
- verify the ISSN is present and correctly formatted
- review the scraped log output for the row
Ensure the input Excel file exists and is accessible. The script expects a valid .xlsx file path.
The project is a practical batch-data enrichment pipeline for journal metadata collection. It combines browser automation, HTML parsing, Excel processing, and a desktop GUI into a single workflow that can be run locally or packaged as a Windows executable.