feat(l2g): pathway enrichment features

[addramir]

Adds two opt-in L2G features built on a catalogue of pathways enriched among the genetically associated genes of each Open Targets phenotype.

What the feature is

For a credible set, take every gene whose TSS is within 500 kb of the study locus. Resolve the study to its disease identifiers, collect the pathways enriched for those diseases, and score each gene by

pathwayEnrichment500kb = (pathways containing the gene that are enriched) / (pathways containing the gene that were tested)

The value is 0 for a gene no enriched pathway contains, 1 for a gene whose every pathway is enriched, and it is normalised so that promiscuous genes are not favoured just for belonging to many gene sets.

• pathwayEnrichment500kb — the score above
• pathwayEnrichment500kbNeighbourhood — the same score divided by the largest score at the locus

The second one exists because the raw score is dense rather than sparse: for a typical disease around half of the library's genes belong to at least one enriched pathway, so nearly every gene in a window is non-zero and what separates genes is how they compare with their neighbours.

Neither feature is in the default features_list. They have to be asked for explicitly, and the step raises if the inputs they need are missing.

New datasets

PathwayIndex — the gene sets of a pathway library, one row per pathway. from_gmt parses a GMT file, so the library the enrichment was computed against can be passed straight through.

PathwayEnrichment — the enriched pathways of each disease. Read with an explicit schema, so extra columns in the source (effect sizes, gene set sizes, provenance) are simply not projected.

with_corrected_fdr fills in a missing FDR with a Benjamini-Hochberg step-up value recomputed from the p-values of that disease. This is not cosmetic: in the catalogue this was developed against, 249 of 3,766 diseases have a null FDR on every single row, and those are the gene-rich ones — body mass index, systemic lupus erythematosus, psoriasis, ulcerative colitis, haematocrit among them. Roughly 15% of GWAS studies point at one. Their p-values are intact, so recomputing recovers them; without it an FDR filter deletes them silently. Where the published FDR is present it is kept as is.

Implementation notes

Studies are grouped by their set of diseases rather than handled one at a time. A pathway enriched for more than one of a study's diseases has to count once, which cannot be done from per-disease gene counts, and there are far fewer distinct disease sets than studies (94% of GWAS studies carry exactly one disease). That keeps the pathway-level explosion bounded by the number of distinct sets instead of the number of studies.

Gene symbols in the library are mapped onto Ensembl identifiers through TargetIndex.symbols_lut, which covers approved and obsolete symbols. On the library used here that resolves 16,733 of 17,246 symbols; the remainder are tRNA and immunoglobulin segment names.

The enrichment dataset is read without recursiveFileLookup, since it is partitioned by diseaseId and recursing would stop Spark from reading that partition column back.

Testing

• Unit tests for GMT parsing, gene set explosion, the BH correction (values, monotonicity under ties, and that a published FDR is left alone) and the enriched-pathway filter.
• Feature tests asserting exact scores for a single-disease study, for a study carrying two diseases where the pathway union matters, and for the neighbourhood normalisation.
• Both feature classes added to the feature factory return-type test.
• Smoke tested end to end against the real catalogue and Open Targets 26.03: scores stay within [0, 1], and psoriasis goes from 0 to 426 enriched pathways once the FDR is repaired.

For review

The catalogue ranks genes by the Open Targets genetic_association datatype score, which is itself built largely on L2G output. A feature derived from it therefore feeds L2G predictions back into L2G input: gold standard performance will overstate what the feature adds, and it couples each model to its predecessor. That is why the features are opt-in here, and it is worth settling before either reaches a released model.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds two opt-in L2G features that score genes near a GWAS locus by the fraction of their pathway memberships that are enriched for the study’s disease(s), backed by new pathway library + enrichment datasets (including BH-based FDR repair).

Changes:

• Introduces PathwayIndex (GMT-parsed pathway→gene set membership) and PathwayEnrichment (disease→enriched pathways, with FDR correction).
• Adds pathwayEnrichment500kb and pathwayEnrichment500kbNeighbourhood features to the L2G feature factory and wiring in l2g.py/config.
• Adds unit + feature-level tests and API docs for the new datasets/features.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/gentropy/dataset/test_pathway.py	Adds unit tests for GMT parsing, BH correction, and enriched-pathway filtering.
tests/gentropy/dataset/test_l2g_feature.py	Adds feature tests for pathway enrichment scoring and neighbourhood normalization; updates factory return-type test.
tests/gentropy/conftest.py	Adds fixtures for mock pathway index and enrichment datasets.
src/gentropy/method/l2g/feature_factory.py	Registers the two new pathway features in the factory mapping.
src/gentropy/l2g.py	Loads optional pathway inputs and enforces required inputs when pathway features are requested.
src/gentropy/dataset/pathway_index.py	New dataset for pathway library membership with GMT parsing and gene-membership explosion.
src/gentropy/dataset/pathway_enrichment.py	New dataset for disease-pathway enrichment with corrected-FDR logic and enriched-pathway extraction.
src/gentropy/dataset/l2g_features/pathway.py	Implements pathway enrichment feature logic + neighbourhood normalization.
src/gentropy/config.py	Adds config paths for pathway library and enrichment inputs.
src/gentropy/assets/schemas/pathway_index.json	Adds schema for PathwayIndex.
src/gentropy/assets/schemas/pathway_enrichment.json	Adds schema for PathwayEnrichment.
docs/python_api/datasets/pathway_index.md	Adds API docs page for PathwayIndex.
docs/python_api/datasets/pathway_enrichment.md	Adds API docs page for PathwayEnrichment.
docs/python_api/datasets/l2g_features/pathway.md	Adds API docs page for the pathway enrichment features and shared logic.
docs/assets/schemas/pathway_index.md	Adds rendered schema doc snippet for PathwayIndex.
docs/assets/schemas/pathway_enrichment.md	Adds rendered schema doc snippet for PathwayEnrichment.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.