Export metadata and feature data tables - #34
Conversation
Added ability to filter BV-BRC genomes based on genome stat metadata, including CheckM completeness and CheckM contamination as defaults. Also enabled optional checking of genome length, CDS count, and GC content based on deviations from median distribution per bug. Updated genome downloader functions to use ftpes_download_one instead of old ftp_download_one, and moved a dot function helper to inside the ftpes_download_one loop to avoid issues with BiocParallel workers not understanding what that helper is.
Uncommenting a helper that was still needed for several calls. Fixing stuff I broke myself!
Updating to remove BiocParallel implementation that caused bugs. Patching back to prior Future implementation instead. CLI version of genome downloading appears to have issues that may be separate and requires further troubleshooting.
data_processing.R runDataProcessing() missing(ref_file_path) check ignores if the default path is used. Updated to accept the default path if a user doesn't specify rather than erroring at the end of the processing steps.
Updated default workers to 8 for CLI and FTP downloads, made two-try FTP download option the default, added a helper to purge broken file sets (i.e., if .faa is missing for an AccID but .fna and .gff download successfully).
ftpes --> ftps is_complete --> .is_complete
… consolidate on furrr
Cleaning up parallel implementation in data_curation.R, standardizing with furrr and dropping old duplicated within-loop helper function that was needed for earlier BiocParallel implementation. Standardized values for contamination and completeness across functions. Added a parameter in prepareGenomes to allow users to set number of workers. Reduced console clutter for final "out" print.
…to qc_filtering
…to qc_filtering
In data_curation.R, adding ability to export genome stats and QC metadata into human readable CSVs. `load_tables` allows you to run "example <- exportTables("data/Staphylococcus_aureus/Sau.duckdb", load_tables=TRUE)" and import the tables into R so you can play around with them directly. By default, it just saves the tables as exported CSVs. In data_processing.R, added a similar exporter that allows you to save out all the feature data and other DuckDB-connected tables, with the ability to output sequences too, and append AST phenotype data to the feature tables. Tables can be saved out as csv (default), tsv, Parquet, and xlsx. Added `writexl` to Suggests in DESCRIPTION. Fixed some reversions where I broke things in the PR this branch is based on. Will add those same fixes to that PR momentarily to keep things aligned.
Ran data processing workflow for Campylobacter coli. This can be unzipped and used as the test data for exportProcessedData() rather than having to run it all beforehand. Data curation step still requires that prepareGenomes() be run for a test bug, and Staphylococcus argenteus is a good use case for this.
jananiravi
left a comment
jananiravi
left a comment
There was a problem hiding this comment.
Few quick comments. Will test it out tomorrow.
Clarified a few names based on code review feedback.
@epbrenner prepareGenomes + exportTables --> they run locally with Sar. |
|
After running the campy --> 
what's TET doing in |
|
jananiravi
left a comment
jananiravi
left a comment
There was a problem hiding this comment.
is this going to be merged to qc_filtering and then qc_filtering --> main? that might be helpful to avoid duplicate reviews.
| count_if_present("metadata"), | ||
| count_if_present("genome_data"), | ||
| count_if_present("amr_phenotype"), | ||
| count_if_present("metadata_qc"), | ||
| count_if_present("metadata_qc_rejections"), | ||
| count_if_present("filtered"), | ||
| count_if_present("files"), | ||
| count_if_present("bac_data") |
There was a problem hiding this comment.
how do these checks tie back to basic_tables? is the latter not exported? so should "files" be added to basic_tables?
There was a problem hiding this comment.
Refactored to use a default_tables list rather than duplicating this way. There was logic for the checks, but upon careful review, the logic belonged in the garbage.
eboyer221
left a comment
eboyer221
left a comment
There was a problem hiding this comment.
When I tested this, it looks like genome IDs such as 263.1, 263.10, and 263.100 are being written to the CSV as plain numbers rather than text, since they happen to look numeric. When I read that export back (with readr::read_csv(), pandas, or Excel's defaults) all three collapsed to the same value, since trailing zeros after a decimal don't count (263.1 and 263.10 parse as numerically identical). So the three distinct genomes merged under one ID, and it'll happen for any taxon with more than ~9 genomes. I also tried forcing quotes around every value (quote = "all") to see if that would prevent it, but the collision still happened on read. Quoting only protects against commas breaking the file format, it doesn't tell the reader "treat this as text."
This affects every genome_id-keyed table both export functions produce (genome_data, metadata, amr_phenotype_wide, gene_count, etc.).
Two options I see: (1) add a clear warning telling users to force col_types/dtype=str on ID columns when reading CSV/TSV back, or (2) recommend Parquet as the default/safe export format for programmatic reuse, since arrow preserves the real column type - I verified this did not occur with Parquet.
epbrenner
left a comment
epbrenner
left a comment
There was a problem hiding this comment.
Oh I have to submit my own review to my own PR to actually post these comments, huh? Oops.
@epbrenner @AbhirupaGhosh didnt we have this fix in already? |
Co-authored-by: Emily Boyer <130874527+eboyer221@users.noreply.github.com>
Yeah, relatively easy solution, just requires special handling of the columns of interest. Patched in the latest commit. |
4 participants
Testing:
For this PR, I have included bundled test data in
/inst/extdata/Campy_testdata.zip. Extract this into a single directory of your choosing. This bundled data is not intended to the ultimate example data for final package release.For
data_curation.R:Run
prepareGenomes("Staphylococcus argenteus")then
exportTables("data/Staphylococcus_argenteus/Sar.duckdb")For
data_processing.R:Unzip the Campy test data first, then run
exportProcessedData("inst/extdata/Campy_testdata/Cco_parquet.duckdb")There are several thrilling parameters to fiddle around with.
What this does:
Adds the ability to export basic stats into human readable CSVs from data_curation.R workflow.
Adds the ability to export all the processed feature data into a variety of formats from data_processing.R workflow.
What's the point?
Makes amRdata more independently useful to people even if they don't run the full amR suite. For example, you could export feature matrices for custom modeling workflows outside of ours, or just get an idea of what distributions of different QC and AST labels are across different taxa.