Join WQP data with the TADA criteria table

[wokenny13]

This is the first PR step for joining the criteria table to the WQP data for analysis. This new function will automatically join the WQP data frame with the TADA compatible criteria table dependent on what has been filled out in the criteria table.

Join WQP results to a criteria table by the best available key:

1. TADA.ComparableDataIdentifier (if present in both and non-NA in criteria)
2. TADA.CharacteristicName + TADA.ResultSampleFractionText + TADA.MethodSpeciationName
3. TADA.CharacteristicName + TADA.ResultSampleFractionText
4. TADA.CharacteristicName + TADA.MethodSpeciationName
5. TADA.CharacteristicName (or when byChar = TRUE)

Users can make the criteria table join only by the TADA.CharacteristicName regardless of what has been filled out in the criteria table.

For testing, please look at the example tests and test out example WQP data retrievals with criteria table filled out. You can use the autoassign method in TADA_DefineCriteriaMethodology() if desired for testing or use a criteria table from the TADACommunityHub.

Ex.

Data_Nutrients_UT2 <- TADA_RunKeyFlagFunctions(Data_Nutrients_UT)
Data_Nutrients_UT_Harmonized <- TADA_HarmonizeSynonyms(Data_Nutrients_UT2)
UTAH_Criteria <- TADA_DefineCriteriaMethodology(Data_Nutrients_UT_Harmonized, org_id = "UTAHDWQ", auto_assign = TRUE, displayUniqueId = TRUE)[[1]]
UTAH_with_criteria <- TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized, UTAH_Criteria)

[github-actions]

coverage-report

File	Coverage		Missing
All files	67%	✅
R/ATTAINSCrosswalks.R	53%	✅	32-876 1102-1106 1111-1116 1121-1124 1129-1160 1166-1169 1175-1178 1192-1195 1206-1221 1267-1270 1514-1515 1522-1523 1528-1529 1556-1563 1694 2018-2021 2026 2055-2059 2071-2074 2082 2091-2098 2113-2122 2127-2135 2141-2164 2184-2199 2204-2207 2212-2243 2256-2259 2332-2336 2431-2440 2656-2657 2726 2897-2903 3124-3706 3874 3877-3882 3900-3903 3941-3953 4002-4140 4144-4147 4210-4211 4247 4323-4329 4448 4525-4527 4531-4533 4569-4614 4671-4673
R/CriteriaAnalysis.R	71%	✅	112 119-132 164-168 252 255 268 278 289 393-395 397 406 419 445-483 508-515 540-567
R/CriteriaMethods.R	96%	✅	312 316-320 336-365 589 753-756 773-776 921-924 933-936 1007-1015 1185-1191 1195-1201 1385-1386 1391-1392 1465-1471 1484 1563 1716-1718 1749 2156
R/TADARefTables.R	91%	✅	75-78 306-308 370-383 394 869-881 969 979-981 995 1016-1018 1023-1040 1054 1205-1209 1402-1408 1425-1438
R/Utilities.R	69%	✅	547 665-666 670 675-677 772 901-912 1056-1058 1126-1127 1181-1185 1295-1296 1300-1301 1311-1315 1320-1325 1367-1522 1613 1669 1677 1737 1745-1765 1791-1792 1801-1815 1839-1842 1906-1907 1924-1926 1931 1934-1936 2020-2098 2129-2131 2167-2335 2362-2366 2375-2379 2596 2649 2655-2658 2662 2673 2692-2693 2695-2710 2712 2714 2720 2723-2725 2727 2762 2777 2783 2791 2796 2804-2809 2838 2848-2850 2871-2875 2885-2887 2890-2895 2899-2904 2912 2922-2923 2934 2950 2982-3016 3131 3140 3179-3183

Minimum allowed coverage is 20%

Generated by 🐒 cobertura-action against 664281d

[cristinamullin]

The ATTAINS and CST uses crosswalk function, TADA_GetTADAUsesAliasRef, needs review/edits. Too many rows are being generated with the auto assign method.

Currently, for the ATTAINS.UseName there is also a UseGroup that these uses would fall under. The CST has an Aquatic Life or Human Health indicator column. If a CST use is NA for that Aquatic Life or Human Health indicator column or if that ATTAINS.UseName is NA, it will get assigned to all unique CST uses. For example, this means many CST uses are getting matched to "Domestic Source" for UT even though they are not all related to domestic source...

To do:

1. Suggestion from Cristina: Greg provided a crosswalk that was manually reviewed. These matches should be labeled as approved.
2. Suggestion from Kenny: The current TADA_GetTADAUsesAliasRef only leverages the CRITERIATYPEAQUAHUMHLTH in its crosswalk. I think additional labeling of these ATTAINS Use Groups with the CRITERIATYPE_WATERORG from the CST could be very useful to reduce this list. Ex. Labeling Domestic Source (which falls under the ATTAINS Use Group of DRINKINGWATER_USE) to W could help narrow this list down.
3. How many are left for manual review after completing 1 and 2 above... anything else we can do to automate the matching so there are less left to review manually?
4. Finally, review and modify the TADAUsesAlias table to specify which rows should be rejected. Make this a a team priority!

[wokenny13]

I compared the "like match" method currently used in TADA vs Greg's table. Greg's table after filtering only had 156 ATTAINS Uses that matched to CST Uses (filtering out NAs, filtering out any changes in ATTAINS.Uses domain as well as CST uses domain.)

From there, the TADA method was able to match 147 out of those 156 matches in Greg's table. The TADA method is looking at % like matches to avoid any manual review needed, and the tolerance is meant to be adjustable. TADA identifies 7,708 close matches out of the 17281 total matches. The remaining 10k matches are based on the Human health/Aquatic Life and Organism & water or water only indicator crosswalk of ATTAINS Use Groups.

2. From adding O & W indicator, this reduced the table from over 30k aliases to 17281 aliases. Reduced table comparison for Domestic Source:

3. We could increase the tolerance to the maximum and have these as "very certain matches" if we would like. These very certain matches would be if there is an exact match of word in any order.

[wokenny13]

@cristinamullin This PR is ready to review. We can work on additional approval or rejection of rows in a future PR by adding ways to differentiate the methods of the uses alias (percent match method vs A/H and O/W categories crosswalk).

The use alias from Greg's table was added as approved and adding on the O/W indicator to the ATTAINS Use Group was added to help reduce the list of potential use alias match to those that are more certain. I added arg inputs to ensure that the csv file does not replace any rows that got approved (reviewing these additional inputs would be nice to test).

[hillarymarler]

So far I have tested your example workflow with some Colorado data using the criteria table from TADACommunityHub. It appears like it all run correctly - I will take a closer look at each of the outputs in the workflow tomorrow or Monday but so far so good as far as a different example from the community hub.

[hillarymarler]

I realized I did not run the final step in your example workflow with the CO data and I am seeing a failure there:

This is where the error comes from (found with rlang::last_trace()):

To reproduce:
`#' # load example data.frame
CO_data <- TADA_DataRetrieval(statecode = "CO",
startDate = "2023-01-01",
endDate = "2023-12-31",
characteristicType = "Physical",
ask = FALSE)
#'
#' # load example criteria table from community hub
criteria_CO <- EPATADA::TADA_GetCriteriaFile(org_id = "21COL001")
#'
#' # join the table by best match from what is filled out from the criteria table
CO_data_criteria <- TADA_Analysis_Join_WQP_Criteria(CO_data, criteria_CO)

#' # create the MLSummaryRef (ML only - no AU or other spatial columns)
CO_params <- TADA_ParametersForAnalysis(
CO_data, org_id = "21COL001", auto_assign = "Org")
#'
CO_uses <- TADA_UsesForAnalysis(CO_data,
org_id = "21COL001", paramRef = CO_params, auto_assign = TRUE)
#'
CO_mlsummary <- TADA_MLSummary(
CO_data,
org_id = "21COL001",
usesRef = CO_uses)
#'
#' # join the table by best match, along with the MLSummaryRef
CO_data_criteria2 <- TADA_Analysis_Join_WQP_Criteria(
CO_data,
CO_data_criteria,
MLSummaryRef = CO_mlsummary)`

I also see the same error if I run it with the example UT nutrients data set with UTDWQ as the org id. Would there be some reason that I would not want to run TADA_Analysis_Join_WQP_Critieria a second time on some data sets? Let's chat about this on Monday during our call and I can continue helping troubleshoot after that if you'd like.

[wokenny13]

#' # join the table by best match, along with the MLSummaryRef
CO_data_criteria2 <- TADA_Analysis_Join_WQP_Criteria(
CO_data,
CO_data_criteria,
MLSummaryRef = CO_mlsummary)`

The argument input for criteria is meant to only be for the criteria table itself and not the data_with_criteria already combined.
So my thought was that it should be filled out as:

CO_data_criteria2 <- TADA_Analysis_Join_WQP_Criteria(
CO_data,
criteria_CO,
MLSummaryRef = CO_mlsummary)`

[hillarymarler]

#' # join the table by best match, along with the MLSummaryRef CO_data_criteria2 <- TADA_Analysis_Join_WQP_Criteria( CO_data, CO_data_criteria, MLSummaryRef = CO_mlsummary)`

The argument input for criteria is meant to only be for the criteria table itself and not the data_with_criteria already combined. So my thought was that it should be filled out as:

CO_data_criteria2 <- TADA_Analysis_Join_WQP_Criteria( CO_data, criteria_CO, MLSummaryRef = CO_mlsummary)`

Thanks for clarifying! Maybe that could be explained in a comment with the example? I'll test again know that I understand what I should be doing there.

[hillarymarler]

The first time I tried with the UT nutrients data (following your instructions to use criteria not the data_with_criteria in TADA_Analysis_Join_WQP_Criteria, no matches were found. I think this is because UT has not entered TADA.ComparableDataIdentifier in their criteria table.

When I set byChar = TRUE, matches are found. In a situation where byChar = FALSE, but the criteria table does not contain any TADA.ComparableDataIdentifier values I would suggest either (1) error with message explaining failure or (2) switch to following byChar = TRUE workflow for the rest of the function with a printed message explaining why.

Additionally, we may want to discuss how a criteria table with some values and some NAs in the TADA.ComparableDataIdentifier column should be handled.

To recreate:
` # load example data.frame
UT_data <- Data_Nutrients_UT

load example criteria table from community hub

criteria_UT <- EPATADA::TADA_GetCriteriaFile(org_id = "UTAHDWQ")

get uses

UT_uses <- TADA_UsesForAnalysis(UT_data,
org_id = "UTAHDWQ", paramRef = UT_params, auto_assign = TRUE)

create ML summary

UT_mlsummary <- TADA_MLSummary(
UT_data,
org_id = "UTAHDWQ1",
usesRef = UT_uses)

join the table by best match, along with the MLSummaryRef

when byChar = FALSE

UT_data_criteria2 <- TADA_Analysis_Join_WQP_Criteria(
UT_data,
criteria_UT,
MLSummaryRef = UT_mlsummary,
byChar =FALSE)

when byChar = TRUE

UT_data_criteria3<- TADA_Analysis_Join_WQP_Criteria(
UT_data,
criteria_UT,
MLSummaryRef = UT_mlsummary,
byChar =TRUE)
`

[wokenny13]

The function will automatically join the criteria table by the best match of what has been filled out from the criteria table. (If TADA.ComparableDataIdentifier is missing, it will proceed with joining by TADA.Characteristic with any combo of fraction or speciation)

I believe the problem arises when using a combination of the criteria table (sourced from the community hub) and the MLSummaryRef table.

Ideally the TADA_CommunityHub should be fully compatible, but if a user does not go through the full recommended Mod 3 workflow and jumps straight to filling out the criteria table, these mismatches occurred. For example, TT filled out the communityhub criteria table using the shortcut method.

If a user has gone through the full work with creating the MLSummaryRef, they should also create the criteria table sourced from the MLSummaryRef

criteria_UT_final <- TADA_DefineCriteriaMethodology(Data_Nutrients_UT, org_id = "UTAHDWQ", MLSummaryRef = UT_mlsummary)[[1]]
criteria_UT_w_CH <- TADA_DefineCriteriaMethodology(Data_Nutrients_UT, org_id = "UTAHDWQ", criteriaMethods = criteria_UT)[[1]]

# compare this with UT_data_criteria3 and other tables
UT_data_criteria4<- TADA_Analysis_Join_WQP_Criteria(
    UT_data,
    criteria_UT_final,
    MLSummaryRef = UT_mlsummary,
    byChar =TRUE)

[wokenny13]

The first time I tried with the UT nutrients data (following your instructions to use criteria not the data_with_criteria in TADA_Analysis_Join_WQP_Criteria, no matches were found. I think this is because UT has not entered TADA.ComparableDataIdentifier in their criteria table.

When I set byChar = TRUE, matches are found. In a situation where byChar = FALSE, but the criteria table does not contain any TADA.ComparableDataIdentifier values I would suggest either (1) error with message explaining failure or (2) switch to following byChar = TRUE workflow for the rest of the function with a printed message explaining why.

Additionally, we may want to discuss how a criteria table with some values and some NAs in the TADA.ComparableDataIdentifier column should be handled.

To recreate: ` # load example data.frame UT_data <- Data_Nutrients_UT

load example criteria table from community hub

criteria_UT <- EPATADA::TADA_GetCriteriaFile(org_id = "UTAHDWQ")

get uses

UT_uses <- TADA_UsesForAnalysis(UT_data, org_id = "UTAHDWQ", paramRef = UT_params, auto_assign = TRUE)

create ML summary

UT_mlsummary <- TADA_MLSummary( UT_data, org_id = "UTAHDWQ1", usesRef = UT_uses)

join the table by best match, along with the MLSummaryRef

when byChar = FALSE

UT_data_criteria2 <- TADA_Analysis_Join_WQP_Criteria( UT_data, criteria_UT, MLSummaryRef = UT_mlsummary, byChar =FALSE)

when byChar = TRUE

UT_data_criteria3<- TADA_Analysis_Join_WQP_Criteria( UT_data, criteria_UT, MLSummaryRef = UT_mlsummary, byChar =TRUE) `

@hillarymarler I took a deeper look. The Criteria table from the CH has a FRESH label for the SaltFresh indicator. The MLSummaryRef does not have any salt indicator, so no matches were found if we were to use the CommunityHub criteria table (as we are joining by those column names below)

The case for byChar = T should not have resulted in any matches either. This has been addressed.

To handle these mismatches from either

1. .data with criteria
2. criteria with MLSummaryRef
   3)* in theory .data with MLSummaryRef should always match (but if modification is made in MLSummaryRef a mismatch could occur)

the plan is to handle this with a validation message error (either a function in EPATADA itself or in the TADACommunityHub package)

unique(criteria_UT$SaltFresh)
[1] "Fresh" NA
unique(UT_mlsummary$SaltFresh)
[1] NA

[hillarymarler]

the plan is to handle this with a validation message error (either a function in EPATADA itself or in the TADACommunityHub package)

unique(criteria_UT$SaltFresh)
[1] "Fresh" NA
unique(UT_mlsummary$SaltFresh)
[1] NA

Have you added the validation message error yet? I re ran the UT example and got the same result (criteria not matched due to SaltFresh indicator) and did not see any message about it.

Would it make sense to give users the option to exclude SaltFresh as a joining column? For example, w/ the UT criteria table all the rows are "Fresh". So I can imagine a scenario where a user goes through mod 2, skips the SaltFresh step (because they know their criteria are only for freshwater) and is then surprised when no criteria are joined because the criteria table does have the SaltFresh column filled.

[wokenny13]

The validation messages will be done as a separate PR (still trying to decide if that is best for a TADACommunityHub validation function or as part of the EPATADA package).

We can allow users to skip the SaltFresh Indicator step if it is not relevant, and therefore exclude it as a column in the final MLSummaryRef output. So in the scenario when the function tries to join the MLSummaryRef and criteria table, it will get joined by all columns except the SaltFresh Indicator. Users will still get a warning message (will be done as a future PR) about a mismatch if a user has filled out the criteria table with a saltfresh indicator but it was not included in the MLSummaryRef. Would that make sense as an approach @hillarymarler?

[hillarymarler]

The validation messages will be done as a separate PR (still trying to decide if that is best for a TADACommunityHub validation function or as part of the EPATADA package).

We can allow users to skip the SaltFresh Indicator step if it is not relevant, and therefore exclude it as a column in the final MLSummaryRef output. So in the scenario when the function tries to join the MLSummaryRef and criteria table, it will get joined by all columns except the SaltFresh Indicator. Users will still get a warning message (will be done as a future PR) about a mismatch if a user has filled out the criteria table with a saltfresh indicator but it was not included in the MLSummaryRef. Would that make sense as an approach @hillarymarler?

That approach makes sense to me and I think will be useful for states without any saltwater criteria.

Other than adding some additional helpful messages (future PR), this all seems to be working well for the examples I've tested.

I am still wondering what would happen in a situation where a user fills out some, but not all TADA.ComparableDataIdentifier values in a criteria table. Would this be caught by a validation test? Would users be prompted to join by TADA.CharacteristicName? Or a message indicating which criteria could not be joined due to no value for TADA.ComparableDataIdentifier?

[wokenny13]

The function will automatically split the criteria table into 5 categories and joins it to the WQP data based on what has been filled out. All rows will get binded at the end. So users are allowed to fill some rows out by the TADA.ComparableDataIdentifier and some rows with just the characteristic name (or by its fraction and/or speciation) and the function will join it based on its best match.

1. Rows that have been filled out by TADA.ComparableDataIdentifier
2. By Char, fraction and speciation
3. By char and fraction only
4. By char and speciation only
5. By char only.

If an invalid char, fraction and/or speciation is provided, a warning message will be returned (does not currently do so but will create a function to return this warning message)