Extending output checking guidance to make requests easier to review

[LFISHER7]

Below are some behaviours that make output checking more difficult that we should discourage in the docs.

• Incomplete context. Some of the context descriptions I see only make sense if you have prior knowledge about the project. We should extend this list or context requirements to make it clear that the context should make the request understandable to someone not familiar with the project

• Not using file groups. Some requests group all files into a single group and then write describe groups of files within a single, long context. As a reviewer, this means I have to describe what subset of files i'm referring to when I make any comments. If the files were appropriately grouped in the first place, this would be obvious.

• Producing wide tables that don't have to be wide. Wide tables where all of the columns don't fit into view are difficult to review. Sometimes this is unavoidable, but there are times where it's not necessary. For example:

  	count_numerator_age_18_39	count_denominator_age_18_39	count_numerator_age_40_59	count_denominator_age_40_59	count_numerator_age_60_plus	count_denominator_age_60_plus	count_numerator_sex_F	count_denominator_sex_F	count_numerator_sex_M	count_denominator_sex_M	count_numerator_imd_1	count_denominator_imd_1	count_numerator_imd_5	count_denominator_imd_5
  value	120	1000	340	2500	510	3000	480	3200	490	3300	95	800	140	900

  This is easier to check (visually, and with the built-in summary stats) if it's structured like:

  demographic	category	count_numerator	count_denominator
  age	18–39	120	1000
  age	40–59	340	2500
  age	60+	510	3000
  sex	F	480	3200
  sex	M	490	3300
  imd	1	95	800
  imd	5	140	900

[alschaffer]

Incomplete context. Some of the context descriptions I see only make sense if you have prior knowledge about the project. We should extend this list or context requirements to make it clear that the context should make the request understandable to someone not familiar with the project

Is there a way that the project summary (i.e. what's on the project page) could be pulled into Airlock? Would that help do you think?

[LFISHER7]

Yes, that's a good idea, although they're light on methods. The applications may be more detailed.

[marrpesce]

• Avoid combining multiple statistics into a single table cell (e.g. 45.3% (453/1000)). Present percentages, numerators, and denominators in separate columns wherever possible so that each value can be checked independently.

• Avoid presenting multiple tables when a single table would suffice. For example, rather than producing separate tables for overall results and each subgroup (e.g. overall, IMD, ethnicity), combine them into a single long-format table with an additional column identifying the subgroup. Exceptions may be necessary when a single table exceeds the output row limit.

[alschaffer]

Re: context, if there are minimum pieces of info that we agree that all output requests require, we could turn those into fields. E.g.,

• Study purpose:
• Population:
• Variable descriptions:
• etc.

Same for disclosure control, we could have:

• Small counts redacted? yes/no and to what value, and if not why not?
• Counts rounded? yes/no and to what value, and if not why not?
• Rates / % calculated from rounded counts?

I don't necessarily want to be too prescriptive, and it might get complicated for different types of outputs, but it might make it easier for researchers (in addition to reviewers) if it's clearer what is expected of them.

[emprestige]

it would be useful to be able to be able to perform column-wide calculations, e.g. to check two columns form the percentage/proportion reported next to them