Problem
MHC needs a certian minimum amount of available disk space to function properly, since the HealthKit/SensorKit background data collection features first encode and compress ingested data into zip files, which are then scheduled for eventual upload (during later background sessions), but stay on device until then.
the exact amount here is difficult to say, as it depends on how much data the user generates over time.
for example, if someone has SensorKit disabled, and doesn't have a watch, they will generate significantly less data than someone with a watch and with SensorKit enabled.
the big issue here is that if a user generates a lot of data, and the rate at which the app ingests the data is larger than the rate at which its uploads succeed, the app will effectively fall into a death spiral where the local upload buffer (i.e., the staging directory containing the zip files that should be uploaded) will keep growing in size over time, and the app can't keep up with actually uploading the data.
Solution
we should address and fix this, somehow
at the very least, we should add some kind of simple report mechanism to the app that lets us know what the average user's total size of staged uploads is, and also that somehow lets us know how many (if any) users actually end up in the above-described data ingestion and upload death spiral.
we then should consider the following:
- reduce upload batch size if the issue is that the individual upload files are too large for the device to handle in a single background session (i.e., if each launch will restart the uploads but never actually manages to complete them bc they're too large (and there's too many of them) for that any single one of them could complete w/in the background session)
- limit the number of concurrent file uploads (we currently don't have a limit and effectively delegate this to firebase, hoping that it will deal with this correctly)
- add, in addition to the regular "upload file" step we already have, additional steps (before initiating and after completing the upload), that inform the server that the app is about to start uploading some specific file for a user, and that the app completed the upload
- the server would not keep these data points around for too long, but it could use them to get a picture of how many failed downloads there are, and that might help us figure out how to best fix this / work around it
additionally, we might even want to add a mechanism that disables HealthKit data ingestion entirely if the device's storage is running too low (or if MHC itself is above a certain threshold; this would arguably be a better approach here).
in this case, the app would simply keep the HealthKit query anchors at their old value, which would essentially buy it time to process through the backlog of pending uploads before starting to schedule additional new uploads.
this would only work if there is a guarantee that the app can somehow churn through the backlog; otherwise it would effectively disable health data donation for the user (though arguably not having such a feature also will effectively end up disabling data sharing for the user)
ultimately, the way the app's data ingestion and upload system should work, is that over time (e.g., over a 5-day period) it should always upload, without issues, all data it ingested, and it should never end up in a situation where it enters this spiral of doom caused by ingesting data quicker than it can upload it
Additional context
there's a lot of guesswork and speculation in the text above; the purpose of this issue is more to start a general investigation into whether this even is an issue to begin with (and if yes to which extent and under which circumstances); we then should derive concrete steps from the insights gained
Code of Conduct
Problem
MHC needs a certian minimum amount of available disk space to function properly, since the HealthKit/SensorKit background data collection features first encode and compress ingested data into zip files, which are then scheduled for eventual upload (during later background sessions), but stay on device until then.
the exact amount here is difficult to say, as it depends on how much data the user generates over time.
for example, if someone has SensorKit disabled, and doesn't have a watch, they will generate significantly less data than someone with a watch and with SensorKit enabled.
the big issue here is that if a user generates a lot of data, and the rate at which the app ingests the data is larger than the rate at which its uploads succeed, the app will effectively fall into a death spiral where the local upload buffer (i.e., the staging directory containing the zip files that should be uploaded) will keep growing in size over time, and the app can't keep up with actually uploading the data.
Solution
we should address and fix this, somehow
at the very least, we should add some kind of simple report mechanism to the app that lets us know what the average user's total size of staged uploads is, and also that somehow lets us know how many (if any) users actually end up in the above-described data ingestion and upload death spiral.
we then should consider the following:
additionally, we might even want to add a mechanism that disables HealthKit data ingestion entirely if the device's storage is running too low (or if MHC itself is above a certain threshold; this would arguably be a better approach here).
in this case, the app would simply keep the HealthKit query anchors at their old value, which would essentially buy it time to process through the backlog of pending uploads before starting to schedule additional new uploads.
this would only work if there is a guarantee that the app can somehow churn through the backlog; otherwise it would effectively disable health data donation for the user (though arguably not having such a feature also will effectively end up disabling data sharing for the user)
ultimately, the way the app's data ingestion and upload system should work, is that over time (e.g., over a 5-day period) it should always upload, without issues, all data it ingested, and it should never end up in a situation where it enters this spiral of doom caused by ingesting data quicker than it can upload it
Additional context
there's a lot of guesswork and speculation in the text above; the purpose of this issue is more to start a general investigation into whether this even is an issue to begin with (and if yes to which extent and under which circumstances); we then should derive concrete steps from the insights gained
Code of Conduct