Skip to content

Add get_archival_data() function #409

Open
Feature
3 of 3 issues completed
#589
Open
Add get_archival_data() function#409
Feature
3 of 3 issues completed
#589
Assignees
PietrH
Labels
enhancementNew feature or request
Milestone
STRAITS 2026

Description

@PietrH
PietrH
opened on Oct 1, 2025

STRAITS

Add a new function get_archival_data() to retrieve processed tag archival data. The user can define tag (tag_serial_number) and data aggregation level (summarized or all). The function retrieves relevant data (files) and returns it to the user as a single data frame.

summarized: only return data from whole hours. Drop all other data points. I could easily add support for other steps in aggregation. Are we sure no averaging or similar takes place?

  • fetch csv's and combine them
  • fetch them where?
  • decompression
  • how does the user select the archival data, on what parameters?
  • Is a etn database query necessary? Or can it be done purely client based?
  • What postgres table contains the links to the csv files?
  • What is the best way to programmatically read the csv files without having to manage an authentication cookie.
  • Read csv with readr or via httr2? -> httr2 for throttling and retries
  • Add lazy evaluation support via duckdb? arrow
  • Fix bug: col classes need to match while merging: add schema?
`purrr::list_rbind()` at etn/R/get_archival_data.R:49:3:
! Can't combine `..1$timestamp_utc` <datetime<UTC>> and `..2$timestamp_utc` <character>.
  • Adjust throttling
  • BUG: running out of working memory when loading multliple large datasets into memory for parsing; Store onto disk and parse one by one?
  • Add progress bar argument
  • Idea: Break up progress in two steps: fetching data, reading/parsing csv
  • Idea: Fetch files sequentially (big files saturate network connections), and use httr2::req_progress() to show downloaded bytes.
  • Add aggregation and aggregation argument: via duckdb? need to group by the floor of the datetime, and then aggregate. duckdb is much faster. Let's not take this as extra maintenance for now.
  • Test on animal_project_code case sensitivitly
  • Test on invalid values for animal_project_code
  • Add caching via httr2 for return values: very stable
  • Add tests for col classes: hard to predict what cols we'll get back
  • Add limit argument for tests?
  • Add argument so users can store csv files: path or dir
  • Allow partial resume from previously downloaded files
  • Name the local files: uuid for now
  • BUG: case sensitivity in animal_project_code: BUG: animal_project_code is case sensitive in etnservice::get_archival_data_uuid() etnservice#154 (comment)

Minimal Viable Product

  • Describe changes in NEWS
  • Add extension to local files
  • Deploy etnservice::get_archival_data_uuid() to OpenCPU production
  • Require etnservice <= v0.7.0
  • Add test skips on data domain offline and on no credentials
  • Add documentation of returned object, what cols to expect
  • Decide: Should invalid values for arguments be checked on the client or via the service? Currently all on service, is this robust?, to client, but postponed see Check for invalid values on the client side #588
  • Offer friendly abort option when suggesting out of memory call, similar to usethis::ui_yeah()

Postponed/Extend goals

  • Should users be able to read local files without a database query? Without database credentials?
  • Don't allow tibble return on very large datasets? How to efficiently count rows of returned csv's? Decide purely on filesize?
  • Stop users from trying to read all archival data available to them, this will be millions of records
  • Add local caching of uuid values? very stable: cachem over memoise
  • return helpful error if a passed path (dir) doesn't exist
  • Abstract opening arrow dataset from online hosted files: reusable for public data read from disk in this implementation
  • Abstract downloading a file with package defaults (declaring user agent, retries, path handling): can be reused for public data, and for get_acoustic_detections() feather and parquet files
  • rename temp_file_paths: not always temporary files
  • Abstract out extrating a uuid from a filepath: more readable
  • tests: cache http responses?: faster, more consistent, less load on VLIZ, but possibly unacceptably increases package size
  • Perform requests in parallel?

Questions

How to detect if a project has archival data -> Will be useful for get_package(), it'll sometimes need to get achival data, how to know that you need to get it.

Requirements

  • Add a new function get_archival_data() to retrieve processed tag archival data.
  • The user can define tag (tag_serial_number) ~~and data aggregation level (summarized or all). ~~ -> users can aggregate themselves for now, if there is demand we can add aggregation functions later on. You'd need to read all the data anyway.
  • The function retrieves relevant data (files) and returns it to the user as a single data frame.

Activity

  1. PietrH
    added this to the STRAITS 2026 milestone on Oct 1, 2025
  2. PietrH
    self-assigned this
    on Oct 1, 2025
  3. PietrH
    added the
    Feature
    issue type on Oct 1, 2025
  4. PietrH
    added
    enhancementNew feature or request
    on Oct 1, 2025
  5. PietrH
    marked this as blocking on Oct 1, 2025

  6. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor
    get_archival_data <- function(tag_serial_number = NULL, data_aggregation_level = c("all", "summarized")){}

  7. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor

    Status 20260203: etn development roadmap, document by Lotte

    • What: data pipeline for archival data collected by Data Storage Tags.
    • How: Data will be stored as raw (original files), processed (readable files) and summary
      (database table). Summary data are a subset of the raw data.
    • Structure: similar to Darwin Core Measurements or Facts:
    • date_time
    • tag_serial_number
    • sensor_type
    • sensor_value
    • sensor_unit
    • Scope: STRAITS, Deadline: December 2026

  8. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor

    Notes form discussion with Peter 20250606:

    get_archival_data() is a difficult one, the raw data fetching should be pretty easy. Because the link to these files is present in the database. They should always be in the same format. Then convert to data.frame and return to the user.

    Aggregated data will be more difficult, because it's calculated with a query with a mongo-db database. Package will need to be able to run a mongo_db query, not sure how this'll go. = currently blocked.

    The mongo-db also contains all of the processed data, so we might not need to go via the file download at all.

  9. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor

    Notes from ETN meeting 20250909

    Archival data: first release in production: only works for cfas. 

summary is calculated when requested, a query on the mongo db

the standardized csv is available, stored in mongo. Mongo -> easy path for parquet export. Mongo is built in same partitioning as the parquet files will be.

contract: 
 - add function r package: to get processed tag archival data (not summary data), tag, tag_serial_number, from contract: 
 - > Add a new function get_archival_data() to retrieve processed tag archival data. The user can define tag (tag_serial_number) and data aggregation level (summarized or all). The function retrieves relevant data (files) and returns it to the user as a single data frame.
 -  Vision: query postgres database tags one by one, then combine them one by one. If the user requests summarized data: query mongo db database via the R package.
 - the summaries take around 30 seconds to be calculated via the mongodb at the moment
 - Peter doesn't think the R package should talk to mongo-db
 - there is no summary file, they are calculated on the spot. Stijn: it's possible to calculate them and store them, but how useful would that be?

  10. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor

    Does a user need to log into MDA (marine data archive: https://mda.vliz.be/archive.php) to download the csv files?

  11. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor

    Does a user need to log into MDA (marine data archive: https://mda.vliz.be/archive.php) to download the csv files?

    No, I think these can be fetched from: "https://www.lifewatch.be/etn/archival-data/", files are selected via a GUID: eg: https://www.lifewatch.be/etn/archival-data/file/1980c349-b4df-4b58-a95d-0a6c77a31fc9

    But an authentication cookie is passed in the GET request, so you need to login to access this data.

    Is there a way around this? What is the preferred way of authentication on the ETN data platform?

  12. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor

    Let's not do data aggregation to start with. Mongo-db could aggregate, but let's not depend on mongo-db. We are not sure it'll still be here on the long term.

    The user can aggregate themselves.

  13. PietrH commented on Mar 19, 2026

    @PietrH
    PietrH
    on Mar 19, 2026
    MemberAuthor

    Only support one tag_serial_number at a time, supporting a vector makes it too easy to ask for much much more data than you'd be able to process. Even a single tag is already 1GB+ of data.

    Users can call the function in a loop.

  14. Stijn-VLIZ commented on Mar 26, 2026

    @Stijn-VLIZ
    Stijn-VLIZ
    on Mar 26, 2026
    Collaborator

    Does a user need to log into MDA (marine data archive: https://mda.vliz.be/archive.php) to download the csv files?

    No, I think these can be fetched from: "https://www.lifewatch.be/etn/archival-data/", files are selected via a GUID: eg: https://www.lifewatch.be/etn/archival-data/file/1980c349-b4df-4b58-a95d-0a6c77a31fc9

    But an authentication cookie is passed in the GET request, so you need to login to access this data.

    Is there a way around this? What is the preferred way of authentication on the ETN data platform?

    ETN can make these links unprotected by login, but we will need to add checks if people are allowed to retrieve certain files, question remains how we will do that...

  15. PietrH commented on Mar 26, 2026

    @PietrH
    PietrH
    on Mar 26, 2026
    MemberAuthor

    Does a user need to log into MDA (marine data archive: https://mda.vliz.be/archive.php) to download the csv files?

    No, I think these can be fetched from: "https://www.lifewatch.be/etn/archival-data/", files are selected via a GUID: eg: https://www.lifewatch.be/etn/archival-data/file/1980c349-b4df-4b58-a95d-0a6c77a31fc9
    But an authentication cookie is passed in the GET request, so you need to login to access this data.
    Is there a way around this? What is the preferred way of authentication on the ETN data platform?

    ETN can make these links unprotected by login, but we will need to add checks if people are allowed to retrieve certain files, question remains how we will do that...

    If there is some sort of secret or token I can pass with the request, great, but we need to correctly estimate what the chances are a bad agent can guess the uuid of a file they might be interested in. Personally I believe this to be quite unlikely, but not impossible.

    I suppose that depends on when/on what system the uuids were generated, but generally they are not considered a cryptographically safe token.

    I'll think about a solution.

  16. 8 remaining items

  17. PietrH commented on May 5, 2026

    @PietrH
    PietrH
    on May 5, 2026
    MemberAuthor

    Is it always?

    • timestamp_utc : datetime
    • measurement_type : string
    • measurement_value : double precision floating point
    • measurement_unit : string

  18. PietrH commented on May 7, 2026

    @PietrH
    PietrH
    on May 7, 2026
    MemberAuthor

    I've come across a tag_serial_number: A09119 that results in a csv file that only has a header, no rows.

  19. CLAUMEMO commented on May 7, 2026

    @CLAUMEMO
    CLAUMEMO
    on May 7, 2026
    Collaborator

    Is it always?

    * timestamp_utc : datetime
* measurement_type : string
* measurement_value : double precision floating point
* measurement_unit : string

    Yes, they should be constant.

    I've come across a tag_serial_number: A09119 that results in a csv file that only has a header, no rows.

    @Stijn-VLIZ maybe we should exclude from digital_twin.archival_files files that gave an error: https://www.lifewatch.be/etn/conversionlogs/253

  20. Stijn-VLIZ commented on May 11, 2026

    @Stijn-VLIZ
    Stijn-VLIZ
    on May 11, 2026
    Collaborator

    Good Idea, view was changed.

  21. PietrH
    added a sub-issue on May 12, 2026

  22. PietrH commented on May 12, 2026

    @PietrH
    PietrH
    on May 12, 2026
    MemberAuthor

    I'll cache csv files per R Session, not per function call.

  23. PietrH
    linked a pull request that will close this issueread archival data: add `get_archival_data()` function #589on May 21, 2026
  24. PietrH
    removed
    blocked
    on May 22, 2026

  25. PietrH commented on Jul 17, 2026

    @PietrH
    PietrH
    on Jul 17, 2026
    MemberAuthor

    I've added a warning with helpful suggestions if you are trying to return over 1GB of data as a tibble, suggesting you use out of memory processing instead.

    Should I also give the option to abort at this step?

    Image

  26. peterdesmet commented on Jul 23, 2026

    @peterdesmet
    peterdesmet
    on Jul 23, 2026
    Member

    Should I also give the option to abort at this step?

    You mean a nicer way than using Ctrl + C to abort? Yeah, asking for user input (continue / abort) at that point might be useful.

  27. PietrH commented on Jul 23, 2026

    @PietrH
    PietrH
    on Jul 23, 2026
    MemberAuthor

    Will do.

  28. PietrH
    added a sub-issue on Aug 6, 2026

  29. PietrH commented on Aug 6, 2026

    @PietrH
    PietrH
    on Aug 6, 2026
    MemberAuthor

    Requested deployment of etnservice v0.7.0

  30. Stijn-VLIZ commented on Aug 10, 2026

    @Stijn-VLIZ
    Stijn-VLIZ
    on Aug 10, 2026
    Collaborator

    Deployed

    • Version number starts with a capital this time.
    • Installation of libuv1-dev needed to build the r code.

  31. PietrH commented on Aug 11, 2026

    @PietrH
    PietrH
    on Aug 11, 2026
    MemberAuthor

    The capital was a typo, would you like me to change it?

    I wonder why the IO library is needed, there is a new dependency withr, but I don't see why it would need libuv1-dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Feature

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions