Online media processing architecture

[thekid]

Purpose

This issue serves as a concept for an architecture which enables us to process uploaded media online. Key requirements:

• Separate from main web application
• Make use of MongoDB change streams
• Horizontally scalable

Overview and flow

User uploads

Implemented inside the backstage area, see #26. Uploaded images are stored directly in the correct location (see below), and an entry in the processing collection is created:

{
  "slug":"rueppur_alb_2022-11-13",
  "file":"20221113_151718315_iOS.jpg",
  "at":{"$date":{"$numberLong":"1668873515000"}},
  "state":"new"
}

Media processor

Implemented in #46. Watches processing collection for entries with state = new, fetches uploaded images, extracts meta data and converts images to the correct sizes, then uploads all of that.

Media storage

Currently, media is stored right inside the folder for a given entry. Each variant has a prefix:

• full - Full-size image (WebP, 3840 * x)
• thumb - Thumbnail image (WebP, 1024 * x)
• preview - Preview image (JPEG, 720 * x)
• video - Video file (MPEG-4, H.264, 1920 * x)

/space/image
 |- rueppur_alb_2022-11-13
 |  |- full-20221113_151718315_iOS.jpg.webp
 |  |- preview-20221113_151718315_iOS.jpg.jpg
 |  `- thumb-20221113_151718315_iOS.jpg.webp
 |
 `- philippines-2006
    |- alona_white_beach-2006-09-06
    |  |- full...
    |  `- ...
    |- full...
    `- ...

The original uploads will be stored alongside these without any prefix, e.g. 20221113_151718315_iOS.jpg.