DIVYANSHU.md

Truth document for the local yt-dlp audit and installation decision.

Generated from local inspection on 2026-06-28 in:

Workspace: <local-workspace-root>
Cloned repo: <local-workspace-root>/work/yt-dlp
Upstream: https://github.com/yt-dlp/yt-dlp.git
Local git HEAD inspected: c13e2f8a20fc1fafb31cba4e6287c874bc0c0cc0
Last commit inspected: c13e2f8a2 2026-06-27 23:31:07 +0000 [ie] Fix request logging (#17072)

This file separates confirmed facts from recommendation. It is not a claim that every extractor works against every website today; extractor success depends on remote sites that change constantly.

Executive Verdict

yt-dlp is a serious, production-grade downloader framework, not a small CLI script. It is capable enough to become the standard media-ingestion tool for this computer and workspace, provided it is installed with the right optional dependencies and operational discipline.

Best setup for this machine:

Keep one clean global command for daily/manual use.
Add a JavaScript runtime path for full YouTube support.
Use Docker for bulk/project jobs where isolation, repeatability, and rate control matter.
Do not use the raw source checkout as the global executable for normal work.
Do not rely on the currently installed global yt-dlp without updating it; it is older than 90 days and already warns about staleness.

Recommended global lane:

python3 -m pip install -U --pre "yt-dlp[default,curl-cffi]"
brew install deno

Alternative if avoiding pip-managed globals:

brew install pipx deno ffmpeg
pipx install --force "yt-dlp[default,curl-cffi]"

For bulk:

docker run --rm \
  -v "$PWD/downloads:/downloads" \
  -v "$PWD/cookies:/cookies:ro" \
  yt-dlp-bulk \
  yt-dlp --cookies /cookies/cookies.txt -P /downloads "URL"

Current Machine Truth

Confirmed installed tools:

Homebrew exists: /opt/homebrew/bin/brew
Global yt-dlp exists: /opt/homebrew/bin/yt-dlp
Node exists: /opt/homebrew/bin/node
ffmpeg exists: /opt/homebrew/bin/ffmpeg
Docker exists: Docker 29.4.2
Docker Compose exists: v5.1.3

Confirmed missing or not found in PATH:

pipx was not found.
deno was not found.

Current global yt-dlp truth:

yt-dlp --version reported 2026.03.17.
Verbose output identified it as a pip install from yt-dlp/yt-dlp.
It warns that the version is older than 90 days.
It has ffmpeg/ffprobe 8.1.1.
It has optional networking libraries: certifi, curl_cffi, requests, sqlite3, urllib3, websockets.
It reported JS runtimes: none.
It reported no plugin directories loaded.

Current cloned source truth:

python3 -m yt_dlp --version from the clone reported 2026.06.09.
Verbose output identified source git HEAD c13e2f8a2.
It loaded request handlers: urllib, requests, websockets, curl_cffi.
It reported JS runtimes: none.
It loaded 1740 extractors in the source-run smoke.
It warned that no supported JavaScript runtime was available.

Important observed failure:

A YouTube smoke command against https://www.youtube.com/watch?v=TEST_VIDEO_ID failed with Video unavailable after warning that no supported JavaScript runtime was available.
This does not prove yt-dlp is broken. It proves this machine's current global runtime setup is incomplete for modern YouTube support.

Repository Scale

Local file counts from the clone:

Python source files under yt_dlp: 1046
Python extractor files under yt_dlp/extractor: 971
Python test files under test: 56
GitHub workflow files: 12
Requirement lock/input files under bundle/requirements: 17

This scale matters. yt-dlp is a framework with:

Hundreds of site-specific extractors.
Multiple networking backends.
Native and external downloaders.
ffmpeg postprocessors.
Plugin loading.
Browser cookie extraction.
Self-update and release channel logic.
Docker build support.
CI, CodeQL, static checks, and release workflows.

Feature Truth

Confirmed major feature families from README and code:

Site extraction through many InfoExtractor modules.
YouTube-specific client selection, PO token framework, and JS challenge logic.
Format selection and sorting.
Merged audio/video downloads through ffmpeg.
Audio extraction and transcoding.
Subtitle download and embedding.
Thumbnail download, conversion, and embedding.
Metadata writing and embedding.
Chapter splitting.
Partial downloads by section.
SponsorBlock integration.
Browser-cookie loading with --cookies-from-browser.
Netscape cookies file support with --cookies.
External downloader support: aria2c, axel, curl, ffmpeg, httpie, wget.
Proxy and SOCKS support.
Source-address binding support.
Concurrent fragment downloads via --concurrent-fragments / -N.
Download archive support via --download-archive.
Config files and portable configuration.
Output templates and multiple output paths via --paths.
Stable parseable integration outputs via -J, --print, --progress-template, and controlled hooks.
Plugin support for external extractors and postprocessors.
Release update channels: stable, nightly, master.
Self-update for supported release binaries.

Architecture Truth

Core modules inspected:

yt_dlp/YoutubeDL.py: orchestration, option application, extraction, download, postprocessing, archive, output, and logging behavior.
yt_dlp/options.py: CLI surface and validation.
yt_dlp/networking/*: request director, urllib/requests/curl-cffi/websocket handlers, SSL/proxy behavior.
yt_dlp/downloader/*: native and protocol-specific downloaders.
yt_dlp/downloader/external.py: external downloader command construction.
yt_dlp/postprocessor/*: ffmpeg and metadata postprocessing.
yt_dlp/postprocessor/exec.py: user command execution postprocessor.
yt_dlp/cookies.py: browser and cookies-file loading.
yt_dlp/plugins.py: plugin discovery/import.
yt_dlp/update.py: release-channel and update mechanism.

The design is pragmatic and mature:

Shared core handles common behavior.
Extractors are many and intentionally specialized.
Networking is abstracted through request handlers.
External tools are integrated through explicit argument arrays where possible.
Dangerous power features exist, but many are explicit user-requested surfaces.

Security Truth

Good signs:

GitHub Actions workflows use pinned action SHAs.
Checkout steps use persist-credentials: false where inspected.
CodeQL workflow exists for actions, JavaScript/TypeScript, and Python.
Dependency files under bundle/requirements use hashed installs in CI/build workflows.
Self-update blocks automatic restart into unverified builds.
Update code warns for unofficial repositories and disables automatic restart into custom builds.
Redirect handling removes Cookie headers across redirects in the urllib path.
Cache filenames quote keys and restrict section names.
Verbose-output tests cover hiding password values.
The --exec surface now restricts unsafe output template expansion unless allow-unsafe-exec-expansion compatibility mode is explicitly enabled.

Power/risk surfaces:

Plugins are arbitrary Python code. Only install plugins you trust.
--exec runs shell commands. Treat it as deliberate code execution.
--downloader-args, --postprocessor-args, and ffmpeg args can create powerful side effects.
--cookies-from-browser can access sensitive authenticated sessions.
--cookies files are secrets.
--netrc-cmd and login/password options are sensitive.
External downloaders inherit trust and behavior from installed binaries.
Docker containers need careful volume mounts to avoid writing into the wrong host path.
Updating to an arbitrary <owner>/<repo> is explicitly risky because normal verification does not protect custom repositories the same way.

Test Truth

Initial broad test command:

python3 -m pytest -Werror -m "not download"

Result:

Failed globally because a host-installed pytest_asyncio plugin emitted a Python 3.14 deprecation warning during pytest setup.
This was an environment/plugin problem, not evidence that all tests were broken.

Cleaned test command:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -Werror -m "not download"

Result:

1242 passed
38 skipped
2 failed
5801 deselected
Runtime: about 2 minutes 45 seconds

The two failures observed:

TestDownloaderCookieBehavior.test_cookie_behavior[CurlFD]
- Failed to bind a generated loopback address on macOS: OSError: [Errno 49] Can't assign requested address
- This looks host/network-interface-specific.
TestConvertThumbnail.test_escaping
- ffmpeg failed generating a WebP fixture: subprocess.CalledProcessError from an ffmpeg command using libwebp.
- This looks local ffmpeg-build/codec behavior, not a broad Python failure.

Meaning:

The core offline test signal is strong.
Local environment still needs cleanup if we want full developer-grade test repeatability.

Dependency Truth

Python:

Project supports Python 3.10+ for CPython.
This machine is using Python 3.14.3 through Homebrew.

Strongly recommended runtime dependencies:

ffmpeg and ffprobe: needed for merging and postprocessing.
yt-dlp-ejs: required for full YouTube support.
A JS runtime/engine such as deno, node, bun, or QuickJS.
curl-cffi: recommended for browser/TLS impersonation.

Important local fact:

Node is installed, but yt-dlp did not automatically use it.
README says deno is recommended.
Current yt-dlp warning says only deno is enabled by default, and other runtimes require --js-runtimes RUNTIME[:PATH].

Therefore the cleanest setup is to install deno globally, or configure yt-dlp to use the existing Node path.

Global Installation Decision

Do not use sudo pip install or unmanaged system Python pollution.

Do not make the source checkout itself the primary global yt-dlp; it is best kept for audit, patching, and development.

Good global options:

Option A: Update Current Pip-Based Global Install

This matches the current machine state most directly because global yt-dlp already comes from a pip-installed package under Homebrew Python.

python3 -m pip install -U --pre "yt-dlp[default,curl-cffi]"
brew install deno

Pros:

Fastest path.
Keeps the existing /opt/homebrew/bin/yt-dlp style.
Enables fresh nightly/pre-release behavior if --pre is used.

Cons:

Still uses a Python environment shared with other Homebrew Python packages.
Less isolated than pipx.

Option B: pipx Global CLI

brew install pipx deno ffmpeg
pipx ensurepath
pipx install --force "yt-dlp[default,curl-cffi]"

Pros:

Clean global CLI.
Dependencies isolated from system/project Python.
Best long-term host setup for many projects.

Cons:

Requires installing pipx and ensuring PATH order.
Existing /opt/homebrew/bin/yt-dlp may need to be removed or placed behind pipx in PATH to avoid ambiguity.

Option C: Homebrew yt-dlp

brew install yt-dlp ffmpeg deno

Pros:

Simple and Mac-native.
Easy upgrades through brew upgrade.

Cons:

Homebrew version can lag.
Optional Python extras may not match yt-dlp's fastest-moving needs.

Best host recommendation:

If you want clean long-term operations: Option B.
If you want immediate repair with minimum disruption: Option A.

Docker Decision

Use Docker for bulk jobs, project pipelines, CI-like tasks, and repeatable ingestion.

Do not use Docker as the only lane for interactive authenticated browser-cookie downloads, because browser profiles, macOS Keychain, and local session state are host-native and awkward/risky to mount.

Docker is best when:

Inputs are URL lists, files, or queues.
Cookies are exported into cookies.txt.
Outputs go to mounted project directories.
You need one reproducible toolchain across projects.
You need to throttle, parallelize, retry, and log systematically.

Docker is weaker when:

You need --cookies-from-browser safari or Chrome profile decryption from the host.
You need direct macOS Keychain access.
You need ad-hoc manual diagnosis with browser sessions.

Recommended pattern:

mkdir -p downloads cookies logs

docker run --rm \
  -v "$PWD/downloads:/downloads" \
  -v "$PWD/cookies:/cookies:ro" \
  -v "$PWD/logs:/logs" \
  yt-dlp-bulk \
  yt-dlp \
    --cookies /cookies/cookies.txt \
    --download-archive /downloads/archive.txt \
    --paths /downloads \
    --concurrent-fragments 4 \
    --sleep-requests 1 \
    --retries 10 \
    --fragment-retries 10 \
    --newline \
    "URL"

For URL lists:

docker run --rm \
  -v "$PWD:/job" \
  yt-dlp-bulk \
  yt-dlp -a /job/urls.txt -P /job/downloads --download-archive /job/archive.txt

Workspace Usage Doctrine

For other projects on this computer:

Use the global yt-dlp command for one-off and authenticated work.
Use a shared Docker image for bulk project work.
Keep project-specific configs inside each project, not in the source checkout.
Keep cookies outside git.
Keep download archives per project.
Keep logs per project.
Avoid global plugins unless they are trusted and useful everywhere.

Suggested project structure:

project/
  media-ingest/
    urls.txt
    cookies/
      cookies.txt
    downloads/
    logs/
    yt-dlp.conf
    archive.txt

Suggested conservative yt-dlp.conf for bulk:

--ignore-errors
--no-overwrites
--continue
--download-archive archive.txt
--concurrent-fragments 4
--retries 10
--fragment-retries 10
--sleep-requests 1
--restrict-filenames
--paths downloads
--newline

For YouTube after installing deno:

--js-runtimes deno

If using existing Node instead of deno:

--js-runtimes node:/opt/homebrew/bin/node

Capability Assessment

Satva:

Mature architecture.
Strong test suite.
Strong CI posture.
Broad protocol and site coverage.
Good update-channel design.
Good local integration options.
Good parseable output modes for automation.
Explicit handling for sensitive command/cookie/plugin surfaces.

Rajas:

Very large extractor surface means constant churn.
YouTube is adversarial and requires fresh runtime support.
Many options create operational complexity.
Multiple installation lanes can conflict on PATH.
Host Python plugins can poison test runs unless isolated.

Tamas:

Current global install is stale.
No JS runtime is detected by yt-dlp on this machine.
Global Python site-packages currently contains yt-dlp directly.
No pipx isolation is present.
Bulk workflow is not yet standardized in this workspace.

Final Recommendation

Use a two-lane installation:

Host lane for daily/manual/authenticated use:

brew install pipx deno ffmpeg
pipx ensurepath
pipx install --force "yt-dlp[default,curl-cffi]"

Then confirm:

which yt-dlp
yt-dlp --version
yt-dlp -v --ignore-config --simulate "https://www.youtube.com/watch?v=TEST_VIDEO_ID"

Docker lane for bulk:

Build or pull a controlled image.
Mount only input/output/cookies/log directories.
Use per-project archives.
Use sleep/retry settings by default.
Do not mount browser profile directories unless absolutely necessary.

Source lane for development/audit only:

Keep work/yt-dlp as the inspected source.
Use it for patches, tests, and understanding.
Do not point other projects at this checkout as the default runtime.

Next Proof Gate

Before declaring this computer fully ready:

Install or enable a JS runtime for yt-dlp (deno preferred, or configured Node).
Update the stale global yt-dlp.
Rerun the YouTube smoke.
Create one Docker bulk image and run one URL-list dry run.
Decide whether global plugins are needed; default answer should be no.

Only after those gates pass should this be considered fully operational for bulk workspace-wide use.

Installed State Proof - 2026-06-28

The global lane was upgraded and tested after the original audit.

What changed:

Installed Homebrew pipx 1.15.0.
Installed Homebrew deno 2.9.0.
Installed Homebrew yt-dlp 2026.6.9.
Homebrew yt-dlp could not be kept as the active command because its Homebrew Python runtime had a broken pyexpat linkage on this machine.
Downloaded the official upstream yt-dlp_macos standalone binary from the latest GitHub release.
Verified yt-dlp_macos against the upstream SHA2-256SUMS file: yt-dlp_macos: OK.
Replaced /opt/homebrew/bin/yt-dlp with the verified standalone binary.
Preserved the Homebrew symlink as: /opt/homebrew/bin/yt-dlp.homebrew-symlink-20260628.

Active global command:

/opt/homebrew/bin/yt-dlp

Active global version:

2026.06.09

Active runtime proof:

yt-dlp version stable@2026.06.09 from yt-dlp/yt-dlp [821bef0f0] (darwin_exe)
Python 3.14.5 bundled in standalone executable
ffmpeg 8.1.1
ffprobe 8.1.1
yt_dlp_ejs 0.8.0
JS runtimes: deno-2.9.0
Request Handlers: urllib, requests, websockets, curl_cffi
Loaded 1739 extractors

Functional proofs:

YouTube extraction succeeded for https://www.youtube.com/watch?v=jNQXAC9IVRw.
YouTube extraction succeeded for https://www.youtube.com/watch?v=aqz-KE-bpKQ.
Deno/EJS JS challenge solving was observed in verbose output.
Generic XML/MPD extraction succeeded for the test MPD URL after switching to the standalone binary.
Real download-and-merge succeeded:
- URL: https://www.youtube.com/watch?v=jNQXAC9IVRw
- Format: 395+251
- Output: work/global-test/downloads/jNQXAC9IVRw.mkv
- Size: 474564 bytes
- Duration: 19.021000
- Streams: AV1 video 320x240, Opus audio
Archive skip succeeded with: work/global-test/archive.txt
yt-dlp -U reported: yt-dlp is up to date (stable@2026.06.09 from yt-dlp/yt-dlp).

Checksum proof:

b82c3626952e6c14eaf654cc565866775ffd0b9ffb7021628ac59b42c2f4f244  /opt/homebrew/bin/yt-dlp
b82c3626952e6c14eaf654cc565866775ffd0b9ffb7021628ac59b42c2f4f244  work/global-install/yt-dlp_macos

Remaining caveat:

Homebrew Python 3.14.6 on this machine currently has broken pyexpat behavior. That affects Python-based yt-dlp installs from Homebrew/Python for XML parsing. The active standalone binary avoids this issue.

System Upgrade And Repair Proof - 2026-06-28

After the installed-state proof, the machine was further cleaned and brought to current Homebrew state.

Actions completed:

Ran broad brew upgrade.
Upgraded 57 outdated packages/casks, including:
- ffmpeg 8.1.1 -> 8.1.2
- node 25.8.2 -> 26.4.0
- python@3.13 3.13.12_1 -> 3.13.14
- uv 0.11.1 -> 0.11.25
- gemini-cli 0.35.3 -> 0.46.0
- codex 0.128.0 -> 0.142.3
- expo-orbit 2.3.0 -> 2.8.0
- libreoffice 26.2.1 -> 26.2.4
Unlinked and then uninstalled the Homebrew yt-dlp formula to remove the confusing broken/Python-based lane.
Kept /opt/homebrew/bin/yt-dlp as the verified official standalone binary.
Removed the dead backup symlink: /opt/homebrew/bin/yt-dlp.homebrew-symlink-20260628.
Installed Homebrew expat 2.8.2.
Patched Homebrew Python pyexpat extensions for both Python 3.14 and Python 3.13 to load: /opt/homebrew/opt/expat/lib/libexpat.1.dylib instead of the incompatible macOS /usr/lib/libexpat.1.dylib.
Ran brew autoremove.
Ran brew cleanup --prune=all.
Cleanup freed approximately 1.8GB.

Final key versions at this checkpoint:

yt-dlp 2026.06.09 at /opt/homebrew/bin/yt-dlp
deno 2.9.0
expat 2.8.2
ffmpeg 8.1.2
ffprobe 8.1.2
node 26.4.0
pipx 1.15.0
python@3.13 3.13.14
python@3.14 3.14.6
uv 0.11.25

Final Homebrew outdated check:

FORMULA_OUTDATED: none
CASK_OUTDATED: none

Python XML repair proof:

python3_xml_ok ('26.2', ('', '', ''), 'arm64') root
pyexpat OK on Homebrew Python 3.14
xml.parsers.expat OK on Homebrew Python 3.14
pyexpat OK on Homebrew Python 3.13
xml.parsers.expat OK on Homebrew Python 3.13

Deeper yt-dlp proof after full upgrade:

yt-dlp -U:
Latest version: stable@2026.06.09 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2026.06.09 from yt-dlp/yt-dlp)

YouTube metadata batch:
youtube|jNQXAC9IVRw|Me at the zoo|19|not_live
youtube|aqz-KE-bpKQ|Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film|10:35|not_live

Generic XML/MPD:
generic|float_duration|mp4

Real download/merge:
work/deep-proof/downloads/jNQXAC9IVRw.mkv
size: 474564 bytes
duration: 19.021000
video: av1 320x240
audio: opus

Archive skip:
youtube jNQXAC9IVRw

Final yt-dlp checksum:

b82c3626952e6c14eaf654cc565866775ffd0b9ffb7021628ac59b42c2f4f244  /opt/homebrew/bin/yt-dlp

Homebrew warning cleanup after this checkpoint:

Removed deprecated gemini-cli 0.46.0.
Inspected the untrusted anomalyco/tap; it only mattered because opencode was installed from that tap.
Removed anomalyco/tap/opencode 1.3.13.
Untapped anomalyco/tap.
Final brew doctor result: Your system is ready to brew.
Final Homebrew outdated checks: formulae: none; casks: none.

The Homebrew warning sources are now removed. The yt-dlp lane remains current and proven as the official standalone binary at /opt/homebrew/bin/yt-dlp.

yt-dlp Feature And Subtitle Truth - 2026-06-28

This section is source-grounded from the checked-out repo and live-grounded from the installed global binary.

Exact Feature Families

Site extraction: 1739 extractors in the active binary, with almost all site logic under yt_dlp/extractor.
Playlist and channel handling: full playlist traversal, slicing, reversing, item matching, lazy extraction, and archive-based duplicate skipping.
Search and URL dispatch: direct URLs, site-specific extractors, generic extractor fallback, and default search providers when configured.
Format selection: -f selectors, format sorting, codec/container filters, resolution/FPS/HDR preferences, filesize filters, audio-only/video-only, and merge behavior.
Download engine: fragment downloads, retries, sleep intervals, rate limits, concurrent fragments, live stream behavior, external downloaders, and archive recording.
Network layer: proxy support, source address binding, IPv4/IPv6 preference, cookies, headers, impersonation support through curl_cffi, and multiple request handlers.
JavaScript support: active install has deno 2.9.0; yt-dlp can use JS runtimes for extractor logic that needs JavaScript interpretation.
Subtitle/caption support: list, select, download, convert, embed, automatic captions, normal captions, language regex, exclusions, and format preference.
Metadata sidecars: info JSON, thumbnails, descriptions, annotations where supported, comments where supported, URL/link output, and metadata printing.
Postprocessing: ffmpeg merge, extract audio, remux/recode video, embed subtitles, embed thumbnails, embed metadata, split chapters, SponsorBlock operations, and custom exec hooks.
Automation: batch files, --print, JSON dumping, progress templates, download archives, match filters, and hooks usable from scripts.
Plugin surface: local plugin discovery exists, but plugins are an explicit trust boundary because they execute Python code inside the yt-dlp process.

Subtitle Flow From Source

YoutubeDL.process_subtitles is the decision center:

Extractors return two dictionaries: subtitles for normal subtitles and automatic_captions for generated captions.
yt-dlp only considers normal subtitles when --write-subs is active.
yt-dlp only considers automatic captions when --write-auto-subs is active.
If both are active, normal subtitles take precedence over automatic captions for the same language key.
--all-subs selects every available language.
--sub-langs supports regex and exclusions, so patterns like en.* can match more than plain English.
Without explicit language selection, yt-dlp uses a practical English-first heuristic: exact normal en, then normal languages beginning with en, then any en, then any language beginning with en, then other normal languages, then all remaining languages.
--sub-format is a preference list split by /; the first available match wins, and best means the last listed format in that site's subtitle entry.
_write_subtitles writes inline subtitle data when already present, or downloads the subtitle URL when the extractor only provides a URL.
ffmpeg postprocessors handle conversion and embedding after download.

Language Detection Truth

yt-dlp generally does not infer language from audio or text by doing its own AI/NLP language detection. It trusts language metadata exposed by the site, manifest, or extractor:

YouTube subtitle language keys come from YouTube caption track metadata.
DASH/HLS/SMIL/ISM language values come from manifest attributes such as lang, LANGUAGE, systemLanguage, or Language.
Format/audio stream language fields come from extractor and manifest data.
Auto-translation keys such as en-de are site-provided caption/translation identifiers, not proof that yt-dlp itself detected the spoken language.

For bulk work, treat subtitle language selection as metadata selection, not semantic language recognition. If true language identification is required, add a separate ASR/language-identification pipeline after download.

Live Subtitle Proof

Command family tested:

yt-dlp --ignore-config --list-subs https://www.youtube.com/watch?v=jNQXAC9IVRw
yt-dlp --ignore-config --skip-download --write-auto-subs --sub-langs 'en.*,-live_chat' --sub-format 'vtt/best' ...

Observed truth:

YouTube exposed automatic captions for jNQXAC9IVRw.
The subtitle table included en, de, and many auto-translation language keys with formats including vtt, srt, ttml, srv3, srv2, srv1, and json3.
Corrected JSON probe succeeded after fixing the shell pipeline: subtitles ['de', 'en'], automatic_count 2, auto_sample ['de', 'en'].
yt-dlp downloaded:
- work/subtitle-proof/jNQXAC9IVRw.en.vtt
- work/subtitle-proof/jNQXAC9IVRw.en-en.vtt
The VTT file begins with WEBVTT, Kind: captions, Language: en, and English cue text from the video.
The broad regex en.* also selected translated tracks such as en-de; one such request hit YouTube HTTP 429. This is a bulk-use warning: use narrower language selectors like --sub-langs en unless translations are truly needed.

Bulk Subtitle Doctrine

Recommended bulk defaults:

yt-dlp --ignore-config \
  --download-archive archive.txt \
  --write-subs --write-auto-subs \
  --sub-langs 'en,-live_chat' \
  --sub-format 'vtt/best' \
  --sleep-subtitles 1 \
  --retries infinite --fragment-retries infinite \
  -P downloads \
  -o '%(extractor_key)s/%(uploader|unknown)s/%(id)s.%(ext)s' \
  URL_OR_BATCH

Use --write-subs when official/manual subtitles matter. Use --write-auto-subs when generated captions are acceptable. Use --embed-subs only when the chosen output container supports embedded subtitle streams cleanly; otherwise keep VTT/SRT sidecars.

Workspace Root Organization - 2026-06-28

Root:

<local-workspace-root>

Observed size:

root: 261M
work: 164M
outputs: 97M

Current root structure:

DIVYANSHU.md
outputs/
outputs/WORKSPACE-MAP.md
outputs/youtube-VIDEO_ID/
work/
work/deep-proof/
work/global-install/
work/global-test/
work/subtitle-proof/
work/yt-dlp/

Role of each area:

DIVYANSHU.md: the main truth ledger for this whole mission.
outputs/: user-facing deliverables that can be opened/tested directly.
outputs/WORKSPACE-MAP.md: quick map and command reference.
outputs/youtube-VIDEO_ID/: requested YouTube audio/transcript output.
work/: internal working area for clone, proofs, checksums, and scratch verification.
work/yt-dlp/: upstream yt-dlp clone, currently clean with no local modifications.
work/global-install/: official standalone binary download, checksum file, and install backup material.
work/global-test/: earlier global install proof downloads/logs.
work/deep-proof/: deeper active-binary proof including real download, ffmpeg merge, archive behavior, and MPD/XML proof.
work/subtitle-proof/: subtitle/caption behavior proof.

Current global toolchain:

/opt/homebrew/bin/yt-dlp
yt-dlp 2026.06.09
ffmpeg 8.1.2
deno 2.9.0
brew doctor: Your system is ready to brew.

User video output:

outputs/youtube-VIDEO_ID/VIDEO_ID.mp3
outputs/youtube-VIDEO_ID/VIDEO_ID.hi-orig.vtt
outputs/youtube-VIDEO_ID/VIDEO_ID.hi-orig.transcript.txt

User video proof:

Audio downloaded and converted to MP3.
MP3 duration: 3422.250667 seconds, approximately 57m 02s.
MP3 size: about 97M.
Original automatic caption VTT downloaded as hi-orig.
Clean transcript TXT generated from VTT.
Clean transcript line count: 1183.
English auto-translated caption endpoint returned HTTP 429; this is a YouTube rate-limit condition, not a local install failure.

End-To-End Workflow

Use global /opt/homebrew/bin/yt-dlp, not a local venv, for normal machine use.
Keep user-facing final files under outputs/.
Keep proof/scratch downloads under work/.
For every video, first check subtitles with:

yt-dlp --ignore-config --list-subs URL

Prefer exact subtitle keys, for example:

--sub-langs 'en,-live_chat'
--sub-langs 'hi-orig,-live_chat'
--sub-langs 'en,hi-orig,-live_chat'

Avoid broad selectors such as en.* in bulk because they can request many translated caption tracks.
For bulk jobs, always use --download-archive so reruns skip completed videos.
Use --sleep-subtitles to reduce subtitle endpoint pressure.
Store each channel/uploader/video in deterministic output templates.

Recommended practical command for audio plus subtitles:

yt-dlp --ignore-config \
  --download-archive archive.txt \
  --extract-audio --audio-format mp3 --audio-quality 0 \
  --write-subs --write-auto-subs \
  --sub-langs 'hi-orig,-live_chat' \
  --sub-format 'vtt/best' \
  --sleep-subtitles 2 \
  --retries infinite --fragment-retries infinite \
  -P outputs/bulk \
  -o '%(extractor_key)s/%(uploader|unknown)s/%(id)s.%(ext)s' \
  URL_OR_BATCH

For transcript-only work:

yt-dlp --ignore-config --skip-download \
  --write-subs --write-auto-subs \
  --sub-langs 'hi-orig,-live_chat' \
  --sub-format 'vtt/best' \
  --sleep-subtitles 2 \
  -P outputs/transcripts \
  -o '%(extractor_key)s/%(uploader|unknown)s/%(id)s.%(ext)s' \
  URL_OR_BATCH

If a translated transcript returns HTTP 429, fall back to the original caption key from --list-subs, then translate downstream if needed.

BOB YouTube Operating Layer - 2026-06-28

Workspace name:

Break Ordinary Boundaries (BOB) YouTube

Friendly alias:

<local-friendly-alias> -> <local-workspace-root>

Name marker:

.workspace-name

New reusable files:

README.md
bin/bob-youtube
scripts/clean_vtt_transcripts.py

Global command installed:

/opt/homebrew/bin/bob-youtube -> <local-workspace-root>/bin/bob-youtube

The first global test exposed an important bug: when launched through the symlink, the script initially resolved its root as /opt/homebrew. The wrapper was corrected to resolve the real symlink target before calculating the workspace root.

Confirmed global proof from /tmp:

bob-youtube root
<local-workspace-root>

bob-youtube info 'https://www.youtube.com/watch?v=jNQXAC9IVRw'
youtube|jNQXAC9IVRw|Me at the zoo|19|public

bob-youtube clean outputs/youtube-VIDEO_ID
VIDEO_ID.hi-orig.vtt -> VIDEO_ID.hi-orig.transcript.txt

Confirmed auto-transcript proof from /tmp with redirected output:

BOB_YOUTUBE_OUTPUT_DIR=/tmp/bob-youtube-smoke bob-youtube transcript 'https://www.youtube.com/watch?v=VIDEO_ID'
Auto-selected subtitle language: hi-orig,-live_chat
/tmp/bob-youtube-smoke/VIDEO_ID/VIDEO_ID.hi-orig.vtt
/tmp/bob-youtube-smoke/VIDEO_ID/VIDEO_ID.hi-orig.transcript.txt
clean transcript lines: 1183

Programmatic surface:

bob-youtube info URL
bob-youtube subs URL
bob-youtube metadata URL
bob-youtube thumbnail URL
bob-youtube audio URL
bob-youtube video URL
bob-youtube transcript URL [SUB_LANGS]
bob-youtube both URL [SUB_LANGS]
bob-youtube batch URLS_TXT [SUB_LANGS]
bob-youtube clean PATH
bob-youtube index
bob-youtube where VIDEO_ID
bob-youtube open VIDEO_ID
bob-youtube doctor
bob-youtube root

Environment controls:

BOB_YOUTUBE_OUTPUT_DIR
BOB_YOUTUBE_ARCHIVE
BOB_YOUTUBE_SUB_LANGS
BOB_YOUTUBE_TEMPLATE

Current verdict:

The workspace is now not only a one-off proof area.
It is a named local tool workspace with a global command.
It can be called from any directory on the computer as bob-youtube.
Default output remains inside this workspace under outputs/.
Output can be redirected per project with BOB_YOUTUBE_OUTPUT_DIR=/path.
Transcript mode now defaults to auto: the wrapper inspects caption metadata and selects one concrete caption key, preferring hi-orig, then en, then hi, then the first available non-live-chat key. This avoids requesting many translated tracks by default.

User Terminal Confirmation - 2026-06-28

The user opened a new terminal from the home directory and ran:

bob-youtube transcript 'https://www.youtube.com/watch?v=EXAMPLE_VIDEO_ID'

Observed result:

Global command was available in the new terminal.
Auto-selected subtitle key: hi-orig,-live_chat.
Downloaded: outputs/EXAMPLE_VIDEO_ID/EXAMPLE_VIDEO_ID.hi-orig.vtt
Generated: outputs/EXAMPLE_VIDEO_ID/EXAMPLE_VIDEO_ID.hi-orig.transcript.txt
Verified transcript size: 58420 bytes.
Verified transcript line count: 672.
First cleaned transcript lines are Hindi text beginning with: आगे मेरे एक आगे लिखा हुआ है इंडोग्रीक.

The run also revealed that the wrapper re-cleaned an older VTT under outputs/youtube-VIDEO_ID/. This was not destructive, but it was noisy. bin/bob-youtube was patched so transcript cleanup now targets the current video folder when the default output template is used.

Cleanup scope proof:

/tmp/bob-youtube-clean-scope/jNQXAC9IVRw/jNQXAC9IVRw.en.vtt
/tmp/bob-youtube-clean-scope/jNQXAC9IVRw/jNQXAC9IVRw.en.transcript.txt
/tmp/bob-youtube-clean-scope/stale/stale.vtt
scope_ok_stale_not_cleaned

Separate shell issue found:

New terminals print ~/.zshrc:98: unmatched ".
Root cause is ~/.zshrc line 4: JFOc7h0SG4ff6X5VCyJZZ"
This is unrelated to bob-youtube; it is a broken shell startup config line.
It has not been edited here because it is in the home config, outside the BOB YouTube workspace.

Feature Expansion - 2026-06-28

The BOB YouTube wrapper was expanded from an audio/transcript helper into a small local corpus tool.

New files:

scripts/index_outputs.py
outputs/BOB-YOUTUBE-FEATURE-MATRIX.md
outputs/library-index.json
outputs/library-index.tsv

New commands:

bob-youtube doctor
bob-youtube metadata URL
bob-youtube thumbnail URL
bob-youtube video URL
bob-youtube index
bob-youtube where VIDEO_ID
bob-youtube open VIDEO_ID

Proof:

bob-youtube doctor
yt-dlp version: 2026.06.09
ffmpeg: ffmpeg version 8.1.2
deno: deno 2.9.0
global symlink: /opt/homebrew/bin/bob-youtube -> <local-workspace-root>/bin/bob-youtube

bob-youtube where EXAMPLE_VIDEO_ID
<local-workspace-root>/outputs/EXAMPLE_VIDEO_ID

bob-youtube metadata 'https://www.youtube.com/watch?v=jNQXAC9IVRw'
outputs/jNQXAC9IVRw/jNQXAC9IVRw.info.json

bob-youtube index
indexed 3 video folders
outputs/library-index.json
outputs/library-index.tsv

Current indexed corpus:

EXAMPLE_VIDEO_ID: transcript 672 lines, subtitles present
jNQXAC9IVRw: metadata JSON present
youtube-VIDEO_ID: MP3, transcript 1183 lines, subtitles present

Docker Packaging - 2026-06-28

Added:

Dockerfile
docker-compose.yml
.dockerignore
docs/DOCKER.md
docs/VISION.md

Docker design:

Image name: bob-youtube:local.
Runtime root inside container: /app.
Output/data contract inside container: /data.
The image includes BOB wrapper/scripts/docs, Python, yt-dlp, ffmpeg, deno.
The image intentionally excludes work/, outputs/, local proof material, downloaded media, and upstream yt-dlp git history.

Docker proof:

docker compose build
Image bob-youtube:local Built

docker compose run --rm bob-youtube doctor
BOB YouTube root: /app
Output root: /data
yt-dlp: /usr/local/bin/yt-dlp
yt-dlp version: 2026.06.09
ffmpeg: ffmpeg version 5.1.9-0+deb12u1
deno: deno 2.9.0

docker compose run --rm bob-youtube index
indexed 3 video folders
/data/library-index.json
/data/library-index.tsv

docker run --rm -v "$PWD/outputs:/data" bob-youtube:local info 'https://www.youtube.com/watch?v=jNQXAC9IVRw'
youtube|jNQXAC9IVRw|Me at the zoo|19|public

docker run --rm -v "$PWD/outputs:/data" bob-youtube:local where EXAMPLE_VIDEO_ID
/data/EXAMPLE_VIDEO_ID

Image proof:

id=sha256:fdf59e23a671f8a84b0c9291b86b7ef2f4c0f8bad07f551c2186624575d437d6
size=274908615

Future product direction is documented in docs/VISION.md.

API Service Mode - 2026-06-28

Added:

scripts/bob_youtube_api.py
bob-youtube api [PORT]
docker-compose service: bob-youtube-api

API endpoints:

GET  /health
GET  /library
GET  /library?refresh=1
GET  /where/VIDEO_ID
GET  /jobs
GET  /jobs/JOB_ID
POST /jobs

Historical allowed synchronous job commands at that point:

info, subs, metadata, thumbnail, transcript, segments, audio, video, both, index, where

Local API proof:

BOB_YOUTUBE_API_PORT=18080 bob-youtube api 18080
GET /health -> ok true, output_root local outputs path
GET /library?refresh=1 -> ok true, 3 items
GET /where/EXAMPLE_VIDEO_ID -> local outputs/EXAMPLE_VIDEO_ID
POST /jobs {"command":"info","url":"https://www.youtube.com/watch?v=jNQXAC9IVRw"}
-> youtube|jNQXAC9IVRw|Me at the zoo|19|public

Docker API proof:

docker run --rm -p 18081:8080 -v "$PWD/outputs:/data" bob-youtube:local api 8080
GET /health -> ok true, output_root /data
GET /library?refresh=1 -> ok true, 3 items
POST /jobs {"command":"info","url":"https://www.youtube.com/watch?v=jNQXAC9IVRw"}
-> youtube|jNQXAC9IVRw|Me at the zoo|19|public

Full Smoke Suite - 2026-06-28

Added:

scripts/smoke_test.sh

Smoke coverage:

bash -n bin/bob-youtube
Python compile checks for helper/API scripts
local bob-youtube doctor
local bob-youtube info
local bob-youtube where
local bob-youtube segments
local bob-youtube index
Docker CLI doctor
Docker CLI info
Docker CLI index
Docker API /health
Docker API /library?refresh=1
Docker API POST /jobs with info
Docker API async POST /jobs with polling through GET /jobs/JOB_ID

Proof result:

scripts/smoke_test.sh
BOB YouTube smoke test passed

Timestamped Segments And Async Jobs - 2026-06-28

Added:

scripts/vtt_to_segments.py
bob-youtube segments PATH
automatic segment generation after bob-youtube transcript
API async jobs with in-memory job status

Segment proof:

bob-youtube segments outputs/EXAMPLE_VIDEO_ID
outputs/EXAMPLE_VIDEO_ID/EXAMPLE_VIDEO_ID.hi-orig.segments.json
segment_count: 1342
first segment text: आगे मेरे एक आगे लिखा हुआ है इंडोग्रीक

Updated index proof:

id                    segment_count
EXAMPLE_VIDEO_ID           1
youtube-VIDEO_ID   1

Async API proof is now included in scripts/smoke_test.sh:

api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
BOB YouTube smoke test passed

Runtime Profiles - 2026-06-28

Added:

config/profiles/default.env
config/profiles/consumer.env
config/profiles/research.env
config/profiles/datacenter.env
config/profiles/agent.env
bob-youtube profiles
BOB_YOUTUBE_PROFILE
API job profile override

Profile intent:

default     current stable behavior
consumer    simple user-facing operation
research    uploader-grouped review folders
datacenter  deterministic worker paths
agent       deterministic paths for tool-calling agents

Proof:

bob-youtube profiles
Active profile: default
Available profiles: agent, consumer, datacenter, default, research

BOB_YOUTUBE_PROFILE=consumer bob-youtube doctor
Profile: consumer

BOB_YOUTUBE_PROFILE=research bob-youtube doctor
Template: %(extractor_key)s/%(uploader|unknown)s/%(id)s/%(id)s.%(ext)s

docker run --rm -e BOB_YOUTUBE_PROFILE=agent -v "$PWD/outputs:/data" bob-youtube:local doctor
Profile: agent

API profile override proof from smoke suite:
api_profile_job_ok agent

The latest full smoke suite passed after profile support:

scripts/smoke_test.sh
BOB YouTube smoke test passed

Docker/API Product Hardening - 2026-06-28

Added:

BOB_YOUTUBE_API_TOKEN
BOB_YOUTUBE_JOB_DIR
durable API job JSON files
API reload of completed job history on startup
sync job records
async job records
token-gated smoke coverage
container restart persistence proof

Truth boundary:

The Docker image does not include upstream yt-dlp git history, local proof
downloads, or this workspace's work/ tree. The image contains the BOB runtime
surface only: wrapper, helper scripts, config profiles, docs, yt-dlp, ffmpeg,
deno, and Python runtime.

Runtime job store:

default local/API path: outputs/.bob-youtube/jobs
default Docker path: /data/.bob-youtube/jobs
override: BOB_YOUTUBE_JOB_DIR

API security:

If BOB_YOUTUBE_API_TOKEN is set, every API request must include either:
Authorization: Bearer TOKEN
or:
X-BOB-YOUTUBE-Token: TOKEN

Restart behavior:

Completed jobs are reloaded from JSON on API startup.
Queued/running jobs found during startup are marked interrupted.
This is durable job history, not yet a distributed worker queue.

Next true datacenter gap:

Add a real queue/worker model for concurrent high-volume jobs, structured logs
per job, webhook completion callbacks, and optional external queue backends.

Proof after Docker/API hardening:

docker compose build
image id: sha256:4bbd60b66c3d5bc7a466d11054319dd2110431170e07b61378172e568557ba01
image size: 274921440
created: 2026-06-28T09:16:59.51489651Z

scripts/smoke_test.sh
api_health_ok /data /data/.bob-youtube/jobs
api_library_items 3
api_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_profile_job_ok agent
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_persisted_job_ok 287c9842-8aaa-44c5-90b5-47b3e6ccb43e
BOB YouTube smoke test passed

docker ps --filter ancestor=bob-youtube:local
no running containers

Post-Restart Alignment Review - 2026-06-28

Computer restart review result:

global command: /opt/homebrew/bin/bob-youtube
global symlink target: <local-workspace-root>/bin/bob-youtube
yt-dlp version: 2026.06.09
ffmpeg version: 8.1.2
deno version: 2.9.0
friendly alias: <local-friendly-alias>

Alignment changes after restart:

Added AGENTS.md with local BOB YouTube agent doctrine.
Copied AGENTS.md into the Docker runtime package.
Updated outputs/WORKSPACE-MAP.md to reflect Docker/API/profile/job-store truth.
Hardened API token comparison with constant-time hmac.compare_digest.
Removed outputs/.DS_Store macOS metadata noise.
Regenerated library index.

Final rebuilt image:

docker compose build
image id: sha256:77e5a8a96901cceaede82cdfe0c9bf345e7288b0414d76b6d513a9e1f0be717e
image size: 274923075
created: 2026-06-28T10:49:51.642165261Z

Final full proof:

scripts/smoke_test.sh
api_health_ok /data /data/.bob-youtube/jobs
api_library_items 3
api_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_profile_job_ok agent
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_persisted_job_ok 1eaf41e9-810c-429c-887a-8afc5d0a0206
BOB YouTube smoke test passed

Persisted job store after the final run:

outputs/.bob-youtube/jobs
job_count: 3
1eaf41e9-810c-429c-887a-8afc5d0a0206 succeeded async default ok
480ce14c-6778-49ed-a085-ca589c74ae7a succeeded sync default ok
5b59336a-7123-4b83-96bf-e8029e06a1d7 succeeded sync agent ok

Clean shutdown state:

docker ps --filter ancestor=bob-youtube:local
no running containers

find outputs -maxdepth 2 -name .DS_Store
no files

Current truth:

BOB YouTube is now a globally callable Mac tool, a Docker CLI package, and a
token-gated Docker API package with durable job records. The remaining gap is
not extraction capability; it is datacenter orchestration: real queue workers,
per-job structured logs, callbacks, and optional STT/embedding exports.

Docker-First Release Review - 2026-06-28

Fresh-user release additions:

docs/RELEASE.md
docs/openapi.json
GET /
GET /openapi.json
.env.example
Docker HEALTHCHECK
BOB_YOUTUBE_CORS_ORIGIN optional CORS response headers
Compose default local token: change-this-token, override with BOB_YOUTUBE_API_TOKEN

Release image proof:

docker compose build
image id: sha256:da2b4b6f5d9daf4a28bc4901b3f5f08cbae0aebb3e0a1c878c20f13a3b65b95c
image size: 274931763
created: 2026-06-28T11:01:55.199466012Z
healthcheck: bob-youtube doctor >/dev/null || exit 1

Release smoke proof:

scripts/smoke_test.sh
api_discovery_ok bob-youtube
api_openapi_ok 0.1.0
api_auth_boundary_ok 401
api_health_ok /data /data/.bob-youtube/jobs
api_library_items 3
api_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_profile_job_ok agent
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_persisted_job_ok 7a85f48a-6396-4cfc-b4a5-57e7d626028e
BOB YouTube smoke test passed

Release verdict:

Ready for local/private Docker-first use and integration into other products.
Not yet public multi-tenant/datacenter complete.

Remaining public-production work:

P0: real queue/worker model, per-job structured logs, request size/rate limits,
TLS reverse-proxy example, published image tags, checksums, and SBOM.

P1: concurrency controls, webhook callbacks, external queue backend, cookie
secret handling, and metrics.

P2: STT fallback, embedding/RAG export bundles, language-normalized metadata,
deduplication, and corpus manifest signing.

GitHub Ecosystem Comparison And USP - 2026-06-28

Added:

docs/COMPARISON.md

Compared lanes:

yt-dlp core engine
raw yt-dlp Docker images
self-hosted Web UIs
REST/Web download queues
API wrappers
MCP transcript/agent servers
n8n workflow nodes

BOB YouTube USP:

BOB YouTube is not primarily a browser downloader. It is a Docker-first
YouTube ingestion service and corpus runtime for products, research workflows,
bulk operators, and AI agents.

Differentiating contract:

/data/VIDEO_ID/VIDEO_ID.mp3
/data/VIDEO_ID/VIDEO_ID.mkv
/data/VIDEO_ID/VIDEO_ID.LANG.vtt
/data/VIDEO_ID/VIDEO_ID.LANG.transcript.txt
/data/VIDEO_ID/VIDEO_ID.LANG.segments.json
/data/library-index.json
/data/library-index.tsv
/data/.bob-youtube/jobs/JOB_ID.json
GET /openapi.json
POST /jobs

Where public alternatives are ahead:

MeTube, YTPTube, and yt-dlp-web-ui are ahead for polished browser UI.
YTPTube is ahead for advanced scheduling, notifications, presets, and live
stream/web UI operations.
Raw Docker images are ahead for public registry simplicity.
MCP/n8n tools are ahead inside their specific host ecosystems.

Best strategic lane:

Double down on OpenAPI + Docker + durable jobs + transcript segments + corpus
indexes + agent/product integration.

BOB YouTube 0.1 Product Gate - 2026-06-28

Added:

docs/PRODUCT-0.1.md
GET /jobs/JOB_ID/log
GET /metrics
/data/.bob-youtube/jobs/JOB_ID.log
BOB_YOUTUBE_MAX_BODY_BYTES

Reason for this work:

The comparison is useful only if it becomes product discipline. BOB should
absorb the best developer-facing parts of other tools: one-command Docker,
clear job visibility, API specs, logs, metrics, and exact output contracts.
BOB should not chase polished Web UI scope for 0.1.

Final 0.1 image proof:

docker compose build
image id: sha256:540dfccc302b1f0c36daf11af56361ccc6cb3669bf910feb0b49ae931a955a5c
image size: 274938987
created: 2026-06-28T11:13:28.090286555Z

Final 0.1 smoke proof:

scripts/smoke_test.sh
api_discovery_ok bob-youtube
api_openapi_ok 0.1.0
api_auth_boundary_ok 401
api_health_ok /data /data/.bob-youtube/jobs
api_library_items 3
api_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_profile_job_ok agent
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_job_log_ok
api_metrics_ok 3
api_persisted_job_ok 296b4bac-bf5a-4e90-80e6-c58638c24f21
api_persisted_log_ok
BOB YouTube smoke test passed

Final 0.1 verdict:

BOB YouTube 0.1 is complete as a local/private Docker developer product.
It is ready to integrate into other projects as a Docker CLI/API ingestion
runtime. Public distribution work remains for 0.2.

Final 0.1 Release Gate - 2026-06-28

Added after completion audit:

scripts/release_gate.sh
durable rewrite of queued/running jobs to interrupted on API startup
invalid BOB_YOUTUBE_MAX_BODY_BYTES fallback
smoke proof for bad token rejection
smoke proof for oversized request body rejection

Authoritative command:

scripts/release_gate.sh

Final gate result:

static validation passed
docker compose config passed
docker compose build passed
image contract passed
full smoke passed
container cleanup passed
BOB YouTube release gate passed

Caption-First STT Fallback Flow - 2026-07-12

Purpose:

Close the 0.1 gap between a manual `bob-youtube stt MEDIA_PATH` sidecar command
and the product need for "try captions first, then run STT when captions do not
produce a transcript."

Added:

bob-youtube transcript-or-stt URL [SUB_LANGS]
POST /jobs {"command":"transcript-or-stt","url":"...","langs":"..."}
BOB_YOUTUBE_STT_FALLBACK_MEDIA for controlled product/test fallback media
docs/openapi.json command enum includes transcript-or-stt
local, Docker CLI, and API smoke coverage

Execution contract:

1. Resolve video ID and target output directory.
2. Run the normal caption transcript flow.
3. If a non-STT caption transcript exists, stop successfully.
4. If no caption transcript is produced, require BOB_YOUTUBE_STT_COMMAND.
5. Use BOB_YOUTUBE_STT_FALLBACK_MEDIA when supplied; otherwise create/find media.
6. Run the trusted STT sidecar and convert VTT to transcript text and segments.

Truth boundary:

This proves caption-missing fallback control flow and sidecar artifact handling.
It does not prove bundled speech-to-text model quality, language accuracy, or
production STT throughput. The smoke stub is still not a speech-to-text model.

Verified proof:

stt_fallback_local_ok
stt_fallback_docker_ok
api_stt_fallback_ok 3
api_metrics_ok 12 12 2 1 0 2879 3
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
image=sha256:3c0b9e5d140bec628c2cf42ceb938be1763feafa08de443f97a476984318ad14 size=274999803 created=2026-07-12T05:56:08.991538962Z
BOB YouTube release gate passed

Final image:

image=sha256:a888b0ac880befdb37f51007819357f4b083b10c6b521c04742135b5f3900a34
size=274940075
created=2026-06-28T11:19:55.918465096Z

Final smoke proof excerpt:

api_discovery_ok bob-youtube
api_openapi_ok 0.1.0
api_auth_boundary_ok 401
api_bad_auth_boundary_ok 401
api_body_limit_ok 413
api_health_ok /data /data/.bob-youtube/jobs
api_library_items 3
api_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_profile_job_ok agent
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_job_log_ok
api_metrics_ok 3
api_persisted_job_ok 3a199e0b-a735-4d31-a226-f8e5259daa55
api_persisted_log_ok
BOB YouTube smoke test passed

Final completion audit:

0.1 product definition exists: docs/PRODUCT-0.1.md
formal API contract exists and validates: docs/openapi.json
Docker image builds: passed
Docker image contains docs/openapi.json: passed
Docker healthcheck exists: passed
local CLI proof: passed
Docker CLI proof: passed
Docker API proof: passed
auth boundary proof: passed
body limit proof: passed
job JSON/log proof: passed
restart persistence proof: passed
no running containers after gate: passed

Post-0.1 Nishkaam Stabilization Review - 2026-06-28

Purpose:

Review the completed 0.1 state again without attachment to the earlier verdict,
identify weak stateful surfaces, fix cheap stability gaps, and re-run the
authoritative release gate.

Added:

scripts/stateful_api_test.sh
release gate now runs stateful restart test
docs updated for request body limit truth
docs updated for plain logs vs future structured JSONL/event logs

Stability fix:

API startup now durably rewrites queued/running persisted jobs to interrupted
on disk, not only in memory.
Invalid BOB_YOUTUBE_MAX_BODY_BYTES now falls back to 1048576.

Final post-0.1 release gate:

scripts/release_gate.sh
static validation passed
docker compose config passed
docker compose build passed
image contract passed
full smoke passed
stateful restart passed
container cleanup passed
BOB YouTube release gate passed

Final post-0.1 image:

image=sha256:6df6b89d6c64d9b2c5e6c495ba4ed390b0e155a3b040a385184371fd96573b5c
size=274941085
created=2026-06-28T12:07:10.642378542Z

Final post-0.1 proof excerpt:

api_auth_boundary_ok 401
api_bad_auth_boundary_ok 401
api_body_limit_ok 413
api_job_log_ok
api_metrics_ok 3
api_persisted_job_ok 4a4271be-a7f2-4725-a5d8-ece2643dad76
api_persisted_log_ok
stateful_interrupted_api_ok 00000000-0000-4000-8000-000000000001
stateful_interrupted_disk_ok 00000000-0000-4000-8000-000000000001

Observed external warning:

During one Docker CLI info check, YouTube returned HTTP 429 warnings, but the
yt-dlp command still succeeded and the release gate passed. This is an external
network/platform warning, not a failed 0.1 gate.

Final post-0.1 state:

running BOB containers: none
stateful test scratch directory: removed
normal job store: 3 succeeded JSON/log pairs

Version Identity And Secret-Context Hardening - 2026-06-28

Purpose:

Make BOB YouTube 0.1 self-identifying across CLI, API, docs, and Docker image
metadata, and prevent local .env secrets from entering the Docker build context.

Added:

VERSION
bob-youtube version
GET /version
Docker OCI label org.opencontainers.image.version=0.1.0
.dockerignore excludes .env and .env.*
release gate verifies /app/.env is absent

Final gate result:

scripts/release_gate.sh
api_discovery_ok bob-youtube 0.1.0
api_version_ok 0.1.0
api_openapi_ok 0.1.0
api_auth_boundary_ok 401
api_bad_auth_boundary_ok 401
api_body_limit_ok 413
api_job_log_ok
api_metrics_ok 3
api_persisted_job_ok 674f8a82-cee9-4451-a613-cfe7c9cff61f
api_persisted_log_ok
stateful_interrupted_api_ok 00000000-0000-4000-8000-000000000001
stateful_interrupted_disk_ok 00000000-0000-4000-8000-000000000001
BOB YouTube release gate passed

Final image:

image=sha256:2b09d9594d85a315e48e36b7cc66dbdfbc6ad2b956241b91e51216ec9175b0b2
size=274942078
created=2026-06-28T12:16:02.271460802Z
version label=0.1.0

Filesystem Queue Lease Backend - 2026-07-10

Purpose:

Move the 0.1 queue truth beyond a purely in-memory Python Queue while keeping
the boundary honest: durable filesystem queue records and atomic worker leases,
not a claimed Redis/SQS/Postgres distributed queue.

Added:

queue_backend = memory+filesystem-lease
queued records: outputs/.bob-youtube/jobs/.queue/*.json
lease records: outputs/.bob-youtube/jobs/.leases/*.json
GET /health exposes queue_backend, queue_dir, queue_depth, lease_depth
GET /metrics exposes queue_backend, queue_depth, lease_depth
startup re-enqueues persisted queued jobs
startup marks persisted running jobs interrupted
stateful_api_test proves both paths

Truth boundary:

This is a real durable queue/lease model for one API process and one shared
filesystem. It is not yet a Redis, SQS, Postgres, or multi-replica distributed
worker backend. That remains a public/high-volume production gap.

Verified proof:

queue_static_ok
api_health_ok /data /data/.bob-youtube/jobs 1
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_job_events_ok 6
api_metrics_ok 11 11 2 1 0 2879 3
stateful_interrupted_api_ok 00000000-0000-4000-8000-000000000001
stateful_queued_recovered_api_ok 00000000-0000-4000-8000-000000000002
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
stateful_queued_recovered_events_ok 00000000-0000-4000-8000-000000000002
stateful_interrupted_disk_ok 00000000-0000-4000-8000-000000000001
stateful_queued_recovered_disk_ok 00000000-0000-4000-8000-000000000002
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
image=sha256:df6fd7c63b3496199990ac538843c0399cae43a5cee4d49c8491aac66ee75887 size=274998544 created=2026-07-10T17:55:47.009636012Z
BOB YouTube release gate passed

Optional Embedding Sidecar Contract - 2026-07-10

Purpose:

Add a real execution point after RAG export so products can turn chunks.jsonl
into embeddings.jsonl without BOB pretending to bundle an embedding model.

Added:

bob-youtube embed [RAG_DIR_OR_CHUNKS_JSONL] [OUT_FILE]
POST /jobs {"command":"embed","path":"...","output_file":"..."}
BOB_YOUTUBE_EMBED_COMMAND
BOB_YOUTUBE_EMBED_MODEL
scripts/embed_contract_stub.py for contract smoke only
scripts/embed_manifest.py for JSONL validation and checksummed manifest

Embedding command contract:

BOB sets BOB_YOUTUBE_EMBED_INPUT to chunks.jsonl.
BOB sets BOB_YOUTUBE_EMBED_OUTPUT to embeddings.jsonl.
BOB sets BOB_YOUTUBE_EMBED_OUTPUT_DIR to the output directory.
BOB sets BOB_YOUTUBE_EMBED_MODEL to the model label.
The sidecar command must write JSONL rows with an embedding array.
BOB validates chunk/embedding row counts and writes embeddings-manifest.json.

Truth boundary:

This proves the sidecar execution, row-count validation, and manifest contract.
It does not prove bundled embedding models, vector database insertion, semantic
quality, or production embedding throughput. The smoke stub is not an
embedding model.

Verified proof after bug fix:

embed_contract_stub_rows 1
embedding_manifest_ok 1
/private/tmp/bob-embed-check/embeddings-manifest.json
/tmp/bob-embed-check/embeddings.jsonl
embed_static_contract_ok
embed_contract_local_ok
embed_contract_docker_ok
api_embed_contract_ok 2
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
image=sha256:4799778cbe905a1acd081ed357936073360c0c56b555d472792aa4560463433d size=274994716 created=2026-07-10T17:30:33.679843548Z
BOB YouTube release gate passed

Optional STT Sidecar Contract - 2026-07-10

Purpose:

Add a real integration point for caption-missing workflows without pretending
BOB bundles a speech-to-text model.

Added:

bob-youtube stt MEDIA_PATH [OUT_DIR]
POST /jobs {"command":"stt","path":"...","output_dir":"..."}
BOB_YOUTUBE_STT_COMMAND
BOB_YOUTUBE_STT_LANG
scripts/stt_contract_stub.py for contract smoke only

STT command contract:

BOB sets BOB_YOUTUBE_STT_INPUT to the media path.
BOB sets BOB_YOUTUBE_STT_OUTPUT to the target .stt.vtt file.
BOB sets BOB_YOUTUBE_STT_OUTPUT_DIR to the output directory.
BOB sets BOB_YOUTUBE_STT_LANG to the language hint, default auto.
The sidecar command must write a non-empty VTT file to BOB_YOUTUBE_STT_OUTPUT.
BOB then creates .stt.transcript.txt and .stt.segments.json.

Truth boundary:

This proves the sidecar execution and artifact conversion contract. It does not
prove bundled Whisper/faster-whisper or any actual speech recognition quality.
The smoke stub is not a transcription model.

Verified proof:

python3 -m json.tool docs/openapi.json
bash -n bin/bob-youtube scripts/smoke_test.sh scripts/stateful_api_test.sh scripts/release_gate.sh scripts/release_manifest.sh scripts/release_artifacts.sh
python3 -m py_compile scripts/clean_vtt_transcripts.py scripts/index_outputs.py scripts/bob_youtube_api.py scripts/rag_export.py scripts/stt_contract_stub.py scripts/webhook_capture.py scripts/vtt_to_segments.py
stt_static_contract_ok
stt_contract_local_ok
stt_contract_docker_ok
api_stt_contract_ok 3
api_metrics_ok 10 10 2 1 0 2073 3
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
BOB YouTube release gate passed

Latest verified image:

image=sha256:277ff5c4e9f438177098036dea2e71e356b6d48a4615161507ce6f134045cbf1
size=274991039
created=2026-07-10T17:00:55.001510794Z
version label=0.1.0

Artifact-Derived Media And Extractor Metrics - 2026-07-10

Purpose:

Close the "deeper media/extractor metrics" portion of the 0.1 observability
gap as far as the current local artifact contract honestly supports.

Added:

scripts/index_outputs.py now parses .info.json files into metadata_summary.
GET /metrics now includes library_media.
scripts/smoke_test.sh asserts parsed metadata, YouTube extractor count, and
duration totals through the Docker API smoke path.

New library_media fields:

metadata_count
metadata_parsed_count
duration_seconds_total
duration_seconds_max
filesize_or_approx_bytes_total
extractors
domains
live_statuses
availabilities
formats
extensions
resolutions
video_codecs
audio_codecs

Truth boundary:

This is artifact-derived media/extractor metadata from yt-dlp .info.json
files. It does not claim fragment-level HTTP byte counters, extractor-internal
timings, per-request network telemetry, or live yt-dlp progress hook metrics.
Those remain future work.

Verified proof:

python3 -m json.tool docs/openapi.json
bash -n bin/bob-youtube scripts/smoke_test.sh scripts/stateful_api_test.sh scripts/release_gate.sh scripts/release_manifest.sh scripts/release_artifacts.sh
python3 -m py_compile scripts/clean_vtt_transcripts.py scripts/index_outputs.py scripts/bob_youtube_api.py scripts/rag_export.py scripts/webhook_capture.py scripts/vtt_to_segments.py
python3 scripts/index_outputs.py outputs
metadata_summary_ok {'youtube': 1} 19.0 {'320x240': 1}
api_metrics_ok 9 9 2 1 0 1625 3
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
BOB YouTube release gate passed

Latest verified image:

image=sha256:2e3bae00eb45645976a76b9e4709e5990fb2ed092346f4c5c9540f47aae34d9c
size=274988805
created=2026-07-10T16:41:47.098415638Z
version label=0.1.0

Final state:

running BOB containers: none
stateful scratch directory: absent
normal job store: 3 succeeded JSON/log pairs

Machine-Readable Events And Release Manifest Hardening - 2026-06-28

Purpose:

Close the 0.1.x operations gap between a human-useful wrapper and an
agent/bulk-integration runtime by adding structured job events, security and
operations runbooks, curl examples, and a checksum/image release manifest.

Added:

/data/.bob-youtube/jobs/JOB_ID.events.jsonl
GET /jobs/JOB_ID/events
docs/SECURITY.md
docs/OPERATIONS.md
docs/examples/curl.md
scripts/release_manifest.sh
outputs/release/bob-youtube-0.1.0-manifest.txt

Verified proof:

scripts/release_gate.sh
api_job_log_ok
api_job_events_ok 3
api_metrics_ok 3
api_persisted_job_ok 32c1f6a4-b75a-43c9-8fe1-97c3476552f3
api_persisted_log_ok
api_persisted_events_ok
stateful_interrupted_api_ok 00000000-0000-4000-8000-000000000001
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
stateful_interrupted_disk_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:f46208d2ec6288f2fc749a7a92ebdfc9424cb88adb806323414c7d0dbde629ae
size=274945846
created=2026-06-28T12:24:26.724683258Z
version label=0.1.0

Docker Product Rebrand - 2026-06-28

Problem observed:

Docker Desktop was grouping the Compose environment under the generated source
checkout name. That made yt-dlp look like the product,
even though it is only the backend extraction engine.

Current product identity:

Product: Break Ordinary Boundaries (BOB) YouTube
Compose project: bob-youtube
Image: bob-youtube:local
Service: bob-youtube-api
CLI: bob-youtube
Backend engine: yt-dlp

Changed:

docker-compose.yml now sets top-level name: bob-youtube.
Docker service labels identify BOB YouTube and Break Ordinary Boundaries.
Dockerfile OCI labels identify BOB YouTube as the product and yt-dlp as
backend, without using yt-dlp/yt-dlp as the image source label.
README, docs/DOCKER.md, docs/PRODUCT-0.1.md, and outputs/WORKSPACE-MAP.md
now explain the product/backend split.

Verified proof:

docker compose config --format json -> name=bob-youtube
scripts/release_gate.sh
api_webhook_signature_ok d121f604-8d49-4168-9d6a-ba62e23bb56b
api_metrics_ok 4 4 1 1 0
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Historical verified image at that time:

image=<historical-local-image-digest>
size=274964148
created=2026-06-28T13:54:41.119976041Z
version label=0.1.0

Truth boundary:

Historical audit sections may mention local checkout origins because they record
real local development history. Docker/Compose/product-facing identity is BOB
YouTube.

Signed Webhooks And Idempotency - 2026-06-28

Purpose:

Make completed jobs callable by external products and AI agents through signed
job.finished webhooks with idempotency headers, without persisting webhook
secrets.

Added:

POST /jobs accepts webhook_url, webhook_secret, and idempotency_key.
BOB_YOUTUBE_WEBHOOK_SECRET provides a default signing secret.
Webhook secret is not persisted in job JSON, job logs, events, or API
responses.
Callback headers:
- X-BOB-YOUTUBE-Event: job.finished
- X-BOB-YOUTUBE-Delivery: DELIVERY_UUID
- X-BOB-YOUTUBE-Signature-256: sha256=HMAC_SHA256(raw_body)
- X-Idempotency-Key: configured key or JOB_ID
Delivery result is persisted under job.webhook.deliveries[].
Delivery emits webhook_delivered or webhook_failed event rows.

Test support:

scripts/webhook_capture.py captures webhook POSTs during smoke tests so the
signature and idempotency key can be verified deterministically.

Truth boundary:

Signed completion delivery is proven. A retry/backoff scheduler for failed
webhook delivery is not yet implemented. External distributed queues, cookie
secret mounts, STT fallback, RAG bundles, byte/throughput metrics, GHCR image
publishing, SBOM, and signing remain future work.

Verified proof:

scripts/release_gate.sh
api_webhook_signature_ok 3fcba597-5a25-4518-9c71-0f2b86bb9069
api_retry_history_ok 2
api_job_events_ok 6
api_metrics_ok 4 4 1 1 0
api_persisted_job_ok 244fb888-d215-4937-b35c-097c72bcca3e
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:8c8d536ded1ab47aa1a10cd60c04c67058c81e9b512db5272d7738d6a107b30c
size=274962969
created=2026-06-28T13:49:45.652648419Z
version label=0.1.0

Truth boundary:

Structured JSONL job events are now proven locally and in Docker smoke tests.
This is not yet a distributed worker queue, webhook callback system, signed
release, or full SBOM.

API Rate Limit Hardening - 2026-06-28

Purpose:

Close the local/shared deployment protection gap with a configurable per-client
fixed-window API rate limiter while preserving Docker-first simplicity.

Added:

BOB_YOUTUBE_RATE_LIMIT_PER_MINUTE=120
HTTP 429 rate-limit response
Retry-After response header
api_rate_limit_ok smoke proof
OpenAPI 429 response documentation

Verified proof:

scripts/release_gate.sh
api_job_events_ok 3
api_persisted_job_ok b43b1641-69e7-446f-b46f-0193ae4f9ee1
api_persisted_log_ok
api_persisted_events_ok
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:24e14651880cc2e0c0728a02d4b4f088e8dde46f4855e585acf7c35bb1261cf4
size=274947826
created=2026-06-28T12:30:39.487486097Z
version label=0.1.0

Truth boundary:

This is an in-process single-container limiter keyed by client IP. Public
horizontal deployments still need reverse-proxy or gateway rate limiting.

Failed-Job Summary Hardening - 2026-06-28

Purpose:

Let agents and bulk orchestrators route failed jobs from structured job JSON
instead of scraping full stderr logs.

Added:

job.failure.category
job.failure.message
job.failure.returncode
job.failure.command
failure field in plain job log
failure field in JSONL finished event

Current categories:

invalid_url
timeout
upstream_rate_limited
captions_unavailable
unsupported_url
command_failed

Verified proof:

scripts/release_gate.sh
api_failure_summary_ok invalid_url
api_job_events_ok 3
api_metrics_ok 4
api_persisted_job_ok 8087384b-343b-4870-b860-5aa97b9346db
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:7c7d1565102470c5322edd8b3ad60756057987da2d6204bd06ef257dbc70478c
size=274949406
created=2026-06-28T12:39:56.391758007Z
version label=0.1.0

Truth boundary:

Failure summaries are compact classifications for routing. Full forensic detail
remains in JOB_ID.log and JOB_ID.events.jsonl.

API Input Confinement Hardening - 2026-06-28

Purpose:

Reject unsafe or ambiguous API inputs before invoking the CLI, while preserving
the trusted local CLI as the broader operator surface.

Added API validation:

profile must exist under config/profiles/*.env
URL jobs require http or https
where video_id is a bounded safe ID and cannot contain ..
segments path must resolve inside BOB_YOUTUBE_OUTPUT_DIR

Verified proof:

scripts/release_gate.sh
api_invalid_profile_ok 400
api_invalid_url_ok 400
api_invalid_video_id_ok 400
api_invalid_segments_path_ok 400
api_failure_summary_ok command_failed
api_persisted_job_ok 03ea5c30-e254-412d-b030-487ec4000934
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:eea9c87b80f281b43d8493a5a2686c075d7b01d1c80fa8e3f2b95bcabb982401
size=274951216
created=2026-06-28T12:45:54.404644506Z
version label=0.1.0

Truth boundary:

This confines the HTTP API surface. The local bob-youtube CLI remains a trusted
operator tool and can still be used directly with broader yt-dlp capabilities.

Request ID Correlation Hardening - 2026-06-28

Purpose:

Let products, agents, and operators correlate API calls, job JSON, job logs,
and JSONL events without guessing from timestamps or stderr.

Added:

X-Request-ID request header support
X-Request-ID response header
request_id in JSON API responses
request_id persisted in JOB_ID.json
request_id written to JOB_ID.log
request_id included in JOB_ID.events.jsonl rows
CORS allow-header includes X-Request-ID

Verified proof:

scripts/release_gate.sh
api_discovery_ok bob-youtube 0.1.0
api_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_job_events_ok 3
api_persisted_job_ok 34c2e454-64aa-42eb-b875-da746b9fce54
api_persisted_events_ok
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:3987611c82c5efaf40b5cb1a28bda073082836ecc2c6860ddb4918171bd51bed
size=274952609
created=2026-06-28T13:07:22.63557538Z
version label=0.1.0

Truth boundary:

Request IDs are correlation metadata only. They are not authentication,
authorization, signing, or idempotency keys.

Workspace Consistency Cleanup - 2026-06-28

Purpose:

Review the whole workspace for stale current-roadmap claims and rebuildable
cache residue after the 0.1.x hardening passes.

Updated current docs:

AGENTS.md
docs/COMPARISON.md
docs/VISION.md
docs/PRODUCT-0.1.md
docs/RELEASE.md

Corrections made:

Per-job JSONL events and failure summaries are no longer listed as future gaps.
Request body limits and in-process API rate limits are no longer listed as
missing 0.1 work.
The metrics endpoint is treated as present, with richer duration/byte/failure
metrics still pending.
Curl examples are treated as present; Python and Node examples remain future
integration examples.
Removed rebuildable upstream audit cache: work/yt-dlp/.pytest_cache
Removed rebuildable upstream audit bytecode caches under work/yt-dlp/

Truth boundary:

Historical proof sections above are intentionally preserved as chronology. The
latest current-state docs now carry the corrected remaining-gap list.

Latest verification after cleanup:

scripts/release_gate.sh
api_invalid_profile_ok 400
api_invalid_url_ok 400
api_invalid_video_id_ok 400
api_invalid_segments_path_ok 400
api_failure_summary_ok command_failed
api_job_events_ok 3
api_metrics_ok 4
api_persisted_job_ok eb788ab6-8c6d-4646-8938-ebbcaa92a50b
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:adc3aa63eeacb0a2125802d2275b0b288fc5ccad1c921300f0ce3225dff3862c
size=274953499
created=2026-06-28T13:18:11.09017525Z
version label=0.1.0

Final post-ledger-correction proof:

scripts/release_gate.sh
api_persisted_job_ok ed26633c-f198-4ddc-bb88-8dc3a6c81d7e
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Metrics Depth Hardening - 2026-06-28

Purpose:

Close the proven observability gap for job durations and failure-category
counts while keeping byte/throughput and deeper media/extractor metrics as
honest future work.

Added:

GET /metrics now reports:
- jobs_total
- jobs_by_status
- jobs_finished_total
- jobs_running_total
- jobs_by_failure_category
- job_durations_seconds {count,min,max,avg}
- library_items
- output_root
- job_store
- rate_limit_per_minute

Updated current docs:

AGENTS.md
docs/COMPARISON.md
docs/DOCKER.md
docs/PRODUCT-0.1.md
docs/RELEASE.md
docs/VISION.md
docs/openapi.json
outputs/BOB-YOUTUBE-FEATURE-MATRIX.md
outputs/WORKSPACE-MAP.md

Truth boundary:

Duration metrics are derived from each job's started_at and finished_at
timestamps. Failure-category metrics are derived from job.failure.category.
This does not yet measure bytes written, throughput, per-extractor counters, or
media-level transfer rates.

Verified proof:

scripts/release_gate.sh
api_invalid_profile_ok 400
api_invalid_url_ok 400
api_invalid_video_id_ok 400
api_invalid_segments_path_ok 400
api_failure_summary_ok command_failed
api_job_events_ok 3
api_metrics_ok 4 4 1
api_persisted_job_ok 5f60f1f4-5382-4343-9a83-57d405572a48
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:28eb04205a530ec34b4cc581526241dcc01dacae46c1adc36b59cdfbdce7ab08
size=274954500
created=2026-06-28T13:24:00.818728593Z
version label=0.1.0

Workspace Runtime Review - 2026-06-28

Checked at:

2026-06-28T13:31:01Z

Docker state:

docker compose ps --all: no containers
docker ps --filter ancestor=bob-youtube:local: no running containers
docker run --rm -v "$PWD/outputs:/data" bob-youtube:local doctor: passed
image=sha256:28eb04205a530ec34b4cc581526241dcc01dacae46c1adc36b59cdfbdce7ab08
size=274954500
created=2026-06-28T13:24:00.818728593Z
version label=0.1.0

Workspace inventory before rebuildable-cache cleanup:

top-level entries: .dockerignore, .env.example, .workspace-name, AGENTS.md,
DIVYANSHU.md, Dockerfile, README.md, VERSION, bin, config,
docker-compose.yml, docs, outputs, scripts, work
files: 2379
directories: 116
symlinks: 0
workspace size: 261M
outputs: 25 files, 7 directories, 99M
work: 2322 files, 102 directories, 161M

Largest material:

outputs/youtube-VIDEO_ID/VIDEO_ID.mp3: 97M
work/yt-dlp/.git/objects pack: 60M
work/global-install/yt-dlp_macos: 35M
work/global-install/yt-dlp-active-backup: 35M

Cleanup performed:

Removed rebuildable __pycache__ directories under work/yt-dlp.

Confirmed Docker image boundary:

.dockerignore excludes work, downloaded outputs, .git, __pycache__, .env, and
cache directories.
The image includes only product files plus outputs/WORKSPACE-MAP.md and
outputs/BOB-YOUTUBE-FEATURE-MATRIX.md.

Stateful engineering still required:

1. Real queue/worker model with bounded concurrency.
2. Per-job retry policy and structured retry/failure history.
3. Webhook callbacks with signing/idempotency.
4. Byte/throughput and deeper media/extractor metrics.
5. Explicit cookie/profile secret mount contract.
6. Optional STT fallback when captions do not exist.
7. Embedding/RAG export bundles beyond timestamped segment JSON.
8. GHCR/versioned public image, SBOM, signatures, and checksum artifacts.

Truth boundary:

The local/private Docker developer product is clean and release-gate proven.
The remaining work is production orchestration and distribution depth, not the
basic yt-dlp extraction wrapper.

Post-review release proof:

scripts/release_gate.sh
api_metrics_ok 4 4 1
api_persisted_job_ok 4dc3b078-eeda-49f6-9cff-df449c8125aa
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed
image=sha256:6c83a8bea2a51cca5df3893ebebca194c4353c3e2d3cf3838bf9020b751f3cc6
size=274955685
created=2026-06-28T13:32:37.503972846Z
version label=0.1.0

Bounded Worker Queue And Retry History - 2026-06-28

Purpose:

Move the API from unbounded one-thread-per-async-job execution to a bounded
worker queue with persisted retry history, while keeping external/distributed
queue backends as future public-production work.

Added:

BOB_YOUTUBE_WORKERS controls in-process async worker count, range 1..16.
BOB_YOUTUBE_MAX_RETRIES controls default per-job retries, range 0..5.
POST /jobs accepts max_retries.
Async jobs emit queued, dequeued, started, attempt_started, attempt_finished,
retry_scheduled, and finished events.
Job JSON persists max_retries, attempt, and attempts[].
Job logs include max_retries, attempt, and attempts.
GET /metrics includes queue_depth, worker_count, and default_max_retries.
GET /health includes worker_count.

GitHub comparison signal:

GitHub yt-dlp wrapper projects commonly compete on queue/Web/API operation,
Docker deployment, and background job visibility. BOB's differentiator remains
the transcript/corpus/agent contract, but bounded workers and retry visibility
are now proven local/private 0.1 capabilities instead of only roadmap text.

Truth boundary:

This is a bounded in-process queue for one API process. It is not yet an
external distributed queue for multi-replica workers. Webhook callbacks,
cookie secret mounts, STT fallback, RAG bundles, SBOM/signing, and public image
publishing remain future work.

Verified proof:

scripts/release_gate.sh
api_health_ok /data /data/.bob-youtube/jobs 1
api_failure_summary_ok command_failed
api_retry_history_ok 2
api_job_events_ok 6
api_metrics_ok 4 4 1 1 0
api_persisted_job_ok c5012f89-d98c-447d-8de6-2d3519349fd0
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:962690856d86fef6424acc5f39166205674521f77fa04c4e219bcbbce9e8ad6b
size=274958242
created=2026-06-28T13:41:27.168356216Z
version label=0.1.0

Byte And Artifact Metrics Hardening - 2026-06-28

Purpose:

Close the byte/throughput portion of the 0.1 observability gap without
claiming extractor-internal telemetry that is not yet implemented.

Added:

Each finished job now persists output_stats in JOB_ID.json.
JOB_ID.log includes output_stats.
The final finished event includes output_stats.
GET /metrics aggregates per-job output bytes and artifact type counters.
GET /metrics reports throughput summaries where job duration is available.
GET /metrics reports library_output byte totals and artifact counts from
library-index.json.

New per-job output_stats fields:

changed_file_count
bytes_written
bytes_per_second
counts_by_type
bytes_by_type
files
files_truncated

New service metrics:

job_output_bytes_total
job_output_changed_files_total
job_output_bytes_by_type
job_output_counts_by_type
job_output_bytes_written
job_output_throughput_bytes_per_second
library_output

Verification design:

scripts/smoke_test.sh creates a smoke-owned local VTT fixture under
outputs/bob-smoke-metrics and runs the API segments job against it. This proves
output_stats and service-level byte metrics without depending on YouTube for
the byte-metrics assertion.

Truth boundary:

This proves durable artifact byte metrics, per-job output deltas, and
throughput summaries derived from job duration. It does not yet prove
yt-dlp extractor-internal counters, fragment-level transfer rates, upstream
HTTP byte counters, or per-site extractor telemetry.

Verified proof:

scripts/release_gate.sh
api_library_items 4
api_webhook_signature_ok 0c4d7f86-ba7e-4230-a4db-b92decd621ec
api_output_stats_ok 322 1
api_retry_history_ok 2
api_job_events_ok 6
api_metrics_ok 5 5 1 1 0 322
api_persisted_job_ok 29d6ea97-018b-45e1-bdcd-73f556b91298
api_rate_limit_ok 429
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:75f003711164754a35c9f8ccb51de6842227e0ccb84e05fc67815f15a0efd018
size=274969115
created=2026-06-28T14:05:35.91937347Z
version label=0.1.0

Cookie Secret Mount Contract - 2026-06-28

Purpose:

Make authenticated YouTube cookie usage explicit, mounted, testable, and
non-leaky without mounting browser profiles or baking secrets into the image.

Added:

BOB_YOUTUBE_COOKIES_FILE points to one Netscape cookies.txt file.
bob-youtube passes the file to yt-dlp with --cookies when configured.
bob-youtube fails early if BOB_YOUTUBE_COOKIES_FILE is set but missing or
unreadable.
bob-youtube doctor reports only configured/readable/path status.
Docker Compose passes BOB_YOUTUBE_COOKIES_FILE through when set.
GET /health reports cookie_contract metadata only.
.dockerignore excludes cookies, secrets, cookies.txt, and *.cookies.

Required operational contract:

Mount cookies read-only, for example:
-v "$PWD/secrets/cookies.txt:/run/secrets/bob-youtube/cookies.txt:ro"
-e BOB_YOUTUBE_COOKIES_FILE=/run/secrets/bob-youtube/cookies.txt

Do not mount browser profile directories into Docker.
Do not place cookies under /data.
Do not commit cookie files.
Rotate/delete the host cookie file when the session should no longer be used.

Truth boundary:

This proves explicit file-based cookie support. It does not implement an
external secret manager, browser-profile extraction inside Docker, automatic
cookie rotation, or SaaS account/session management.

Verified proof:

scripts/release_gate.sh
cookie_mount_local_ok
cookie_mount_docker_ok
api_cookie_contract_ok /run/secrets/bob-youtube/cookies.txt
api_webhook_signature_ok 0be99494-d5a1-4f79-acb2-9e1dcc050a35
api_output_stats_ok 322 1
api_retry_history_ok 2
api_metrics_ok 5 5 1 1 0 322
api_persisted_job_ok 499c3ebc-3771-4df7-9aa9-eba3f9c3c152
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:04fc5c60257920c155bd238b73228eb4188e55a66529722c6b86c8a12d319603
size=274971958
created=2026-06-28T14:15:36.346079637Z
version label=0.1.0

RAG Export Bundle Contract - 2026-06-28

Purpose:

Move beyond per-video segment JSON by producing a direct handoff bundle for
agent retrieval, vector database import, and downstream embedding pipelines.

Added:

scripts/rag_export.py creates RAG-friendly export bundles.
bob-youtube rag [PATH] [OUT_DIR] is the CLI command.
POST /jobs accepts command="rag" with path and output_dir confined to /data.
RAG bundle folders are excluded from library-index video-folder detection.

Bundle files:

chunks.jsonl
corpus.txt
manifest.json

chunks.jsonl rows include:

id
video_id
language
chunk_index
source_segments
source_vtt
start
end
segment_indices
text
char_count

manifest.json includes:

schema=bob-youtube-rag-export.v1
source segment files
chunk_count
max_chars
sha256 checksums for chunks.jsonl and corpus.txt

Truth boundary:

This proves retrieval/export bundle generation. It does not execute an
embedding model, manage a vector database, deduplicate a corpus, or sign the
corpus manifest.

Verified proof:

scripts/release_gate.sh
rag_export_local_ok
rag_export_docker_ok
api_rag_export_ok 1303
api_metrics_ok 6 6 1 1 0 1625
api_persisted_job_ok 7ffe2178-5908-4cff-a8bb-39c620fba369
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
outputs/release/bob-youtube-0.1.0-manifest.txt
BOB YouTube release gate passed

Latest verified image:

image=sha256:17cd76a17da43b1eeb958dc43c60b17547c22267213cabc10519679c2c2ede8f
size=274974770
created=2026-06-28T14:25:43.594353917Z
version label=0.1.0

Release Artifacts, SBOM, Checksums, And Signatures - 2026-06-28

Purpose:

Close the local release-proof gap for versioned image tags, SBOM, checksums,
and signed release artifacts while keeping public GHCR push truth separate.

Added:

scripts/release_artifacts.sh
local image tag: bob-youtube:0.1.0
local GHCR-style tags:
  ghcr.io/breakordinaryboundaries/bob-youtube:0.1.0
  ghcr.io/breakordinaryboundaries/bob-youtube:latest
CycloneDX SBOM from dpkg and pip package inventories
image inspect JSON
SHA256SUMS
OpenSSL Ed25519 signature over SHA256SUMS
signing public key
release artifact summary JSON

Artifacts:

outputs/release/bob-youtube-0.1.0-release-artifacts.json
outputs/release/bob-youtube-0.1.0-image-inspect.json
outputs/release/bob-youtube-0.1.0-sbom.cdx.json
outputs/release/bob-youtube-0.1.0-sbom-dpkg.tsv
outputs/release/bob-youtube-0.1.0-sbom-pip.json
outputs/release/bob-youtube-0.1.0-SHA256SUMS
outputs/release/bob-youtube-0.1.0-SHA256SUMS.sig
outputs/release/bob-youtube-0.1.0-signing-public.pem

Truth boundary:

This proves local release artifacts and local Docker tags. It does not prove a
public GHCR push unless BOB_YOUTUBE_PUSH_GHCR=1 is run with authenticated
Docker credentials and the pushed digest is recorded.

Verified proof:

scripts/release_gate.sh
rag_export_local_ok
rag_export_docker_ok
api_rag_export_ok 1303
api_metrics_ok 6 6 1 1 0 1625
release_artifacts_ok 0.1.0
outputs/release/bob-youtube-0.1.0-release-artifacts.json
outputs/release/bob-youtube-0.1.0-sbom.cdx.json
outputs/release/bob-youtube-0.1.0-SHA256SUMS
outputs/release/bob-youtube-0.1.0-SHA256SUMS.sig
BOB YouTube release gate passed

Latest verified image:

image=sha256:c494e8d2defb83fc595aa0eb5050f448b855365b6d514a1c0e0d205abf44e02d
size=274977755
created=2026-06-28T14:34:26.339914507Z
version label=0.1.0

Webhook Retry And Backoff Scheduler - 2026-06-28

Purpose:

Upgrade signed job.finished callbacks from one-shot delivery to bounded,
persisted retry/backoff delivery while keeping idempotency and HMAC signing.

Added:

BOB_YOUTUBE_WEBHOOK_MAX_ATTEMPTS controls default attempts, range 1..5.
BOB_YOUTUBE_WEBHOOK_BACKOFF_SECONDS controls base exponential backoff.
POST /jobs accepts webhook_max_attempts.
Each delivery attempt has a new delivery_id and signed payload.
Payload and headers include attempt and max_attempts.
Failed attempts persist under job.webhook.deliveries[].
Failed attempts emit webhook_failed events.
Scheduled retries emit webhook_retry_scheduled events.
Successful attempts emit webhook_delivered events.

Truth boundary:

This is bounded in-process post-job retry. It is not a durable external
webhook queue that survives process death mid-backoff, and it is not a
distributed callback scheduler.

Verified proof:

scripts/release_gate.sh
api_webhook_signature_ok df74f155-664c-4764-945c-f083fe7d053b
api_webhook_retry_history_ok 2 200
api_webhook_retry_signature_ok 2
api_metrics_ok 7 7 1 1 0 1625
api_persisted_job_ok 08ddda84-a444-4c2c-976d-10a19f144029
release_artifacts_ok 0.1.0
BOB YouTube release gate passed

Latest verified image:

image=sha256:80057d9a090b9f378e3bbccfd5d98c62edaeade1120aa55d144326ee6a278b68
size=274979853
created=2026-06-28T14:44:10.689501847Z
version label=0.1.0

No-Fake-Work GitHub Research Gate - 2026-06-28

Purpose:

Re-check the current GitHub/yt-dlp ecosystem before adding more claimed
features, and prevent fake STT or cosmetic roadmap work from being counted as
real BOB YouTube capability.

New document:

docs/GITHUB-RESEARCH-2026-06-28.md

Observed public baseline on 2026-06-28:

alexta69/metube: 13983 stars, updated 2026-06-28, self-hosted yt-dlp Web UI.
marcopiovanello/yt-dlp-web-ui: 2491 stars, updated 2026-06-27, Web UI/RPC.
arabcoders/ytptube: 969 stars, updated 2026-06-25, Web UI with concurrency,
presets, scheduling, and live-stream support.
kevinwatt/yt-dlp-mcp: 251 stars, updated 2026-06-26, MCP bridge for media/LLMs.
Vasysik/yt-dlp-host: 30 stars, updated 2026-05-10, dockerized API host.
jim60105/docker-yt-dlp: 17 stars, updated 2026-06-11, raw yt-dlp Docker image.
samson-art/transcriptor-mcp: 12 stars, updated 2026-06-22, transcript MCP.
fvadicamo/yt-dlp-api: 4 stars, updated 2026-05-12, REST API.

Local upstream feature anchors re-checked from work/yt-dlp/README.md:

subtitles: --write-subs, --write-auto-subs
authenticated extraction: --cookies, --cookies-from-browser
deduplication: --download-archive
speed: --concurrent-fragments
partial/chapter workflows: --download-sections, --split-chapters
structured embedding surfaces: --print, --progress-template, JSON output
extractor-specific tuning: --extractor-args
plugin surface: yt_dlp_plugins extractor/postprocessor namespace
SponsorBlock: mark/remove chapters

Truth boundary:

BOB YouTube should not pretend to beat mature Web UI projects on frontend
polish. Its honest lane is Docker/OpenAPI/product integration, a stable /data
corpus contract, transcript segments, durable job proof, webhooks, metrics, and
release artifacts.

STT remains future work until a real speech-to-text engine or sidecar is
integrated and tested against media without captions. A dummy STT command can
test an interface, but it is not actual transcription capability.

Next honest engineering slice:

Prefer an advanced upstream-power lane with output-root confinement and clear
risk boundaries, or real public distribution proof. Do not spend the next slice
on fake STT or cosmetic UI claims.

Advanced Raw yt-dlp Lane - 2026-07-10

Purpose:

Close the honest gap between upstream yt-dlp power and the BOB YouTube product
contract without turning the API into arbitrary shell execution.

Added:

bob-youtube raw YT_DLP_ARGS...
POST /jobs {"command":"raw","args":[...]}
docs/openapi.json command enum includes raw
README, Docker, Product, Vision, Security, Operations, Research, Release, and
feature-matrix docs mention the raw boundary.

Safety contract:

raw uses yt-dlp --ignore-config.
BOB still owns output root, archive, template, configured cookies, and plugins.
External yt-dlp plugins are disabled with YTDLP_NO_PLUGINS=1.
The API accepts an explicit args array, not a shell command string.
Path/archive/config/cookie/plugin/command-execution overrides are rejected.

Blocked examples:

-P / --paths
-o / --output
--download-archive
--config-locations
--exec
--print-to-file
--cookies / --cookies-from-browser
--plugin-dirs / --use-postprocessor
--batch-file
--netrc

Truth boundary:

This is a trusted advanced operator/product lane. It is not hostile-user
sandboxing, not a public SaaS permission model, and not a replacement for
future typed presets around the most common upstream yt-dlp options.

Verified proof:

python3 -m json.tool docs/openapi.json
bash -n bin/bob-youtube scripts/smoke_test.sh scripts/stateful_api_test.sh scripts/release_gate.sh scripts/release_manifest.sh scripts/release_artifacts.sh
python3 -m py_compile scripts/clean_vtt_transcripts.py scripts/index_outputs.py scripts/bob_youtube_api.py scripts/rag_export.py scripts/webhook_capture.py scripts/vtt_to_segments.py
bob-youtube raw --version -> 2026.06.09
bob-youtube raw --simulate --print id https://www.youtube.com/watch?v=jNQXAC9IVRw -> jNQXAC9IVRw
raw_local_ok
raw_docker_ok
api_raw_ok jNQXAC9IVRw
api_raw_block_ok command_failed
api_metrics_ok 9 9 2 1 0 1625
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
BOB YouTube release gate passed

Latest verified image:

image=sha256:30cce0549df39d67af061034a09fe652b504a3f96e7e668daefabe4ecb1cbe83
size=274986459
created=2026-07-10T16:32:30.037605588Z
version label=0.1.0

Prometheus Metrics and Durable Job Read Truth - 2026-07-12

Purpose:

Make BOB YouTube easier to operate from Docker, agents, and monitoring stacks by
adding a Prometheus scrape endpoint while hardening the API's job read path to
prefer durable job JSON over stale process memory.

Added:

GET /metrics.prom
docs/openapi.json /metrics.prom contract
scripts/bob_youtube_api.py render_prometheus_metrics()
scripts/bob_youtube_api.py job_snapshot()
scripts/bob_youtube_api.py jobs_snapshot()

Prometheus metric families now exposed:

bob_youtube_jobs_total
bob_youtube_jobs_by_status
bob_youtube_jobs_finished_total
bob_youtube_jobs_running_total
bob_youtube_queue_depth
bob_youtube_lease_depth
bob_youtube_worker_count
bob_youtube_jobs_by_failure_category
bob_youtube_job_duration_seconds
bob_youtube_job_output_bytes_total
bob_youtube_job_output_changed_files_total
bob_youtube_job_output_bytes_by_type
bob_youtube_job_output_files_by_type
bob_youtube_library_total_size_bytes
bob_youtube_library_artifacts_by_type
bob_youtube_library_media_metadata_parsed_total
bob_youtube_library_media_duration_seconds_total
bob_youtube_library_media_extractors

Durable job read hardening:

GET /jobs
GET /jobs/JOB_ID
GET /jobs/JOB_ID/log
GET /jobs/JOB_ID/events
GET /metrics
GET /metrics.prom

These paths now refresh from outputs/.bob-youtube/jobs/JOB_ID.json or /data/.bob-youtube/jobs/JOB_ID.json where available. This keeps the HTTP API aligned with the durable job store after async worker updates and after API restart.

Truth boundary:

This is Prometheus text exposition for the current single-node filesystem queue
and artifact-derived metrics. It is not yet external Redis/SQS/Postgres
queueing, not fragment-level yt-dlp progress telemetry, and not network-level
per-fragment extractor counters.

Verified proof:

job_snapshot_static_ok
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_job_log_ok
api_job_events_ok 6
api_metrics_ok 12 12 2 1 0 2879 3
api_prometheus_metrics_ok
api_persisted_job_ok 1cd54568-2a72-4066-9c7d-801686653fad
api_persisted_log_ok
api_persisted_events_ok
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
BOB YouTube release gate passed

Latest verified image:

image=sha256:20299345c8127d939da330973871c34cbe7a22707bc36dd175b9e6e2a0f48371
size=275003282
created=2026-07-12T06:10:53.078160552Z
version label=0.1.0

Stable 0.1 Readiness Verdict - 2026-07-12

Purpose:

Complete the 0.1 release-readiness boundary with a machine-readable readiness
report that can be regenerated by the release gate.

Added:

scripts/release_readiness.sh
outputs/release/bob-youtube-0.1.0-readiness.json
scripts/release_gate.sh now runs release_readiness.sh after release artifacts
README.md, docs/OPERATIONS.md, docs/PRODUCT-0.1.md, docs/DOCKER.md updated

Readiness verdict:

stable-0.1-local-private-ready

Readiness checks currently proven:

openapi_required_paths
version_file
truth_docs_name_version
truth_ledger_latest_gate
release_artifacts_exist
ghcr_truth_boundary
release_signature
docker_image_exists
docker_image_version_label
docker_image_brand_label
docs_image_proof_boundary
docker_image_excludes_env
docker_image_has_openapi

Truth boundary:

Stable 0.1 means local/private Docker developer product readiness. It does not
claim public GHCR publication, distributed queueing, bundled STT, bundled
embeddings, external secret manager support, or fragment-level network
telemetry.

Verified proof:

readiness_static_ok
release_readiness_ok stable-0.1-local-private-ready
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_metrics_ok 12 12 2 1 0 2879 4
api_prometheus_metrics_ok
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
outputs/release/bob-youtube-0.1.0-readiness.json
release_readiness_ok stable-0.1-local-private-ready
BOB YouTube release gate passed

Historical verified image at that time:

image=<historical-local-image-digest>
size=275006050
created=<historical-local-image-created-at>
version label=0.1.0

Public GitHub Repository Readiness - 2026-07-12

Purpose:

Prepare BOB YouTube for first public GitHub publication under the BOB product
series: BOB = Break Ordinary Boundaries, product domain = YouTube, public name =
BOB YouTube.

Added:

.gitignore
LICENSE
CODE_OF_CONDUCT.md
CONTRIBUTING.md
SECURITY.md
docs/BRAND.md
docs/LEGAL.md
docs/PUBLIC-RELEASE.md
.github/ISSUE_TEMPLATE/bug_report.md
.github/ISSUE_TEMPLATE/feature_request.md
.github/PULL_REQUEST_TEMPLATE.md
README.md public product/founder language
docs/assets/bob-youtube-banner.webp README banner

Public language:

BOB YouTube
Break Ordinary Boundaries for YouTube

Founder identity source checked:

https://abcsteps.com/about/
https://github.com/div197

Public-safe Git dry-run result:

public_add_count=56
public_git_dry_run_ok
The dry run adds source, docs, scripts, config, and curated output docs only.
It excludes work/, downloaded media, smoke outputs, job ledgers, release
bundles, transcripts, and upstream yt-dlp git history.

Truth boundary:

This workspace is ready to stage a clean public GitHub repository. Do not push
the raw workspace with local proof/download folders manually. Use the root
.gitignore or a clean staging directory so only the intended public source set
is published.

Verified proof:

public_release_static_ok
public_git_dry_run_ok
BOB YouTube smoke test passed
BOB YouTube stateful API test passed
release_artifacts_ok 0.1.0
release_readiness_ok stable-0.1-local-private-ready
BOB YouTube release gate passed

Latest verified image proof:

outputs/release/bob-youtube-0.1.0-image-inspect.json
outputs/release/bob-youtube-0.1.0-readiness.json

FilesExpand file tree

DIVYANSHU.md

Latest commit

History

DIVYANSHU.md

File metadata and controls

DIVYANSHU.md

Executive Verdict

Current Machine Truth

Repository Scale

Feature Truth

Architecture Truth

Security Truth

Test Truth

Dependency Truth

Global Installation Decision

Option A: Update Current Pip-Based Global Install

Option B: pipx Global CLI

Option C: Homebrew yt-dlp

Docker Decision

Workspace Usage Doctrine

Capability Assessment

Final Recommendation

Next Proof Gate

Installed State Proof - 2026-06-28

System Upgrade And Repair Proof - 2026-06-28

yt-dlp Feature And Subtitle Truth - 2026-06-28

Exact Feature Families

Subtitle Flow From Source

Language Detection Truth

Live Subtitle Proof

Bulk Subtitle Doctrine

Workspace Root Organization - 2026-06-28

End-To-End Workflow

BOB YouTube Operating Layer - 2026-06-28

User Terminal Confirmation - 2026-06-28

Feature Expansion - 2026-06-28

Docker Packaging - 2026-06-28

API Service Mode - 2026-06-28

Full Smoke Suite - 2026-06-28

Timestamped Segments And Async Jobs - 2026-06-28

Runtime Profiles - 2026-06-28

Docker/API Product Hardening - 2026-06-28

Post-Restart Alignment Review - 2026-06-28

Docker-First Release Review - 2026-06-28

GitHub Ecosystem Comparison And USP - 2026-06-28

BOB YouTube 0.1 Product Gate - 2026-06-28

Final 0.1 Release Gate - 2026-06-28

Caption-First STT Fallback Flow - 2026-07-12

Post-0.1 Nishkaam Stabilization Review - 2026-06-28

Version Identity And Secret-Context Hardening - 2026-06-28

Filesystem Queue Lease Backend - 2026-07-10

Optional Embedding Sidecar Contract - 2026-07-10

Optional STT Sidecar Contract - 2026-07-10

Artifact-Derived Media And Extractor Metrics - 2026-07-10

Machine-Readable Events And Release Manifest Hardening - 2026-06-28

Docker Product Rebrand - 2026-06-28

Signed Webhooks And Idempotency - 2026-06-28

API Rate Limit Hardening - 2026-06-28

Failed-Job Summary Hardening - 2026-06-28

API Input Confinement Hardening - 2026-06-28

Request ID Correlation Hardening - 2026-06-28

Workspace Consistency Cleanup - 2026-06-28

Metrics Depth Hardening - 2026-06-28

Workspace Runtime Review - 2026-06-28

Bounded Worker Queue And Retry History - 2026-06-28

Byte And Artifact Metrics Hardening - 2026-06-28

Cookie Secret Mount Contract - 2026-06-28

RAG Export Bundle Contract - 2026-06-28

Release Artifacts, SBOM, Checksums, And Signatures - 2026-06-28

Webhook Retry And Backoff Scheduler - 2026-06-28

No-Fake-Work GitHub Research Gate - 2026-06-28

Advanced Raw yt-dlp Lane - 2026-07-10

Prometheus Metrics and Durable Job Read Truth - 2026-07-12

Stable 0.1 Readiness Verdict - 2026-07-12

Public GitHub Repository Readiness - 2026-07-12