Skip to content

RELEASE.md

Latest commit

 

History

History
386 lines (315 loc) · 8.98 KB

RELEASE.md

File metadata and controls

386 lines (315 loc) · 8.98 KB
Raw

BOB YouTube Release Readiness

Date: 2026-06-28

This document reviews BOB YouTube from a Docker-first fresh-user perspective.

Release Position

BOB YouTube is ready as a local/private Docker package for:

  • one-command video inspection,
  • transcript/audio/video extraction,
  • mounted-volume corpus generation,
  • token-gated local HTTP API use,
  • basic async jobs,
  • durable job history in /data/.bob-youtube/jobs,
  • AI-agent integration through JSON and filesystem artifacts.

It is not yet a public multi-tenant SaaS-grade service. The remaining gap is datacenter orchestration, not basic extraction capability.

One-Command Docker CLI

No Homebrew, local yt-dlp, local ffmpeg, or local deno install is required for container users.

docker run --rm \
  -v "$PWD/bob-youtube-data:/data" \
  bob-youtube:local doctor

Download transcript and audio:

docker run --rm \
  -v "$PWD/bob-youtube-data:/data" \
  bob-youtube:local both 'https://www.youtube.com/watch?v=VIDEO_ID'

Index outputs:

docker run --rm \
  -v "$PWD/bob-youtube-data:/data" \
  bob-youtube:local index

One-Command API

docker run --rm \
  -e BOB_YOUTUBE_API_TOKEN='change-this-token' \
  -p 8080:8080 \
  -v "$PWD/bob-youtube-data:/data" \
  bob-youtube:local api 8080

Discovery:

curl -fsS http://127.0.0.1:8080/
curl -fsS http://127.0.0.1:8080/version
curl -fsS http://127.0.0.1:8080/openapi.json

Health:

curl -fsS \
  -H 'Authorization: Bearer change-this-token' \
  http://127.0.0.1:8080/health

Create an async job:

curl -fsS -X POST http://127.0.0.1:8080/jobs \
  -H 'Authorization: Bearer change-this-token' \
  -H 'Content-Type: application/json' \
  -d '{"command":"both","url":"https://www.youtube.com/watch?v=VIDEO_ID","async":true}'

Poll a job:

curl -fsS \
  -H 'Authorization: Bearer change-this-token' \
  http://127.0.0.1:8080/jobs/JOB_ID

API Contract

Formal API spec:

docs/openapi.json
GET /openapi.json

Operational endpoints:

GET  /
GET  /version
GET  /openapi.json
GET  /health
GET  /library
GET  /library?refresh=1
GET  /where/VIDEO_ID
GET  /jobs
GET  /jobs/JOB_ID
GET  /jobs/JOB_ID/log
GET  /jobs/JOB_ID/events
GET  /metrics
GET  /metrics.prom
POST /jobs

Allowed job commands:

info
subs
metadata
thumbnail
transcript
transcript-or-stt
segments
audio
video
both
index
rag
embed
stt
raw
where

Filesystem Contract

Mounted data root:

/data

Per-video artifacts:

/data/VIDEO_ID/VIDEO_ID.info.json
/data/VIDEO_ID/VIDEO_ID.jpg
/data/VIDEO_ID/VIDEO_ID.mp3
/data/VIDEO_ID/VIDEO_ID.mkv
/data/VIDEO_ID/VIDEO_ID.LANG.vtt
/data/VIDEO_ID/VIDEO_ID.LANG.transcript.txt
/data/VIDEO_ID/VIDEO_ID.LANG.segments.json

Corpus indexes:

/data/library-index.json
/data/library-index.tsv
/data/rag-export/chunks.jsonl
/data/rag-export/corpus.txt
/data/rag-export/manifest.json

Job history:

/data/.bob-youtube/jobs/JOB_ID.json
/data/.bob-youtube/jobs/JOB_ID.log
/data/.bob-youtube/jobs/JOB_ID.events.jsonl

Runtime Configuration

BOB_YOUTUBE_OUTPUT_DIR=/data
BOB_YOUTUBE_ARCHIVE=/data/bob-youtube-archive.txt
BOB_YOUTUBE_JOB_DIR=/data/.bob-youtube/jobs
BOB_YOUTUBE_PROFILE=datacenter
BOB_YOUTUBE_API_TOKEN=change-this-token
BOB_YOUTUBE_CORS_ORIGIN=
BOB_YOUTUBE_MAX_BODY_BYTES=1048576
BOB_YOUTUBE_RATE_LIMIT_PER_MINUTE=120

Version identity:

VERSION
bob-youtube version
GET /version
OCI label org.opencontainers.image.version

Profiles:

default
consumer
research
datacenter
agent

Security Truth

Current security is appropriate for local/private deployment:

  • token-gated operational API endpoints,
  • request body size limit through BOB_YOUTUBE_MAX_BODY_BYTES,
  • built-in per-client fixed-window rate limit through BOB_YOUTUBE_RATE_LIMIT_PER_MINUTE,
  • API input confinement for profiles, URL schemes, video IDs, and output paths,
  • request id correlation across responses, jobs, logs, and events,
  • explicit read-only cookie file mount through BOB_YOUTUBE_COOKIES_FILE, with only mount metadata exposed in health output,
  • no secrets baked into the image,
  • mounted /data keeps runtime state outside the image,
  • Docker image excludes work/, outputs/, media, and upstream git history.

Not yet public hostile-network hardened:

  • no TLS termination inside the container,
  • no per-user RBAC,
  • no external secret manager integration.

For public exposure, put it behind a reverse proxy with TLS, keep BOB_YOUTUBE_API_TOKEN secret, restrict network access, and use proxy/gateway rate limits for multi-replica deployments.

Dedicated hardening/runbook docs:

docs/SECURITY.md
docs/OPERATIONS.md
docs/examples/curl.md

Current Proof

Latest local image metadata is generated in:

outputs/release/bob-youtube-0.1.0-image-inspect.json
outputs/release/bob-youtube-0.1.0-readiness.json

Required release proof command:

scripts/release_gate.sh

Focused smoke command:

scripts/smoke_test.sh

This proves:

  • wrapper shell syntax,
  • Python helper compilation,
  • version identity,
  • interrupted job restart persistence,
  • local CLI health,
  • Docker CLI health,
  • Docker API health,
  • public API discovery endpoint,
  • public OpenAPI endpoint,
  • unauthenticated operational API rejection,
  • tokenized API access,
  • invalid profile, URL, video ID, and segments path rejection,
  • request id propagation and persistence,
  • library refresh,
  • sync job,
  • profile-specific API job,
  • async job polling,
  • bounded async worker queue,
  • persisted retry attempt history,
  • signed webhook callback and idempotency proof,
  • job log retrieval,
  • job JSONL event retrieval,
  • failed-job summary classification,
  • API rate-limit rejection,
  • service metrics,
  • persisted job reload after container restart,
  • persisted job log reload after container restart,
  • persisted job event reload after container restart,
  • release manifest generation.

Latest proof excerpt:

api_discovery_ok bob-youtube 0.1.0
api_version_ok 0.1.0
api_openapi_ok 0.1.0
api_auth_boundary_ok 401
api_bad_auth_boundary_ok 401
api_body_limit_ok 413
api_invalid_profile_ok 400
api_invalid_url_ok 400
api_invalid_video_id_ok 400
api_invalid_segments_path_ok 400
api_health_ok /data /data/.bob-youtube/jobs 1
api_library_items 7
cookie_mount_local_ok
cookie_mount_docker_ok
api_cookie_contract_ok /run/secrets/bob-youtube/cookies.txt
rag_export_local_ok
rag_export_docker_ok
embed_contract_local_ok
embed_contract_docker_ok
api_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_webhook_signature_ok b10feb41-d752-4fc9-bfca-268270e822b3
api_webhook_retry_history_ok 2 200
api_webhook_retry_signature_ok 2
api_profile_job_ok agent
api_raw_ok jNQXAC9IVRw
api_raw_block_ok command_failed
api_output_stats_ok 322 1
api_rag_export_ok 1303
api_embed_contract_ok 2
api_stt_contract_ok 3
api_stt_fallback_ok 3
api_failure_summary_ok command_failed
api_retry_history_ok 2
api_async_job_ok youtube|jNQXAC9IVRw|Me at the zoo|19|public
api_job_log_ok
api_job_events_ok 6
api_metrics_ok 12 12 2 1 0 2879 4
api_prometheus_metrics_ok
api_persisted_job_ok f4c6620f-8884-457f-9264-71ae26bae5b9
api_persisted_log_ok
api_persisted_events_ok
api_rate_limit_ok 429
BOB YouTube smoke test passed
stateful_interrupted_api_ok 00000000-0000-4000-8000-000000000001
stateful_queued_recovered_api_ok 00000000-0000-4000-8000-000000000002
stateful_interrupted_events_ok 00000000-0000-4000-8000-000000000001
stateful_queued_recovered_events_ok 00000000-0000-4000-8000-000000000002
stateful_interrupted_disk_ok 00000000-0000-4000-8000-000000000001
stateful_queued_recovered_disk_ok 00000000-0000-4000-8000-000000000002
BOB YouTube stateful API test passed
outputs/release/bob-youtube-0.1.0-manifest.txt
release_artifacts_ok 0.1.0
outputs/release/bob-youtube-0.1.0-readiness.json
release_readiness_ok stable-0.1-local-private-ready
image=<see outputs/release/bob-youtube-0.1.0-image-inspect.json>
BOB YouTube release gate passed

Remaining Work Before Public Release

P0 before public/networked release:

  • Add a Redis/SQS/Postgres-style queue backend for multi-replica workers. Current 0.1 has durable filesystem queue records and atomic leases for one shared filesystem.
  • Add reverse-proxy deployment example with TLS.
  • Prove public GHCR push under the final organization/repository.

P1 for high-volume production:

  • External Redis/SQS/Postgres-style queue backend option.
  • External secret-manager integration beyond the explicit cookie-file mount contract.
  • Fragment-level extractor/network counters beyond current byte, throughput, duration, failure-category, queue, artifact, and artifact-derived media/extractor metadata.

P2 for AI corpus workflows:

  • Bundled speech-to-text model execution. The sidecar contract exists, but an actual STT engine must be supplied by the operator.
  • Bundled embedding model execution. The sidecar contract exists, but an actual embedding engine must be supplied by the operator.
  • Embedding model execution beyond the RAG export bundle.
  • Language-normalized transcript metadata.
  • Deduplication and corpus manifest signing.