Repository navigation
Runpod integration - #70
Closed
victorkjung wants to merge 20 commits into
Closed
victorkjung wants to merge 20 commits into
victorkjung wants to merge 20 commits into
Conversation
Adds a runpod/ directory with everything needed to deploy transcribe-anything as a RunPod Serverless GPU worker, so machines without an NVIDIA card can use --device insane (the only backend producing speaker.json diarization). Contents: - handler.py: Serverless entrypoint wrapping transcribe_anything.transcribe() - Dockerfile: CUDA 12.8 + transcribe-anything + RunPod SDK - client_example.py: reference caller against /runsync - test_local.py: CPU smoke test (no GPU/cloud required) - README.md: deploy walkthrough + I/O contract + cost notes - LICENSE: MIT, dual-attributed to author + upstream zackees Infrastructure pointers (secrets, hostnames, vault references) are kept out of the repo entirely. Local resolver lives in gitignored runpod/load_secrets.sh and runpod/.env per the project's secrets convention. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Dockerfile: remove '|| true' from transcribe-anything-init-insane so build fails loudly if env init breaks, instead of silently producing a worker that can't run --device insane. Upstream's Dockerfile is designed to handle the no-GPU-at-build-time case gracefully without the masking. README: add Known limitations section explaining YouTube URLs don't work from datacenter-hosted workers (yt-dlp gets blocked). Use direct audio URLs (podcast CDN, archive.org, etc.) instead. Validated end-to-end in CPU mode against the deployed Serverless endpoint: audio download, transcribe(), JSON/VTT output, response delivery all clean. Insane mode (CUDA + diarization) is the remaining piece. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
v2 build failed without '|| true' — confirms transcribe-anything-init-insane needs a working CUDA stack to complete, which a GPU-less Docker builder doesn't have. The build-time pre-warm strategy is unworkable here. New approach: initialize the insane backend's isolated env on first import of handler.py, before runpod.serverless.start. Pays a ~30-60 sec one-time cost on each cold worker, then subsequent requests are fast (within the endpoint's idle-timeout window). - handler.py: add _prewarm_insane_env() called at module top. Swallows failures with a warning so CPU-mode requests still work if init breaks. Respects INSANE_PREWARM=0 to skip pre-warming (CPU-only deployments). - Dockerfile: drop the build-time init step; comment explains why. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The insane backend's isolated env fails to build openai-whisper==20240930
because that package imports pkg_resources without declaring it as a build
dependency — and pkg_resources was removed from setuptools 82.
The CPU env builder (whisper.py) already has this fix:
[tool.uv]
build-constraint-dependencies = ["setuptools<82"]
…but insanley_fast_whisper_reqs.py is missing it, so every insane-mode
request fails at env install time with a misleading FileNotFoundError on
/tmp/<tmpdir>/stdout.txt (the failure happens inside iso-env before
cuda_available.py can write its output).
Mirrors the fix verbatim. Good candidate for an upstream PR.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The transcribe-anything insane env (insanley_fast_whisper_reqs.py) doesn't
ship the flash_attn package. Passing --flash True therefore fails at
runtime with:
ImportError: FlashAttention2 has been toggled on, but it cannot be used
due to the following error: the package flash_attn seems to be not installed
Flip client_example.py default to flash=False and document the caveat in
README. To opt back in, add flash-attn to insane env deps and rebuild.
Validated end-to-end against the deployed Serverless endpoint with
flash=False: all output fields (text/srt/vtt/json) populated correctly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
run_insanely_fast_whisper raises OSError with the full subprocess command line on non-zero exit. That command line contains the user's HuggingFace token verbatim (--hf-token hf_xxxx...). The OSError propagates into: - RunPod's job status API response - Worker stdout (visible in Logs tab) - Any error tracking the caller wires up Anyone with access to those surfaces sees the plaintext token. Mask the token before logging-or-raising. Tokens passed via env var to the subprocess are still functional — only the displayed command line is sanitized. Good candidate for an upstream PR to zackees/transcribe-anything. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
RunPod GPU workers run in datacenters; YouTube blocks datacenter IPs, and other sources (Spotify, Apple Podcasts, paywalled hosts) similarly fail or rate-limit on cloud egress. ListenNotes / lnns.co URLs aren't direct media at all — they land on HTML episode pages. This script runs on a residential-IP host (e.g. TRIGGY over Tailscale), classifies the input URL, fetches the audio via the right strategy (yt-dlp / listennotes scraper / direct curl), then scp's the result to the VPS audio host (/var/www/voice-audio/) so RunPod can pull it from a stable public URL. Output is the public URL of the uploaded file. Caller passes that URL to the RunPod endpoint as url_or_file. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous defaults baked the VPS IP and voice.vgh-usa.com into the script. Repo is meant to be publish-safe with zero infrastructure pointers, so replace defaults with required-env-var semantics. Script now also auto-loads ~/.vgh.env (a shell-style KEY=value file) so users can set the AUDIO_* vars once and have the script pick them up transparently across SSH and cron invocations.
Two related changes that make the client usable for production workloads: 1. --resolve flag: when set, the input URL is first piped through $TRANSCRIBE_RESOLVER_HOST:$TRANSCRIBE_RESOLVER_SCRIPT (ssh + python) to pre-fetch the audio from a residential IP and mirror it to a stable public URL. Required for YouTube, Spotify, listennotes, paywalled sources, etc. — anything RunPod's datacenter IPs can't reach directly. 2. Switch from /runsync (90s sync timeout) to /run + polling. The sync endpoint can't complete any real-world podcast since cold-start + model-download alone often exceed 90s. Polling handles cold starts, long audio, and arbitrary worker spin-up up to max_wait_s (default 30 minutes). .env.example documents the new env vars without leaking host info. Validated end-to-end: lnns.co/yizau57-HAL → TRIGGY resolver (yt-dlp + listennotes scraper) → VPS audio host → RunPod insane mode → speaker.json. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The truncate was a leftover from the original scaffold and breaks parsers on any podcast-length output (whisper transcripts of 30+ min audio easily run past 4000 chars). Production callers want the full result; scaffolding callers can still pipe through head if they want to limit terminal noise.
Webapp (runpod/webapp/):
- server.py: FastAPI service that proxies to RunPod with --resolve
support, in-memory job state, and token masking in all responses.
- static/index.html: Single-page UI, Gumroad-inspired (Tailwind CDN,
Inter typography, sun-yellow / blush-pink accents, chunky black
shadows). Paste URL → status polling → diarized transcript with
speaker color coding + .txt/.srt/.vtt/.json downloads.
- requirements.txt + transcribe-webapp.service for systemd deploy.
Designed to bind on a Tailscale-only address (RUNPOD/HF creds never
leave the VPS; resolver host config carried in env/~/.vgh.env).
Upstream PR drafts (runpod/upstream-prs/):
01 - setuptools<82 constraint for insane env (zackees/transcribe-anything)
02 - mask --hf-token in OSError (zackees/transcribe-anything)
03 - handle empty pyannote segments (Vaibhavs10/insanely-fast-whisper)
All three already validated in the runpod-integration branch's runtime
behavior; the docs are PR-body-ready for cherry-pick-and-submit.
Earlier README assumed users would either run the CLI directly against a custom Docker image or fire from RunPod templates. Now that we have a working self-hosted webapp, a residential-IP resolver, and three upstream PR drafts, the docs need to point users at the right surface. New structure: - Three usage surfaces (webapp / CLI / direct API) tabled upfront - ASCII architecture diagram showing the full hybrid pipeline - Per-component descriptions (handler / Dockerfile / client / resolver / webapp / PRs) - 5-step quick start ending with the optional webapp deploy - I/O contract, known limitations, cost, contributing notes
Spotify intentionally DRMs their audio so we can't fetch episode MP3s directly from open.spotify.com. But most podcasts on Spotify also publish a public RSS feed elsewhere. Three-step lookup: 1. Spotify oEmbed (https://open.spotify.com/oembed?url=...) → returns episode title + show author (no API key needed) 2. iTunes Search API (free, no key) → finds the show's canonical RSS feed by name 3. Fetch RSS feed, parse <item> entries, fuzzy-match the episode title to find the right one → return the <enclosure url="..."> (publisher's CDN MP3) Limitation: Spotify-exclusive shows (no public RSS) can't be resolved. The resolver fails with a clear message in that case. Added open.spotify.com and spotify.link to SPOTIFY_HOSTS in classify(). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
On COMPLETED, the webapp now SSHs to a configurable host and writes the
transcript into the configured vault directory:
${OBSIDIAN_VAULT_DIR}/YYYY-MM-DD <title>.md
The note carries frontmatter (source_url, runpod_job_id, speakers,
duration, tags) and a speaker-grouped transcript body. Failure to
write is non-fatal — surfaced in the job state as obsidian_error
without affecting the underlying result.
Config (set in /etc/transcribe-webapp.env):
OBSIDIAN_HOST SSH target, e.g. user@host
OBSIDIAN_VAULT_DIR absolute path to the target dir
Health endpoint reports obsidian_configured: bool, frontend renders
"📓 saved to <path>" or the failure reason under the summary card.
In our mesh, OBSIDIAN_HOST is TRIGGY (vault at /home/.../VKJOS2025/);
Obsidian sync propagates the new note to MACKY/MAXXIE automatically.
When the publisher's CDN URL returns 404 (e.g. episode delisted), plain 'curl -sSL -o dest url' silently writes a 0-byte file because the body is empty and --fail wasn't set. The resolver then uploads that 0-byte file to the audio host, where RunPod tries to fetch it and yt-dlp confusingly exits with status 1. Fix download_direct(): - Pass --fail to curl so HTTP 4xx/5xx exit non-zero with a clear msg - Add --max-time 600 to prevent infinite hangs on slow CDNs - capture_output=True and surface stderr in the raised exception - Post-download size check (<1024 bytes = treated as failure) Real-world trigger: lnns.co/lSPyFGIwTkx → podtrac → pscrb.fm → traffic.omny.fm/.../audio.mp3 → 404. ListenNotes' embedded URL was stale because the publisher pulled the episode.
Three changes to the post-completion view:
1. .md download (server-side, /api/jobs/{id}/markdown):
Returns the same Obsidian-style markdown we write to the vault on
completion — frontmatter + speaker-grouped body. Reuses the existing
_build_obsidian_note() builder. Filename from Content-Disposition,
so the downloaded file mirrors the vault note name exactly.
2. .docx download (server-side, /api/jobs/{id}/docx):
New _build_docx() using python-docx. Real Word XML, not the
HTML-pretending-to-be-doc trick. Proper headings, bold colored
speaker labels, monospace-grey timestamps. Speaker color cycle
loosely matches the webapp UI palette.
3. Show all / Collapse toggle on the transcript view:
Default view keeps the 600px scroll cap (keeps the page short).
Click "Show all" to remove the cap and read the full transcript
inline. Toggles label to "Collapse" when expanded.
Frontend changes:
- currentJobId captured on POST /api/jobs response so server-side
download endpoints can be reached
- Download handler split: client-side for text/srt/vtt/json (built
from currentResult), server-side for md/docx (fetched from API)
- .md and .docx buttons styled in blush-pink to distinguish them
as the richer-format exports
Server requirements bumped: + python-docx>=1.1
Both halves of a bug that bit a user job today: a webapp restart
during an in-flight transcription wiped the in-memory JOBS dict.
The RunPod side kept running but the local mapping was gone, so the
browser polled /api/jobs/<id> forever and got 404s with no clear
end-state. Logs showed 12+ minutes of accumulated 404s.
Server-side:
- JOBS now mirrors to JSON_FILE (default /var/lib/transcribe-webapp/jobs.json,
override via TRANSCRIBE_JOBS_FILE env var)
- _persist_jobs() writes atomically via tmp+rename
- _load_jobs_from_disk() restores on startup
- _persist_jobs() called at every state transition (submit, resolving,
submitting, transcribing, IN_QUEUE/IN_PROGRESS message change,
COMPLETED/FAILED, exception paths)
Frontend:
- pollOnce() treats HTTP 404 as terminal: stop polling, show clear
"Job is no longer on the server. The webapp may have been
restarted... Try resubmitting" error, re-enable submit button.
Other status codes still treated as transient retries.
After this commit, a service restart preserves in-flight jobs;
worker continues polling RunPod from where it left off after the
asyncio task is re-spawned on restart (TODO: that re-spawn is
missing — currently restored jobs are visible but won't progress
until the user manually re-submits. Acceptable for v1; sql-backed
job runner is the proper long-term fix).
OpenSSH joins all remote-command argv elements into a single shell string and evaluates it on the remote side. URLs with ? (YouTube query-string delimiter), *, $, and other shell-special chars get mangled by the remote shell. zsh in particular errors out with 'no matches found' when ? doesn't match any local file. Failure mode: pasting any youtube.com/watch?v=... URL into the webapp produced: RuntimeError: resolver exited 1: zsh:1: no matches found: https://www.youtube.com/watch?v=twHotqVRIBY Fix: use shlex.quote() on each remote-command argument, then send as a single literal string to ssh. The URL is treated as opaque on the remote side. Matches what we already do for ssh into obsidian write paths (which use Python repr quoting, equivalent for our paths). Applied in two places: - webapp/server.py: _run_resolver() - client_example.py: resolve_via_remote()
Closes the persistence loop. Previous commit (167f176) saved JOBS to disk on every state change and restored on startup, but didn't re-spawn the asyncio tasks that drive the resolver→submit→poll pipeline. A restored job would sit in 'transcribing' forever with no worker attached. Two changes: 1. _process_job is now resumption-aware. Checks job state for runpod_id and resolved_url, skips stages that are already done: - has runpod_id → jump straight to polling - has resolved_url → skip resolver, submit to RunPod - else → run resolver (if use_resolver), submit, poll Idempotent on partial-submit failures: a retry creates a new runpod_id; the original (if any) orphans but is bounded by RunPod scale-to-zero economics. 2. @app.on_event("startup") scans restored JOBS and asyncio.create_task for any non-terminal job. _TERMINAL_STATUSES = {"completed", "failed"}. stderr logs how many were resumed. End result: a webapp restart during transcription is invisible to the user. The browser keeps polling; the job continues from where it was.
When TELEGRAM_BOT_TOKEN + TELEGRAM_CHAT_ID are set in the host's env
(auto-loaded from ~/.vgh.env), the webapp now sends a Telegram message
on each job's terminal state. No new secrets touch the repo.
Server-side:
- _send_telegram(text): async httpx POST to api.telegram.org. 4000-char
body limit (Telegram caps at 4096; trim safe), Markdown parse mode,
no web-page previews. Raises on non-200 so caller can log.
- _telegram_message_for_job(job): composes the message
✅ Transcript ready — N speakers · Xm Ys
_Source:_ <url>
_Obsidian:_ `<path>` (if saved)
> 250-char transcript preview
or for failures:
❌ Transcription failed
_Source:_ <url>
```<error text>```
- Wired into both COMPLETED and FAILED branches of _process_job;
sets job.telegram_notified=True on success, job.telegram_error=str
on failure. Best-effort — never blocks the result.
- Health endpoint reports telegram_configured: bool.
Frontend:
- obsidian-status section now also shows "📨 Telegram notification sent"
or "📨 Telegram send failed: <err>" below the Obsidian line.
Both visible together if both are configured.
Docs:
- runpod/.env.example: TELEGRAM_BOT_TOKEN + TELEGRAM_CHAT_ID
documented with quick-start pointers to @Botfather and getUpdates.
Why not shell out to /usr/local/bin/vgh-alert (the existing vgh
Telegram wrapper)? The webapp is async/await throughout. Native
httpx avoids the subprocess hop and surfaces structured errors into
job state. vgh-alert remains the right tool for one-shot bash scripts
(pomodoro, nightly-ingestion alerts) elsewhere on VPS.
Owner
|
Thanks for the substantial work here — having looked through the diff (2,524 / -3 across 20 files), my read is that this is the right direction as a downstream fork, but probably not the right shape for upstream merge. The PR mixes three quite different things: What this PR actually contains
Why item 3 probably shouldn't merge upstream
Recommended pathI'd suggest keeping this PR as the canonical pointer to your fork (which is genuinely useful — folks without a local NVIDIA GPU clearly want this), and instead:
Filed #94 to track the conversation about whether/how to advertise a community RunPod deployment alongside the upstream README (link to your fork is the natural answer). Worth getting maintainer's call before closing. |
This was referenced Jun 3, 2026
Owner
|
Closing — maintainer's call now in. The path forward, captured across the new tracking issues and the merged work:
This PR itself stays as a canonical pointer to your fork from the closed-PR list. Thanks for the substantial work on the RunPod deployment, the upstream-fix discipline (separate (For folks finding this later: @victorkjung's fork — https://github.com/victorkjung/transcribe-anything/tree/runpod-integration/runpod — is a working RunPod Serverless deployment of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For more impressive GPU serverless integration for those on a budget and want power of GPU.