Merge branch 'master' into av/add-byoc-streaming

Author: ad-astra-video

CHANGELOG.md

@@ -1,6 +1,79 @@
 # Changelog
 
-## v0.8.5
+## v0.8.7
+
+### Breaking Changes 🚨🚨
+
+### Features ⚒
+
+#### General
+
+- [#3532](https://github.com/livepeer/go-livepeer/pull/3532) Payment "per time" instead of "pay per pixel" (@leszko)
+- [#3686](https://github.com/livepeer/go-livepeer/pull/3686) Add `comfyui` / `streamdiffusion` pipeline to current-live pipeline (@leszko)
+- [#3705](https://github.com/livepeer/go-livepeer/pull/3705) AI/Live: segmented MediaWriter / MediaReader (@j0sh)
+- [#3641](https://github.com/livepeer/go-livepeer/pull/3641) BYOC: switch billing model to 1 ticket per second of compute (@ad-astra-video)
+- [#3680](https://github.com/livepeer/go-livepeer/pull/3680) AI: allow local Kafka with no auth for local deployment (@ad-astra-video)
+- [#3623](https://github.com/livepeer/go-livepeer/pull/3623) Stop stream if there are no payments from Gateway (@leszko)
+
+#### Broadcaster
+
+- [#3714](https://github.com/livepeer/go-livepeer/pull/3714) Make debug segment count configurable (@mjh1)
+- [#3721](https://github.com/livepeer/go-livepeer/pull/3721) Stream heartbeats (@mjh1)
+- [#3657](https://github.com/livepeer/go-livepeer/pull/3657) Re-encode audio to AAC for non-local (@j0sh)
+- [#3676](https://github.com/livepeer/go-livepeer/pull/3676) Pass full URLs to app for WHEP, status and update (@mjh1)
+- [#3733](https://github.com/livepeer/go-livepeer/pull/3733) Send authorization header to support new API (@mjh1)
+- [#3671](https://github.com/livepeer/go-livepeer/pull/3671) Send trickle errors down to `processStream` (@j0sh)
+- [#3677](https://github.com/livepeer/go-livepeer/pull/3677) Enable Video-to-Video suspension mechanism (penalty = 5) (@leszko)
+- [#3678](https://github.com/livepeer/go-livepeer/pull/3678) Add `LiveSelectionAlgorithm` (@leszko)
+- [#3694](https://github.com/livepeer/go-livepeer/pull/3694) Fix zeroing `ai_container_idle` metric for non-existing Os (@leszko)
+- [#3695](https://github.com/livepeer/go-livepeer/pull/3695) Remove session from pool if selection failed (@leszko)
+- [#3709](https://github.com/livepeer/go-livepeer/pull/3709) Send Kafka event when no orchestrators are available (@mjh1)
+- [#3722](https://github.com/livepeer/go-livepeer/pull/3722) Send Kafka event on orchestrator swaps (@j0sh)
+- [#3634](https://github.com/livepeer/go-livepeer/pull/3634) BYOC: Allow clients to include or exclude specific Orchestrators by serviceAddr in requests (@ad-astra-video)
+- [#3682](https://github.com/livepeer/go-livepeer/pull/3682) AI/Live: store & forward parameter updates (@j0sh)
+- [#3688](https://github.com/livepeer/go-livepeer/pull/3688) AI: options for `/live/video-to-video` update & status routes to support local browser tests (@ad-astra-video)
+  
+#### Orchestrator
+
+- [#3665](https://github.com/livepeer/go-livepeer/pull/3665) Box: give GPUs to orchestrator if running GPU pipeline (@victorges)
+- [#3674](https://github.com/livepeer/go-livepeer/pull/3674) AI: fix startup to allow 0 price on AI models (@ad-astra-video)
+
+#### Transcoder
+
+- [#3734](https://github.com/livepeer/go-livepeer/pull/3734) Update LPMS to `a53e20a` (@j0sh)
+
+#### CLI
+
+- [#3700](https://github.com/livepeer/go-livepeer/pull/3700) cmd: move a few things to starter for easy embedding (@iameli)
+
+### Bug Fixes 🐞
+
+#### General
+
+- [#3711](https://github.com/livepeer/go-livepeer/pull/3711) AI/Live: watchdog for output segments (@j0sh)
+- [#3670](https://github.com/livepeer/go-livepeer/pull/3670) Set trickle idle timeout (@j0sh)
+- [#3594](https://github.com/livepeer/go-livepeer/pull/3594) Fix multiple issues in trickle, improving reliability and stability (@j0sh)
+- [#3650](https://github.com/livepeer/go-livepeer/pull/3650) BYOC: fix payment ticket count calculation (@ad-astra-video)
+- [#3651](https://github.com/livepeer/go-livepeer/pull/3651) BYOC: fix request timeout when removing capability (@ad-astra-video)
+
+#### Broadcaster
+
+- [#3673](https://github.com/livepeer/go-livepeer/pull/3673) AI/Live: WHIP endpoint updates for stream creation (@ad-astra-video)
+- [#3713](https://github.com/livepeer/go-livepeer/pull/3713) Send param updates to Kafka from control handler (@j0sh)
+- [#3699](https://github.com/livepeer/go-livepeer/pull/3699) Metrics: set correct `pipeline` and `model_name` labels (@pwilczynskiclearcode)
+- [#3702](https://github.com/livepeer/go-livepeer/pull/3702) Metrics: `GetCapacity` supports pipeline/modelID filtering (@pwilczynskiclearcode)
+
+#### Orchestrator
+
+- [#3640](https://github.com/livepeer/go-livepeer/pull/3640) Add lock while removing session during O suspension (@leszko)
+- [#3667](https://github.com/livepeer/go-livepeer/pull/3667) Don't delete stream state if `requestID` doesn't match (@mjh1)
+- [#3708](https://github.com/livepeer/go-livepeer/pull/3708) Retry ffmpeg more often in AI/Live to improve stream resilience (@j0sh)
+
+#### Transcoder
+
+- [#3734](https://github.com/livepeer/go-livepeer/pull/3734) Update LPMS to `a53e20a` (@j0sh)
+
+## v0.8.6
 
 ### Features ⚒
 
@@ -1167,4 +1240,4 @@ Thanks everyone that submitted bug reports and assisted in testing!
 - [#1775](https://github.com/livepeer/go-livepeer/pull/1775) Fix transcoder load balancer race condition around session cleanup (@jailuthra)
 - [#1784](https://github.com/livepeer/go-livepeer/pull/1784) Use auth token sessionID to index into sessions map in transcoder load balancer (@jailuthra)
 
-[Full list of changes](https://github.com/livepeer/go-livepeer/compare/v0.5.14...v0.5.15)
+[Full list of changes](https://github.com/livepeer/go-livepeer/compare/v0.5.14...v0.5.15)

CHANGELOG_PENDING.md

@@ -17,13 +17,3 @@
 ### Bug Fixes 🐞
 
 #### CLI
-
-- [#3700](https://github.com/livepeer/go-livepeer/pull/3700) cmd: move a few things to starter for easy embedding (@iameli)
-
-#### General
-
-#### Broadcaster
-
-#### Orchestrator
-
-#### Transcoder

VERSION

@@ -1 +1 @@
-0.8.6
+0.8.7

ai/worker/docker.go

@@ -67,10 +67,12 @@ var pipelineToImage = map[string]string{
 	"llm":                "livepeer/ai-runner:llm",
 }
 var livePipelineToImage = map[string]string{
-	"streamdiffusion":    "livepeer/ai-runner:live-app-streamdiffusion",
-	"comfyui":            "livepeer/ai-runner:live-app-comfyui",
-	"segment_anything_2": "livepeer/ai-runner:live-app-segment_anything_2",
-	"noop":               "livepeer/ai-runner:live-app-noop",
+	"streamdiffusion": "livepeer/ai-runner:live-app-streamdiffusion",
+	// streamdiffusion-sd15 is a utility image that uses a SD1.5 model on the default config of the pipeline. Optimizes startup time.
+	"streamdiffusion-sd15": "livepeer/ai-runner:live-app-streamdiffusion-sd15",
+	"comfyui":              "livepeer/ai-runner:live-app-comfyui",
+	"segment_anything_2":   "livepeer/ai-runner:live-app-segment_anything_2",
+	"noop":                 "livepeer/ai-runner:live-app-noop",
 }
 
 type ImageOverrides struct {

box/build-runner.sh

@@ -3,8 +3,8 @@ set -ex
 
 PIPELINE=${PIPELINE:-noop}
 
-if [[ "$PIPELINE" != "noop" && "$PIPELINE" != "comfyui" && "$PIPELINE" != "streamdiffusion" ]]; then
-  echo "Error: PIPELINE must be either 'noop', 'comfyui' or 'streamdiffusion'"
+if [[ "$PIPELINE" != "noop" && "$PIPELINE" != "comfyui" && "$PIPELINE" != "streamdiffusion" && "$PIPELINE" != "streamdiffusion-sd15" ]]; then
+  echo "Error: PIPELINE must be either 'noop', 'comfyui', 'streamdiffusion' or 'streamdiffusion-sd15'"
   exit 1
 fi
 
@@ -18,8 +18,21 @@ docker build -t livepeer/ai-runner:live-base -f docker/Dockerfile.live-base .
 if [ "${PIPELINE}" = "noop" ]; then
     docker build -t livepeer/ai-runner:live-app-noop -f docker/Dockerfile.live-app-noop --build-arg VERSION=${VERSION} .
 else
-    docker build -t livepeer/ai-runner:live-base-${PIPELINE} -f docker/Dockerfile.live-base-${PIPELINE} .
-    docker build -t livepeer/ai-runner:live-app-${PIPELINE} -f docker/Dockerfile.live-app__PIPELINE__ --build-arg PIPELINE=${PIPELINE} --build-arg VERSION=${VERSION} .
+    BASE_PIPELINE=${PIPELINE}
+    INFERPY_INITIAL_PARAMS=""
+    if [[ "$PIPELINE" == "streamdiffusion-sd15" ]]; then
+        BASE_PIPELINE="streamdiffusion"
+        JSON_FILE=./app/live/pipelines/streamdiffusion_sd15_default_params.json
+        INFERPY_INITIAL_PARAMS=$(tr -d '\n' < "$JSON_FILE")
+    fi
+    docker build -t livepeer/ai-runner:live-base-${BASE_PIPELINE} -f docker/Dockerfile.live-base-${BASE_PIPELINE} .
+    docker build \
+      -f docker/Dockerfile.live-app__PIPELINE__ \
+      -t livepeer/ai-runner:live-app-${PIPELINE} \
+      --build-arg PIPELINE=${BASE_PIPELINE} \
+      --build-arg VERSION=${VERSION} \
+      --build-arg INFERPY_INITIAL_PARAMS="$INFERPY_INITIAL_PARAMS" \
+      .
 fi
 
 docker stop live-video-to-video_${PIPELINE}_8900 || true

box/stream.sh

@@ -5,6 +5,8 @@ PIPELINE=${PIPELINE:-noop}
 STREAM_KEY="my-stream"
 STREAM_ID="my-stream-id"
 RTMP_OUTPUT=${RTMP_OUTPUT:-""}
+FPS=${FPS:-30}
+INPUT_VIDEO=${INPUT_VIDEO:-""}
 
 case "$1" in
   start)
@@ -18,12 +20,16 @@ case "$1" in
       QUERY="${QUERY}\&params=${ENCODED_PARAMS}"
     fi
 
-    ffmpeg -re -f lavfi \
-      -i testsrc=size=1920x1080:rate=30,format=yuv420p \
-      -vf scale=1280:720 \
+    INPUT_FLAGS="-f lavfi -i testsrc=size=1280x720:rate=${FPS},format=yuv420p"
+    if [ -n "$INPUT_VIDEO" ]; then
+      INPUT_FLAGS="-stream_loop -1 -i ${INPUT_VIDEO}"
+    fi
+
+    ffmpeg -re ${INPUT_FLAGS} \
+      -vf "scale=1280:720:force_original_aspect_ratio=increase,crop=1280:720" \
       -c:v libx264 \
       -b:v 1000k \
-      -x264-params keyint=60 \
+      -x264-params keyint=$((FPS * 2)) \
       -f flv rtmp://127.0.0.1:1935/${STREAM_KEY}?${QUERY}
     ;;
   playback)

cmd/livepeer/starter/flags.go

@@ -67,10 +67,14 @@ func NewLivepeerConfig(fs *flag.FlagSet) LivepeerConfig {
 	cfg.MediaMTXApiPassword = fs.String("mediaMTXApiPassword", "", "HTTP basic auth password for MediaMTX API requests")
 	cfg.LiveAITrickleHostForRunner = fs.String("liveAITrickleHostForRunner", "", "Trickle Host used by AI Runner; It's used to overwrite the publicly available Trickle Host")
 	cfg.LiveAIAuthApiKey = fs.String("liveAIAuthApiKey", "", "API key to use for Live AI authentication requests")
+	cfg.LiveAIHeartbeatURL = fs.String("liveAIHeartbeatURL", "", "Base URL for Live AI heartbeat requests")
+	cfg.LiveAIHeartbeatHeaders = fs.String("liveAIHeartbeatHeaders", "", "Map of headers to use for Live AI heartbeat requests. e.g. 'header:val,header2:val2'")
+	cfg.LiveAIHeartbeatInterval = fs.Duration("liveAIHeartbeatInterval", *cfg.LiveAIHeartbeatInterval, "Interval to send Live AI heartbeat requests")
 	cfg.LiveAIAuthWebhookURL = fs.String("liveAIAuthWebhookUrl", "", "Live AI RTMP authentication webhook URL")
 	cfg.LivePaymentInterval = fs.Duration("livePaymentInterval", *cfg.LivePaymentInterval, "Interval to pay process Gateway <> Orchestrator Payments for Live AI Video")
 	cfg.LiveOutSegmentTimeout = fs.Duration("liveOutSegmentTimeout", *cfg.LiveOutSegmentTimeout, "Timeout duration to wait the output segment to be available in the Live AI pipeline; defaults to no timeout")
 	cfg.LiveAICapRefreshModels = fs.String("liveAICapRefreshModels", "", "Comma separated list of models to periodically fetch capacity for. Leave unset to switch off periodic refresh.")
+	cfg.LiveAISaveNSegments = fs.Int("liveAISaveNSegments", 10, "Set how many segments to save to disk for debugging (both input and output)")
 
 	// Onchain:
 	cfg.EthAcctAddr = fs.String("ethAcctAddr", *cfg.EthAcctAddr, "Existing Eth account address. For use when multiple ETH accounts exist in the keystore directory")

cmd/livepeer/starter/starter.go

@@ -177,9 +177,13 @@ type LivepeerConfig struct {
 	KafkaGatewayTopic          *string
 	MediaMTXApiPassword        *string
 	LiveAIAuthApiKey           *string
+	LiveAIHeartbeatURL         *string
+	LiveAIHeartbeatHeaders     *string
+	LiveAIHeartbeatInterval    *time.Duration
 	LivePaymentInterval        *time.Duration
 	LiveOutSegmentTimeout      *time.Duration
 	LiveAICapRefreshModels     *string
+	LiveAISaveNSegments        *int
 }
 
 // DefaultLivepeerConfig creates LivepeerConfig exactly the same as when no flags are passed to the livepeer process.
@@ -232,6 +236,7 @@ func DefaultLivepeerConfig() LivepeerConfig {
 	defaultLivePaymentInterval := 5 * time.Second
 	defaultLiveOutSegmentTimeout := 0 * time.Second
 	defaultGatewayHost := ""
+	defaultLiveAIHeartbeatInterval := 5 * time.Second
 
 	// Onchain:
 	defaultEthAcctAddr := ""
@@ -347,6 +352,7 @@ func DefaultLivepeerConfig() LivepeerConfig {
 		LivePaymentInterval:      &defaultLivePaymentInterval,
 		LiveOutSegmentTimeout:    &defaultLiveOutSegmentTimeout,
 		GatewayHost:              &defaultGatewayHost,
+		LiveAIHeartbeatInterval:  &defaultLiveAIHeartbeatInterval,
 
 		// Onchain:
 		EthAcctAddr:             &defaultEthAcctAddr,
@@ -1673,6 +1679,22 @@ func StartLivepeer(ctx context.Context, cfg LivepeerConfig) {
 	if cfg.LiveAIAuthApiKey != nil {
 		n.LiveAIAuthApiKey = *cfg.LiveAIAuthApiKey
 	}
+	if cfg.LiveAIHeartbeatURL != nil {
+		n.LiveAIHeartbeatURL = *cfg.LiveAIHeartbeatURL
+	}
+	if cfg.LiveAIHeartbeatInterval != nil {
+		n.LiveAIHeartbeatInterval = *cfg.LiveAIHeartbeatInterval
+	}
+	if cfg.LiveAIHeartbeatHeaders != nil {
+		n.LiveAIHeartbeatHeaders = make(map[string]string)
+		headers := strings.Split(*cfg.LiveAIHeartbeatHeaders, ",")
+		for _, header := range headers {
+			parts := strings.SplitN(header, ":", 2)
+			if len(parts) == 2 {
+				n.LiveAIHeartbeatHeaders[parts[0]] = parts[1]
+			}
+		}
+	}
 	n.LivePaymentInterval = *cfg.LivePaymentInterval
 	n.LiveOutSegmentTimeout = *cfg.LiveOutSegmentTimeout
 	if cfg.LiveAITrickleHostForRunner != nil {
@@ -1681,6 +1703,7 @@ func StartLivepeer(ctx context.Context, cfg LivepeerConfig) {
 	if cfg.LiveAICapRefreshModels != nil && *cfg.LiveAICapRefreshModels != "" {
 		n.LiveAICapRefreshModels = strings.Split(*cfg.LiveAICapRefreshModels, ",")
 	}
+	n.LiveAISaveNSegments = cfg.LiveAISaveNSegments
 
 	//Create Livepeer Node
 

core/livepeernode.go

@@ -161,9 +161,13 @@ type LivepeerNode struct {
 	MediaMTXApiPassword        string
 	LiveAITrickleHostForRunner string
 	LiveAIAuthApiKey           string
+	LiveAIHeartbeatURL         string
+	LiveAIHeartbeatHeaders     map[string]string
+	LiveAIHeartbeatInterval    time.Duration
 	LivePaymentInterval        time.Duration
 	LiveOutSegmentTimeout      time.Duration
 	LiveAICapRefreshModels     []string
+	LiveAISaveNSegments        *int
 
 	// Gateway
 	GatewayHost string
@@ -175,6 +179,7 @@ type LivePipeline struct {
 	Pipeline    string
 	ControlPub  *trickle.TricklePublisher
 	StopControl func()
+  ReportUpdate func([]byte)
 
 	StreamCtx     context.Context
 	streamCancel  context.CancelCauseFunc

go.mod

@@ -5,6 +5,8 @@ go 1.23.2
 require (
 	contrib.go.opencensus.io/exporter/prometheus v0.4.2
 	github.com/Masterminds/semver/v3 v3.2.1
+	github.com/bluenviron/gortsplib/v4 v4.12.3
+	github.com/bluenviron/mediacommon/v2 v2.4.0
 	github.com/cenkalti/backoff v2.2.1+incompatible
 	github.com/dustin/go-humanize v1.0.1
 	github.com/ethereum/go-ethereum v1.13.5
@@ -17,14 +19,18 @@ require (
 	github.com/jaypipes/pcidb v1.0.0
 	github.com/livepeer/go-tools v0.3.6-0.20240130205227-92479de8531b
 	github.com/livepeer/livepeer-data v0.7.5-0.20231004073737-06f1f383fb18
-	github.com/livepeer/lpms v0.0.0-20250529161047-c1eefa63ff3a
+	github.com/livepeer/lpms v0.0.0-20250905152032-a53e20a33466
 	github.com/livepeer/m3u8 v0.11.1
 	github.com/mattn/go-sqlite3 v1.14.18
 	github.com/oapi-codegen/nethttp-middleware v1.0.1
 	github.com/oapi-codegen/runtime v1.1.1
 	github.com/olekukonko/tablewriter v0.0.5
 	github.com/patrickmn/go-cache v2.1.0+incompatible
 	github.com/peterbourgon/ff/v3 v3.4.0
+	github.com/pion/interceptor v0.1.37
+	github.com/pion/rtp v1.8.21
+	github.com/pion/sdp/v3 v3.0.10
+	github.com/pion/webrtc/v4 v4.0.11
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.14.0
 	github.com/segmentio/kafka-go v0.4.47
@@ -43,25 +49,18 @@ require (
 require (
 	github.com/asticode/go-astikit v0.30.0 // indirect
 	github.com/asticode/go-astits v1.13.0 // indirect
-	github.com/bluenviron/gortsplib/v4 v4.12.3 // indirect
-	github.com/bluenviron/mediacommon/v2 v2.4.0 // indirect
 	github.com/pion/datachannel v1.5.10 // indirect
 	github.com/pion/dtls/v3 v3.0.4 // indirect
 	github.com/pion/ice/v4 v4.0.7 // indirect
-	github.com/pion/interceptor v0.1.37 // indirect
 	github.com/pion/logging v0.2.3 // indirect
 	github.com/pion/mdns/v2 v2.0.7 // indirect
 	github.com/pion/randutil v0.1.0 // indirect
 	github.com/pion/rtcp v1.2.15 // indirect
-	github.com/pion/rtp v1.8.21 // indirect
 	github.com/pion/sctp v1.8.36 // indirect
-	github.com/pion/sdp/v3 v3.0.10 // indirect
-	github.com/pion/sdp/v4 v4.0.0-20240223200530-fb77fb3c6578 // indirect
 	github.com/pion/srtp/v3 v3.0.4 // indirect
 	github.com/pion/stun/v3 v3.0.0 // indirect
 	github.com/pion/transport/v3 v3.0.7 // indirect
 	github.com/pion/turn/v4 v4.0.0 // indirect
-	github.com/pion/webrtc/v4 v4.0.11 // indirect
 	github.com/wlynxg/anet v0.0.5 // indirect
 	go.opentelemetry.io/auto/sdk v1.1.0 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.33.0 // indirect