Memory & CPU leak

[spetrenko]

Which version are you using?

1.12.3

Which operating system are you using?

Linux arm7 standard

Which hardware are you using?

Raspberry Pi 4 Model B 4Gb

Describe how to replicate the issue

1. Start the server.
2. Register 6 paths: cam1, cam2, cam3, ... cam6.
3. Assign several hooks to these paths.
4. Publish a 20-second video to each path in a loop, with a 10-second interval:
   ffmpeg -re -stream_loop 0 -t 20 -i video.mp4 -c copy -f rtsp -rtsp_transport tcp rtsp://localhost:8554/cam1
5. Access it through the device, which internally uses GStreamer (RTSP), whenever MediaMTX reports that it is ready.

Expected results

1. MediaMTX consumes resources moderately.

Actual results

1. MediaMTX initially consumes a small amount of CPU and memory, but the usage increases over time.
2. After 40000 sessions the CPU consumption is 270% and the memory consumption is 42%.
3. Stopping publishing & accessing the server doesn't help.

Temporary fix

Restart MediaMTX

Server logs

Logs for the last hour: mediamtx.log
CPU/Memory consumption: top.log

Network dump

No response

[pikachu937]

I confirm, there is a problem and it started to appear earlier, I was able to reproduce it on version 1.12.2, but not on arm, but on x86_64

[aler9]

If you want to report a CPU / memory issue, you have to use the integrated pprof reporter and provide the goroutine, heap and profile reports, which clearly show the real occupied memory and CPU among all components. Instructions are in the README: https://github.com/bluenviron/mediamtx?tab=readme-ov-file#pprof

Furthermore, without a sample configuration this is nearly impossible to replicate.

[spetrenko]

Hello @aler9. Thanks. Here is another test run:

• Duration: 4 days
• Number of sessions: ~38000

Test artifacts:
memory_leak_artifacts.tar.gz

Unfortunately I couldn't get "goroutine" data:
pi@rpi:~ $ go tool pprof -text http://localhost:9999/debug/pprof/goroutine Fetching profile over HTTP from http://localhost:9999/debug/pprof/goroutine http://localhost:9999/debug/pprof/goroutine: Get "http://localhost:9999/debug/pprof/goroutine": net/http: timeout awaiting response headers failed to fetch any source profiles

[yllekz]

Noticed today that my standalone raspberry pi running mediamtx is running at 400% CPU usage with mediamtx at the top. Yeah something is up.

[aler9]

i've finally found out the reason of the leak thanks to pprof reports provided by @hoperok5 (#4964) and fixed it with #4967. This issue could have been solved immediately if someone would have provided proper pprof reports as repeatedly asked.

[spetrenko]

@aler9 many thanks!

[github-actions]

This issue is mentioned in release v1.15.0 🚀
Check out the entire changelog by clicking here