Skip to content

mediamtx crashes after a variable amount of time running #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxTheMooshroom opened this issue Jul 6, 2023 · 4 comments · Fixed by #5
Closed

mediamtx crashes after a variable amount of time running #4

MaxTheMooshroom opened this issue Jul 6, 2023 · 4 comments · Fixed by #5
Assignees
Labels
bug Something isn't working critical

Comments

@MaxTheMooshroom
Copy link
Contributor

Description:
mediamtx crashes after a variable amount of time running. What causes the crash is unknown.

How to replicate:
Run the docker image and wait for a bit, it will eventually crash

Relevant details:
Crash report is as follows:

runtime: lfstack.push invalid packing: node=0xffffff7ee60120 cnt=0x1 packed=0xffff7ee601200001 -> node=0xffff7ee60120
fatal error: lfstack.push

runtime stack:
runtime.throw({0xa6555e, 0xc})
        /usr/lib/go-1.18/src/runtime/panic.go:992 +0x58
runtime.(*lfstack).push(0x11e5b18, 0xffffff7ee60120)
        /usr/lib/go-1.18/src/runtime/lfstack.go:30 +0x144
runtime.(*spanSetBlockAlloc).free(...)
        /usr/lib/go-1.18/src/runtime/mspanset.go:292
runtime.(*spanSet).reset(0x11d9558)
        /usr/lib/go-1.18/src/runtime/mspanset.go:265 +0xac
runtime.finishsweep_m()
        /usr/lib/go-1.18/src/runtime/mgcsweep.go:260 +0xc8
runtime.gcStart.func1()
        /usr/lib/go-1.18/src/runtime/mgc.go:664 +0x1c
runtime.systemstack()
        /usr/lib/go-1.18/src/runtime/asm_riscv64.s:133 +0x50

goroutine 2 [running]:
runtime.systemstack_switch()
        /usr/lib/go-1.18/src/runtime/asm_riscv64.s:96 +0x8 fp=0xc00004af38 sp=0xc00004af30 pc=0x78918
runtime.gcStart({0x1, 0x1d68896f04, 0x0})
        /usr/lib/go-1.18/src/runtime/mgc.go:663 +0x500 fp=0xc00004afb0 sp=0xc00004af38 pc=0x2a0e0
runtime.forcegchelper()
        /usr/lib/go-1.18/src/runtime/proc.go:307 +0x64 fp=0xc00004afd8 sp=0xc00004afb0 pc=0x4a314
runtime.goexit()
        /usr/lib/go-1.18/src/runtime/asm_riscv64.s:497 +0x4 fp=0xc00004afd8 sp=0xc00004afd8 pc=0x7a784
created by runtime.init.5
        /usr/lib/go-1.18/src/runtime/proc.go:289 +0x28

goroutine 1 [chan receive]:
github.com/bluenviron/mediamtx/internal/core.(*Core).Wait(...)
        /builds/mediamtx/internal/core/core.go:124
main.main()
        /builds/mediamtx/main.go:15 +0x98

goroutine 8 [select]:
github.com/bluenviron/mediamtx/internal/core.(*pathManager).run(0xc0003b2360)
        /builds/mediamtx/internal/core/path_manager.go:155 +0x304
created by github.com/bluenviron/mediamtx/internal/core.newPathManager
        /builds/mediamtx/internal/core/path_manager.go:134 +0x5ac

goroutine 9 [IO wait]:
internal/poll.runtime_pollWait(0xffffff7d61e138, 0x72)
        /usr/lib/go-1.18/src/runtime/netpoll.go:302 +0xe0
internal/poll.(*pollDesc).wait(0xc000403b98, 0x72, 0x0)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:83 +0x34
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).ReadFromInet4(0xc000403b80, {0xc0004bf800, 0x5c1, 0x5c1}, 0xc000047e10)
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:250 +0x1d8
net.(*netFD).readFromInet4(0xc000403b80, {0xc0004bf800, 0x5c1, 0x5c1}, 0xc000047e10)
        /usr/lib/go-1.18/src/net/fd_posix.go:66 +0x48
net.(*UDPConn).readFrom(0xc00000e2e0, {0xc0004bf800, 0x5c1, 0x5c1}, 0xc000047f90)
        /usr/lib/go-1.18/src/net/udpsock_posix.go:52 +0x1a0
net.(*UDPConn).readFromUDP(0xc00000e2e0, {0xc0004bf800, 0x5c1, 0x5c1}, 0xc000047f90)
        /usr/lib/go-1.18/src/net/udpsock.go:149 +0x4c
net.(*UDPConn).ReadFromUDP(...)
        /usr/lib/go-1.18/src/net/udpsock.go:141
github.com/bluenviron/gortsplib/v3.(*serverUDPListener).run(0xc00049c370)
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_udp_listener.go:187 +0xd4
created by github.com/bluenviron/gortsplib/v3.newServerUDPListener
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_udp_listener.go:164 +0x434

goroutine 10 [IO wait]:
internal/poll.runtime_pollWait(0xffffff7d61e048, 0x72)
        /usr/lib/go-1.18/src/runtime/netpoll.go:302 +0xe0
internal/poll.(*pollDesc).wait(0xc000403c98, 0x72, 0x0)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:83 +0x34
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).ReadFromInet4(0xc000403c80, {0xc0004bf200, 0x5c1, 0x5c1}, 0xc000048610)
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:250 +0x1d8
net.(*netFD).readFromInet4(0xc000403c80, {0xc0004bf200, 0x5c1, 0x5c1}, 0xc000048610)
        /usr/lib/go-1.18/src/net/fd_posix.go:66 +0x48
net.(*UDPConn).readFrom(0xc00000e2e8, {0xc0004bf200, 0x5c1, 0x5c1}, 0xc000048790)
        /usr/lib/go-1.18/src/net/udpsock_posix.go:52 +0x1a0
net.(*UDPConn).readFromUDP(0xc00000e2e8, {0xc0004bf200, 0x5c1, 0x5c1}, 0xc000048790)
        /usr/lib/go-1.18/src/net/udpsock.go:149 +0x4c
net.(*UDPConn).ReadFromUDP(...)
        /usr/lib/go-1.18/src/net/udpsock.go:141
github.com/bluenviron/gortsplib/v3.(*serverUDPListener).run(0xc00049c3c0)
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_udp_listener.go:187 +0xd4
created by github.com/bluenviron/gortsplib/v3.newServerUDPListener
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_udp_listener.go:164 +0x434

goroutine 11 [runnable]:
syscall.Syscall6(0xf2, 0x8, 0xc000014cc8, 0xc000014cbc, 0x80800, 0x0, 0x0)
        /usr/lib/go-1.18/src/syscall/asm_linux_riscv64.s:37 +0xc
syscall.accept4(0x8, 0xc000014cc8, 0xc000014cbc, 0x80800)
        /usr/lib/go-1.18/src/syscall/zsyscall_linux_riscv64.go:1378 +0x5c
syscall.Accept4(0x8, 0x80800)
        /usr/lib/go-1.18/src/syscall/syscall_linux.go:567 +0x5c
internal/poll.accept(0x8)
        /usr/lib/go-1.18/src/internal/poll/sock_cloexec.go:17 +0x3c
internal/poll.(*FD).Accept(0xc000403d00)
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:605 +0x170
net.(*netFD).accept(0xc000403d00)
        /usr/lib/go-1.18/src/net/fd_unix.go:172 +0x2c
net.(*TCPListener).accept(0xc00043ce88)
        /usr/lib/go-1.18/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc00043ce88)
        /usr/lib/go-1.18/src/net/tcpsock.go:288 +0x30
github.com/bluenviron/gortsplib/v3.(*serverTCPListener).run(0xc00043cea0)
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_tcp_listener.go:39 +0x84
created by github.com/bluenviron/gortsplib/v3.newServerTCPListener
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_tcp_listener.go:26 +0x16c

goroutine 12 [select]:
github.com/bluenviron/gortsplib/v3.(*Server).runInner(0xc00012cd80)
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server.go:354 +0x124
github.com/bluenviron/gortsplib/v3.(*Server).run(0xc00012cd80)
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server.go:337 +0x5c
created by github.com/bluenviron/gortsplib/v3.(*Server).Start
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server.go:315 +0xb5c

goroutine 13 [select]:
github.com/bluenviron/mediamtx/internal/core.(*rtspServer).run(0xc000396000)
        /builds/mediamtx/internal/core/rtsp_server.go:186 +0x154
created by github.com/bluenviron/mediamtx/internal/core.newRTSPServer
        /builds/mediamtx/internal/core/rtsp_server.go:156 +0x628

goroutine 14 [select]:
github.com/bluenviron/mediamtx/internal/core.(*rtmpServer).run(0xc0003cab60)
        /builds/mediamtx/internal/core/rtmp_server.go:193 +0x228
created by github.com/bluenviron/mediamtx/internal/core.newRTMPServer
        /builds/mediamtx/internal/core/rtmp_server.go:141 +0x444

goroutine 15 [IO wait]:
internal/poll.runtime_pollWait(0xffffff7d61dd78, 0x72)
        /usr/lib/go-1.18/src/runtime/netpoll.go:302 +0xe0
internal/poll.(*pollDesc).wait(0xc000403e18, 0x72, 0x0)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:83 +0x34
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000403e00)
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:614 +0x208
net.(*netFD).accept(0xc000403e00)
        /usr/lib/go-1.18/src/net/fd_unix.go:172 +0x2c
net.(*TCPListener).accept(0xc00043cf18)
        /usr/lib/go-1.18/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc00043cf18)
        /usr/lib/go-1.18/src/net/tcpsock.go:288 +0x30
net/http.(*Server).Serve(0xc0003962a0, {0xbaf018, 0xc00043cf18})
        /usr/lib/go-1.18/src/net/http/server.go:3039 +0x360
created by github.com/bluenviron/mediamtx/internal/core.newHTTPServer
        /builds/mediamtx/internal/core/http_server.go:63 +0x450

goroutine 16 [select]:
github.com/bluenviron/mediamtx/internal/core.(*hlsManager).run(0xc0003960e0)
        /builds/mediamtx/internal/core/hls_manager.go:160 +0x164
created by github.com/bluenviron/mediamtx/internal/core.newHLSManager
        /builds/mediamtx/internal/core/hls_manager.go:139 +0x54c

goroutine 18 [IO wait]:
internal/poll.runtime_pollWait(0xffffff7d61dc88, 0x72)
        /usr/lib/go-1.18/src/runtime/netpoll.go:302 +0xe0
internal/poll.(*pollDesc).wait(0xc000403e98, 0x72, 0x0)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:83 +0x34
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000403e80)
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:614 +0x208
net.(*netFD).accept(0xc000403e80)
        /usr/lib/go-1.18/src/net/fd_unix.go:172 +0x2c
net.(*TCPListener).accept(0xc00043cf90)
        /usr/lib/go-1.18/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc00043cf90)
        /usr/lib/go-1.18/src/net/tcpsock.go:288 +0x30
net/http.(*Server).Serve(0xc000396460, {0xbaf018, 0xc00043cf90})
        /usr/lib/go-1.18/src/net/http/server.go:3039 +0x360
created by github.com/bluenviron/mediamtx/internal/core.newHTTPServer
        /builds/mediamtx/internal/core/http_server.go:63 +0x450

goroutine 19 [select]:
github.com/bluenviron/mediamtx/internal/core.(*webRTCManager).run(0xc0000da780)
        /builds/mediamtx/internal/core/webrtc_manager.go:268 +0x180
created by github.com/bluenviron/mediamtx/internal/core.newWebRTCManager
        /builds/mediamtx/internal/core/webrtc_manager.go:245 +0x874

goroutine 20 [IO wait]:
internal/poll.runtime_pollWait(0xffffff7d61db98, 0x72)
        /usr/lib/go-1.18/src/runtime/netpoll.go:302 +0xe0
internal/poll.(*pollDesc).wait(0xc000134cd8, 0x72, 0x1)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:83 +0x34
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000134cc0, {0xc00028bf80, 0x10000, 0x10000})
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:167 +0x1f0
os.(*File).read(...)
        /usr/lib/go-1.18/src/os/file_posix.go:31
os.(*File).Read(0xc00000e2f8, {0xc00028bf80, 0x10000, 0x10000})
        /usr/lib/go-1.18/src/os/file.go:119 +0x70
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc00049c4b0)
        /root/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:356 +0x11c
created by github.com/fsnotify/fsnotify.NewWatcher
        /root/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:150 +0x200

goroutine 21 [select]:
github.com/bluenviron/mediamtx/internal/confwatcher.(*ConfWatcher).run(0xc00031c270)
        /builds/mediamtx/internal/confwatcher/confwatcher.go:85 +0x11c
created by github.com/bluenviron/mediamtx/internal/confwatcher.New
        /builds/mediamtx/internal/confwatcher/confwatcher.go:66 +0x2c4

goroutine 22 [semacquire]:
sync.runtime_Semacquire(0xc00012ce60)
        /usr/lib/go-1.18/src/runtime/sema.go:56 +0x34
sync.(*WaitGroup).Wait(0xc00012ce58)
        /usr/lib/go-1.18/src/sync/waitgroup.go:136 +0x64
github.com/bluenviron/gortsplib/v3.(*Server).Wait(...)
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server.go:330
github.com/bluenviron/mediamtx/internal/core.(*rtspServer).run.func1()
        /builds/mediamtx/internal/core/rtsp_server.go:182 +0x3c
created by github.com/bluenviron/mediamtx/internal/core.(*rtspServer).run
        /builds/mediamtx/internal/core/rtsp_server.go:181 +0xe0

goroutine 23 [IO wait]:
internal/poll.runtime_pollWait(0xffffff7d61de68, 0x72)
        /usr/lib/go-1.18/src/runtime/netpoll.go:302 +0xe0
internal/poll.(*pollDesc).wait(0xc000403d98, 0x72, 0x0)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:83 +0x34
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000403d80)
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:614 +0x208
net.(*netFD).accept(0xc000403d80)
        /usr/lib/go-1.18/src/net/fd_unix.go:172 +0x2c
net.(*TCPListener).accept(0xc00043ceb8)
        /usr/lib/go-1.18/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc00043ceb8)
        /usr/lib/go-1.18/src/net/tcpsock.go:288 +0x30
github.com/bluenviron/mediamtx/internal/core.(*rtmpServer).run.func1.1(0xc0003cab60, 0xc000083860)
        /builds/mediamtx/internal/core/rtmp_server.go:172 +0x3c
github.com/bluenviron/mediamtx/internal/core.(*rtmpServer).run.func1()
        /builds/mediamtx/internal/core/rtmp_server.go:183 +0x74
created by github.com/bluenviron/mediamtx/internal/core.(*rtmpServer).run
        /builds/mediamtx/internal/core/rtmp_server.go:168 +0x124

goroutine 24 [select]:
github.com/bluenviron/mediamtx/internal/core.(*Core).run(0xc0000cc420)
        /builds/mediamtx/internal/core/core.go:147 +0x160
created by github.com/bluenviron/mediamtx/internal/core.New
        /builds/mediamtx/internal/core/core.go:111 +0x61c

goroutine 26 [syscall]:
os/signal.signal_recv()
        /usr/lib/go-1.18/src/runtime/sigqueue.go:151 +0x38
os/signal.loop()
        /usr/lib/go-1.18/src/os/signal/signal_unix.go:23 +0x1c
created by os/signal.Notify.func1.1
        /usr/lib/go-1.18/src/os/signal/signal.go:151 +0x2c

goroutine 67 [runnable]:
syscall.Syscall(0x40, 0x1, 0xc0004f0000, 0x62)
        /usr/lib/go-1.18/src/syscall/asm_linux_riscv64.s:13 +0xc
syscall.write(0x1, {0xc0004f0000, 0x62, 0xc8})
        /usr/lib/go-1.18/src/syscall/zsyscall_linux_riscv64.go:924 +0x50
syscall.Write(...)
        /usr/lib/go-1.18/src/syscall/syscall_unix.go:216
internal/poll.ignoringEINTRIO(...)
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:794
internal/poll.(*FD).Write(0xc0000760c0, {0xc0004f0000, 0x62, 0xc8})
        /usr/lib/go-1.18/src/internal/poll/fd_unix.go:383 +0x2e8
os.(*File).write(...)
        /usr/lib/go-1.18/src/os/file_posix.go:48
os.(*File).Write(0xc00000e020, {0xc0004f0000, 0x62, 0xc8})
        /usr/lib/go-1.18/src/os/file.go:176 +0x74
github.com/bluenviron/mediamtx/internal/logger.(*destinationStdout).log(0xc0001753b0, 0x2, {0xc0002d6280, 0x1a}, {0xc00035cb10, 0x3, 0x3})
        /builds/mediamtx/internal/logger/destination_stdout.go:21 +0xd8
github.com/bluenviron/mediamtx/internal/logger.(*Logger).Log(0xc000175380, 0x2, {0xc0002d6280, 0x1a}, {0xc00035cb10, 0x3, 0x3})
        /builds/mediamtx/internal/logger/logger.go:154 +0x138
github.com/bluenviron/mediamtx/internal/core.(*Core).Log(0xc0000cc420, 0x2, {0xc0002d6280, 0x1a}, {0xc00035cb10, 0x3, 0x3})
        /builds/mediamtx/internal/core/core.go:129 +0x58
github.com/bluenviron/mediamtx/internal/core.(*rtspServer).Log(0xc000396000, 0x2, {0xc0004d6f60, 0x15}, {0xc000447ca0, 0x2, 0x2})
        /builds/mediamtx/internal/core/rtsp_server.go:168 +0x1b8
github.com/bluenviron/mediamtx/internal/core.(*rtspConn).Log(0xc00012f440, 0x2, {0xa64054, 0xb}, {0xc000013e60, 0x1, 0x1})
        /builds/mediamtx/internal/core/rtsp_conn.go:93 +0x1a8
github.com/bluenviron/mediamtx/internal/core.(*rtspConn).onClose(0xc00012f440, {0xba8028, 0xc00033cdf0})
        /builds/mediamtx/internal/core/rtsp_conn.go:111 +0x74
github.com/bluenviron/mediamtx/internal/core.(*rtspServer).OnConnClose(0xc000396000, 0xc00043d1b8)
        /builds/mediamtx/internal/core/rtsp_server.go:233 +0xb8
github.com/bluenviron/gortsplib/v3.(*ServerConn).run(0xc0000ed480)
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_conn.go:179 +0x40c
created by github.com/bluenviron/gortsplib/v3.newServerConn
        /root/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_conn.go:107 +0x468
@MaxTheMooshroom MaxTheMooshroom added bug Something isn't working critical labels Jul 6, 2023
@MaxTheMooshroom
Copy link
Contributor Author

Found a link to the function that's erroring's source:
https://go.dev/src/runtime/lfstack.go line 29

will continue looking into it

@MaxTheMooshroom MaxTheMooshroom self-assigned this Jul 6, 2023
@MaxTheMooshroom
Copy link
Contributor Author

MaxTheMooshroom commented Jul 6, 2023

this appears to be a bug with go, rather than with mediamtx, but it's also been patched already. Currently investigating the go version installed by the dockerfile, will likely need to manually install go or build go from source.

See the following issue on Go's github: golang/go#54104

@MaxTheMooshroom
Copy link
Contributor Author

go version of reported bug:

$ go version
go version go1.18.4 darwin/arm64

go version installed in the docker container:

root@1f1b9960e06b:/# go version
go version go1.18.1 linux/amd64

So we're even before the bug report. Will be addressing the bug shortly

@rummik
Copy link
Collaborator

rummik commented Jul 6, 2023

So the good news is the packaged version of Go in nixpkgs-23.05 is 1.20.4, so the flake should be good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working critical
Projects
None yet
2 participants