CRASH ffmpeg on ARM crashes on dr_app_setup_and_start()

[suyashmahar]

Describe the bug
When attaching drmemtrace to ffmpeg using dr_app_setup_and_start(), DR crashes with this error message:

# benchmarks/video_transcode_bench/ffmpeg -y -i Netflix_Boat_4096x2160_60fps_10bit_420.y4m -crf 23 -preset 6 -g 80 -threads 1 -c:v libsvtav1 -f ivf -svtav1-params lp=1 /tmp/ffmpeg-svt_M6_NETFLIX_ElFuente_1920x1080to1280x720_lanc_14296frames_2997fps_003724_003802_Q23c.bin
[..]
<Application …/DCPerf/benchmarks/video_transcode_bench/ffmpeg_build/bin/ffmpeg (902470) has no directory specified for forensics files>
<log dir=>
<Starting application …/DCPerf/benchmarks/video_transcode_bench/ffmpeg_build/bin/ffmpeg (902470)>
<Initial options = -loglevel 3 -client_lib '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395//tools/lib64/debug/libdrmemtrace.so' -code_api -stack_size 64K -signal_stack_size 64K -max_elide_jmp 0 -max_elide_call 0 -vmm_block_size 64K -initial_heap_unit_size 64K -initial_heap_nonpers_size 64K -initial_global_heap_unit_size 512K -max_heap_unit_size 4M -heap_commit_increment 64K -cache_commit_increment 64K -cache_bb_unit_init 64K -cache_bb_unit_max 64K -cache_bb_unit_quadruple 64K -cache_trace_unit_init 64K -cache_trace_unit_max 64K -cache_trace_unit_quadruple 64K -cache_shared_bb_unit_init 512K -cache_shared_bb_unit_max 512K -cache_shared_bb_unit_quadruple 512K -cache_shared_trace_unit_init 512K -cache_shared_trace_unit_max 512K -cache_shared_trace_unit_quadruple 512K -cache_bb_unit_upgrade 64K -cache_trace_unit_upgrade 64K -cache_shared_bb_unit_upgrade 512K -cache_shared_trace_unit_upgrade 512K -no_inline_ignored_syscalls -no_per_thread_guard_pages -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<WARNING! symbol lookup error: libdrsyscall.so undefined symbol tolower>
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395//tools/lib64/debug/libdrmemtrace.so' 0x0000fffb5401d8b0
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrwrap.so' 0x0000fffb54283610
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrmgr.so' 0x0000fffb542b4c00
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrutil.so' 0x0000fffb542f11b0
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrstatecmp.so' 0x0000fffb54321a10
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrcovlib.so' 0x0000fffb54351f60
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrx.so' 0x0000fffb54384b80
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrreg.so' 0x0000fffb543c3130
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrbbdup.so' 0x0000fffb543f3900
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrsyscall_record_lib.so' 0x0000fffb54420f50
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrsyscall.so' 0x0000fffb54456d30
add-symbol-file '/usr/lib64/libsnappy.so.1' 0x0000fffce73c2410
add-symbol-file '/usr/lib64/libstdc++.so.6' 0x0000fffce721d800
add-symbol-file '/usr/lib64/libm.so.6' 0x0000fffce70cbae0
add-symbol-file '/usr/lib64/libc.so.6' 0x0000fffce6f27180
add-symbol-file '/usr/lib/ld-linux-aarch64.so.1' 0x0000fffce6ea1bc0
add-symbol-file '/usr/lib64/libgcc_s.so.1' 0x0000fffce6e52d10
add-symbol-file '/usr/lib64/liblz4.so.1' 0x0000fffce6e02c00
add-symbol-file '/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/ext/lib64/debug/libdrsyms.so' 0x0000fffb544c8870
>
<Attached to 1/23 threads in application .../DCPerf/benchmarks/video_transcode_bench/ffmpeg_build/bin/ffmpeg (902470)>
[1]    902470 segmentation fault (core dumped)  benchmarks/video_transcode_bench/ffmpeg -y -i  -crf 23 -preset 6 -g 80  1 -c:

To Reproduce

1. Define variables needed for building ffmpeg with DynamoRIO. The build script assumes that the DR_ROOT is a tarball from https://github.com/DynamoRIO/dynamorio/releases

# # Ensure that $DR_ROOT/tools/lib64/debug/;$DR_ROOT/lib64/debug/;$DR_ROOT/ext/lib64/debug/;$DR_ROOT/tools/lib64/debug/libdrmemtrace.so are valid paths, adjust if necessary.
# export DR_ROOT=/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/
# export DYNAMORIO_OPTIONS="-client_lib $DR_ROOT/tools/lib64/debug/libdrmemtrace.so -loglevel 3;;-offline -code_api -loglevel 3"
# export LD_LIBRARY_PATH="$DR_ROOT/tools/lib64/debug/;$DR_ROOT/lib64/debug/;$DR_ROOT/ext/lib64/debug/;[DCPerf root directory]/benchmarks/video_transcode_bench/ffmpeg_build/lib64/"

2. A modified ‘n8.0’ version of ffmpeg with support for DynamoRIO’s dr_app_setup_and_start() and scripts to build it is available here: https://github.com/suyashmahar/DCPerf . To build this version, run git clone https://github.com/suyashmahar/DCPerf. and follow the steps here: https://github.com/suyashmahar/DCPerf/blob/export-D86680182-to-v2-beta/packages/video_transcode_bench/README.md .

Note

In case the ffmpeg build fails, please check if these paths are valid: $DR_ROOT/tools/lib64/debug/, $DR_ROOT/lib64/debug/, and $DR_ROOT/ext/lib64/debug/. Once verified, clean the existing build with rm -rf benchmarks benchmark_installs.txt and try to build it again.

3. Once built, grab the input netflix sequence using: wget http://download.opencontent.netflix.com.s3.amazonaws.com/ElFuente/Netflix_Boat_4096x2160_60fps_10bit_420.y4m
4. Run ffmpeg: benchmarks/video_transcode_bench/ffmpeg -y -i Netflix_Boat_4096x2160_60fps_10bit_420.y4m -crf 23 -preset 6 -g 80 -threads 1 -c:v libsvtav1 -f ivf -svtav1-params lp=1 /tmp/ffmpeg-svt_M6_NETFLIX_ElFuente_1920x1080to1280x720_lanc_14296frames_2997fps_003724_003802_Q23c.bin

Please also answer these questions:

• What happens when you run without any client?
  Crashes with a SIGSEGV, same behavior
• What happens when you run with debug build ("-debug" flag to drrun/drconfig/drinject)?
  Not sure how to do this. Passing -debug before or after ;; in DYNAMORIO_OPTIONS results in an error message saying unknown flag -debug. I’m already the debug version of the shared objects if that is more relevant for internal attach/detach:

# ldd benchmarks/video_transcode_bench/ffmpeg | grep dy
        libdynamorio.so => /home/smahar/DynamoRIO-AArch64-Linux-11.90.20395//lib64/debug/libdynamorio.so (0x0000000071000000)

Please provide a symbolized callstack of any crash or assert so we do not have to speculate or guess as to where it is occurring (see our Linux gdb instructions and Windows windbg instructions to see how to load DynamoRIO symbols for a callstack).

Running ffmpeg under gdb:

[After continuing SIGILL messages…]
(gdb) c
Continuing.

Thread 17 "vf#0:0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xfffbe850f100 (LWP 1746086)]
(gdb) bt
#0  0x0000000071453fb0 in get_thread_private_dcontext () at /home/runner/work/dynamorio/dynamorio/core/unix/os.c:3177
#1  0x000000007147aa34 in main_signal_handler_C (sig=4, siginfo=0xfffbe850d320, ucxt=0xfffbe850d3a0, xsp=0xfffbe850d320 "\004") at /home/runner/work/dynamorio/dynamorio/core/unix/signal.c:5712
#2  0x000000007144f7a0 in _dynamorio_runtime_resolve () at /home/runner/work/dynamorio/dynamorio/core/arch/aarch64/aarch64.asm:534
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Expected behavior
FFMPEG should not crash.

Versions

• What version of DynamoRIO are you using? DynamoRIO-AArch64-Linux-11.90.20395
• Does the latest build from https://github.com/DynamoRIO/dynamorio/releases solve the problem? This is the latest version.
• What operating system version are you running on? ("Windows 10" is not sufficient: give the release number.) CentOS 9 on aarch64. Linux hostname 6.13.2-custom_string #1 SMP Tue Sep 2 10:38:48 PDT 2025 aarch64 aarch64 aarch64 GNU/Linux
• Is your application 32-bit or 64-bit? 64-bit aarch64

Additional context

The test suite for attach detach is passing for this version of DR:

# ctest -j 5 -R client.attach
Test project /home/smahar/dynamorio/build
    Start 138: code_api|client.attach_test
    Start 140: code_api|client.attach_blocking
    Start 141: code_api|client.attach_state
    Start 143: code_api|client.attach_memory_dump_test
    Start 359: code_api|client.attach-memory-dump-syscall-test
1/5 Test #143: code_api|client.attach_memory_dump_test ...........   Passed    0.21 sec
2/5 Test #138: code_api|client.attach_test .......................   Passed    0.32 sec
3/5 Test #140: code_api|client.attach_blocking ...................   Passed    0.32 sec
4/5 Test #359: code_api|client.attach-memory-dump-syscall-test ...   Passed    0.32 sec
5/5 Test #141: code_api|client.attach_state ......................   Passed    5.01 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) =   5.04 sec

# ctest -j 5 -R client.detach
Test project /home/smahar/dynamorio/build
    Start 139: code_api|client.detach_test
1/1 Test #139: code_api|client.detach_test ......   Passed    0.22 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.24 sec

[1] Patch for enabling DynamoRIO in ffmpeg:

diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
index de607cac93..7b3468f882 100644
--- a/fftools/ffmpeg.c
+++ b/fftools/ffmpeg.c
@@ -23,6 +23,24 @@
  * multimedia converter based on the FFmpeg libraries
  */

+// DynamoRIO required start
+#define LINUX
+#if defined(__x86_64__) || defined(__i386__)
+#define X86_64
+#elif defined(__aarch64__) || defined(__arm__)
+#define ARM_64
+#endif
+
+#include "dr_api.h"
+
+#define TRACE_DURATION_S_DEFAULT 2
+#define TRACE_DURATION_S_ENV "TRACE_DURATION_S"
+
+#define ITERATION_WAIT_S_DEFAULT 10
+#define ITERATION_WAIT_S_ENV "ITERATION_WAIT_S"
+
+// DynamoRIO required end
+
 #include "config.h"

 #include <errno.h>
@@ -960,10 +978,60 @@ static int64_t getmaxrss(void)
 #endif
 }

+
+void *tracing_thread(void *arg);
+
+void *tracing_thread(void* arg) {
+  // Do nothing for now
+
+  int trace_duration_s = TRACE_DURATION_S_DEFAULT;
+  int iteration_wait_s = ITERATION_WAIT_S_DEFAULT;
+
+  char *trace_dur_raw = getenv(TRACE_DURATION_S_ENV);
+  if (trace_dur_raw) {
+    errno = 0;
+    trace_duration_s = strtol(trace_dur_raw, NULL, 0);
+    if (errno != 0) {
+      perror("Unable to parse TRACE_DURATION_S env var");
+      exit(1);
+    }
+  }
+
+  char *iter_wait_raw = getenv(ITERATION_WAIT_S_ENV);
+  if (iter_wait_raw) {
+    errno = 0;
+    iteration_wait_s = strtol(iter_wait_raw, NULL, 0);
+    if (errno != 0) {
+      perror("Unable to parse ITERATION_WAIT_S env var");
+      exit(1);
+    }
+  }
+
+  printf("Tracing duration set to %d, wait between iteration set to %d.\n",
+         trace_duration_s, iteration_wait_s);
+
+  sleep(1);
+  int trace_iter = 0;
+  while (1) {
+    dr_app_setup_and_start();
+    printf("Sleeping for %d seconds\n", trace_duration_s);
+    sleep(trace_duration_s);
+    dr_app_stop_and_cleanup();
+
+    printf("Completed %d iteration\n", trace_iter);
+    trace_iter++;
+    sleep(iteration_wait_s);
+  }
+}
+
+
 int main(int argc, char **argv)
 {
     Scheduler *sch = NULL;

+    pthread_t tracing_thread_handler;
+    pthread_create(&tracing_thread_handler, NULL, &tracing_thread, NULL);
+
     int ret;
     BenchmarkTimeStamps ti;

[suyashmahar]

Trying an older version of the build fixed the crashes for runs with no tool:

	With memtrace	No client
cronbuild-11.90.20395	Crash	Crash
cronbuild-11.90.20356	Crash	Success

---

I've also been testing on both Grace and Graviton machines, and the initial results were from a Grace machine.

Here are the observations I made regarding the ffmpeg runs with different DynamoRIO builds:

Machine Type	DR Build	Tool Attached?	Observation
Graviton	11.90.20395 (Newer)	Yes (Tracing)	Crashes after completing at least one tracing iteration.
Grace	11.90.20395 (Newer)	Yes (Tracing)	Crashes on the first tracing iteration.
Grace	11.90.20356 (Older)	Yes (Tracing)	Completes at least one tracing iteration before crashing (unlike the newer build).
Grace	11.90.20395 (Newer)	No	DR crashes even with no tool attached, preventing ffmpeg completion.
Grace	11.90.20356 (Older)	No	ffmpeg runs to completion successfully.

[suyashmahar]

With version 11.90.20356, I see crashes with very similar stacktrace as #7669.

With commit dcad4c5f7 and changes mentioned in #7669, ffmpeg runs to completion using the debug version, but the release version still crashes.