console breaks when running in parallel

[kylemcdonald]

I am calling the following code inside the job of an pool.imap_unordered:

    data, _ = (ffmpeg
               .input(filename)
               .output('pipe:', format=format, acodec=acodec, ac=ac, ar=sr)
               .run(capture_stdout=True, capture_stderr=True))

in order to load a lot of audio in parallel. It seems to work correctly, but when it's done running I can't type in the console anymore. Or rather, the text I type does not appear, but when I hit return the commands are still executed.

If I run the script inside screen and then exit the screen, the original console works.

Is there another way that I should be running ffmpeg-python when using multiprocessing?

Looking at this issue https://askubuntu.com/questions/171449/shell-does-not-show-typed-in-commands-reset-works-but-what-happened my situation matches it very closely. Before I run the script the result of stty --all is:

speed 9600 baud; rows 33; columns 197; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = M-^?; eol2 = M-^?; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O;
min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon -iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc

and after it is:

speed 9600 baud; rows 33; columns 197; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = M-^?; eol2 = M-^?; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O;
min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig -icanon -iexten -echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc

echo is one of the options that switches from true to false. But I can't find what in the ffmpeg-python library causes this.

Previously I had written my own wrapper to ffmpeg https://github.com/kylemcdonald/python-utils/blob/master/ffmpeg_load_audio.py I didn't notice this problem before. But I just did some more testing and it has the same bug. Which means it's something pretty basic about running Popen in threads.

[kkroening]

I am calling the following code inside the job of an pool.imap_unordered in order to load a lot of audio in parallel.

Sounds like a good use case.

Hmm, yeah that is strange. There's nothing in ffmpeg-python that does any sort of terminal manipulation.

ffmpeg though does seem to do weird terminal escape sequences, and I've had situations where running ffmpeg directly causes my terminal to get messed up, requiring something like stty sane or a terminal reset. But we're capturing both stdout and stderr via pipes so it doesn't make sense that such escape sequences could leak out from Popen.

My guess is that ffmpeg has some other way of manipulating the terminal besides stdout/stderr, e.g. through /dev/tty*. We should try running strace on it and see if this is the case. If so, I'm not at all surprised that there would be race conditions around multiple processes doing terminal manipulation concurrently.

The solution is probably to find a way to sandbox ffmpeg to not try to mess with the terminal. I'm not familiar enough with all the TTY intricacies to know exactly how to do this, but there's almost certainly a way. I think it's just a question of how to run a subprocess in python without allowing it to have access to the parent process' tty. Or perhaps there's an ffmpeg option to prevent it from messing with the terminal settings.

(Honestly I wish ffmpeg wouldn't do any tty manipulation. From what I've seen, tty manipulation is a terrible idea in almost every situation, besides interactive text editors or command prompts/shells)

I don't know that I can figure this out at the moment, but I'll keep it in mind and come back to whenever possible.

[kofalt]

What would you think about using pty.openpty() and using that in the subprocess.Popen call? I'm not sure if that would actually achieve a full separation, I have not used the former before.

Also, complete shot in the dark, but maybe -nostdin would help with this?

-stdin

Enable interaction on standard input. On by default unless standard input is used as an input. To explicitly disable interaction you need to specify -nostdin.

Disabling interaction on standard input is useful, for example, if ffmpeg is in the background process group. Roughly the same result can be achieved with ffmpeg ... < /dev/null but it requires a shell.

Yeah, I know it's about input, not output, but perhaps worth a go.

[kkroening]

Great ideas @kofalt. I bet you're onto something.

I guess the next step is to creating a failing test scenario so we can reproduce the issue.

I'm curious whether this is happening on OS X or Linux or both. I doubt it's happening on Windows (but I haven't really worked with Windows in the past decade or so other than a period of a couple of weeks of giving it a chance and then running away screaming "never again")

[kofalt]

If someone has an easy script to reproduce (big buck bunny, anyone?) I'll happily give it a shot on Linux & Windows.

[kylemcdonald]

Hi 👋I'm back again.

I found an example that doesn't use any threads outside the library, just run_async. I think this could give a clue of what's going on:

import ffmpeg
import numpy as np

in1_filename = 'ch03.mp4'
in2_filename = 'ch04.mp4'
out_filename = 'ch03+04.mp4'
width = 3072
height = 3072

process1 = (
    ffmpeg
    .input(in1_filename)
    .output('pipe:', format='rawvideo', pix_fmt='rgb24', vframes=8)
    .global_args('-hide_banner', '-nostats', '-loglevel', 'panic')
    .run_async(pipe_stdout=True)
)

process2 = (
    ffmpeg
    .input(in2_filename)
    .output('pipe:', format='rawvideo', pix_fmt='rgb24', vframes=8)
    .global_args('-hide_banner', '-nostats', '-loglevel', 'panic')
    .run_async(pipe_stdout=True)
)

process3 = (
    ffmpeg
    .input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width*2, height))
    .output(out_filename, pix_fmt='yuv420p')
    .global_args('-hide_banner', '-nostats', '-loglevel', 'panic')
    .overwrite_output()
    .run_async(pipe_stdin=True)
)

print('Running')

while True:
    in1_bytes = process1.stdout.read(width * height * 3)
    in2_bytes = process2.stdout.read(width * height * 3)

    if not in1_bytes:
        break
    in1_frame = (
        np
        .frombuffer(in1_bytes, np.uint8)
        .reshape([height, width, 3])
    )

    if not in2_bytes:
        break
    in2_frame = (
        np
        .frombuffer(in2_bytes, np.uint8)
        .reshape([height, width, 3])
    )

    out_frame = np.hstack((in1_frame, in2_frame))

    process3.stdin.write(
        out_frame
        .astype(np.uint8)
        .tobytes()
    )

process1.wait()
process2.wait()
process3.stdin.close()
process3.wait()

This code stacks two videos side by side (you wouldn't want to do it this way of course, it's just an example). The snippet above works. But if you move this line down:

    in2_bytes = process2.stdout.read(width * height * 3)

So it appear after the break from not in2_bytes, then the terminal gets messed up. Even though we know process2.stdout.read is going to return None, we need to call it for it to properly make the terminal go back to normal.

If you have some script that is too complicated to figure out where this is happening, you can also add this to the end of your script:

import subprocess
subprocess.run(['stty', 'echo'])

And it will return the terminal to normal even if there's some async code that has messed it up somehow.

[vguzov]

Hi,
Just wanted to say that adding '-nostdin' to .global_args() in reader processes fixed the issue with terminal for me.

[bernardohenz]

Hello, I'm having a similar problem when using pydub. I have the following script running on multithread for converting many audio files:

from pydub import AudioSegment

def ogg_to_wav(ogg_file_path, wav_file_path, sample_rate=16000,channels=1):
    audio_data = AudioSegment.from_file(ogg_file_path, format='ogg')
    audio_data = AudioSegment.set_frame_rate(audio_data,sample_rate)
    audio_data = AudioSegment.set_channels(audio_data,channels)
    audio_data = AudioSegment.set_sample_width(audio_data, 2)
    duration_seconds = audio_data.duration_seconds
    if audio_data.export(wav_file_path, format='wav'):
        return True, duration_seconds
    return False, duration_seconds

After that, the console freezes and I cannot input anything. any ideas?

[qistoph]

I am also having this issue effectively calling

multiprocessing.Pool.imap -> subprocess.run(..., capture_output = True) -> ffmpeg

The -nostdin work-around works for me too