Github Windows VMs, alarm() sometimes fails to fire at all during long-running regexes

[toddr]

I've noticed this failure on several CI tests which were completely unrelated to this test.

2020-09-13T19:19:46.2191182Z sh: line 1:  6654 Killed                  /home/runneradmin/work/perl.exe "-I../lib" "-I." "tmp6570A" 2>&1
2020-09-13T19:19:47.1516660Z # Failed test 955 - Test Perl 73464 at ./test.pl line 1065
2020-09-13T19:19:47.1517926Z #      got "# Test process timed out - terminating\nkill: 6654: No such process"
2020-09-13T19:19:47.1518573Z # expected "Timeout"
2020-09-13T19:19:47.1518994Z # PROG: 
2020-09-13T19:19:47.1519346Z # 
2020-09-13T19:19:47.1519921Z #                 BEGIN{require q(test.pl);}
2020-09-13T19:19:47.1520517Z #                 watchdog(3);
2020-09-13T19:19:47.1521025Z #                 $SIG{ALRM} = sub {print "Timeout\n"; exit(1)};
2020-09-13T19:19:47.1521437Z #                 alarm 1;
2020-09-13T19:19:47.1521838Z #                 $_ = "a" x 1000 . "b" x 1000 . "c" x 1000;
2020-09-13T19:19:47.1522235Z #                 /.*a.*b.*c.*[de]/;
2020-09-13T19:19:47.1522593Z #             
2020-09-13T19:19:47.1522984Z # STATUS: 35072
2020-09-13T19:19:58.3677295Z t/re/pat ....................................................... FAILED at test 955

Re-running the entire workflow usually does not fail.

[toddr]

Seen in a chat. I think this was the same problem:

I have a test VM reproducing the cygwin failures that keep showing up on github. It’s the test on line 1905 of t/re/pat.t. I believe it’s either the signals from the alarms showing up out of order, or the kill of the watcher process just unblocking it from its sleep(), or the print statements showing up out of order. Increasing the timeout of the watchdog process to 10 seconds decreases the liklihood significantly, but doesn’t eliminate it entirely. Need to dig into it more when I have free time.

It might fix the problem if the watchdog process used a select() loop for it’s timeout rather than sleep() + alarm(). I’d bet it’s just an oddity of the way signals are implemented on cygwin.

[wchristian]

There's two bugs here:

1. For some reason sometimes alarm doesn't fire in time, and the watchdog fires instead. This is the actual bug. I'm presuming performance reasons right now.

2. For some reason the cygwin kill logic is silly. As you can see, the system kill is called unconditionally on cygwin, without checking if the previous kill worked or not. This results in the watchdog termination sometimes ALSO having a spurious error message which should however either: Be ignored, or the logic be augmented to check if the process still exists before trying to kill it. NOTE: This doesn't address why the process is being eaten by the watchdog, it just addresses the kill error, which is really ... irrelevant.

            # Kill test process if still running
            if (kill(0, $pid_to_kill)) {
                _diag($timeout_msg);
                kill('KILL', $pid_to_kill);
                if ($is_cygwin) {
                    # sometimes the above isn't enough on cygwin
                    sleep 1; # wait a little, it might have worked after all
                    system("/bin/kill -f $pid_to_kill");
                }
            }

[lightsey]

I spent some time a while back trying to work around this buggy behavior. I believe the introduction of ConPTY support in Cygwin made it unreliable. It seems to disappear so long as the cygwin server is enabled and the cygwin terminal process is started with CYGWIN=disable_pcon set.

There's also a seemingly related problem in the way S_exit_warning() vetos the Perl interpreter cleanup and emits warnings when a child thread calls exit(). In a nutshell, there's no reason to veto the cleanup if all of the threads are also in a joinable state. This seems to happen frequently on cygwin when a child thread calls exit().

The top three commits in this branch have the code I was experimenting with to fix this problem, but I didn't figure out a way to get the cygwin shell to start properly with disable_pcon set.

https://github.com/lightsey/perl5/tree/cygwin-threads

[toddr]

The relevant attempt seems to have been 26eacad

[toddr]

I feel like we should put a skip in this unless someone wants to come up with a fix. IMO it's not cool when all our commits are decorated red because of this failure. Opinions?

[wchristian]

You could try increasing the timeout first to see if that helps.

Also please do fix the secondary bug regardless. I thought my explanation was fairly thorough?

[toddr]

You could try increasing the timeout first to see if that helps.

It helps but only changes the rate of failure. It does not fix anything. See: #18129 (comment)

Also please do fix the secondary bug regardless. I thought my explanation was fairly thorough?

Can you produce a pull request?

[wchristian]

I saw the comment. It refers to their test VM. We have no data on what, e.g. a 30 second watchdog would do on the github VM.

Can you produce a pull request?

I'd be happier to do it if i had less rocks in my way, but sure, i'll produce initial PRs.