popen crashes in resty but not in luajit

[mikz]

I'm writing simple script to install exact versions of dependencies into a folder (3scale/APIcast@e6d5e27).

There is surprising behaviour of luarocks (luarocks/luarocks#732) that is different than running it with openresty interpreted compiled with openresty.

When running with resty I'm getting errors like:

ERROR: /usr/local/share/lua/5.1/luarocks/fs/unix/tools.lua:114: Interrupted system call
stack traceback:
	/usr/local/share/lua/5.1/luarocks/fs/unix/tools.lua:114: in function 'find'
	/usr/local/share/lua/5.1/luarocks/manif.lua:59: in function 'make_rock_manifest'
	/usr/local/share/lua/5.1/luarocks/build.lua:317: in function 'build_rockspec'
	/usr/local/share/lua/5.1/luarocks/build.lua:359: in function 'build_rock'

or

Failed producing checksum: Failed to compute MD5 hash for file ...

The second one actually fails because reading from a pipe returns nil: https://github.com/mikz/luarocks/blob/22a7b81946edfe3e25f914c9c672b6705153714d/src/luarocks/fs/tools.lua#L146-L147

All this works when I run the script with the luajit interpreter compiled by openresty.

OS: macOS High Sierra 10.13 (17A365)

[agentzh]

@mikz We fixed some issues with pipes in the latest resty utility just very recently. Will you try the following preview version on your side?

https://openresty.org/download/openresty-1.13.5.1rc0.tar.gz

[agentzh]

@mikz BTW, it would be much easier for us to investigate if you can prepare a minimal and standalone example that can easily reproduce the problem on our side. Luarocks is HUGE :)

[mikz]

@agentzh Just tried 1.13.5.1rc0 and got the same error.

I'll try to make smaller example, but I don't see how loading luarocks could affect io.popen in a way calling :lines() would error Interrupted system call or :read('*a') return nil.

I wasn't been able to reproduce this on Linux inside Docker on Mac, so I guess this might be related to macOS. Will try older versions and other machines.

[mikz]

@agentzh found the minimal reproducible case.

luajit -e 'print(assert(io.popen("openssl dgst -md5 cpanfile"):read("*a")))'

MD5(cpanfile)= 2c0f2b4d3ae46f41ab3b79ed4ca5f572

resty -e 'print(assert(io.popen("openssl dgst -md5 cpanfile"):read("*a")))'

ERROR: (command line -e):1: Interrupted system call
stack traceback:
	(command line -e):1: in function 'inline_gen'
	init_worker_by_lua:45: in function <init_worker_by_lua:44>
	[C]: in function 'xpcall'
	init_worker_by_lua:53: in function <init_worker_by_lua:51>

[agentzh]

@mikz This is the SIGCHLD signal from your shell command openssl dgst -md5 cpanfile interrupts the read syscall of the pipe reading operation. The luajit utility does not exhibit this problem since it does not handle the SIGCHLD signal like nginx (used by resty under the hood). I think the Lua VM's read("*a") should be smarter about such interrupted syscalls, since it is completely normal when the host application handles signals themselves. For the current work-around, you (or luarocks) should try using read("*l") here instead (I see no reason to use *a in this case).

[mikz]

@agentzh interesting. That was the workaround that I tried to use in this case.

Unfortunately this also breaks functions like :lines(). It is easily reproducible with cd && find:

resty -e 'for file in io.popen("cd lua_modules && find * 2> /dev/null"):lines() do print(file) end'

Seems quite hard to unroll all those calls to while && read. Especially in a library like luarocks.
Do you have any ideas for some global workaround? Like wrapping it in signal trapping shell or something like that?

[agentzh]

@mikz As I've said above, the best way to address this is to patch the LuaJIT/Lua VM's io module's C implementation to handle the EINTR error returned by read() syscalls properly for UNIX-style systems. You can contribute a patch for it.

[mikz]

I worked around it by patching lines and capturing the error. Of course this is not a long term solution but the code executes as it should:

local lines = getmetatable(io.output()).lines

getmetatable(io.output()).lines = function(self, ...)
    local iter = lines(self, ...)

    return function()
        local ok, ret = pcall(iter)

        if ok then return ret
        else return nil end
    end
end