Blocking mode of existing fds opened with `IO::FileDescriptor.new` is changed starting with crystal 1.17

[digitalshow]

Using https://github.com/xtokio/inotify I noticed that starting with Crystal 1.17 an fd opened with IO::FileDescriptor.new(fd) is set to blocking mode if it is not a pipe/socket/character device.

The fd that was opened non-blocking by inotify_init1(LibC::O_NONBLOCK) is set to blocking mode.

https://github.com/crystal-lang/crystal/pull/15753/files#diff-b84846c2d72cf735f99c4dbc76c81299794b240a7efea623626f344cf62597b3L37-R48

I've got a notion that this one changed behavior on unix: Non-blocking fds that are not a pipe/socket/character device will be set to blocking mode whereas they were left untouched before (because setting value to true did nothing).

Is that an expected change? I couldn't find anything in the changelogs.

For reference some strace logs:

$ crystal --version
Crystal 1.17.1 (2025-07-22)

LLVM: 21.1.2
Default target: x86_64-unknown-linux-gnu

inotify_init1(IN_NONBLOCK)              = 11
...
fcntl(11, F_GETFL)                      = 0x800 (flags O_RDONLY|O_NONBLOCK)
fstat(11, {st_mode=0600, st_size=0, ...}) = 0
fcntl(11, F_GETFL)                      = 0x800 (flags O_RDONLY|O_NONBLOCK)
fcntl(11, F_SETFL, O_RDONLY)            = 0
...
inotify_add_watch(11, "/var/lib/<redacted>", IN_CLOSE_WRITE|IN_DELETE) = 1
...
read(11		<-- blocking

$ crystal --version
Crystal 1.16.3 (2025-05-12)

LLVM: 20.1.8
Default target: x86_64-unknown-linux-gnu

inotify_init1(IN_NONBLOCK)              = 11
...
fcntl(11, F_GETFL)                      = 0x800 (flags O_RDONLY|O_NONBLOCK)
fstat(11, {st_mode=0600, st_size=0, ...}) = 0
...
inotify_add_watch(11, "/var/lib/<redacted>", IN_CLOSE_WRITE|IN_DELETE) = 1
...
read(11, 0x7fb69d9ec000, 40960)         = -1 EAGAIN (Resource temporarily unavailable)

[ysbaddaden]

The whole blocking/non-blocking has been completely overhauled in 1.17 (as per the 1.17.0 changelog) and we musn't assume that the event loop will use O_NONBLOCK anymore.

Note: the shard shouldn't set IN_NONBLOCK anymore (but should always set IN_CLOEXEC) and let the evloop configure the fd.

Now, the issue is that inotify creates file descriptors with an unknown type, which leads to set the blocking mode because the polling evloops check the file type, but as explained in #15652 (comment) we could drop the check and the polling evloops could always set O_NONBLOCK (and io_uring always remove it). We'd spare a call to fstat and we'd avoid this bug.

[ysbaddaden]

A temporary workaround until the bug is fixed is to call IO::FileDescriptor.set_blocking(fd, false).

[straight-shoota]

This change was introduced in #15753 as part of a bigger refactor to move responsibility for deciding the blocking mode of newly opened file descriptors to the event loop (#15652).
Unfortunately, it has an effect on external file descriptors as well. We didn't fully understand these consequences at the time, so they were not properly considered.

We're working on further improvements for handling blocking state of file descriptors. Basically, the current state is very broken with interoperable file descriptor state management (see #16353, in particular #16353 (comment)).
So some changes are to be expected in the future, which should improve interoperability and handling of blocking file descriptors (#15871).

However, we should consider how libraries can use file descriptors that are explicitly non-blocking.
When creating an IO::FileDescriptor instance from an existing fd, we should be able to detect when it's opened in non-blocking mode and respect that.
Perhaps we could recommend aksing the event loop whether it even want's non-blocking mode?

Anyway, it's currently possible to give blocking mode information to the constructor explicitly: IO::FileDescriptor.new(fd, blocking: false). This should be a good workaround for your use case.

[ysbaddaden]

@straight-shoota The STDIOs are special (the mode leaks to external processes) and they must be handled specifically by the stdlib. That's another bug.

C bindings that need the fd to be in a specific mode must either:

• not adopt the fd into Crystal, which means never instantiate an IO::FileDescriptor (recommended);
• or must explicitly call IO::FileDescriptor.set_blocking to set the specific mode, and risk breaking the evloop (not recommended, obviously).

☝️ we shall put that in the docs ☝️

Here, inotify wants to adopt the fd into the event loop (so we can read events). It shouldn't configure the blocking mode and merely instantiate an IO::FileDescriptor. The problem is the stdlib behavior of "set O_NONBLOCK depending on the file type" which is pointless and buggy: the evloop must always set the non/blocking state, regardless of the actual mode of the fd (again: STDIOs are special).

[straight-shoota]

I don't think there is anything special about STDIO. The same applies to any fd which gets adopted into the event loop or leaves the event loop. We can never be sure whether the file description is shared by other processes and what effects a change of blocking mode can have.

The standard mode for interoperable file descriptors is blocking. So we must make sure to support that for any incoming or outgoing file descriptors.
The event loop must not change the blocking mode of existing fds, unless we can be sure it's safe to do so.

Detachable threads should enable support for blocking file descriptors in polling event loops.

When an incoming fd is in non-blocking mode, the event loop should recognize that and make use of it, if possible. It must never change the mode to blocking, though.

[ysbaddaden]

I disagree. It must change it and inotify is a great example: the shard would have to know what the runtime event loop expects and have to configure it on behalf of the event loop, or have to suffer through sub-par behavior (block a whole thread/scheduler for up to 10ms every time it would block).

[straight-shoota]

The default behaviour should prioritze safety and correctnes.
If we want efficiency, we need to adapt to the needs of the event loop. But this should be an explicit choice. We mustn't implicitly change the blocking state of a file descriptor that we don't own.

The event loop already exposes its preference so we can open the file descriptor accordingly:

fd = LibInotify.init(Crystal::EventLoop.default_file_blocking? ? 0 : LibC::O_NONBLOCK)
io = IO::FileDescriptor.new(fd)

Or we tell FileDescriptor.new that it can treat the fd as its own and do whatever with it.

fd = LibInotify.init(LibC::O_NONBLOCK)
io = IO::FileDescriptor.new(fd, owned: true)

The latter might not work well on IOCP where we cannot change the blocking state of an existing handle.
So the former is probably better.

[yxhuvud]

I agree with @straight-shoota that changing the blocking status is very unexpected, in a bad way. One thing that could be an option would be to add something like Crystal::EventLoop.preferred_file_options to get both O_NONBLOCK and O_CLONOEXEC. I'd rather have the runtime raise than automagically change incoming file descriptors. Perhaps raising on the default constructor and having an explicit 'I-know-I'm-doing-something-weird'-option for the ones that really want to use one in the wrong way could make sense?

Adding a documentation block to the C binding info seems good though, spelling out the options. And the raise-message could also give pointers for what to look at.

[ysbaddaden]

We can't have a general "preferred options": the O_NONBLOCK or O_CLOEXEC flags are only used by open, the rest always uses specific flags such as FD_CLOEXEC, SOCK_CLOEXEC, EPOLL_CLOEXEC, NI_CLOEXEC, ... that may be identical (linux) or not (netbsd, solaris) to O_CLOEXEC (same for O_NONBLOCK).

Round up of external libraries:

• inotify: needs to instantiate an IO::FileDescriptor for reading events, and must adopt the fd into the evloop (to fix: musn't set non-blocking);
  I.e. io = IO::FileDescriptor.new(fd)

• libssh: fully delegates the fd (create, read, write) to the library, but needs to instantiate a raw IO::FileDescriptor instance for waiting on readiness when the library reports it to not be ready (to fix: must set the fd as non-blocking);
  I.e. io = IO::FileDescriptor.raw(fd)

• wayland (xkb): needs to instantiate an IO::FileDescriptor to read, and must adopt the fd into the evloop (to fix: musn't set non-blocking);
  I.e. io = IO::FileDescriptor.new(fd)

• wayland (display, decor): fully delegates the fd to the library, but needs to instantiate a raw IO::FileDescriptor for waiting on readiness (to fix: either the libs support O_NONBLOCK and report EAGAIN or zero timeout, or the bindings shall start a level-triggered epoll and register/wait on the epoll fd, or start an isolated fiber to block on dispatch; because waiting before every read with edge-triggered epoll is undefined behavior).
  I.e. either io = IO::FileDescriptor.new(fd) or io = IO::FileDescriptor.raw(fd).

• wayland (wl_pool): only needs the fd itself (to fix: musn't instantiate an IO::FileDescriptor nor set blocking or non-blocking).

• ... (something else?)

I'm really trying to be convinced... but to no avail 😞

I agree that we need an IO::FileDescriptor.raw(fd : Handle) that won't reconfigure the fd, but I disagree that we should change the current default behavior (albeit inconsistent) of always changing the blocking mode (Crystal 0.7.0) and we explicited the behavior in the docs: "adopts the fd into the internal IO subsystem that will reconfigure it" (same for Socket.new(fd)). There are no surprises here 😞

The surprises are that we assumed the fd to be in blocking mode without even checking (fixed in 1.17), and that we do it inconsistently (must be fixed).

[yxhuvud]

Oh, too bad about the flags :(

I disagree - there are plenty of surprise in anything that modifies the given argument, even if it is documented. I agree that silently trying to work with something that won't work well is wrong, but modification of given fds created outside the event loop is just as bad. At least for a constructor like new - naming does play a very big part here.

and we explicited the behavior in the docs: "adopts the fd into the internal IO subsystem that will reconfigure it" (same for Socket.new(fd)). There are no surprises here

Documentation is needed for nonobvious or surprising behavior that cannot be avoided otherwise or has some other really strong upsides that is a lot stronger than the downsides of the surprise, but it is NEVER in itself a valid excuse for adding or keeping it.