Adding epoll Poller

[aleph0naught]

Hi!

Are there any plans to add epoll to pollers list?
If so, when will it happen? And if not, is it worth adding its support on my own? Are there any pitfalls here?

---

background:
We found high CPU consumption (our java application 70%, supervisor 30%) when using supervisor
After running strace, it turned out that this consumption is associated with redirecting application's stdout to log file - supervisor using system calls read/write/select in python2 and poll in python3 - and apparently the main cpu time is spent on select syscall. Also we can't use kqueue since we use ubuntu

We need logs, so we want to reduce CPU consumption without turning them off
So far, the only thing that comes to mind is using epoll instead of select/poll to wait for descriptors in the hope that it will help

Or maybe you can suggest how to solve this problem in some other way?

Its not #807 ticket dublicate! There is no answer to my question there

[ollofx]

indeed it is a poll issue this is the patch that I applied to avoid continuous looping, but it extend a bit minimum response time (sleep 1s but you could put much less)
I use this patch in production since more then a year.

added time.sleep(1), you might chose less, the only purpose is to avoid continuous loop

/usr/local/lib/python3.7/site-packages/supervisor/poller.py:116
/usr/lib/python2.7/dist-packages/supervisor/poller.py:116

   def poll(self, timeout):
        fds = self._poll_fds(timeout)
        readables, writables = [], []
        time.sleep(1)
        for fd, eventmask in fds:
            if self._ignore_invalid(fd, eventmask):
                continue
            if eventmask & self.READ:
                readables.append(fd)
            if eventmask & self.WRITE:
                writables.append(fd)
        return readables, writables

[mnaberez]

added time.sleep(1), you might chose less, the only purpose is to avoid continuous loop

I'm hiding this response as off-topic only because it's not related to epoll. This solution is already proposed on #807 and I'm hoping all discussions related to #807 can stay on #807. I'm leaving this ticket (#1516) open just for the epoll question.

[mnaberez]

Are there any plans to add epoll to pollers list?
If so, when will it happen?

There are not currently any plans to add epoll.

And if not, is it worth adding its support on my own? Are there any pitfalls here?

Aside from merging fixes contibuted by users, I haven't been involved in the poller code, so I can't really speak to this. All of the polling code is found in poller.py.

We need logs, so we want to reduce CPU consumption without turning them off
So far, the only thing that comes to mind is using epoll instead of select/poll to wait for descriptors in the hope that it will help

Or maybe you can suggest how to solve this problem in some other way?

Its not #807 ticket dublicate! There is no answer to my question there

In #1581, it has been suggested that high CPU usage is caused by polling a wrong file descriptor. If that is also happening in your deployment, then an upcoming version of Supervisor should fix it after #1581 or a variant of it is merged. We are looking for people to test the patch in #1581 if you want to try it.

I'm going to close this because I've attempted to answer the question about epoll and we still have #807 and #1581 open.