Start Scripts incompatible with systemd in OEL9 (252)

[dnorthup-ums]

I am not asking for support for OEL9. It just happens to be our supported server platform at $dayjob.

Jetty version(s)
12.1.9

Jetty Environment
ee11
(doesn't get far enough to matter)

HTTP version
Whatever Apache HTTPd wants to use for reverse proxy. Unrelated to what I'm reporting.

Java version/vendor
Corretto 21

OS type/version
OEL9

Description
I have tried the stock systemd.service unit, but the scripts in /etc/rc.d/init.d tends to cause problems as systemd tries to create SYSVINIT compatibility units, gives up, and pukes all over the place. (I'm actually symlinking to the real ones from there, as updating Jetty and having the wrong startup script is damnation nobody should inflict upon themselves. Thankfully, when the systemd SYSVINIT compatibility is short-circuited by using actual SYSVINIT, started by systemd, then it works ok-ish.)
I took the stock systemd.service unit and attempted to modify it directly to run the actual start script. It refuses to run claiming "Permission denied" (SELINUX perhaps?). (It does the same thing if I use a symlink.)
I took the stock systemd.service unit and pointed it at a wrapper of jetty.sh in /usr/local/bin. Jetty starts ok, but then systemd kills it because it was started in a way that systemd doesn't understand and it decides that the running program isn't supposed to be there.

This happens with and without systemd using the same PID file as Jetty itself.
This happens with all of the "Type=" values that make any sense.

May I gently suggest that ignoring systemd and using a complicated startup script is not working out in a long-term sustainable way? Eventually chkconfig will not be available on servers, or it will start doing undesired things.

How to reproduce?
Put Jetty in any of the normal places, EG: JETTY_BASE=/opt/jetty; JETTY_HOME=/opt/jetty-dist (symlink to actual version directory)
Attempt to use the stock jetty.service file. Observe errors causing failures when attempting to let it start via files in /etc/rc.d/init.d (literal or symlink).
Modify the systemd unit in whatever way seems like a good idea. Stumble upon something that doesn't immediately puke an error. Find that systemd started Jetty just fine and then promptly killed it.

I hate to be a smartass, but this took up more than 6 hours of $dayjob time across several days. The only sane conclusion I'm left with is that at some point either Jetty will work properly with systemd (as much as I hate it, and boy do I—servers are not laptops) or we'll have to abandon using Jetty completely at $dayjob.
I may have time to come back and add some sanitized console / log output, but I don't know yet.

[sbordet]

We don't ship a "stock systemd.service unit", so no idea what you're referring to.

We deploy Jetty under Ubuntu as a systemd service for our own websites, and everything works like a charm, no sysvinit.

I also don't understand your reference to chkconfig: while it is true that jetty.sh has an old comment about it, it is not actually used in the bash script.

I'm sorry you wasted those 6 hours, but Jetty works fine with systemd, at least on Ubuntu.
I have no experience on Oracle Linux, but I doubt it'll be a problem.

This is the systemd file we use:

$ cat /etc/systemd/system/websites.service 
[Unit]
Description=Jetty Web Application Server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Environment=JAVA=/opt/java/bin/java
Environment=JETTY_HOME=/opt/jetty
Environment=JETTY_BASE=/srv/www/websites
Environment=JETTY_RUN=/srv/www/websites
Type=forking
PIDFile=/srv/www/websites/jetty.pid
ExecStart=/opt/jetty/bin/jetty.sh start
ExecStop=/opt/jetty/bin/jetty.sh stop
ExecReload=/opt/jetty/bin/jetty.sh restart
User=www-data
Group=www-data

[Install]
WantedBy=multi-user.target

[dnorthup-ums]

We don't ship a "stock systemd.service unit", so no idea what you're referring to.

Using the TGZ downloaded via the official download page on jetty.org:

$ tar -ztf jetty-home-12.1.9.tar.gz | grep ".service"
jetty-home-12.1.9/bin/jetty.service

That's the "stock" systemd .service file.

We deploy Jetty under Ubuntu as a systemd service for our own websites, and everything works like a charm, no sysvinit.

Good. This is what I want too.

I also don't understand your reference to chkconfig: while it is true that jetty.sh has an old comment about it, it is not actually used in the bash script.

The chkconfig program is the utility used to enable SYSVINIT services. It is not run by the bash script, it enables running the bash script to run as an init.d script in the SYSVINIT environment.
This part works just fine, for the time being. However, it is deprecated in RHEL / OEL and derivatives in generation 9 and above.

I'm sorry you wasted those 6 hours, but Jetty works fine with systemd, at least on Ubuntu. I have no experience on Oracle Linux, but I doubt it'll be a problem.

Unfortunately, it is a problem.

This is the systemd file we use:

What version systemd are your systems using? (OR: What generation / base version Ubuntu?)

$ cat /etc/systemd/system/websites.service

....

[Service]

I presume you're setting the environment here instead of using /etc/default/jetty ?

Environment=JAVA=/opt/java/bin/java
Environment=JETTY_HOME=/opt/jetty
Environment=JETTY_BASE=/srv/www/websites
Environment=JETTY_RUN=/srv/www/websites

...

Is this the same PID file as used by Jetty in your setup?

PIDFile=/srv/www/websites/jetty.pid

This is the part that seems to cause the most difficulty for our systems. They don't seem to "want" to allow anything in /opt to be run directly by systemd.

ExecStart=/opt/jetty/bin/jetty.sh start
ExecStop=/opt/jetty/bin/jetty.sh stop
ExecReload=/opt/jetty/bin/jetty.sh restart

[sbordet]

My bad about jetty.service.

What version systemd are your systems using? (OR: What generation / base version Ubuntu?)

We are on Ubuntu 24.04.4 LTS (GNU/Linux 6.8.0-111-generic x86_64).

$ systemd --version
systemd 255 (255.4-1ubuntu8.15)

I presume you're setting the environment here instead of using /etc/default/jetty ?

Correct, we don't have a /etc/default/jetty anymore (less files to manage).

Is this the same PID file as used by Jetty in your setup?

Not sure I understand this question.
It is the PID file saved when Jetty starts, containing the PID of the JVM process that runs Jetty.

This is the part that seems to cause the most difficulty for our systems. They don't seem to "want" to allow anything in /opt to be run directly by systemd.

This I can't answer, but Claude says it's SELinux.
At this point, your Claude-ing is equivalent to mine, but I'm confident that once you figure out SELinux, all will work fine.

Let us know what magic worked for you, we may incorporate it in the documentation.

[dnorthup-ums]

Is this the same PID file as used by Jetty in your setup?

Not sure I understand this question. It is the PID file saved when Jetty starts, containing the PID of the JVM process that runs Jetty.

That is exactly what I wanted to know. Thank You.

This is the part that seems to cause the most difficulty for our systems. They don't seem to "want" to allow anything in /opt to be run directly by systemd.

This I can't answer, but Claude says it's SELinux. At this point, your Claude-ing is equivalent to mine, but I'm confident that once you figure out SELinux, all will work fine.

Let us know what magic worked for you, we may incorporate it in the documentation.

I hope magic isn't required, but I had already started to suspect SELINUX as well.