Make RabbitMQ a regular Erlang/OTP application #2180

dumbbell · 2019-12-05T13:39:12Z

Today, to start RabbitMQ, we depend on several Bourne shell scripts (for Unix-based OSes) and Batch scripts (for Microsoft Windows). They have several downsides:

The code is duplicated between the two languages used for those scripts.
Maintenance is obviously increased.
Debugging what happens before the Erlang VM is launched is difficult.
The Batch version for Windows gets less testing and may not have the same features as the Bourne shell version.

From an Erlang/OTP point of view, RabbitMQ is not a regular Erlang/OTP application because it relies on several things happening before the actual rabbit application can be started. This complicates developers' life because RabbitMQ does not embrace Erlang/OTP system principles and a few things are reinvented internally.

We want to move most of the scripts to plain Erlang code and change RabbitMQ so it is built as a standard Erlang/OTP application.

The initial description of the problem was written down in a gist.

The text was updated successfully, but these errors were encountered:

dumbbell · 2019-12-11T17:05:21Z

All pull requests were QA'd and merged.

No longer applies after merge of rabbitmq/rabbitmq-server#2180

Before `rabbit` startup code was rewritten as part of #2180 to make it closer to a regular Erlang application, plugins' boot steps were executed before plugins were started. This commit restores this behavior. Indeed the initial patch inverted them by starting the plugins first, then executed the boot steps. It also bring another improvement in the process: a dependency has its boot steps exeucted and is started before a plugin which depends on it is considered. This should improve consistency. Note that the `start_apps/2` function, which is ran when a user enables a plugin at runtime must be improved as well. There is a work in progress in #2219.

Before `rabbit` startup code was rewritten as part of #2180 to make it closer to a regular Erlang application, plugins' boot steps were executed before plugins were started. This commit restores this behavior. Indeed the initial patch inverted them by starting the plugins first, then executed the boot steps. It also brings another improvement in the process: a dependency has its boot steps executed and is started before a plugin which depends on it is considered. This should improve consistency. Note that the `start_apps/2` function, which is run when a user enables a plugin at runtime must be improved as well. There is a work in progress in #2219.

…lated-changes Fix logging after #2180-related changes

Various fixes post #2180

…lure ... in `erlang_config. Since #2180, a failed start_app does not take the node down anymore. Restarting the node just after was failing since, but this remained unnoticed so far because the return value of `start_node()` is not checked. However, since rabbitmq/rabbitmq-ct-helpers#c033d9272afaf3575505533c81f1c0c7cfcb6206, the Make recipe which starts the node automatically stops it if the start failed somewhere. This is in order to not leave an unwanted node around. This means that after the failing `rabbit_ct_broker_helpers:start_node()`, the node was effectively stopped this time, leading to the rest of the testcase to fail.

…lure ... in `erlang_config`. Since #2180, a failed `start_app` does not take the node down anymore. Trying to restart the node just after was failing since (because the node is still there), but this remained unnoticed so far because the return value of `start_node()` is not checked. However, since rabbitmq/rabbitmq-ct-helpers@c033d92, the Make recipe which starts the node automatically stops it if the start failed somewhere. This is in order to not leave an unwanted node around. This means that after the failing `rabbit_ct_broker_helpers:start_node()`, the node was effectively stopped this time, leading to the rest of the testcase to fail.

…lure ... in `erlang_config`. Since #2180, a failed `start_app` does not take the node down anymore. Trying to restart the node just after was failing since (because the node is still there), but this remained unnoticed so far because the return value of `start_node()` is not checked. However, since rabbitmq/rabbitmq-ct-helpers@c033d92, the Make recipe which starts the node automatically stops it if the start failed somewhere. This is in order to not leave an unwanted node around. This means that after the failing `rabbit_ct_broker_helpers:start_node()`, the node was effectively stopped this time, leading to the rest of the testcase to fail. (cherry picked from commit 17968ee)

This has several benefits: 1. It simplifies the code, all configuration being handled by the same code path (no more condition on Erlang-term-based vs. Cuttlefish). `rabbit_config` shrinks quite a lot in the process. 2. We can use additional configuration files AND an Erlang-term-based configuration file. In other words, it is possible to use the same existing Erlang-term-based file and introduce Cuttlefish files when needed. It allows a user to run RabbitMQ with: RABBITMQ_CONFIG_FILE=/path/to/rabbitmq.config \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf \ ./sbin/rabbitmq-server A developer can do the same with `make run-broker`: make run-broker \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf In the example above, the main configuration file generated by rabbitmq-run.mk is an Erlang-term-based one. This is implemented by calling Cuttlefish with a (possibly empty) list of additional files and the Erlang-term-based file as the advanced configuration file. References #2180.

It is unused in RabbitMQ or tier-1 plugins, and the previously returned value made no sense since the switch to Cuttlefish as a library (as part of #2180).

This has several benefits: 1. It simplifies the code, all configuration being handled by the same code path (no more condition on Erlang-term-based vs. Cuttlefish). `rabbit_config` shrinks quite a lot in the process. 2. We can use additional configuration files AND an Erlang-term-based configuration file. In other words, it is possible to use the same existing Erlang-term-based file and introduce Cuttlefish files when needed. It allows a user to run RabbitMQ with: RABBITMQ_CONFIG_FILE=/path/to/rabbitmq.config \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf \ ./sbin/rabbitmq-server A developer can do the same with `make run-broker`: make run-broker \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf In the example above, the main configuration file generated by rabbitmq-run.mk is an Erlang-term-based one. This is implemented by calling Cuttlefish with a (possibly empty) list of additional files and the Erlang-term-based file as the advanced configuration file. References #2180. (cherry picked from commit 7edfe22)

It is unused in RabbitMQ or tier-1 plugins, and the previously returned value made no sense since the switch to Cuttlefish as a library (as part of #2180). (cherry picked from commit 5f5fccf)

This has several benefits: 1. It simplifies the code, all configuration being handled by the same code path (no more condition on Erlang-term-based vs. Cuttlefish). `rabbit_config` shrinks quite a lot in the process. 2. We can use additional configuration files AND an Erlang-term-based configuration file. In other words, it is possible to use the same existing Erlang-term-based file and introduce Cuttlefish files when needed. It allows a user to run RabbitMQ with: RABBITMQ_CONFIG_FILE=/path/to/rabbitmq.config \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf \ ./sbin/rabbitmq-server A developer can do the same with `make run-broker`: make run-broker \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf In the example above, the main configuration file generated by rabbitmq-run.mk is an Erlang-term-based one. This is implemented by calling Cuttlefish with a (possibly empty) list of additional files and the Erlang-term-based file as the advanced configuration file. References #2180. (cherry picked from commit 7edfe22)

It is unused in RabbitMQ or tier-1 plugins, and the previously returned value made no sense since the switch to Cuttlefish as a library (as part of #2180). (cherry picked from commit 5f5fccf)

…tderr redirections Historically we were using $RABBITMQ_LOG_BASE to configure the redirection. The variable default value was set in rabbitmq-env(8) which made it to the SysV init scripts because they sourced it in the past. This was removed in commit 4b70482 as part of the transition to rabbit_env/rabbitmq_prelaunch to handle the environment in the Erlang code (see rabbitmq/rabbitmq-server#2180). Instead, the value of $RABBITMQ_LOG_BASE was hard-coded. Unfortunately, this caused a regression because users couldn't configure it from rabbitmq-env.conf anymore (only /etc/default/rabbitmq-server). Anyway, the semantic was slightly incorrect: $RABBITMQ_LOG_BASE is used in the configuration of log files RabbitMQ is responsible for. Console redirection is the responsibility of the SysV init scripts and the package which creates the directory and set ownership. This patch introduces a new $RABBITMQ_SERVER_CONS_OUTPUT_DIR variable which is specific to the SysV init scripts. For backward compatibility, we still look at the value of $RABBITMQ_LOG_BASE if the user set it in /etc/default/rabbitmq-server. While here, align the Debian SysV init script behavior with the RPM version of the script: console redirection is always configured in the SysV init script, not in `rabbitmq-script-wrapper`. A subsequent commit will take care of cleaning `rabbitmq-script-wrapper`. References #131.

…tderr redirections Historically we were using $RABBITMQ_LOG_BASE to configure the redirection. The variable default value was set in rabbitmq-env(8) which made it to the SysV init scripts because they sourced it in the past. This was removed in commit 4b70482 as part of the transition to rabbit_env/rabbitmq_prelaunch to handle the environment in the Erlang code (see rabbitmq/rabbitmq-server#2180). Instead, the value of $RABBITMQ_LOG_BASE was hard-coded. Unfortunately, this caused a regression because users couldn't configure it from rabbitmq-env.conf anymore (only /etc/default/rabbitmq-server). Anyway, the semantic was slightly incorrect: $RABBITMQ_LOG_BASE is used in the configuration of log files RabbitMQ is responsible for. Console redirection is the responsibility of the SysV init scripts and the package which creates the directory and set ownership. This patch introduces a new $RABBITMQ_SERVER_CONS_OUTPUT_DIR variable which is specific to the SysV init scripts. For backward compatibility, we still look at the value of $RABBITMQ_LOG_BASE if the user set it in /etc/default/rabbitmq-server. While here, align the Debian SysV init script behavior with the RPM version of the script: console redirection is always configured in the SysV init script, not in `rabbitmq-script-wrapper`. A subsequent commit will take care of cleaning `rabbitmq-script-wrapper`. The solution to redirect output when using start-stop-daemon(8) was taking from the following post: https://stackoverflow.com/questions/8251933/how-can-i-log-the-stdout-of-a-process-started-by-start-stop-daemon References #131.

…t/stderr redirections Historically we were using $RABBITMQ_LOG_BASE to configure the redirection. The variable default value was set in rabbitmq-env(8) which made it to the SysV init scripts because they sourced it in the past. This was removed in commit 4b70482 as part of the transition to rabbit_env/rabbitmq_prelaunch to handle the environment in the Erlang code (see rabbitmq/rabbitmq-server#2180). Instead, the value of $RABBITMQ_LOG_BASE was hard-coded. Unfortunately, this caused a regression because users couldn't configure it from rabbitmq-env.conf anymore (only /etc/default/rabbitmq-server). Anyway, the semantic was slightly incorrect: $RABBITMQ_LOG_BASE is used in the configuration of log files RabbitMQ is responsible for. Console redirection is the responsibility of the SysV init scripts and the package which creates the directory and set ownership. This patch introduces a new $RABBITMQ_SERVER_CONSOLE_OUTPUT_DIR variable which is specific to the SysV init scripts. For backward compatibility, we still look at the value of $RABBITMQ_LOG_BASE if the user set it in /etc/default/rabbitmq-server. While here, align the Debian SysV init script behavior with the RPM version of the script: console redirection is always configured in the SysV init script, not in `rabbitmq-script-wrapper`. A subsequent commit will take care of cleaning `rabbitmq-script-wrapper`. The solution to redirect output when using start-stop-daemon(8) was taking from the following post: https://stackoverflow.com/questions/8251933/how-can-i-log-the-stdout-of-a-process-started-by-start-stop-daemon References #131.

As said in the comment, this is normally done at runtime now (see #2180). However, we must configure the distribution on the command line specifically for the Windows service. Until this patch, the distribution was enabled with the correct nodename, but the TCP port configuration was not set. This caused a regression because the selected TCP port was random instead of the expected default of 25672. This patch restores the previous behavior. Fixes #2387.

michaelklishin · 2020-06-26T16:34:50Z

This was backported for 3.8.4 and resulted in a number of relatively minor regressions (mostly around environment/configuration discovery) addressed in 3.8.5 and 3.8.6.

No longer applies after merge of rabbitmq/rabbitmq-server#2180 (cherry picked from commit 1013717)

This is unused by RabbitMQ after the completion of rabbitmq/rabbitmq-server#2180. (cherry picked from commit c6b0d1a)

Between RabbitMQ 3.8.3 and 3.8.4 logging to `syslog` broke because the `syslog` OTP application was no longer started when needed. This is probably due to changes in #2180. This change explicitly starts the `syslog` app if the backend is configured. Found while investigating this issue: https://groups.google.com/forum/?oldui=1#!topic/rabbitmq-users/-aqS2BiHm4k

…t/stderr redirections Historically we were using $RABBITMQ_LOG_BASE to configure the redirection. The variable default value was set in rabbitmq-env(8) which made it to the SysV init scripts because they sourced it in the past. This was removed in commit c080747 as part of the transition to rabbit_env/rabbitmq_prelaunch to handle the environment in the Erlang code (see rabbitmq/rabbitmq-server#2180). Instead, the value of $RABBITMQ_LOG_BASE was hard-coded. Unfortunately, this caused a regression because users couldn't configure it from rabbitmq-env.conf anymore (only /etc/default/rabbitmq-server). Anyway, the semantic was slightly incorrect: $RABBITMQ_LOG_BASE is used in the configuration of log files RabbitMQ is responsible for. Console redirection is the responsibility of the SysV init scripts and the package which creates the directory and set ownership. This patch introduces a new $RABBITMQ_SERVER_CONSOLE_OUTPUT_DIR variable which is specific to the SysV init scripts. For backward compatibility, we still look at the value of $RABBITMQ_LOG_BASE if the user set it in /etc/default/rabbitmq-server. While here, align the Debian SysV init script behavior with the RPM version of the script: console redirection is always configured in the SysV init script, not in `rabbitmq-script-wrapper`. A subsequent commit will take care of cleaning `rabbitmq-script-wrapper`. The solution to redirect output when using start-stop-daemon(8) was taking from the following post: https://stackoverflow.com/questions/8251933/how-can-i-log-the-stdout-of-a-process-started-by-start-stop-daemon References rabbitmq/rabbitmq-server-release#131.

dumbbell added the enhancement label Dec 5, 2019

dumbbell added this to the 3.9.0 milestone Dec 5, 2019

dumbbell self-assigned this Dec 5, 2019

dumbbell changed the title ~~Move rabbitmq-server(8) script(s) to Erlang code~~ Move rabbitmq-server(8) scripts to Erlang code Dec 5, 2019

dumbbell closed this as completed Dec 11, 2019

lukebakken added a commit to rabbitmq/rabbitmq-website that referenced this issue Dec 16, 2019

Remove RABBITMQ_GENERATED_CONFIG_DIR

257eead

No longer applies after merge of rabbitmq/rabbitmq-server#2180

lukebakken added a commit to rabbitmq/rabbitmq-common that referenced this issue Dec 16, 2019

Remove RABBITMQ_GENERATED_CONFIG_DIR

6b3bc5c

No longer applies after merge of rabbitmq/rabbitmq-server#2180

This was referenced Dec 16, 2019

Remove RABBITMQ_GENERATED_CONFIG_DIR rabbitmq/rabbitmq-common#345

Merged

Remove RABBITMQ_GENERATED_CONFIG_DIR rabbitmq/rabbitmq-website#914

Merged

This was referenced Jan 21, 2020

rabbit_env: Record variables' origin rabbitmq/rabbitmq-common#352

Merged

Move all RabbitMQ-specific environment variables to rabbit_env #2216

Merged

rabbit: Change start_apps/2 to call application:ensure_all_started/2 #2219

Merged

dumbbell mentioned this issue Jan 24, 2020

rabbit: Fix plugins' run_boot_steps() vs. start order #2220

Merged

dumbbell changed the title ~~Move rabbitmq-server(8) scripts to Erlang code~~ Make RabbitMQ a regular Erlang/OTP application Feb 4, 2020

This was referenced Feb 4, 2020

rabbit_env_SUITE: Check if $RABBITMQ_FEATURE_FLAGS is overridden rabbitmq/rabbitmq-common#353

Merged

Fix logging after #2180-related changes #2236

Merged

dumbbell added a commit that referenced this issue Feb 10, 2020

Merge pull request #2236 from rabbitmq/fix-logging-after-issue2180-re…

bfc9aca

…lated-changes Fix logging after #2180-related changes

This was referenced Feb 10, 2020

unit_log_config_SUITE: Adapt after fixes to rabbit_lager #2237

Merged

Various fixes post #2180 #2238

Merged

dumbbell added a commit that referenced this issue Feb 10, 2020

Merge pull request #2238 from rabbitmq/various-fixes-post-issue2180

a922e91

Various fixes post #2180

This was referenced Feb 10, 2020

Convert systemd notification to prelaunch steps #2231

Merged

Remove systemd_notify_socket rabbitmq/rabbitmq-common#354

Merged

This was referenced Mar 20, 2020

Backport "Make RabbitMQ a regular Erlang/OTP application" to RabbitMQ 3.8.x #2282

Merged

Backport "Make RabbitMQ a regular Erlang/OTP application" to RabbitMQ 3.8.x rabbitmq/rabbitmq-server-release#121

Merged

dumbbell mentioned this issue Mar 31, 2020

clustering_management_SUITE: No need to stop node after start_app failure in erlang_config #2294

Merged

dumbbell added a commit that referenced this issue Apr 23, 2020

rabbit_config: Deprecate schema_dir/0

5f5fccf

It is unused in RabbitMQ or tier-1 plugins, and the previously returned value made no sense since the switch to Cuttlefish as a library (as part of #2180).

dumbbell mentioned this issue Jun 19, 2020

scripts/rabbitmq-service.bat: Configure Erlang distribution port #2389

Merged

michaelklishin modified the milestones: 3.9.0, 3.8.4 Jun 26, 2020

HoloRin pushed a commit to rabbitmq/rabbitmq-monorepo that referenced this issue Aug 11, 2020

Remove RABBITMQ_GENERATED_CONFIG_DIR

6620eb1

No longer applies after merge of rabbitmq/rabbitmq-server#2180 (cherry picked from commit 1013717)

HoloRin pushed a commit to rabbitmq/rabbitmq-monorepo that referenced this issue Aug 11, 2020

rabbitmq-run.mk: Remove uses of $RABBITMQ_SCHEMA_DIR

c6d4981

This is unused by RabbitMQ after the completion of rabbitmq/rabbitmq-server#2180. (cherry picked from commit c6b0d1a)

lukebakken mentioned this issue Sep 28, 2020

Explicitly start syslog application #2456

Merged

This was referenced Dec 16, 2020

rabbit: Run plugins' boot steps during rabbit start/2 #2656

Merged

rabbitmq-server failed to start after enabling the rabbitmq_email plugin gotthardp/rabbitmq-email#42

Closed

michaelklishin mentioned this issue May 12, 2021

Setting RABBITMQ_NODE_PORT doesn't seem to effect docker-library/rabbitmq#480

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make RabbitMQ a regular Erlang/OTP application #2180

Make RabbitMQ a regular Erlang/OTP application #2180

dumbbell commented Dec 5, 2019

dumbbell commented Dec 11, 2019 •

edited

Loading

Uh oh!

michaelklishin commented Jun 26, 2020 •

edited

Loading

Uh oh!

Make RabbitMQ a regular Erlang/OTP application #2180

Make RabbitMQ a regular Erlang/OTP application #2180

Comments

dumbbell commented Dec 5, 2019

dumbbell commented Dec 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelklishin commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dumbbell commented Dec 11, 2019 •

edited

Loading

michaelklishin commented Jun 26, 2020 •

edited

Loading