Broker ID's can get out of sync with metadata ID

[tmelhiser]

In our environment, we do not have shared storage and need to persist volumes on local storage for the kafka logs. As such when/if a kafka node goes down and restarts, the broker and index may not always return to the same host. Leaving the /opt/kafka/config/server.properties mismatched with the /opt/kafka/logs/meta.properties file.

Our work around was to extend the functionality of gen.conf.tmpl.sh to look for meta.properties first, extract its broker ID and use it to populate the values for server.properties. If the meta.properties file is not found, or if it had no broker.id value set, it would use the original method of querying the rancher meta-data service for the /self/container/service_index value and populating both the meta.properties and server.properties with matching values.

--- gen.conf.tmpl.sh.orig
+++ gen.conf.tmpl.sh
@@ -36,9 +36,23 @@
 reload_cmd = "${SERVICE_HOME}/bin/kafka-service.sh restart"
 EOF

+[ -e ${KAFKA_LOG_DIRS}/meta.properties ] && {
+   BROKER_ID=$(grep ^broker.id ${KAFKA_LOG_DIRS}/meta.properties|awk -F= '{print $2}');
+}
+
+[ -z "${BROKER_ID}" ] && {
 cat << EOF > ${SERVICE_VOLUME}/confd/etc/templates/server.properties.tmpl
 ############################# Server Basics #############################
 broker.id={{getv "/self/container/service_index"}}
+EOF
+} || {
+cat << EOF > ${SERVICE_VOLUME}/confd/etc/templates/server.properties.tmpl
+############################# Server Basics #############################
+broker.id=${BROKER_ID}
+EOF
+}
+
+cat << EOF >> ${SERVICE_VOLUME}/confd/etc/templates/server.properties.tmpl
 ############################# Socket Server Settings #############################
 listeners=${KAFKA_LISTENER}
 advertised.listeners=${KAFKA_ADVERTISE_LISTENER}