Large increase in RAM usage from 21.12.0 to 22.4.0

[Daniel15]

Version

22.4.0

Steps to Reproduce

I upgraded from Sentry 21.12.0 to 22.4.0 and now the RAM usage is significantly higher. The below graph shows available RAM on the server.

The upgrade was performed around May 16, where you can see a steep drop in available RAM. I received a critical alert when the system dropped to ~2.4GB RAM available.

Most of the RAM usage on the server is from Snuba, Clickhouse and Kafka:

There's other services running on this server, but they consume a much smaller portion of RAM.

Did something change from 21.12.0 to 22.4.0 that would result in significantly higher RAM usage?

I'm running Sentry on a dedicated server with 16GB RAM. My usage is not very heavy - I'm only monitoring two smallish apps with it.

Expected Result

Sentry should comfortably fit on a server with 8GB RAM as per the minimum requirements in the d oc.

Actual Result

As above

[aminvakil]

@Daniel15 This may be related to https://github.com/getsentry/self-hosted/discussions/1453, can you upgrade to 21.5.0 ?

[chadwhitacre]

From the Discussions version of this thread:

@Daniel15:

Thanks @aminvakil! I'm about to go to sleep but I'll try that soon (probably tomorrow) and let you know.

@Daniel15:

@aminvakil I'm trying to upgrade but it gives me this output:

root@lv02:/opt/sentry# ./install.sh
▶ Parsing command line ...

▶ Initializing Docker Compose ...

▶ Setting up error handling ...

▶ Checking for latest commit ...

▶ Checking minimum requirements ...
Found Docker version 20.10.12
Found Docker Compose version 1.29.2

▶ Turning things off ...
The HEALTHCHECK_INTERVAL variable is not set. Defaulting to a blank string.
The HEALTHCHECK_TIMEOUT variable is not set. Defaulting to a blank string.
The HEALTHCHECK_RETRIES variable is not set. Defaulting to a blank string.
Error while attempting to convert service.memcached.healthcheck.retries to appropriate type: "" is not a valid integer
An error occurred, caught SIGERR on line 8
Cleaning up...

Should I make a separate post about that? I can't find any other posts about it.

@aminvakil:

@Daniel15 Is your issue related to #1482 ?

Until the PR is merged and included in next release, I'd suggest testing with latest docker-compose (which is currently 2.5.1).

Although AFAICS you're using 1.29.2, right? Upgrading to latest docker-compose release is worth to try anyway.

@Daniel15:

@aminvakil - I upgraded to the latest versions of Docker and Docker-compose, and edited the docker-compose.yml file as per that issue. Now I'm getting a different error:

▶ Checking minimum requirements ...
Found Docker version 20.10.16
Found Docker Compose version 2.5.1

▶ Turning things off ...
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"HEALTHCHECK_INTERVAL\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"HEALTHCHECK_TIMEOUT\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"HEALTHCHECK_RETRIES\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"SENTRY_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"SNUBA_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"SNUBA_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"SYMBOLICATOR_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"SNUBA_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"HEALTHCHECK_INTERVAL\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"HEALTHCHECK_TIMEOUT\" variable is not set. Defaulting to a blank string."
time="2022-05-28T23:43:07-07:00" level=warning msg="The \"HEALTHCHECK_RETRIES\" variable is not set. Defaulting to a blank string."
error while interpolating services.kafka.healthcheck.retries: failed to cast to expected type: strconv.Atoi: parsing "": invalid syntax
An error occurred, caught SIGERR on line 8

@aminvakil:

There is no need to change anything in docker-compose.yml if you're using the latest docker-compose version, use git restore docker-compose.yml to make sure there is not a change in your docker-compose.yml and try again.

@Daniel15:

@aminvakil I reverted docker-compose.yml but hit the same issue. There are no other files modified:

root@lv02:/opt/sentry# git status
HEAD detached at 22.5.0
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        sentry/backup.json

nothing added to commit but untracked files present (use "git add" to track)

The only thing I've customised in .env.custom is SENTRY_BIND:

SENTRY_BIND=127.0.0.1:9000

@aminvakil:

Stop and remove containers manually before running ./install.sh, then please the output of ./install.sh.

docker-compose down -v --remove-orphans
./install.sh

@Daniel15:

@aminvakil Same error:

root@lv02:/opt/sentry# docker-compose ps --all
NAME                COMMAND             SERVICE             STATUS              PORTS
root@lv02:/opt/sentry# ./install.sh
▶ Parsing command line ...

▶ Initializing Docker Compose ...

▶ Setting up error handling ...

▶ Checking for latest commit ...

▶ Checking minimum requirements ...
Found Docker version 20.10.16
Found Docker Compose version 2.5.1

▶ Turning things off ...
time="2022-05-29T00:15:13-07:00" level=warning msg="The \"SENTRY_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-29T00:15:13-07:00" level=warning msg="The \"SNUBA_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-29T00:15:13-07:00" level=warning msg="The \"SYMBOLICATOR_IMAGE\" variable is not set. Defaulting to a blank string."
time="2022-05-29T00:15:13-07:00" level=warning msg="The \"HEALTHCHECK_INTERVAL\" variable is not set. Defaulting to a blank string."
time="2022-05-29T00:15:13-07:00" level=warning msg="The \"HEALTHCHECK_TIMEOUT\" variable is not set. Defaulting to a blank string."time="2022-05-29T00:15:13-07:00" level=warning msg="The \"HEALTHCHECK_RETRIES\" variable is not set. Defaulting to a blank string."error while interpolating services.memcached.healthcheck.retries: failed to cast to expected type: strconv.Atoi: parsing "": invalid syntax
An error occurred, caught SIGERR on line 8
Cleaning up...	

If I run the same command turn-things-off.sh should be running, I don't get an error:

docker-compose rm -fsv $(docker-compose config --services | grep -v -E '^(nginx|relay)$')
No stopped containers

However, while debugging it, I worked out the issue: ALL the entries from .env need to be copied to .env.custom, not just the modified ones (which is usually the case with an overrides file). I did that and the installation is running now. I'll update the readme to make it clearer.

@aminvakil:

Ah, readme could definitely be improved, glad you got to the bottom of this.

[Daniel15]

I forgot to reply (couldn't find the discussion again) but I can confirm that the upgrade fixed this.