Use a distroless based runtime image #18208

AndrewFerr · 2025-03-03T21:21:02Z

Based on #18039

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct
(run the linters)

as there is no risk of pulling these from a public registry

AndrewFerr · 2025-03-04T13:20:17Z

docker/conf-workers/supervisord.conf.j2

-command=/usr/local/bin/prefix-log /usr/local/bin/redis-server --unixsocket /tmp/redis.sock
+command=/usr/local/bin/prefix-log /usr/bin/redis-server --unixsocket /tmp/redis.sock
 {% else %}
-command=/usr/local/bin/prefix-log /usr/local/bin/redis-server
+command=/usr/local/bin/prefix-log /usr/bin/redis-server


This should have been its own commit to make this fix more clear, but this is needed because of the changes to how Dockerfile-workers installs Redis. This fixes the Complement run for workers+Postgres.

This reverts commit b9fab43.

AndrewFerr · 2025-03-04T21:06:41Z

The failed Complement run in SQLite is because of the application-defined rank SQL function not working:

synapse/synapse/storage/engines/sqlite.py

Line 112 in 08c56c3

db_conn.create_function("rank", 1, _rank)

synapse/synapse/storage/engines/sqlite.py

Lines 190 to 204 in 08c56c3

    
           def _rank(raw_match_info: bytes) -> float: 
        
               """Handle match_info called w/default args 'pcx' - based on the example rank 
        
               function http://sqlite.org/fts3.html#appendix_a 
        
               """ 
        
               match_info = _parse_match_info(raw_match_info) 
        
               score = 0.0 
        
               p, c = match_info[:2] 
        
               for phrase_num in range(p): 
        
                   phrase_info_idx = 2 + (phrase_num * c * 3) 
        
                   for col_num in range(c): 
        
                       col_idx = phrase_info_idx + (col_num * 3) 
        
                       x1, x2 = match_info[col_idx : col_idx + 2] 
        
                       if x1 > 0: 
        
                           score += float(x1) / x2 
        
               return score

For whatever reason, when queries call that function (as is done by search_rooms in synapse/storage/databases/main/search.py), it appears as if the function was never called. I say this after having added some log lines to be printed in the _rank implementation, which appear when using a non-distroless Synapse image, but not when using a distroless one.

sandhose · 2025-03-05T07:38:33Z

Sadly, this might be because of how sqlite is compiled in the builds we're grabbing…

That particular call uses matchinfo function and MATCH keyword from the FTS3 extension. That one isn't enabled by default:

it's explicitly enabled on the build you find on Ubuntu for example: https://git.launchpad.net/ubuntu/+source/sqlite3/tree/debian/rules?h=applied/ubuntu/noble-devel#n45
it's not enabled on the build done by the python-build-standalone project: https://github.com/astral-sh/python-build-standalone/blob/2a9167e8098ef74f2752eb59636182d0d967c5e9/cpython-unix/build-sqlite.sh#L25

Edit: nevermind, tried executing a simple FTS query with the Python we have in this image and it works

sandhose · 2025-03-05T07:59:09Z

Alright I figured out the difference!

Using the current image:

$ docker run -it --rm --entrypoint='' matrixdotorg/synapse /usr/local/bin/python3
Python 3.12.9 (main, Feb 25 2025, 08:58:51) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> con = sqlite3.connect(":memory:")
>>> con.execute("CREATE VIRTUAL TABLE docs USING fts4(message);").fetchall()
[]
>>> con.execute("INSERT INTO docs VALUES ('Message number 4');").fetchall()
[]
>>> con.execute("SELECT * FROM docs WHERE docs MATCH 'Message AND 4';").fetchall()
[('Message number 4',)]

With the one from this branch:

$ docker run -it --rm --entrypoint='' syn /usr/local/bin/python3
Python 3.12.9 (main, Feb 12 2025, 14:50:54) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> con = sqlite3.connect(":memory:")
>>> con.execute("CREATE VIRTUAL TABLE docs USING fts4(message);").fetchall()
[]
>>> con.execute("INSERT INTO docs VALUES ('Message number 4');").fetchall()
[]
>>> con.execute("SELECT * FROM docs WHERE docs MATCH 'Message AND 4';").fetchall()
[]

Which is ultimately what the test is doing: https://github.com/matrix-org/complement/blob/81c0d9ad96823e92cef2a9d146a23bf32d186126/tests/csapi/apidoc_search_test.go#L93

sandhose · 2025-03-05T08:29:58Z

I've opened an issue on their side: astral-sh/python-build-standalone#550

The Python distribution from uv comes with the SQLite extension compiled without the `ENABLE_FTS3_PARENTHESIS` flag, which Synapse requires for its full-text search queries. So, revert to using the Python from the "python-slim" image, which comes with SQLite compiled with that flag.

Downstream container images that use the Synapse image as a base will break due to the change to distroless.

Exclude pip from the runtime image, and don't copy unused binaries & unused/empty directories. This also requires to go back to using uv in a build stage to install supervisor in the workers image.

as it is useless without a shell anyways

Also add a comment explaining why it's used, and install it

Add back zlib1g, and add libs that uv's python builds statically

Don't append an arch to package names that already have an arch

COPYing a symlink resolves the link, so explicitly make links instead

Instead of manually creating links to the python binary in the final runtime image, copy them from a builder image.

Do this instead of appending package names with ":{arch}", which fails when downloading architecture-independent packages for a non-host arch.

AndrewFerr · 2025-03-07T20:57:24Z

The last round of commits should wrap this up, barring further review.

sandhose · 2025-03-18T08:48:45Z

@AndrewFerr ftr, I've pushed a patch to python-build-standalone to enable the flag we needed, and it just got merged, so if you want to switch back to the uv-based python install, you could :)

Now that uv's prebuilt Python uses the `ENABLE_FTS3_PARENTHESIS` flag for its SQLite extension, it is once again usable for Synapse (see astral-sh/python-build-standalone#550). Still skip copying unused binaries & directories into the runtime image.

reivilibre

I had some comments but not sure all of them still apply because the history is a little windy and I think some of them got handled since ...

docker/complement/Dockerfile

reivilibre · 2025-03-18T16:22:15Z

docker/Dockerfile-workers

+    | grep '^\w' > /tmp/pkg-list && \
+  mkdir -p /tmp/debs && \
+  cat /tmp/pkg-list && \
+  cd /tmp/debs && \
+  xargs apt-get download </tmp/pkg-list
+
+# Extract the debs for each architecture
+RUN \
+    mkdir -p /install/var/lib/dpkg/status.d/ && \
+    for deb in /tmp/debs/*.deb; do \
+      package_name=$(dpkg-deb -I ${deb} | awk '/^ Package: .*$/ {print $2}'); \
+      echo "Extracting: ${package_name}"; \
+      dpkg --ctrl-tarfile $deb | tar -Ox ./control > /install/var/lib/dpkg/status.d/${package_name}; \
+      dpkg --extract $deb /install; \
+    done;


this is not super elegant. Not sure if we can do better though.

FWIW the same strategy is done in stage 2 of docker/Dockerfile.

scripts-dev/complement.sh

reivilibre · 2025-03-18T16:29:19Z

docker/prefix-log

+prefixer = ['mawk', '-W', 'interactive', f'{{print "{os.environ['SUPERVISOR_PROCESS_NAME']} | "$0; fflush() }}']
+
+r_out, w_out = os.pipe()
+if os.fork() == 0:
+    os.close(w_out)
+    os.dup2(r_out, sys.stdin.fileno())
+    os.execvp(prefixer[0], prefixer)
+os.close(r_out)
+
+r_err, w_err = os.pipe()
+if os.fork() == 0:
+    os.close(w_out)
+    os.close(w_err)
+    os.dup2(r_err, sys.stdin.fileno())
+    os.dup2(sys.stderr.fileno(), sys.stdout.fileno())
+    os.execvp(prefixer[0], prefixer)
+os.close(r_err)
+
+os.dup2(w_out, sys.stdout.fileno())
+os.dup2(w_err, sys.stderr.fileno())
+os.execvp(sys.argv[1], sys.argv[1:])


I have to say I'd prefer to use the shell for this, as this has really suffered in clarity.

I am also not confident in vouching for its correctness here.

To help with clarity, I'll add some comments to this.

It's a lot more code than a sh script, but it really just boils down to redirecting output streams.

These changes help with local builds, but aren't needed for distroless

AndrewFerr · 2025-03-19T19:57:15Z

This PR has grown to be quite large, and after having played with it some more, I realize now that the impacts of switching to distroless could be more than originally anticipated (like breaking downstream image builds & environments that expect more tools to be installed). So, closing this until it's potentially revisited later.

There are some changes in here that aren't strictly related to distroless, though, which I intend to split into other PRs.

sandhose and others added 16 commits February 19, 2025 12:11

Produce a docker image based on distroless

ab3f4dc

Newsfile

799bd77

TEMP: build the docker image in the PR

b9fab43

Fix complement build

47e4c6e

Adapt the complement scripts to not depend on bash & curl

c4746a3

Use the same strategy to install dependencies in complement

f2188cf

Explicitely use mawk & fflush in the prefix-log script

0535f63

Merge branch 'develop' into quenting/docker/distroless

f08f8f7

start_for_complement.sh: fix Shellcheck SC2145

d27423b

start_for_complement.sh: fix Shellcheck SC2113

0211df2

start_for_complement.sh: fix Shellcheck SC3010

3c2246a

Ensure local Synapse images are used

6441728

Build Docker images without a shell

3c4f7a6

Remove local tags for unpushed images

bfbbd5d

as there is no risk of pulling these from a public registry

Minor tweaks

a174b1b

Pass args to start_for_complement.sh

1d8ded2

AndrewFerr requested a review from a team as a code owner March 3, 2025 21:21

Rename changelog file

9ffa230

AndrewFerr force-pushed the docker/distroless branch from 5f38cfe to 9ffa230 Compare March 3, 2025 21:22

Lint

fd0aa3c

AndrewFerr commented Mar 4, 2025

View reviewed changes

AndrewFerr added 3 commits March 4, 2025 08:24

Reduce diff of Dockerfile-workers

14cb560

Revert "TEMP: build the docker image in the PR"

13d280b

This reverts commit b9fab43.

Remove misplaced comment

1295da3

AndrewFerr added 2 commits March 5, 2025 09:06

Add breaking change notice to changelog

37631f4

Downstream container images that use the Synapse image as a base will break due to the change to distroless.

AndrewFerr added 13 commits March 5, 2025 13:43

Copy less from python-slim into runtime image

e2efaee

Exclude pip from the runtime image, and don't copy unused binaries & unused/empty directories. This also requires to go back to using uv in a build stage to install supervisor in the workers image.

Don't copy python-config into runtime image

f444e4d

as it is useless without a shell anyways

Copy only postgresql libs into complement image

699656d

Go back to using wget instead of curl

fe50786

Also add a comment explaining why it's used, and install it

Add missing dynamic libraries

955c56d

Add back zlib1g, and add libs that uv's python builds statically

Lint

428e769

Add libreadline8 and fix apt script

7acad52

Don't append an arch to package names that already have an arch

Don't create multiple copies of python

b724beb

COPYing a symlink resolves the link, so explicitly make links instead

Remove unused ARG

ea1c859

Merge branch 'develop' into af/docker/distroless

9e10a56

Copy python symlinks as they are

38dd521

Instead of manually creating links to the python binary in the final runtime image, copy them from a builder image.

Don't rely on PYTHON_VERSION to copy binaries

d2ac174

Set package download arch via APT config option

31938f5

Do this instead of appending package names with ":{arch}", which fails when downloading architecture-independent packages for a non-host arch.

AndrewFerr added 2 commits March 18, 2025 11:47

Merge branch 'develop' into af/docker/distroless

547969d

reivilibre requested changes Mar 18, 2025

View reviewed changes

Revert changes to complement.sh

5d9fe30

These changes help with local builds, but aren't needed for distroless

AndrewFerr closed this Mar 19, 2025

AndrewFerr deleted the docker/distroless branch March 19, 2025 20:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use a distroless based runtime image #18208

Use a distroless based runtime image #18208

Uh oh!

AndrewFerr commented Mar 3, 2025 •

edited

Loading

Uh oh!

AndrewFerr Mar 4, 2025

Uh oh!

AndrewFerr commented Mar 4, 2025

Uh oh!

sandhose commented Mar 5, 2025 •

edited

Loading

Uh oh!

sandhose commented Mar 5, 2025

Uh oh!

sandhose commented Mar 5, 2025

Uh oh!

AndrewFerr commented Mar 7, 2025

Uh oh!

sandhose commented Mar 18, 2025

Uh oh!

reivilibre left a comment

Uh oh!

Uh oh!

reivilibre Mar 18, 2025

Uh oh!

AndrewFerr Mar 18, 2025

Uh oh!

Uh oh!

reivilibre Mar 18, 2025

Uh oh!

AndrewFerr Mar 18, 2025

Uh oh!

AndrewFerr commented Mar 19, 2025

Uh oh!

Uh oh!

Use a distroless based runtime image #18208

Use a distroless based runtime image #18208

Uh oh!

Conversation

AndrewFerr commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

AndrewFerr Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

AndrewFerr commented Mar 4, 2025

Uh oh!

sandhose commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sandhose commented Mar 5, 2025

Uh oh!

sandhose commented Mar 5, 2025

Uh oh!

AndrewFerr commented Mar 7, 2025

Uh oh!

sandhose commented Mar 18, 2025

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reivilibre Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

AndrewFerr Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reivilibre Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

AndrewFerr Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

AndrewFerr commented Mar 19, 2025

Uh oh!

Uh oh!

AndrewFerr commented Mar 3, 2025 •

edited

Loading

sandhose commented Mar 5, 2025 •

edited

Loading