Docker image builds hang on apt-get update

[zanieb]

I'm seeing extremely slow container builds on Linux that hang at 100% CPU during apt-get update. I see very little network IO, disk IO, and memory consumption. I've let it run for >30 minutes to no avail.

Specifically, it looks like this is occurring for the image-gcc, image-xcb, and image-build targets.

I can't figure out how to get verbose output from apt here, any recommendations would be helpful I've been spinning my wheels on it for a while.

Is it possible I'm misinterpreting the output?

e.g.

❯ ./build-linux.py
...
image-gcc> Step 9/10 : RUN apt-get update
image-xcb> ---> Running in 64f6de138eb1
image-build> ---> Running in 3eb38e067d05
image-gcc> ---> Running in 2e0cd76e45bc

❯ docker stats
CONTAINER ID   NAME                 CPU %     MEM USAGE / LIMIT     MEM %     NET I/O       BLOCK I/O       PIDS
64f6de138eb1   gifted_khayyam       99.94%    1.145MiB / 31.25GiB   0.00%     18.6kB / 0B   0B / 0B         3
2e0cd76e45bc   infallible_carver    100.15%   1.188MiB / 31.25GiB   0.00%     18.6kB / 0B   0B / 0B         3
3eb38e067d05   flamboyant_hellman   100.14%   1.121MiB / 31.25GiB   0.00%     18.6kB / 0B   0B / 0B         3

❯ docker inspect 2e0cd76e45bc
[
    {
        "Id": "2e0cd76e45bce897a34c096651987a0e373e06ede8e7922fac2cbe0d31d6af31",
        "Created": "2024-03-20T17:52:31.490588566Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "apt-get update"
        ],
        ...

❯ docker exec -it 2e0cd76e45bc top


top - 17:57:24 up  1:55,  0 users,  load average: 3.21, 3.33, 3.15
Tasks:   4 total,   2 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.5 us, 12.3 sy,  0.0 ni, 80.0 id,  0.0 wa,  0.1 hi,  0.1 si,  0.0 st
KiB Mem:  32771652 total, 16323288 used, 16448364 free,  1327952 buffers
KiB Swap:        0 total,        0 used,        0 free.  6118300 cached Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                
      8 root      20   0   29100    656    384 R 100.0  0.0   4:52.27 apt-get                                                                                                                                                                
      1 root      20   0    4344   1280   1280 S   0.0  0.0   0:00.01 sh                                                                                                                                                                     
      7 root      20   0   29104   4352   4096 S   0.0  0.0   0:00.00 apt-get                                                                                                                                                                
     15 root      20   0   21980   2432   2048 R   0.0  0.0   0:00.01 top                                                                                                                                                                    

I can docker build a derived Dockerfile like

# Debian Jessie.
FROM debian@sha256:32ad5050caffb2c7e969dac873bce2c370015c2256ff984b70c1c08b3a2816a0
MAINTAINER Gregory Szorc <[email protected]>

RUN groupadd -g 1000 build && \
    useradd -u 1000 -g 1000 -d /build -s /bin/bash -m build && \
    mkdir /tools && \
    chown -R build:build /build /tools

ENV HOME=/build \
    SHELL=/bin/bash \
    USER=build \
    LOGNAME=build \
    HOSTNAME=builder \
    DEBIAN_FRONTEND=noninteractive

CMD ["/bin/bash", "--login"]
WORKDIR '/build'

# Jessie's signing keys expired in late 2022. So need to add [trusted=yes] to force trust.
# Jessie stopped publishing snapshots in March 2023.
RUN for s in debian_jessie debian_jessie-updates debian-security_jessie/updates; do \
      echo "deb [trusted=yes] http://snapshot.debian.org/archive/${s%_*}/20230322T152120Z/ ${s#*_} main"; \
    done > /etc/apt/sources.list && \
    ( \
      echo 'APT::Get::Assume-Yes "true";'; \
      echo 'APT::Install-Recommends "false";'; \
      echo 'Acquire::Check-Valid-Until "false";'; \
      echo 'Acquire::Retries "5";'; \
    ) > /etc/apt/apt.conf.d/99cpython-portable

RUN ( echo 'amd64'; \
      echo 'i386'; \
    ) > /var/lib/dpkg/arch

RUN apt-get update

RUN apt-get install \
      autoconf \
      automake \
      bison \
      build-essential \
      gawk \
      gcc \
      gcc-multilib \
      libtool \
      make \
      tar \
      texinfo \
      xz-utils \
      unzip

with no issues.

[zanieb]

I've tried things like...

❯ gd main -- base.Dockerfile
diff --git a/cpython-unix/base.Dockerfile b/cpython-unix/base.Dockerfile
index 76811a5..0401a9a 100644
--- a/cpython-unix/base.Dockerfile
+++ b/cpython-unix/base.Dockerfile
@@ -22,7 +22,8 @@ WORKDIR '/build'
 RUN for s in debian_jessie debian_jessie-updates debian-security_jessie/updates; do \
       echo "deb [trusted=yes] http://snapshot.debian.org/archive/${s%_*}/20230322T152120Z/ ${s#*_} main"; \
     done > /etc/apt/sources.list && \
-    ( echo 'quiet "true";'; \
+    ( echo 'Debug::Acquire::http "true";'; \
+      echo 'Debug::pkgAcquire::Worker "true";'; \
       echo 'APT::Get::Assume-Yes "true";'; \
       echo 'APT::Install-Recommends "false";'; \
       echo 'Acquire::Check-Valid-Until "false";'; \
@@ -33,4 +34,4 @@ RUN ( echo 'amd64'; \
       echo 'i386'; \
     ) > /var/lib/dpkg/arch
 
-RUN apt-get update
+RUN apt-get update 2>&1 | tee /tmp/output.txt

to get my hands on output from apt, works great with docker build but not with the build spawned by ensure_docker_image.