Image building fails with latest Paketo base builder and additional buildpacks configured

[backjo]

Reproducible example: https://github.com/backjo/broken-buildpack-example/blob/master/build.gradle.kts#L34 (see associated GitHub Action for failure)

Earlier discussion: https://github.com/orgs/paketo-buildpacks/discussions/29#discussioncomment-2868631

The bootBuildImage task is failing after a new base builder was released. It fails with an obscure error, generated in DockerApi.java "Invalid response received when loading image".

After some further investigation, it appears that the archive that the Spring plugin is sending to the Docker Engine API has too many layers in it, resulting in the following error from the Docker Engine API.

The main change that we've observed was the increase in the Docker layer count of the base image from paketobuildpacks/builder:0.2.69-base -> paketobuildpacks/builder:0.2.70-base, which added two new layers to the base image. This appears to cause the gradle plugin implementation here to start hitting the max layer limit (127).

{"errorDetail":{"message":"max depth exceeded"},"error":"max depth exceeded"}

[scottfrederick]

Thanks very much for the detailed description, example, and link to the discussion. I've confirmed that this is indeed related to the total count of layers in the builder image and the specified buildpacks. With paketobuildpacks/builder:0.2.69-base the total combined layer count for these images was just under the 127 layer limit. The two additional layers in paketobuildpacks/builder:0.2.70-base pushed it over the limit.

I tested the pack CLI with similar parameters. pack ends up with a few less layers than the Boot plugins, so it does not fail with the same combination of builder and buildpacks. I was able to make pack fail in a similar manner by adding a few more buildpacks. I've started a discussion with the Paketo team to see what our options are for dealing with this.

In the meantime, if what you need is to use an alternate JRE/JDK then you can work around this limitation.

For example, with Gradle configuration like this:

tasks.named("bootBuildImage") {
  buildpacks = [
    "gcr.io/paketo-buildpacks/amazon-corretto:latest", 
    "gcr.io/paketo-buildpacks/java:latest"
  ]
}

or Maven configuration like this:

<configuration>
  <image>
    <buildpacks>
       <buildpack>gcr.io/paketo-buildpacks/amazon-corretto:latest</buildpack>
       <buildpack>gcr.io/paketo-buildpacks/java:latest</buildpack>
    </buildpacks>
  </image>
</configuration>

you can replace the reference to the Java buildpack image gcr.io/paketo-buildpacks/java:latest with urn:cnb:builder:paketo-buildpacks/java. This uses the Java buildpack that already exists in the builder image instead of pulling in a Java buildpack image and the 25 or so layers that it contains.

If you need more control over the buildpacks and versions used, you can create a custom builder as described in the CNB documentation.

[backjo]

@scottfrederick Appreciate the response and detail here - I think we've got quite a few workaround options here now and a much better understanding of what is going on!

I wonder if it makes sense longer term for the Spring buildpack support to default to a different builder image - maybe a new builder based on gcr.io/paketo-buildpacks/builder:buildpackless-base with just the Java buildpack included. It seems like there's a lot of cruft in the base builder around supporting other languages/toolchains that is not really relevant to the Spring Boot stack.

One other improvement it's probably worth making here is surfacing the Docker daemon error back to the build process in a more actionable way instead of swallowing it - the error message around Invalid response received when loading image is fairly confusing and not really helpful for resolution.

[scottfrederick]

@backjo Thanks for mentioning the buildpackless-base version of the Paketo builder. That's another good work-around. With the example configurations above, you can use the Java buildpack image reference and change the builder to gcr.io/paketo-buildpacks/builder:buildpackless-base as shown in the Gradle and Maven docs.

I wonder if it makes sense longer term for the Spring buildpack support to default to a different builder image - maybe a new builder based on gcr.io/paketo-buildpacks/builder:buildpackless-base with just the Java buildpack included.

We can consider that, and explore it with the Paketo team.

One other improvement it's probably worth making here is surfacing the Docker daemon error back to the build process

I agree completely. The lack of detail made it much harder to get to the bottom of this than it should have been. I'm looking into that now, and have created #31243 to track it.

[heesuk-ahn]

Thank you for yours contribution. :)

I followed your advice to keep the builder up to date and use cnb ​​java and build the image successfully.

bootBuildImage {
    buildpacks = ["gcr.io/paketo-buildpacks/adoptium:latest", "urn:cnb:builder:paketo-buildpacks/java"]
}

If some developers use a different JDK from paketo while still using the default base builder, they will have this problem.
🤔

It looks like you might need a guide document for this. 🤔

[heesuk-ahn]

@backjo

One thing I am curious about is where did you check the error log below? I'm going to check it out too. :)

{"errorDetail":{"message":"max depth exceeded"},"error":"max depth exceeded"}

[backjo]

@heesuk-ahn I ended up running the Integration Tests for Buildpack support and put a breakpoint and extra logging around the interaction with the docker-engine API.

[TomBeckett]

@scottfrederick. You were right in the Slack thread.

I've updated our POM as you mentioned above.

This seems to work for me:

 <plugin>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-maven-plugin</artifactId>
     <configuration>
         <mainClass>app.xxxxxxx.api.web.ApiApplication</mainClass>
         <image>
             <name>${image}</name>
             <env>
                 <JAVA_TOOL_OPTIONS>-Xms2048m -Xmx2048m</JAVA_TOOL_OPTIONS>
             </env>
             <buildpacks>
                 <buildpack>gcr.io/paketo-buildpacks/amazon-corretto:latest</buildpack>
                 <buildpack>urn:cnb:builder:paketo-buildpacks/java</buildpack>
             </buildpacks>
         </image>
     </configuration>
 </plugin>

[scottfrederick]

Spring Boot can't do anything about the number of layers in a CNB builder and buildpacks, or maximum number of layers allowed in an image.

One thing that the pack CLI does that Spring Boot does not do is to check the layers contained in any provided buildpacks against the layers in the builder, and skip adding any buildpack layers that already exist in the builder. This would help in the case where a buildpack is specified and the buildpack is exactly the same as one bundled in the builder. It would not help when buildpacks are not in the builder - in that case one of the work-arounds above is required.

Tagging this for team attention to discuss whether this layer de-duplication should be done in maintenance releases or only in the next major release.

[wilkinsona]

I think we should consider it a bug that we don't match pack's behaviour. Unless it's too risky, I think we should fix it in 2.6.x.

[philwebb]

+1 for treating as a bug