Entrypoint base64 chunking makes pipeline commands fail randomly

[jopvanraaij]

Component

agent

Describe the bug

The pipeline fails, because the list of commands is cut at a random place.

I estimate it happens about 10 to 20% of my runs, which is really annoying. I'm using images like amazoncorretto:24.0.2-alpine3.22 to run a list of 10 to 20 commands (gradle build commands). Some of the pipelines report /bin/sh: syntax error: unexpected end of file and some report an error the gradle task cannot be executed because it cannot find the specific task, but multiple. Upon close inspection, it turns out is only tries to execute part of the line, like ./gradlew com instead of ./gradlew compileJava. The com maps to multiple tasks, so Gradle is not sure what to execute.
Running the same job again, clicking rerun in Woodpecker, often results in a passing build.

In an attempt to debug and fix it myself, I tried to print out the complete script (list of commands) at the beginning of the pipeline, which seems to be fine. Also for the times when the pipeline failed. I updated the entrypoint to put the script into a file first, before executing, instead of pipeing it directly to /bin/sh. I use:

entrypoint: ["/bin/sh", "-c", "echo $CI_SCRIPT | base64 -d > script.sh && pwd && ls -alh && cat script.sh && /bin/sh -xe script.sh"]

I also added some debug items to see files, current directory and the contents of the script. Up to now, it seems the update of the entrypoint fixes the issue, but as used to fail randomly, I might just have been lucky up to now.

Key difference between the original entrypoint (listed here https://woodpecker-ci.org/docs/usage/workflow-syntax#entrypoint), ["/bin/sh", "-c", "echo $CI_SCRIPT | base64 -d | /bin/sh -e"], is the replacement of the pipe with a pass to the file script.sh before execution. Depending on the image used to run the pipeline, the base64 -d might do some chunking and put a newline at a random point in the script (sometimes), which triggers execution of only a part of the command, instead of the complete command.

Steps to reproduce

Hard to reproduce, as it is happens only some of the runs.

Expected behavior

A passing or failing build 100% of the time.

System Info

Running Woodpecker-ci 3.9.0 (also had this behavior using 3.8.0, was hoping it was fixed in 3.9.0, but it isn't).

Additional context

No response

Validations

• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/versions]

[qwerty287]

@jopvanraaij sorry to not come back to you for a long time.

Has your solution with a custom entrypoint fixed the issue? If yes, we can integrate that into the core. The only question is where we store the script as it shouldn't be in the main workspace.

[jopvanraaij]

@qwerty287 no problem, I understand everyone is busy.

Up to now, none of the builds failed because of this issue, so the 'fix' works for me. It is a pity all my woodpecker scripts now need this line to work 100% of the time. I don't understand why other people don't suffer from the same issue. Especially when running nightly Renovate-bot it is rather annoying when pipelines fail randomly and mess up the renovate auto-merge.

Maybe, if the root-cause is known, it could be fixed without piping the input to a file first? My guess would be something related to gorpc. Is the data for the script passed to the agent/build-container using gorpc? And could there chunking or EOF/newline characters sometimes withing the rpc data-stream? I'm not that good/familiar with the protocol.

[qwerty287]

About RPC I'm also not an expert, but I wouldn't expect this. The encoding to base64 happens on the agent. Afaik the data is passed as JSON to the agent so the commands should be in there as a JSON array. If there something is wrong the agent would fail because it can't parse the JSON. And as it works with the file it seems to always have the full script in the container. So I'd rather expect the problem to be in the container, so likely when decoding the base64.