Add Agent-level Tolerations setting

[scottshotgg]

This PR adds the ability to specify tolerations at the agent level similar to annotations, labels, node selector, etc. I know this ability is available from within a pipeline but as an administrator I should be able to set this at a global level as well as prevent any pipeline from overriding this.
In similar vein to how annotations are done, I have introduced two new env variables and updated the documentation to match:

• WOODPECKER_BACKEND_K8S_POD_TOLERATIONS
• WOODPECKER_BACKEND_K8S_POD_TOLERATIONS_ALLOW_FROM_STEP

I initially started this feature because I have a "runner" node in my k8s cluster specifically for jobs like this (crons, ci/cd, etc) so that the core compute on the worker nodes isn't effected by random variadic scaling patterns. I felt that the ability to define these tolerations and node selectors at the individual step level was really nice and could come in handy, but the inability to define them at a global level was necessary; I needed a simple way to say that every pipeline should be scheduled to a particular node, which WOODPECKER_BACKEND_K8S_POD_NODE_SELECTOR was good for - however it is a bit underwhelming because you are then left with a pod that has been told to schedule onto a node for which it has no toleration for the nodes taint unless you include this in every pipeline, every step, etc.

I think this could be a useful feature that others could benefit from. Overall I found it simple to implement - thanks for the easy code to work in, and thanks for Woodpecker!

[qwerty287]

Hello and sorry that it took a long for us to answer.
This looks good with my very limited k8s knowledge, but you added the docs also for the old versions. Only the files in docs/docs should get changed.

[scottshotgg]

Ahh okay, so don't change the versioned docs - that makes sense. I'll fix that up

[pat-s]

With proper rules applied, this might be a solution to #3070.

[scottshotgg]

@pat-s
I didn't thoroughly read through your issue, but if your goal is to have a builder for arm based images hosted on an arm node, and a corollary one on amd - it could definitely be a solution for that.

If you need/want an example or two I can try to help you out.

[scottshotgg]

@qwerty287
I've restored the docs/versioned_docs folder back to main. I would really like to merge this in soon if that's possible. Right now I am running this branch but would like to run the official image, etc

[qwerty287]

This looks good to me, but could you add the backend options parameter to the schema? https://github.com/woodpecker-ci/woodpecker/blob/main/pipeline/frontend/yaml/linter/schema/schema.json

And I'd like to have a review from someone else, I don't have much knowledge of k8s. @woodpecker-ci/maintainers

[scottshotgg]

@qwerty287
Done!

[pat-s]

I didn't thoroughly read through your issue, but if your goal is to have a builder for arm based images hosted on an arm node, and a corollary one on amd - it could definitely be a solution for that.

No, that's not the scope of the linked issue at all.

[pat-s]

As this is a quite dynamic change, especially to the option to deny/allow override on the user-level, this needs tests. @scottshotgg can you add some that test (ideally all) behaviors of this PR?

[scottshotgg]

@pat-s
Do you have any tests that you could point me to? I'd preferably like to replicate the tests for annotations since this feature is 99% the same as that one with the exception of the default value

[qwerty287]

I guess this test is the one you're looking for:

woodpecker/pipeline/backend/kubernetes/pod_test.go

Line 198 in 5de2d26

func TestFullPod(t *testing.T) {

[xoxys]

LGTM after tests were added.

[qwerty287]

@scottshotgg do you plan to add the tests?

[scottshotgg]

I will try to work on the tests at some point

[marcusramberg]

I'm interested in getting this merged. Here's a patch to add tests for tolerations (as well as fixing a failing existing test)

https://gist.github.com/marcusramberg/c18a22a191efba8810151c1d2d7fbb34

[scottshotgg]

Thank you @marcusramberg for implementing those tests - I've been pretty busy as of late.

@xoxys @qwerty287
Are we g2g now?

[xoxys]

Thank you both.

[woodpecker-bot]

Surge PR preview deployment was removed

[xoxys]

@scottshotgg @marcusramberg Tests failed, can you take a look?