[dashboard] Replace nginx with caddy

[aledbf]

Reasons:

• Using nginx for a static server is overkill
• Caddy is only one static binary without external dependencies
• Configuration is simpler

[aledbf]

/werft run

👎 cannot start job - please talk to whoever's in charge of your Werft installation

[aledbf]

/werft run

👍 started the job as gitpod-build-aledbf-dashboardcaddy.3

[AlexTugarev]

that would only mean "the http service is running", the previous check means "the dashboard is served". in-between there might be error cases.

@geropl, do you know why we started to use actual resources from the dashboard to be tested for in the readiness probe?

[csweichel]

I can imagine several scenarios why that was:

• convenience: by testing well known dashboard resources, we don't have to think about other paths that we might want to use
• previous breakage: we may have had some misconfiguration at some point and then moved to checking a dashboard resource

Either way, once caddy is running it would serve the dashboard. It makes sense to me to use a readiness endpoint of caddy here.

[AlexTugarev]

/werft run

👍 started the job as gitpod-build-aledbf-dashboardcaddy.12

[AlexTugarev]

That worked nicely for me!
I checked that caching works as it used to and known redirects are still in place.
Thanks 🙏🏻

[geropl]

@aledbf @AlexTugarev @csweichel Did anybody verify that caddy emits "ETag" headers based on content-hash of the static files in this configuration? Something we rely upon here.

[geropl]

@aledbf Until now caching was the responsibility of the "front-facing" proxy (cmp. here).

This looks fine generally, but we should double check the results, and ideally either:
a) cross-link with comments, or better
b) merge into one place.

[AlexTugarev]

@geropl, ups. I've seen the ETags, but didn't check for the implementation. It's indeed based on file size and date.
cf.
https://github.com/caddyserver/caddy/blob/3f6283b385642c56f34b479d1275095379b062d3/modules/caddyhttp/fileserver/staticfiles.go#L545