[Feat]: HTTP Timeout from 3984 cause image upload fails

[dragetd]

Is your feature request related to a problem? Please describe.

This PR introduced HTTP Timeouts: https://github.com/project-zot/zot/pull/3984/changes

I did not bisect it down to this, but I know since using this version (.17), some of our very large images fail to upload. I understand the timeouts are a good thing, but debugging was a bit difficult. I eventually found the go exception about TCP timeout. But I wonder if it was possible to have a clearer error message. Or loosed the timeouts a little bit.

It was a ~3gb OCI image uploaded with about 10mb/s.

Ot have a clearer note on the .17 Changelog that hints at such potential problems.

If you disagree, feel free to close this issue!

I will try to reconfigure the timeouts and update zot again, tho I have to set up a quick test environment some time next weeks since I do not want to test it on our production zot. Will report if my analysis was completely wrong.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[rchincha]

| Or loosed the timeouts a little bit.

@dragetd these timeouts are configurable, so in your case, you can choose much larger values.

[ieugen]

Hi, I've had some issues with large images and Zot behind Traefik proxy.
Playing with timeouts on Traefik side was not enough.
I had 10m timeouts on both Traefik and zot.
We hava a 10GB image (LLM models).

My solution was to use traefik as a TCP proxy, not HTTP proxy and use TLS on Zot side (via acme.sh) and use only the timeouts from Zot.

Setting the timeouts on zot with Traefik in HTTP proxy mode did not work.
Traefik had 10m timeouts and increased maxRequestBytes but, was spilling to disk and disk was getting full since image push was in a retry loop.

I believe there might be an issue in how Zot / Traefik work together with regards to timeouts and buffering.
I did not try with other proxy engines to see if this is Traefik only.

[datadot]

I'd like to pick this up.

The timeout from #3984 also affects the download side, not just uploads. Blob GETs stream through WriteDataFromReader, which copies the blob in 10MB chunks but never extends the connection's write deadline, so the whole transfer is bounded by http.writeTimeout. A large or slow download that runs past the timeout gets cut off mid-stream:

failed to copy data into http response: write tcp ...: i/o timeout   (pkg/api/routes.go)

The client then sees a truncated body / unexpected EOF. The same thing shows up as an occasional failure in the GC-on-S3 stress job, where a blob read under load tips past the deadline.

http.Server.WriteTimeout and ReadTimeout are deadlines on the whole request/response, set when it starts, not idle timeouts, so streaming a large body can hit them while data is still flowing. Setting them to 0 avoids it but drops the protection #3984 added.

Proposed fix: reset the deadline per chunk with http.NewResponseController on both the blob-receive (upload) and blob-serve (download) paths, so a transfer that keeps making progress isn't cut off while a genuinely stalled one still times out. That keeps #3984's protection and covers both directions. I'll extend TestWriteDataFromReader to cover the serve path.