Log display on very large logs from Dagu takes minutes to load in browser without clear indication it will work

[zph]

I have a use case where the logs can be very large (example is 216MB on frontend once it fully loads).

The user experience is that the page fails to load while trying and loading the data. You can see it loading in network tab gradually.

Once the 216MB log is delivered clientside, there's a processing phase before it's rendered.

Options:

1. Improve messaging/loading screen to show progress of load (simpler?)
2. Improve performance

• Pagination
• Compression (example using chi: https://stackoverflow.com/questions/57740979/how-enable-gzip-compression-middleware-in-go-chi#60943336)
• Lazy loading
• Ensure polling ends or that polling is very light operation which only compares SHAs vs returning full log

I haven't looked into a solution much yet beyond discovering the issue. If I discover an approach that doesn't require major changes I'll submit as a PR.

Version: around 9ef8153 from about ~4wk ago

As a secondary factor, I see the frontend continuing to poll and each polling appears to fetch full content of log which works in small cases but not large logs.

[yottahmd]

Thank you for raising the issue. I agree that it is a problem. It appears that Airflow seems displays only a portion of log when the the full-log isn't necessary. This approach could serve as a potential solution to the problem. Additionally, I think that we should consider separating the API for polling and downloading the log file to address the the secondary factor.
https://stackoverflow.com/questions/74832250/airflow-logs-break-long-logs-into-smaller-multiple-files

[ghansham]

This has been implemented I suppose. It is closeable