BUG: Progressbar does not update on streamed `stdout`

[tvwenger]

Describe the issue:

I'm running pymc in a condor HTC environment, where my models are sampled on compute nodes and the output is streamed or copied back to the host.

Without any options to checkpoint sampling with pymc, I want a way to check the status of the sampling even though I don't have access to stdout on the compute nodes.

Currently, we only pass refresh=True on the last iteration, so the only time condor streams stdout back to the host machine is after the last iteration. This makes me sad because no updates are streamed back to the host during sampling, so I have no way to satisfy my curiosity on the expected runtime.

In summary, the issue is that progressbar=True doesn't actually print the progress bar to stdout in such a way that condor can stream it until sampling is completed.

I want refresh=True here for all iterations, not just the last:

pymc/pymc/util.py

Lines 893 to 908 in 2842401

	self._progress.update(
	self.tasks[chain_idx],
	completed=draw,
	draws=draw,
	sampling_speed=speed,
	speed_unit=unit,
	**more_updates,
	)
	if is_last:
	self._progress.update(
	self.tasks[chain_idx],
	draws=draw + 1 if not self.combined_progress else draw,
	**more_updates,
	refresh=True,
	)

Reproduceable code example:

pm.sample() # on HTC

Error message:

PyMC version information:

Latest

Context for the issue:

No response

[Hashcode-Ankit]

@tvwenger @ricardoV94 is this need to be fixed?, Let me know will raise a pr for it.
Thanks

[st-bender]

Hi,
I observed that also in notebooks the progress bar does not update anymore until one of the chains is finished. I locally added refresh=True to all iterations to fix it.
PyMC v5.27.0.

Cheers.

[rawlins]

Hi, I observed that also in notebooks the progress bar does not update anymore until one of the chains is finished. I locally added refresh=True to all iterations to fix it. PyMC v5.27.0.

I can confirm that in recent Jupyter (Edit: Jupyter lab 4.5.7) + pymc 6.0.0, I'm seeing the progress bars not update consistently, and that manually adding refresh=True to nonfinal updates still fixes the behavior. The place to fix is now located in pymc/progress_bar/rich_progress.py rather than in the file mentioned in the original report.

(I'm not actually sure this is the right solution, as it's causing scrollbar flicker and probably updating too often. But it works enough for a quick solution.)

[ricardoV94]

there's an issue with jupyter notebook 7, and downgrading to 6 fixes it. What versions are you running?

[rawlins]

there's an issue with jupyter notebook 7, and downgrading to 6 fixes it. What versions are you running?

Thanks! This is in Jupyter Lab 4.5.7, which is aligned with notebook version 7.5. I don't think downgrading is an option for me, unfortunately.

[ricardoV94]

Here is the issue: Textualize/rich#3868

[rawlins]

Thanks, I've replicated this in a fresh env without pymc present, so I think you're right that it's not pymc's fault. I posted the replication conditions in the linked report in hopes to prod it a bit.

(The manual refresh=True workaround from earlier in this issue is definitely not ideal: it tends to trigger io update limits and cause output freezes/errors if sampling is going well. I haven't come up with another workaround, though.)

[ricardoV94]

Thanks, I've replicated this in a fresh env without pymc present, so I think you're right that it's not pymc's fault. I posted the replication conditions in the linked report in hopes to prod it a bit.

(The manual refresh=True workaround from earlier in this issue is definitely not ideal: it tends to trigger io update limits and cause output freezes/errors if sampling is going well. I haven't come up with another workaround, though.)

Need to gate the refresh=True on a timer. I think rich only at every 0.1s by default or smth like that