Refactor evaluation script

[maxjeblick]

Feature

Improve the evaluation script code.

Motivation

Over time, the script has been updated to handle several special cases. It is now in a state where it is a bit hard to understand; thus it is a good time to refactor it.

Relates to #87

[JoelSeniorLiang]

It's useful to add some features in the evaluation script. For example, if someone want to compare different presses, it should prefill once and compress with different presses and ratios then decode in one batch. This way can save a lot of time.

[giulio98]

Probably we could disaggregate the prefill stage from the decode stage in the API, for batching we could use packed tensor format?

huggingface/transformers#39194