[RLlib] Evaluation do-over: Make parallel evaluation to training the default behavior and deprecate async eval option.

[sven1977]

Evaluation do-over:
This PR aims at simplifying our evaluation code a little bit.

• It deprecates the enable_async_evaluation option entirely. This should be replaced by a combination of existing options, evaluation_parallel_to_training=True AND evaluation_duration="auto" AND evaluation_interval=1 (AND evaluation_force_reset_envs_before_iteration=True, which is the default any way). These settings combined should be enough to 100% replace the old async behavior in the sense:
• • that there is always an eval job running under the hood (parallel to training).
• • the eval track is fault tolerant, restarting/ignoring worker failures as configured in config.fault_tolerance()
• • the eval track does NOT report episode metrics from episodes that were run with an outdated weight set.
• • no eval worker is ever idle, even if it run through a very short episode (it'll start right away with the next one and include all finished episodes - as always - in the resorted metrics).

The PR in particular:

• Splits up the self.evaluation method into various sub-methods, depending on the logic configured: _evaluate_with_auto_duration, _evaluate_with_fixed_duration, _evaluate_on_local_worker, _evaluate_with_custom_function.
• • Some of these methods require dissemination between old stack (RolloutWorkers return batches) and new API stack (EnvRunners return lists of episodes), however, these additional if-else blocks will go away once we have moved entirely to the new stack.
• Speeds up the parallel evaluation step with auto duration by: Computing a better rollout_fragment_length dynamically depending on the estimated time it takes for the parallel training step to finish. This used to be a fixed 10 timesteps, which was causing lots of (expensive) remote calls on the eval workers.
• Deprecates the async eval option (see above for an analysis on why this should NOT lead to diminished functionality).

TODOs for future PRs:

• Unify reported metrics (this is currently quite a mess). This PR does NOT change the current behavior.

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

No longer needed, imo.

[simonsays1980]

At the end it is in the responsibility of the user to override a Trainable method or not. Erroring out is too much of a consequence, agreed. We could leave it as a warning, but better imo is to refer more explicitly in the documentation to the tune.trainable.Trainable as the base class - so, if a user wants to use tune she should not override such methods.

[sven1977]

This was a bug, both in code and docs. Can also be 0 to cause NO evaluation.

[sven1977]

Slight examples folder cleanup. Will have to do more of these :)

[sven1977]

Same script as before, just moved here and cleaned up a little.