CLI Flags, JSON configuration files, and default values

[lemonzi]

The current setup with some parameters specified as CLI flags and others in the JSON file is not optimal. There are two competing needs:

• We would like to group related parameters in a JSON file so that it's easy to share model configurations with the community. Attaching the JSON file to a saved checkpoint makes it trivial to replicate experiments and test networks that others with more resources have trained, thus helping democratize the platform.
• Changing individual parameters during development should be very, very easy.

We should find a way to accommodate both. I propose:

• All parameters are defined as flags.
• A list of JSON files are read sequentially and merged, and finally the supplied flags are merged with the combined JSON dictionary for manual overrides.
• We would have a JSON file with model parameters, a JSON file with training parameters (learning rate, etc.), and users could add another one with their specific overrides.

It still doesn't feel optimal, though. Any suggestions?

[lemonzi]

As mentioned in #80, whatever we do should additionally enable us to "unit-test" (or end-to-end test, at this point) the train and generation scripts.

[jyegerlehner]

@lemonzi

All parameters are defined as flags.

I like this (I'm assuming "flag" is synonymous with a command-line argument). I observe that with json, there's no place to specify what the possible allowed values are. Whereas with command-line arguments, one can specify an entire description of what it does and the allowed values are, which is useful.

It still doesn't feel optimal, though. Any suggestions?

Is there a need for a list of json files? A list suggests there's a need of a plurality of sources of information, and a single .json file already allows for a hierarchy of objects within it. Seems it could be a bit simpler if we used the hierarchy of json files to simplify the structure and allow a single json file to specify all command-line arguments.

Until we converge on an "ultimate" solution, I'll add any configurable option as a command-line argument