Merge branch 'master' into imitationlearning

* master:
  Add some development tips to documentation. (ray-project#1426)
  Add link to github from documentation. (ray-project#1425)
  [rllib] Update docs with api and components overview figures (ray-project#1443)
  Multiagent model using concatenated observations (ray-project#1416)
  Load evaluation configuration from checkpoint (ray-project#1392)
  [autoscaling] increase connect timeout, boto retries, and check subnet conf (ray-project#1422)
  Update wheel in autoscaler example. (ray-project#1408)
  [autoscaler] Fix ValueError: Missing required config keyavailability_zoneof type str
  [tune][minor] Fixes (ray-project#1383)
  [rllib] Expose PPO evaluator resource requirements (ray-project#1391)
  fix autoscaler test (ray-project#1411)
  [rllib] Fix incorrect documentation on how to use custom models ray-project#1405
  Added option for availability zone (ray-project#1393)
  Adding all DataFrame methods with NotImplementedErrors (ray-project#1403)
  Remove pyarrow version check. (ray-project#1394)

# Conflicts:
#	python/ray/rllib/eval.py

Author: royf

build.sh

@@ -32,7 +32,7 @@ popd
 bash "$ROOT_DIR/src/thirdparty/download_thirdparty.sh"
 bash "$ROOT_DIR/src/thirdparty/build_thirdparty.sh" $PYTHON_EXECUTABLE
 
-# Now build everything.
+# Now we build everything.
 pushd "$ROOT_DIR/python/ray/core"
   # We use these variables to set PKG_CONFIG_PATH, which is important so that
   # in cmake, pkg-config can find plasma.

doc/source/autoscaling.rst

@@ -41,7 +41,7 @@ Autoscaling
 
 Ray clusters come with a load-based auto-scaler. When cluster resource usage exceeds a configurable threshold (80% by default), new nodes will be launched up the specified ``max_workers`` limit. When nodes are idle for more than a timeout, they will be removed, down to the ``min_workers`` limit. The head node is never removed.
 
-The default idle timeout is 5 minutes. This is because in AWS there is a minimum billing charge of 5 minutes per instance, after which usage is billed by the second.
+The default idle timeout is 5 minutes. This is to prevent excessive node churn which could impact performance and increase costs (in AWS there is a minimum billing charge of 1 minute per instance, after which usage is billed by the second).
 
 Monitoring cluster status
 -------------------------

doc/source/development.rst

@@ -0,0 +1,82 @@
+Development Tips
+================
+
+If you are doing development on the Ray codebase, the following tips may be
+helpful.
+
+1. **Speeding up compilation:** Be sure to install Ray with
+
+   .. code-block:: shell
+
+     cd ray/python
+     python setup.py develop
+
+   (as opposed to ``python setup.py install``). When you do the "install"
+   version, files will be copied from the Ray directory to a directory of Python
+   packages (often something like
+   ``/home/ubuntu/anaconda3/lib/python3.6/site-packages/ray``). This means that
+   changes you make to files in the Ray directory will not have any effect.
+   However, when you run the "develop" version, no files will be copied and so
+   any changes you make to Python files will immediately take effect without
+   rerunning ``setup.py``.
+
+   If you run into **Permission Denied** errors when running ``setup.py``, you
+   can try doing ``python setup.py develop --user``. You may also need to run
+   something like ``sudo chown -R $USER /home/ubuntu/anaconda3`` (substituting
+   in the appropriate path).
+
+   If you make changes to the C++ files, you will need to recompile them.
+   However, you do not need to rerun ``setup.py``. Instead, you can recompile
+   much more quickly by doing
+
+   .. code-block:: shell
+
+     cd ray/python/ray/core
+     make -j8
+
+2. **Starting processes in a debugger:** When processes are crashing, it is
+   often useful to start them in a debugger (``gdb`` on Linux or ``lldb`` on
+   MacOS). See the latest discussion about how to do this `here`_.
+
+3. **Running tests locally:** Suppose that one of the tests (e.g.,
+   ``runtest.py``) is failing. You can run that test locally by running
+   ``python test/runtest.py``. However, doing so will run all of the tests which
+   can take a while. To run a specific test that is failing, you can do
+
+   .. code-block:: shell
+
+     cd ray
+     python test/runtest.py APITest.testKeywordArgs
+
+   When running tests, usually only the first test failure matters. A single
+   test failure often triggers the failure of subsequent tests in the same
+   script.
+
+4. **Running linter locally:** To run the Python linter on a specific file, run
+   something like ``flake8 ray/python/ray/worker.py``. You may need to first run
+   ``pip install flake8``.
+
+5. **Inspecting Redis shards by hand:** To inspect the primary Redis shard by
+   hand, you can query it with commands like the following.
+
+   .. code-block:: python
+
+     r_primary = ray.worker.global_worker.redis_client
+     r_primary.keys("*")
+
+   To inspect other Redis shards, you will need to create a new Redis client.
+   For example (assuming the relevant IP address is ``127.0.0.1`` and the
+   relevant port is ``1234``), you can do this as follows.
+
+   .. code-block:: python
+
+     import redis
+     r = redis.StrictRedis(host='127.0.0.1', port=1234)
+
+   You can find a list of the relevant IP addresses and ports by running
+
+   .. code-block:: python
+
+     r_primary.lrange('RedisShards', 0, -1)
+
+.. _`here`: https://github.com/ray-project/ray/issues/108

doc/source/index.rst

@@ -1,8 +1,18 @@
 Ray
 ===
 
+.. raw:: html
+
+  <embed>
+    <a href="https://github.com/ray-project/ray"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://assets-git-camo.f3mw1.com/365986a132ccd6a44c23a9169022c0b5c890c387/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"></a>
+  </embed>
+
 *Ray is a flexible, high-performance distributed execution framework.*
 
+View the `codebase on GitHub`_.
+
+.. _`codebase on GitHub`: https://github.com/ray-project/ray
+
 Ray comes with libraries that accelerate deep learning and reinforcement learning development:
 
 - `Ray.tune`_: Hyperparameter Optimization Framework
@@ -95,4 +105,5 @@ Example Program
    :caption: Help
 
    troubleshooting.rst
+   development.rst
    contact.rst

doc/source/rllib-api.svg

@@ -10,49 +10,23 @@ Recipe for an RLlib algorithm
 
 Here are the steps for implementing a new algorithm in RLlib:
 
-1. Define an algorithm-specific `Evaluator class <#evaluators-and-optimizers>`__ (the core of the algorithm). Evaluators encapsulate framework-specific components such as the policy and loss functions. For an example, see the `A3C Evaluator implementation <https://github.com/ray-project/ray/blob/master/python/ray/rllib/a3c/a3c_evaluator.py>`__.
+1. Define an algorithm-specific `Policy evaluator class <#policy-evaluators-and-optimizers>`__ (the core of the algorithm). Evaluators encapsulate framework-specific components such as the policy and loss functions. For an example, see the `A3C Evaluator implementation <https://github.com/ray-project/ray/blob/master/python/ray/rllib/a3c/a3c_evaluator.py>`__.
 
 
-2. Pick an appropriate `RLlib optimizer class <#evaluators-and-optimizers>`__. Optimizers manage the parallel execution of the algorithm. RLlib provides several built-in optimizers for gradient-based algorithms. Advanced algorithms may find it beneficial to implement their own optimizers.
+2. Pick an appropriate `Policy optimizer class <#policy-evaluators-and-optimizers>`__. Optimizers manage the parallel execution of the algorithm. RLlib provides several built-in optimizers for gradient-based algorithms. Advanced algorithms may find it beneficial to implement their own optimizers.
 
 
 3. Wrap the two up in an `Agent class <#agents>`__. Agents are the user-facing API of RLlib. They provide the necessary "glue" and implement accessory functionality such as statistics reporting and checkpointing.
 
 To help with implementation, RLlib provides common action distributions, preprocessors, and neural network models, found in `catalog.py <https://github.com/ray-project/ray/blob/master/python/ray/rllib/models/catalog.py>`__, which are shared by all algorithms. Note that most of these utilities are currently Tensorflow specific.
 
-Defining a custom model
------------------------
-
-Often you will want to plug in your own neural network into an existing RLlib algorithm.
-This can be easily done by defining your own `Model class <#models-and-preprocessors>`__ and registering it in the RLlib catalog, after which it will be available for use by all RLlib algorithms.
-
-An example usage of a custom model looks like this:
-
-::
-
-    from ray.rllib.models import ModelCatalog, Model
-
-    class MyModelClass(Model):
-        def _init(self, inputs, num_outputs, options):
-            layer1 = slim.fully_connected(inputs, 64, ...)
-            layer2 = slim.fully_connected(inputs, 64, ...)
-            ...
-            return layerN, layerN_minus_1
-
-    ModelCatalog.register_custom_model("my_model", MyModelClass)
-
-    alg = ppo.PPOAgent(env="CartPole-v0", config={
-        "custom_model": "my_model",
-    })
-
-
-Note that if you need to reference large data objects as part of the computation, e.g. weights, you can put them into the Ray object store with ``ray.put`` and then retrieve them from inside your model class.
+.. image:: rllib-api.svg
 
 
 The Developer API
 -----------------
 
-The following APIs are the building blocks of RLlib algorithms. Note that they are not yet considered stable.
+The following APIs are the building blocks of RLlib algorithms (also take a look at the `user components overview <rllib.html#components-user-customizable-and-internal>`__).
 
 Agents
 ~~~~~~
@@ -65,8 +39,8 @@ a common base class:
 .. autoclass:: ray.rllib.agent.Agent
     :members:
 
-Evaluators and Optimizers
-~~~~~~~~~~~~~~~~~~~~~~~~~
+Policy Evaluators and Optimizers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. autoclass:: ray.rllib.optimizers.evaluator.Evaluator
     :members:
@@ -123,7 +97,7 @@ Currently we support the following action distributions:
 The Model Catalog
 ~~~~~~~~~~~~~~~~~
 
-The Model Catalog is the mechanism for algorithms to get preprocessors, models, and action distributions for varying gym environments. It enables sharing of these components across different algorithms.
+The Model Catalog is the mechanism for algorithms to get canonical preprocessors, models, and action distributions for varying gym environments. It enables easy reuse of these components across different algorithms.
 
 .. autoclass:: ray.rllib.models.ModelCatalog
     :members: