go-queue for incoming events

[bzz]

Queue implementation https://github.com/src-d/go-queue should be used for decoupling review creating and it's processing inside the lookout.

Default configuration should work as is, without extra configuration so lookoutd serve runs using in-memory implementation. Other environments will use RabbitMQ.

Deployment envs would use something like Borges does

• lookoutd enqueue responsible only for interacting \w Github and putting new Review events in queue
• lookout dequeue process events from the queue sequentially, transactionally updating DB with event state, posting results back, etc

Most probably, should include few queues:

• queue for new events
• queue for re-processing failures

[carlosms]

Current status of the steps we defined in the initial discussions:

• go-queue: serve uses a memory queue, new watch and work commands use RabbitMQ
• Add a memory cache in watcher to avoid sending repeated jobs; current polling retrieves all the github events and sends them to the queue again and again (Queue memory cache for watcher #317).
• parallel processing of jobs in worker, with config parameter to control how many goroutines to use. Right now each event is processed sequentially (Concurrent queue job processing #321).
• docs (Add distributed lookout docs #315)
• open issue to request changes in the staging deployment charts.

[carlosms]

All tasks initially planed are merged, I think we can close this one.
Future improvements will have new issues.