Initial Open Source Commit

This existed as an internal project before, but the history contained
access keys and the like. Currently at 1.3

Author: packrat386

.travis.yml

@@ -0,0 +1,20 @@
+language: go
+
+go:
+  - 1.6
+  - 1.7
+  - tip
+
+before_script:
+  - go get github.com/GeertJohan/fgt
+  - go get github.com/golang/lint
+
+script:
+  - fgt gomft -l .
+  - fgt golint ./..
+  - go vet ./...
+  - go test -v ./...
+
+matrix:
+  allow_failures:
+    - go: tip

Changelog.md

@@ -0,0 +1,17 @@
+# Scout Changes
+
+1.3
+----------
+- Rewrite the SQS integration to the use the AWS SDK instead of goamz
+
+1.2
+----------
+- Save jobs in Redis with the Sidekiq `retry` flag set to `true`
+
+1.1
+----------
+
+- Remove the `--quiet` flag in favor of `--log-level` which defaults to `INFO`
+- Move some of the more verbose logging to `DEBUG` level logs
+- Log full message body after parsing it
+

LICENSE.txt

@@ -0,0 +1,8 @@
+Copyright (c) 2016 Enova International
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

README.md

@@ -0,0 +1,96 @@
+# Scout
+
+Scout is a daemon for listening to a set of SNS topics and enqueuing anything it
+finds into sidekiq jobs. It's meant to extract processing of SQS from the rails
+apps that increasingly need to do so.
+
+## Usage
+
+```
+NAME:
+   scout - SQS Listener
+Poll SQS queues specified in a config and enqueue Sidekiq jobs with the queue items.
+It gracefully stops when sent SIGTERM.
+
+USAGE:
+   scout [global options] command [command options] [arguments...]
+
+VERSION:
+   1.3
+
+COMMANDS:
+     help, h  Shows a list of commands or help for one command
+
+GLOBAL OPTIONS:
+   --config FILE, -c FILE       Load config from FILE, required
+   --freq N, -f N               Poll SQS every N milliseconds (default: 100)
+   --log-level value, -l value  Sets log level. Accepts one of: debug, info, warn, error
+   --json, -j                   Log in json format
+   --help, -h                   show help
+   --version, -v                print the version
+```
+
+## Configuration
+
+The configuration requires 3 distinct sets of information. It needs information
+about how to connect to redis to enqueue jobs, credentials to talk to AWS and
+read SQS, and a mapping from SNS topics to sidekiq worker classes in the
+application. The structure looks like this.
+
+```yaml
+redis:
+  host: "localhost:9000"
+  namespace: "test"
+  queue: "background"
+aws:
+  access_key: "super"
+  secret_key: "secret"
+  region: "us-best"
+queue:
+  name: "myapp_queue"
+  topics:
+    foo-topic: "FooWorker"
+    bar-topic: "BazWorker"
+```
+
+None of this information is actually an example of anything other than the
+strucure of the file, so if you copy paste it you'll probably be disappointed.
+
+## Versioning
+
+Scout uses tagged commits to be compatible with gopkg.in. To pin to version 1,
+you can import it as `gopkg.in/enova/scout.v1`. The "first" version is version
+1.3, since all other versions were before this project was made open source.
+Version 2 is possible at some point and may contain breaking changes, so pinning
+to version 1 is recommended unless you want to work with the bleeding edge.
+
+## Development
+
+To get set up make sure to run `go get -t -u ./...` to get all the dependencies.
+
+### Testing
+
+The normal test suite can be run as expected with go test. There are also two
+tagged files with expensive integration tests that require external services.
+They can be run as follows
+
+```
+ [FG-386] scout > go test -run=TestSQS -v -tags=sqsint
+=== RUN   TestSQS_Init
+--- PASS: TestSQS_Init (3.84s)
+=== RUN   TestSQS_FetchDelete
+--- PASS: TestSQS_FetchDelete (3.58s)
+    PASS
+ok     github.com/enova/scout  7.422s
+ [FG-386] scout > go test -run=TestWorker -v -tags=redisint
+=== RUN   TestWorker_Init
+--- PASS: TestWorker_Init (0.00s)
+=== RUN   TestWorker_Push
+--- PASS: TestWorker_Push (0.00s)
+PASS
+ok      github.com/enova/scout  0.013s
+```
+
+The tests themselves (found in `sqs_client_test.go` and `worker_client_test.go`)
+explain what is required to run them. In particular, the SQS integration tests
+require that you provide AWS credentials to run them.

config.go

@@ -0,0 +1,53 @@
+package main
+
+import (
+	"io/ioutil"
+
+	"gopkg.in/yaml.v2"
+)
+
+// Config is the internal representation of the yaml that determines what
+// the app listens to an enqueues
+type Config struct {
+	Redis RedisConfig `yaml:"redis"`
+	AWS   AWSConfig   `yaml:"aws"`
+	Queue QueueConfig `yaml:"queue"`
+}
+
+// RedisConfig is a nested config that contains the necessary parameters to
+// connect to a redis instance and enqueue workers.
+type RedisConfig struct {
+	Host      string `yaml:"host"`
+	Namespace string `yaml:"namespace"`
+	Queue     string `yaml:"queue"`
+}
+
+// AWSConfig is a nested config that contains the necessary parameters to
+// connect to AWS and read from SQS
+type AWSConfig struct {
+	AccessKey string `yaml:"access_key"`
+	SecretKey string `yaml:"secret_key"`
+	Region    string `yaml:"region"`
+}
+
+// QueueConfig is a nested config that gives the SQS queue to listen on
+// and a mapping of topics to workeers
+type QueueConfig struct {
+	Name   string            `yaml:"name"`
+	Topics map[string]string `yaml:"topics"`
+}
+
+// ReadConfig reads from a file with the given name and returns a config or
+// an error if the file was unable to be parsed. It does no error checking
+// as far as required fields.
+func ReadConfig(file string) (*Config, error) {
+	data, err := ioutil.ReadFile(file)
+	if err != nil {
+		return nil, err
+	}
+
+	config := new(Config)
+
+	err = yaml.Unmarshal(data, config)
+	return config, err
+}

config_test.go

@@ -0,0 +1,87 @@
+package main
+
+import (
+	"io/ioutil"
+	"os"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	"github.com/stretchr/testify/suite"
+)
+
+func TestConfig(t *testing.T) {
+	suite.Run(t, new(ConfigTestSuite))
+}
+
+type ConfigTestSuite struct {
+	suite.Suite
+	tempfile *os.File
+	assert   *require.Assertions
+}
+
+func (c *ConfigTestSuite) SetupTest() {
+	c.assert = require.New(c.T())
+
+	var err error
+	c.tempfile, err = ioutil.TempFile("", "config")
+	c.assert.NoError(err)
+}
+
+func (c *ConfigTestSuite) TearDownTest() {
+	os.Remove(c.tempfile.Name())
+}
+
+func (c *ConfigTestSuite) WriteTemp(content string) {
+	_, err := c.tempfile.Write([]byte(content))
+	c.assert.NoError(err)
+	ReadConfig(c.tempfile.Name())
+	err = c.tempfile.Close()
+	c.assert.NoError(err)
+}
+
+var validConfig = `
+redis:
+  host: "localhost:9000"
+  namespace: "test"
+  queue: "background"
+aws:
+  access_key: "super"
+  secret_key: "secret"
+  region: "us_best"
+queue:
+  name: "myapp_queue"
+  topics:
+    foo_topic: "FooWorker"
+    bar_topic: "BazWorker"`
+
+func (c *ConfigTestSuite) TestConfig_Valid() {
+	c.WriteTemp(validConfig)
+	config, err := ReadConfig(c.tempfile.Name())
+	c.assert.NoError(err)
+
+	// More to convince myself that the yaml package works than anything
+	c.assert.Equal(config.Redis.Host, "localhost:9000")
+	c.assert.Equal(config.Redis.Queue, "background")
+	c.assert.Equal(config.AWS.Region, "us_best")
+	c.assert.Equal(config.Queue.Name, "myapp_queue")
+	c.assert.Equal(config.Queue.Topics["foo_topic"], "FooWorker")
+}
+
+var sparseConfig = `
+redis:
+  host: "localhost:9000"
+aws:
+  access_key: "super"
+  secret_key: "secret"
+  region: "us_best"`
+
+// It's ok for stuff to be missing, we'll check that elsewhere
+func (c *ConfigTestSuite) TestConfig_Sparse() {
+	c.WriteTemp(sparseConfig)
+	config, err := ReadConfig(c.tempfile.Name())
+	c.assert.NoError(err)
+
+	c.assert.Equal(config.Redis.Namespace, "")
+	c.assert.Equal(config.AWS.Region, "us_best")
+	c.assert.Equal(len(config.Queue.Topics), 0)
+}