Skip to content

Commit e357e44

Browse files
pierotofypierotofy
authored
Merge pull request #109 from terraframe/master
Support for S3 ACL's, private ip addressing on autoscaled nodes and an additional setup command
2 parents 68b8cc8 + 43408de commit e357e44

File tree

3 files changed

+83
-30
lines changed

3 files changed

+83
-30
lines changed

‎docs/aws.md‎

Lines changed: 25 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ In order to use ClusterODM with AWS:
88
* Select/Create a VPC in which the resources will operate.
99
* Select/Create a subnet within the VPC.
1010
* Create a security group in this region, VPC, and subnet which allows inbound access from your ClusterODM master instance on TCP port 3000. Note the name of the security group not the ID.
11-
* Create an S3 bucket in this region to handle results. Don't configure this bucket to block public access.
11+
* Create an S3 bucket in this region to handle results. If you don't specify an ACL, we will default to 'public-read', which requires public read access enabled for your bucket.
1212
* Select an AMI (machine image) to run - Ubuntu has a [handy AMI finder](https://cloud-images.ubuntu.com/locator/ec2/).
1313
* Create an IAM account for ClusterODM to use, which has EC2 and S3 permissions.
1414
* Create a ClusterODM configuration json file as below.
@@ -31,7 +31,8 @@ the on-demand instance cost - you'll always pay the current market price, not yo
3131
"secretKey": "CHANGEME!",
3232
"s3":{
3333
"endpoint": "s3.us-west-2.amazonaws.com",
34-
"bucket": "bucketname"
34+
"bucket": "bucketname",
35+
"acl": "none"
3536
},
3637
"vpc": "",
3738
"subnet": "",
@@ -64,26 +65,28 @@ the on-demand instance cost - you'll always pay the current market price, not yo
6465
}
6566
```
6667

67-
| Field | Description |
68-
|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
69-
| accessKey | AWS Access Key |
70-
| secretKey | AWS Secret Key |
71-
| s3 | S3 bucket configuration. Note that the bucket should *not* be configured to block public access. |
72-
| vpc | The virtual private cloud in which the instances operate. Not providing this assumes a default setting for VPC within the AWS environment. |
73-
| subnet | The subnet supporting the instances. Not providing this assumes a default setting for the subnet within the AWS environment. |
74-
| securityGroup | AWS Security Group name (not ID). Must exist and allow incoming connections from your ClusterODM host on port TCP/3000. |
75-
| createRetries | Number of attempts to create a droplet before giving up. Defaults to 1. |
76-
| maxRuntime | Maximum number of seconds an instance is allowed to run ever. Set to -1 for no limit. |
77-
| maxUploadTime | Maximum number of seconds an instance is allowed to receive file uploads. Set to -1 for no limit. |
78-
| monitoring | Set to true to enable detailed Cloudwatch monitoring for the instance. |
79-
| region | Region identifier where the instances should be created. |
80-
| zone | Zone identifier where the instances should be created. |
81-
| ami | The AMI (machine image) to launch this instance from. |
82-
| tags | Comma-separated list of key,value tags to associate to the instance. |
83-
| spot | Whether to request spot instances. If this is true, a `spotPrice` needs to be provided in the `imageSizeMapping`. |
84-
| imageSizeMapping | Max images count to instance size mapping. (See below.) |
85-
| addSwap | Optionally add this much swap space to the instance as a factor of total RAM (`RAM * addSwap`). A value of `1` sets a swapfile equal to the available RAM. |
86-
| dockerImage | Docker image to launch |
68+
| Field | Description |
69+
|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
70+
| accessKey | AWS Access Key |
71+
| secretKey | AWS Secret Key |
72+
| s3 | S3 bucket configuration. |
73+
| vpc | The virtual private cloud in which the instances operate. Not providing this assumes a default setting for VPC within the AWS environment. |
74+
| subnet | The subnet supporting the instances. Not providing this assumes a default setting for the subnet within the AWS environment. |
75+
| usePrivateAddress | Set to true to use the private IP address when communicating with auto-scaled nodes. Useful if ClusterODM is on the same vpc as the auto-scaled nodes. |
76+
| securityGroup | AWS Security Group name (not ID). Must exist and allow incoming connections from your ClusterODM host on port TCP/3000. |
77+
| createRetries | Number of attempts to create a droplet before giving up. Defaults to 1. |
78+
| maxRuntime | Maximum number of seconds an instance is allowed to run ever. Set to -1 for no limit. |
79+
| maxUploadTime | Maximum number of seconds an instance is allowed to receive file uploads. Set to -1 for no limit. |
80+
| monitoring | Set to true to enable detailed Cloudwatch monitoring for the instance. |
81+
| region | Region identifier where the instances should be created. |
82+
| zone | Zone identifier where the instances should be created. |
83+
| ami | The AMI (machine image) to launch this instance from. |
84+
| tags | Comma-separated list of key,value tags to associate to the instance. |
85+
| spot | Whether to request spot instances. If this is true, a `spotPrice` needs to be provided in the `imageSizeMapping`. |
86+
| imageSizeMapping | Max images count to instance size mapping. (See below.) |
87+
| addSwap | Optionally add this much swap space to the instance as a factor of total RAM (`RAM * addSwap`). A value of `1` sets a swapfile equal to the available RAM. |
88+
| dockerImage | Docker image to launch |
89+
| nodeSetupCmd | Can be optionally used to run a setup command on auto-scaled nodes right before we run ODM. |
8790

8891
## Image Size Mapping
8992

‎libs/asr-providers/aws.js‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,12 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{
2828
"secretKey": "CHANGEME!",
2929
"s3":{
3030
"endpoint": "CHANGEME!",
31-
"bucket": "CHANGEME!"
31+
"bucket": "CHANGEME!",
32+
"acl": "public-read"
3233
},
3334
"vpc": "",
3435
"subnet": "",
36+
"usePrivateAddress": false,
3537
"securityGroup": "CHANGEME!",
3638
"maxRuntime": -1,
3739
"maxUploadTime": -1,
@@ -50,12 +52,13 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{
5052

5153
"addSwap": 1,
5254
"dockerImage": "opendronemap/nodeodm",
53-
"iamrole": ""
55+
"iamrole": "",
56+
"nodeSetupCmd": ""
5457
}, userConfig);
5558
}
5659

5760
async initialize(){
58-
this.validateConfigKeys(["accessKey", "secretKey", "s3.endpoint", "s3.bucket", "securityGroup"]);
61+
this.validateConfigKeys(["accessKey", "secretKey", "s3.endpoint", "s3.bucket", "s3.acl", "securityGroup"]);
5962

6063
// Test S3
6164
const { endpoint, bucket } = this.getConfig("s3");
@@ -118,12 +121,19 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{
118121
const secretKey = this.getConfig("secretKey");
119122
const s3 = this.getConfig("s3");
120123
const webhook = netutils.publicAddressPath("/commit", req, token);
124+
125+
const setupCmd = this.getConfig("nodeSetupCmd");
126+
if (setupCmd != null && setupCmd.length > 0)
127+
{
128+
await dm.ssh(setupCmd);
129+
}
121130

122131
await dm.ssh([`sudo docker run -d -p 3000:3000 ${dockerImage} -q 1`,
123132
`--s3_access_key ${accessKey}`,
124133
`--s3_secret_key ${secretKey}`,
125134
`--s3_endpoint ${s3.endpoint}`,
126135
`--s3_bucket ${s3.bucket}`,
136+
`--s3_acl ${s3.acl}`,
127137
`--webhook ${webhook}`,
128138
`--token ${nodeToken}`].join(" "));
129139
}
@@ -178,6 +188,10 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{
178188
args.push(this.getConfig("tags").join(","));
179189
}
180190

191+
if (this.getConfig("usePrivateAddress")) {
192+
args.push("--amazonec2-use-private-address");
193+
}
194+
181195
if (this.getConfig("engineInstallUrl")){
182196
args.push("--engine-install-url")
183197
args.push(this.getConfig("engineInstallUrl"));

‎libs/proxy.js‎

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ const async = require('async');
3636
const odmOptions = require('./odmOptions');
3737
const asrProvider = require('./asrProvider');
3838
const floodMonitor = require('./floodMonitor');
39+
const AWS = require('aws-sdk');
3940

4041
module.exports = {
4142
initialize: async function(cloudProvider){
@@ -521,11 +522,46 @@ module.exports = {
521522

522523
const s3Url = url.parse(asrProvider.downloadsPath());
523524
s3Url.pathname = path.join(taskId, assetPath);
524-
res.writeHead(301, {
525-
'Location': url.format(s3Url)
526-
});
527-
res.end();
528-
return;
525+
526+
const s3Config = asrProvider.get().getConfig("s3");
527+
528+
// If URL requires authentication, fetch the object on their behalf and then stream it to them
529+
// If our aws library gets updated to v3, then we could return a redirect to a presigned url instead
530+
if (s3Config != null && s3Config.acl !== "public-read") {
531+
let key = path.join(taskId, assetPath)
532+
533+
const s3 = new AWS.S3({
534+
endpoint: new AWS.Endpoint(s3Config.endpoint),
535+
signatureVersion: 'v4',
536+
accessKeyId: asrProvider.get().getConfig("accessKey"),
537+
secretAccessKey: asrProvider.get().getConfig("secretKey")
538+
});
539+
540+
s3.getObject({ Bucket: s3Config.bucket, Key: key }, (err, data) => {
541+
if (err) {
542+
logger.error(`Error encountered downloading object ${err}`);
543+
res.statusCode = 500;
544+
res.end('Internal server error');
545+
return;
546+
}
547+
548+
// Set the content-type and content-length headers
549+
res.setHeader('Content-Type', data.ContentType);
550+
res.setHeader('Content-Length', data.ContentLength);
551+
552+
// Write the object data to the response
553+
res.write(data.Body);
554+
res.end();
555+
});
556+
return;
557+
558+
} else {
559+
res.writeHead(301, {
560+
'Location': url.format(s3Url)
561+
});
562+
res.end();
563+
return;
564+
}
529565
}
530566
}
531567

0 commit comments

Comments
 (0)