Bug fixes and features for S3 support

[StarlightIbuki]

fix: template incorrect handling of query string
fix: blob param should be raw body
feat: add support for credential from file
fix: escaping for param in uri
fix: handling raw body conflict with body param
fix: crash when no type check designated
fix: missing support for "headers" location
fix: compatibility for S3

[StarlightIbuki]

We could add a unit test, to mock an HTTP service and inspect the request generated.
To have a determinative result for signatures, we could hook functions to have a constant timestamp.

[StarlightIbuki]

Also, the "config".global is pretty slow in my environment, maybe we should address it.

[StarlightIbuki]

Also, the "config".global is pretty slow in my environment, maybe we should address it.

It turns out to be that the utils.getCurrentRegion() is making request:

lua-resty-aws/src/resty/aws/utils.lua

Line 77 in b0a7d06

local token, err = make_request(IDMS_URI.."/latest/api/token", "PUT", headers)

and the request failed in my env.

So it's better to set AWS_DEFAULT_REGION to avoid sending the request. And I wonder why we don't use AWS_REGION to initialize when AWS_DEFAULT_REGION is not present.

[windmgc]

Also, the "config".global is pretty slow in my environment, maybe we should address it.

Are you testing this on your localhost without any related config set(like environment var 'AWS_REGION' or profile config)?
I think it's pretty okay for now not to deal with it because the detect_region will find the correct way if the environment is in the real AWS

So it's better to set AWS_DEFAULT_REGION to avoid sending the request. And I wonder why we don't use AWS_REGION to initialize when AWS_DEFAULT_REGION is not present.

I think the AWS_DEFAULT_REGION is some kind of a fallback value when AWS_REGION is not present, check the
https://github.com/aws/aws-sdk-go/blob/aeeca02c4a3603ec615fa9a011e2e05c9d40e1f8/aws/session/env_config.go#L45-L47

[windmgc]

Failed test on AWS environment due to deprecation of putting bucket name in request URI:

{
  body = '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>suikatestbucket.s3.amazonaws.com</Endpoint><Bucket>XXXXXXXXX</Bucket><RequestId>XXXXXXXXXXX</RequestId><HostId>XXXXXXXXXXX</HostId></Error>',
  headers = {
    Connection = "close",
    ["Content-Type"] = "application/xml",
    Date = "Thu, 03 Nov 2022 07:35:20 GMT",
    Server = "AmazonS3",
    ["Transfer-Encoding"] = "chunked",
    ["x-amz-bucket-region"] = "ap-northeast-1",
    ["x-amz-id-2"] = "XXXXXXX",
    ["x-amz-request-id"] = "XXXXXXX",
    <metatable> = {
      __index = <function 1>,
      __newindex = <function 2>,
      normalised = {
        connection = "Connection",
        ["content-type"] = "Content-Type",
        date = "Date",
        server = "Server",
        ["transfer-encoding"] = "Transfer-Encoding",
        ["x-amz-bucket-region"] = "x-amz-bucket-region",
        ["x-amz-id-2"] = "x-amz-id-2",
        ["x-amz-request-id"] = "x-amz-request-id"
      }
    }
  },
  reason = "Moved Permanently",
  status = 301
}nil

Maybe we should do a similar thing https://github.com/aws/aws-sdk-go/blob/aeeca02c4a3603ec615fa9a011e2e05c9d40e1f8/service/s3/host_style_bucket.go#L132-L136

[StarlightIbuki]

Failed test on AWS environment due to deprecation of putting bucket name in request URI:

{
  body = '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>suikatestbucket.s3.amazonaws.com</Endpoint><Bucket>XXXXXXXXX</Bucket><RequestId>XXXXXXXXXXX</RequestId><HostId>XXXXXXXXXXX</HostId></Error>',
  headers = {
    Connection = "close",
    ["Content-Type"] = "application/xml",
    Date = "Thu, 03 Nov 2022 07:35:20 GMT",
    Server = "AmazonS3",
    ["Transfer-Encoding"] = "chunked",
    ["x-amz-bucket-region"] = "ap-northeast-1",
    ["x-amz-id-2"] = "XXXXXXX",
    ["x-amz-request-id"] = "XXXXXXX",
    <metatable> = {
      __index = <function 1>,
      __newindex = <function 2>,
      normalised = {
        connection = "Connection",
        ["content-type"] = "Content-Type",
        date = "Date",
        server = "Server",
        ["transfer-encoding"] = "Transfer-Encoding",
        ["x-amz-bucket-region"] = "x-amz-bucket-region",
        ["x-amz-id-2"] = "x-amz-id-2",
        ["x-amz-request-id"] = "x-amz-request-id"
      }
    }
  },
  reason = "Moved Permanently",
  status = 301
}nil

Maybe we should do a similar thing https://github.com/aws/aws-sdk-go/blob/aeeca02c4a3603ec615fa9a011e2e05c9d40e1f8/service/s3/host_style_bucket.go#L132-L136

I strongly doubt if aws-sdk-* are officials. How could they not fix the API template but do a patch like that?

[StarlightIbuki]

Should we really calculate the host in v4.lua? @Tieske

[windmgc]

Tested with @StarlightIbuki that the old request URI format also works well, so it is our decision whether to "put bucket into host instead of path".

Already tested this PR with SecretsManager:getValue and S3:putObject as well as getObject.

[StarlightIbuki]

Tested with @StarlightIbuki that the old request URI format also works well, so it is our decision whether to "put bucket into host instead of path".

Already tested this PR with SecretsManager:getValue and S3:putObject as well as getObject.

I made it an option and default to "in host".

[mayocream]

I am good with these changes.
ping @Tieske for another look.

[fffonion]

We don't have automatic CI and I assume the test suites have been ran locally. Since this
is now a blocker for new release I'm merging this in to unblock upstream PRs. Manual
test is required post feature freeze.