Add support for "requester pays" operations

[tseaver]

The storage docs don't really explain this, but the JSON API methods have begun to accept a `userProject' query parameter (or is it a header? only its yak-shaver knows for sure). The point of it looks to be to bill any charges associated with the request to a different project than the one which owns the bucket in which it takes place. I can't tell how a bucket owner is supposed to configure things such that foreign projects can only access objects by passing this parameter/header.

See the g-c-dotnet PR implementing preliminary support.

And hre is the g-c-php PR.

[jdpedrie]

@tseaver, the bucket owner will enable requester pays by setting billing.requesterPays to true in the bucket settings. If this setting is toggled, any requests made to the bucket or an object in the bucket which originate from a non-bucket-owner will throw an error unless the userProject is sent.

https://github.com/GoogleCloudPlatform/google-cloud-php/pull/527/files#diff-703c9fa16eb822e79062a7936fceffe5R103

[tseaver]

@jdpedrie Thanks. That seems less elegant than I had imaginged (e.g., there would be some ACL / IAM tweak which would assign permissions to a "user-is-paying" role).

The whole feature needs both narrative and API docs.

[tseaver]

@jdpedrie Is there anywhere in the API docs which defines the billing field for buckets? Or how / when to populate the userProject header / query parameter / body field?

[jdpedrie]

Not that I've seen in the public docs. I shared a google doc with more information. (Check your email).

The REST discovery document will show which resources accept the userProject parameter (it's in the query string). Our implementation simply adds this field in to each request, if the bucket is constructed with requesterPays toggled on.

[tseaver]

@jdpedrie Thanks muchly!

[tseaver]

Implementation plan:

• Add a Bucket.requester_pays boolean property, mapped onto the billing/requesterPays subfield defined in the REST discovery document. Also include requester_pays as an optional argument to Client.create_bucket. PR Add 'Bucket.requester_pays' property. #3488
• Add an optional user_project constructor parameter to Bucket.__init__: we cannot know whether it is valid / appropriate to be set during the constructor, so we just trust the user. Define Blob.user_project in terms of its bucket's value. PR Add '{Bucket,Blob}.user_project' properties. #3490
• If the bucket's user_project is set, all API requests made by its methods will pass the value as the userProject query parameter. PRs Add '{Bucket,Blob}.user_project' properties. #3490, Pass 'user_project' if set for Bucket API requests #3492
• Blob instances reuse their buckets' user_project: likewise, they send the userProject query parameter to all API requests if set. PR Pass 'user_project' if set for Blob API requests #3495, Pass 'user_project' if set for blob downloads w/ 'mediaLink' set #3500
• Add support to acl.ACL to pass the host bucket / blob's user_project, if set, when making API reqeusts. PR Pass 'user_project' if set for BucketACL/BlobACL API requests #3499
• For both Bucket and Blob API requests, we rely on the back-end to raise errors for cases where userProject should not be passed (i.e., the bucket is not configured for requesterPays), and for cases where it is required but not passed.

@lukesneeringer does that seem correct to you?

[tseaver]

Note that system testing for this feature requires that a second, valid project be accessible to the account running the tests. @dhermes, @lukesneeringer how do you think we should arrange for that, both for local testing and for CI?

Update: we can test the creation of a bucket w/ requester_pays set without the extra project.

[tseaver]

Does this feature require whitelisting? I just tried adding a system test which passed the billing/requesterPays property along as true and it fails with an unhelpfully-terse 400.

[jdpedrie]

@tseaver yes -- tested to verify. works on my whitelisted project, fails on personal project.

[dhermes]

@tseaver We should probably just allow the system test to be skipped locally. It's not too difficult to get a 2nd project for CI.

The plan seems OK 👍