Skip to content

Commit d3f4f37

Browse files
lyu571kmoe
lyu571
and
kmoe
authored
backend/cos: Support endpoint (#33656)
* support endpoint for the COS backend * update go.sum * 1.add domain for tag client used for lock file. 2.adjust endpoint formation. * Apply suggestions from code review Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com> --------- Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
1 parent 12fdf1f commit d3f4f37

File tree

5 files changed

+138
-5
lines changed

5 files changed

+138
-5
lines changed

‎go.mod‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ require (
7777
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.588
7878
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sts v1.0.588
7979
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/tag v1.0.233
80-
github.com/tencentyun/cos-go-sdk-v5 v0.7.29
80+
github.com/tencentyun/cos-go-sdk-v5 v0.7.42
8181
github.com/tombuildsstuff/giovanni v0.15.1
8282
github.com/xanzy/ssh-agent v0.3.1
8383
github.com/xlab/treeprint v0.0.0-20161029104018-1d6e34225557
@@ -162,6 +162,7 @@ require (
162162
github.com/bmatcuk/doublestar/v4 v4.6.0 // indirect
163163
github.com/bradleyfalzon/ghinstallation/v2 v2.1.0 // indirect
164164
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
165+
github.com/clbanning/mxj v1.8.4 // indirect
165166
github.com/cli/go-gh v1.0.0 // indirect
166167
github.com/cli/safeexec v1.0.0 // indirect
167168
github.com/cli/shurcooL-graphql v0.0.2 // indirect

‎go.sum‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -388,6 +388,8 @@ github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e h1:fY5BOSpyZCqRo5O
388388
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
389389
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1 h1:q763qf9huN11kDQavWsoZXJNW3xEE4JJyHa5Q25/sd8=
390390
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
391+
github.com/clbanning/mxj v1.8.4 h1:HuhwZtbyvyOw+3Z1AowPkU87JkJUSv751ELWaiTpj8I=
392+
github.com/clbanning/mxj v1.8.4/go.mod h1:BVjHeAH+rl9rs6f+QIpeRl0tfu10SXn1pUSa5PVGJng=
391393
github.com/cli/go-gh v1.0.0 h1:zE1YUAUYqGXNZuICEBeOkIMJ5F50BS0ftvtoWGlsEFI=
392394
github.com/cli/go-gh v1.0.0/go.mod h1:bqxLdCoTZ73BuiPEJx4olcO/XKhVZaFDchFagYRBweE=
393395
github.com/cli/safeexec v1.0.0 h1:0VngyaIyqACHdcMNWfo6+KdUYnqEr2Sg+bSP1pdF+dI=
@@ -856,6 +858,7 @@ github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0Qu
856858
github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
857859
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
858860
github.com/mitchellh/mapstructure v1.3.3/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
861+
github.com/mitchellh/mapstructure v1.4.3/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
859862
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
860863
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
861864
github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
@@ -981,16 +984,16 @@ github.com/stretchr/testify v1.7.4/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO
981984
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
982985
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
983986
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
984-
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.194/go.mod h1:7sCQWVkxcsR38nffDW057DRGk8mUjK1Ing/EFOK8s8Y=
987+
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.563/go.mod h1:7sCQWVkxcsR38nffDW057DRGk8mUjK1Ing/EFOK8s8Y=
985988
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.588 h1:DYtBXB7sVc3EOW5horg8j55cLZynhsLYhHrvQ/jXKKM=
986989
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.588/go.mod h1:7sCQWVkxcsR38nffDW057DRGk8mUjK1Ing/EFOK8s8Y=
987-
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/kms v1.0.194/go.mod h1:yrBKWhChnDqNz1xuXdSbWXG56XawEq0G5j1lg4VwBD4=
990+
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/kms v1.0.563/go.mod h1:uom4Nvi9W+Qkom0exYiJ9VWJjXwyxtPYTkKkaLMlfE0=
988991
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sts v1.0.588 h1:PlkFOALQZ9BLUyX8EalATUQD5xEn1Sz34C+Rw5VSpvk=
989992
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sts v1.0.588/go.mod h1:vPvXNb+zBZVJfZCIKWcYxLpGzgScKKgiPUArobWZ+nU=
990993
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/tag v1.0.233 h1:5Tbi+jyZ2MojC6GK8V6hchwtnkP2IuENUTqSisbYOlA=
991994
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/tag v1.0.233/go.mod h1:sX14+NSvMjOhNFaMtP2aDy6Bss8PyFXij21gpY6+DAs=
992-
github.com/tencentyun/cos-go-sdk-v5 v0.7.29 h1:uwRBzc70Wgtc5iQQCowqecfRT0OpCXUOZzodZHOOEDs=
993-
github.com/tencentyun/cos-go-sdk-v5 v0.7.29/go.mod h1:4E4+bQ2gBVJcgEC9Cufwylio4mXOct2iu05WjgEBx1o=
995+
github.com/tencentyun/cos-go-sdk-v5 v0.7.42 h1:Up1704BJjI5orycXKjpVpvuOInt9GC5pqY4knyE9Uds=
996+
github.com/tencentyun/cos-go-sdk-v5 v0.7.42/go.mod h1:LUFnaqRmGk6pEHOaRmdn2dCZR2j0cSsM5xowWFPTPao=
994997
github.com/thanhpk/randstr v1.0.4 h1:IN78qu/bR+My+gHCvMEXhR/i5oriVHcTB/BJJIRTsNo=
995998
github.com/thanhpk/randstr v1.0.4/go.mod h1:M/H2P1eNLZzlDwAzpkkkUvoyNNMbzRGhESZuEQk3r0U=
996999
github.com/thlib/go-timezone-local v0.0.0-20210907160436-ef149e42d28e h1:BuzhfgfWQbX0dWzYzT1zsORLnHRv3bcRcsaUk0VmXA8=

‎internal/backend/remote-state/cos/backend.go‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,11 @@ package cos
66
import (
77
"context"
88
"fmt"
9+
"log"
910
"net/http"
1011
"net/url"
1112
"os"
13+
"regexp"
1214
"strconv"
1315
"strings"
1416
"time"
@@ -28,6 +30,8 @@ const (
2830
PROVIDER_SECRET_KEY = "TENCENTCLOUD_SECRET_KEY"
2931
PROVIDER_SECURITY_TOKEN = "TENCENTCLOUD_SECURITY_TOKEN"
3032
PROVIDER_REGION = "TENCENTCLOUD_REGION"
33+
PROVIDER_ENDPOINT = "TENCENTCLOUD_ENDPOINT"
34+
PROVIDER_DOMAIN = "TENCENTCLOUD_DOMAIN"
3135
PROVIDER_ASSUME_ROLE_ARN = "TENCENTCLOUD_ASSUME_ROLE_ARN"
3236
PROVIDER_ASSUME_ROLE_SESSION_NAME = "TENCENTCLOUD_ASSUME_ROLE_SESSION_NAME"
3337
PROVIDER_ASSUME_ROLE_SESSION_DURATION = "TENCENTCLOUD_ASSUME_ROLE_SESSION_DURATION"
@@ -49,6 +53,7 @@ type Backend struct {
4953
key string
5054
encrypt bool
5155
acl string
56+
domain string
5257
}
5358

5459
// New creates a new backend for TencentCloud cos remote state.
@@ -87,6 +92,18 @@ func New() backend.Backend {
8792
Required: true,
8893
Description: "The name of the COS bucket",
8994
},
95+
"endpoint": {
96+
Type: schema.TypeString,
97+
Optional: true,
98+
Description: "The custom endpoint for the COS API, e.g. http://cos-internal.{Region}.tencentcos.cn. Both HTTP and HTTPS are accepted.",
99+
DefaultFunc: schema.EnvDefaultFunc(PROVIDER_ENDPOINT, nil),
100+
},
101+
"domain": {
102+
Type: schema.TypeString,
103+
Optional: true,
104+
DefaultFunc: schema.EnvDefaultFunc(PROVIDER_DOMAIN, nil),
105+
Description: "The root domain of the API request. Default is tencentcloudapi.com.",
106+
},
90107
"prefix": {
91108
Type: schema.TypeString,
92109
Optional: true,
@@ -231,6 +248,33 @@ func (b *Backend) configure(ctx context.Context) error {
231248
return err
232249
}
233250

251+
if v, ok := data.GetOk("domain"); ok {
252+
b.domain = v.(string)
253+
log.Printf("[DEBUG] Backend: set domain for TencentCloud API client. Domain: [%s]", b.domain)
254+
}
255+
// set url as endpoint when provided
256+
// "http://{Bucket}.cos-internal.{Region}.tencentcos.cn"
257+
if v, ok := data.GetOk("endpoint"); ok {
258+
endpoint := v.(string)
259+
260+
re := regexp.MustCompile(`^(http(s)?)://cos-internal\.([^.]+)\.tencentcos\.cn$`)
261+
matches := re.FindStringSubmatch(endpoint)
262+
if len(matches) != 4 {
263+
return fmt.Errorf("Invalid URL: %v must be: %v", endpoint, "http(s)://cos-internal.{Region}.tencentcos.cn")
264+
}
265+
266+
protocol := matches[1]
267+
region := matches[3]
268+
269+
// URL after converting
270+
newUrl := fmt.Sprintf("%s://%s.cos-internal.%s.tencentcos.cn", protocol, b.bucket, region)
271+
u, err = url.Parse(newUrl)
272+
log.Printf("[DEBUG] Backend: set COS URL as: [%s]", newUrl)
273+
}
274+
if err != nil {
275+
return err
276+
}
277+
234278
secretId := data.Get("secret_id").(string)
235279
secretKey := data.Get("secret_key").(string)
236280
securityToken := data.Get("security_token").(string)
@@ -333,6 +377,8 @@ func (b *Backend) NewClientProfile(timeout int) *profile.ClientProfile {
333377
cpf.HttpProfile.ReqMethod = "POST"
334378
// request timeout
335379
cpf.HttpProfile.ReqTimeout = timeout
380+
// request domain
381+
cpf.HttpProfile.RootDomain = b.domain
336382

337383
return cpf
338384
}

‎internal/backend/remote-state/cos/backend_test.go‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,27 @@ func TestRemoteClientWithEncryption(t *testing.T) {
119119
remote.TestClient(t, rs.Client)
120120
}
121121

122+
func TestRemoteClientWithEndpoint(t *testing.T) {
123+
t.Parallel()
124+
125+
bucket := bucketName(t)
126+
127+
be := setupBackendWithEndpoint(t, bucket, defaultPrefix, defaultKey, false)
128+
defer teardownBackend(t, be)
129+
130+
ss, err := be.StateMgr(backend.DefaultStateName)
131+
if err != nil {
132+
t.Fatalf("unexpected error: %s", err)
133+
}
134+
135+
rs, ok := ss.(*remote.State)
136+
if !ok {
137+
t.Fatalf("wrong state manager type\ngot: %T\nwant: %T", ss, rs)
138+
}
139+
140+
remote.TestClient(t, rs.Client)
141+
}
142+
122143
func TestRemoteLocks(t *testing.T) {
123144
t.Parallel()
124145

@@ -201,6 +222,44 @@ func TestBackendWithEncryption(t *testing.T) {
201222
backend.TestBackendStateLocks(t, be0, be1)
202223
}
203224

225+
func TestBackendWithEndpoint(t *testing.T) {
226+
t.Parallel()
227+
228+
bucket := bucketName(t)
229+
230+
be0 := setupBackendWithEndpoint(t, bucket, defaultPrefix, defaultKey, false)
231+
defer teardownBackend(t, be0)
232+
233+
be1 := setupBackendWithEndpoint(t, bucket, defaultPrefix, defaultKey, false)
234+
defer teardownBackend(t, be1)
235+
236+
backend.TestBackendStates(t, be0)
237+
backend.TestBackendStateLocks(t, be0, be1)
238+
backend.TestBackendStateForceUnlock(t, be0, be1)
239+
}
240+
241+
func setupBackendWithEndpoint(t *testing.T, bucket, prefix, key string, encrypt bool) backend.Backend {
242+
t.Helper()
243+
244+
skip := os.Getenv("TF_COS_APPID") == ""
245+
if skip {
246+
t.Skip("This test requires setting the TF_COS_APPID environment variable")
247+
}
248+
249+
if os.Getenv(PROVIDER_REGION) == "" {
250+
os.Setenv(PROVIDER_REGION, "ap-guangzhou")
251+
}
252+
253+
appId := os.Getenv("TF_COS_APPID")
254+
region := os.Getenv(PROVIDER_REGION)
255+
256+
if os.Getenv(PROVIDER_ENDPOINT) == "" {
257+
os.Setenv(PROVIDER_ENDPOINT, fmt.Sprintf("http://%s.cos-internal.%s.tencentcos.cn", bucket+appId, region))
258+
}
259+
260+
return setupBackend(t, bucket, prefix, key, encrypt)
261+
}
262+
204263
func setupBackend(t *testing.T, bucket, prefix, key string, encrypt bool) backend.Backend {
205264
t.Helper()
206265

@@ -223,6 +282,10 @@ func setupBackend(t *testing.T, bucket, prefix, key string, encrypt bool) backen
223282
"key": key,
224283
}
225284

285+
if os.Getenv(PROVIDER_ENDPOINT) != "" {
286+
config["endpoint"] = os.Getenv(PROVIDER_ENDPOINT)
287+
}
288+
226289
b := backend.TestBackendConfig(t, New(), backend.TestWrapConfig(config))
227290
be := b.(*Backend)
228291

‎website/docs/language/settings/backends/cos.mdx‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ The following configuration options or environment variables are supported:
6464
- `encrypt` - (Optional) Whether to enable server side encryption of the state file. If it is true, COS will use 'AES256' encryption algorithm to encrypt state file.
6565
- `acl` - (Optional) Object ACL to be applied to the state file, allows `private` and `public-read`. Defaults to `private`.
6666
- `accelerate` - (Optional) Whether to enable global Acceleration. Defaults to `false`.
67+
- `endpoint` - (Optional) The Custom Endpoint for the COS backend. It supports the environment variable `TENCENTCLOUD_ENDPOINT`.
68+
- `domain` - (Optional) The root domain of the API request. Defaults to `tencentcloudapi.com`. It supports the environment variable `TENCENTCLOUD_DOMAIN`.
6769

6870
### Assume Role
6971
If provided with an assume role, Terraform will attempt to assume this role using the supplied credentials.
@@ -106,4 +108,22 @@ $ export TENCENTCLOUD_ASSUME_ROLE_ARN="qcs::cam::uin/xxx:roleName/yyy"
106108
$ export TENCENTCLOUD_ASSUME_ROLE_SESSION_NAME="my-session-name"
107109
$ export TENCENTCLOUD_ASSUME_ROLE_SESSION_DURATION=3600
108110
$ terraform plan
111+
```
112+
113+
### Endpoint
114+
If provided with an endpoint URL, Terraform will attempt to access the COS backend by the `endpoint` configuration or the environment variable `TENCENTCLOUD_ENDPOINT`.
115+
116+
A typical endpoint looks like this: `http://cos-internal.{Region}.tencentcos.cn`. Both HTTP and HTTPS are accepted.
117+
118+
Usage:
119+
120+
```hcl
121+
terraform {
122+
backend "cos" {
123+
region = "ap-guangzhou"
124+
bucket = "bucket-for-terraform-state-1258798060"
125+
prefix = "terraform/state"
126+
endpoint = "http://cos-internal.ap-guangzhou.tencentcos.cn"
127+
}
128+
}
109129
```

0 commit comments

Comments
 (0)