command: terraform providers mirror#25084
Conversation
Codecov Report
|
apparentlymart
force-pushed
the
f-cmd-providers-mirror
branch
from
db3fe91 to
a848c0d
Compare
| // The tests in this file are for the "terraform providers schema" command, | ||
| // which is tested in an e2etest mode rather than a unit test mode because it | ||
| // interacts directly with Terraform Registry and the full details of that are | ||
| // tricky to mock. Such a mock is _possible_, but we're using e2etest as a |
There was a problem hiding this comment.
I am not at all opposed to this e2e test, but just so you know we recently added a testRegistrySource helper function in the command package:
terraform/command/command_test.go
Line 919 in 020084f
There was a problem hiding this comment.
This particular one is expecting to be talking always to upstream registries (so you don't have to unconfigure your mirrors just to populate them), so it wasn't very amenable to that sort of mocking. I did have an earlier version that created a mock registry on a fake domain and forced use of it via service discovery, but that required the fake providers to be properly signed and that was a little too complex to set up in a reasonable amount of time to get this done right now. Maybe in a later commit!
alisdair
left a comment
alisdair
left a comment
There was a problem hiding this comment.
I think I spotted a couple of typos, otherwise looks great! Thanks as always for the excellent inline comments.
| "github.com/hashicorp/terraform/e2e" | ||
| ) | ||
|
|
||
| // The tests in this file are for the "terraform providers schema" command, |
There was a problem hiding this comment.
| // The tests in this file are for the "terraform providers schema" command, | |
| // The tests in this file are for the "terraform providers mirror" command, |
Typo?
| that will be used as a local filesystem mirror in the provider installation | ||
| configuration. | ||
|
|
||
| -> `terraform providers schema` is available only in Terraform v0.13 or later. |
There was a problem hiding this comment.
| -> `terraform providers schema` is available only in Terraform v0.13 or later. | |
| -> `terraform providers mirror` is available only in Terraform v0.13 or later. |
|
Heh... thanks for catching those typos @alisdair! I think by the end of the day on Friday I had lost the ability to distinguish those two command lines. I'll fix those before merging. |
We previously had this functionality available for cached packages in the providercache package. This moves the main implementation of this over to the getproviders package and then implements it also for PackageMeta, allowing us to compute hashes in a consistent way across both of our representations of a provider package. The new methods on PackageMeta will only be effective for packages in the local filesystem because we need direct access to the contents in order to produce the hash. Hopefully in future the registry protocol will be able to also provide hashes using this content-based (rather than archive-based) algorithm and then we'll be able to make this work for PackageMeta referring to a package obtained from a registry too, but hashes for local packages only are still useful for some cases right now, such as generating mirror directories in the "terraform providers mirror" command.
This is the equivalent of UnpackedDirectoryPathForPackage when working with the packed directory layout. It returns a path to a .zip file with a name that would be detected by SearchLocalDirectory as a PackageLocalArchive package.
This new command is intended to make it easy to create or update a mirror directory containing suitable providers for the current configuration, producing a layout that is appropriate both for a filesystem mirror or, if copied into the document root of an HTTP server, a network mirror. This initial version is not customizable aside from being able to select multiple platforms to install packages for. Future iterations of this could include commands to turn the JSON index generation on and off, or to instruct it to produce the unpacked directory layout instead of the packed directory layout as it currently does. Both of those options would make the generated directory unsuitable to be a network mirror, but it would still work as a filesystem mirror. In the long run this will hopefully form part of a replacement workflow to terraform-bundle as a way to put copies of providers somewhere so we don't need to re-download them every time, but some other changes will be needed outside of just this command before that'd be true, such as adding support for network and/or filesystem mirrors in Terraform Enterprise.
apparentlymart
force-pushed
the
f-cmd-providers-mirror
branch
from
a848c0d to
c9dc864
Compare
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
3 participants
This new command is intended to make it easy to create or update a mirror directory containing suitable providers for the current configuration, producing a layout that is appropriate both for a filesystem mirror or, if copied into the document root of an HTTP server, a network mirror. (Network mirrors are not yet supported by Terraform for installation, so that part is more aspirational but the directory structures are compatible in any case.)
This initial version is not customizable aside from being able to select multiple platforms to install packages for.
Future iterations of this could include commands to turn the JSON index generation on and off, or to instruct it to produce the unpacked directory layout instead of the packed directory layout as it currently does. Both of those options would make the generated directory unsuitable to be a network mirror, but it would still work as a filesystem mirror.
In the long run this will hopefully form part of a replacement workflow to
terraform-bundleas a way to put copies of providers somewhere so we don't need to re-download them every time, but some other changes will be needed outside of just this command before that'd be true, such as adding support for network and/or filesystem mirrors in Terraform Enterprise.(I originally wrote this as a way to make it easier for me to test filesystem and network mirrors during development, so I had originally intended it to land as part of adding support for network mirrors. However, in retrospect it seems useful in already as a helper for building filesystem mirrors, so I'm submitting it now to avoid it getting lost and forgotten in a random branch.)