#28/2.7.0 Missing dependency

[atward]

Version 2.7.0 dependencies broken

terraform_validate.py", line 9, in <module>
    import git
ModuleNotFoundError: No module named 'git'

setup.py does not reference requirements.txt. https://github.com/elmundio87/terraform_validate/blob/master/setup.py#L13

[elmundio87]

Thanks for catching this @atward - I've released 2.7.1 to fix this. Can you let me know if upgrading solves your issue?

[atward]

2.7.1 is now producing errors of the form:

>           raise GitCommandError(command, status, stderr_value, stdout_value)
E           git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
E             cmdline: git clone ../../modules/efs /tmp/terraform_validate/efs
E             stderr: 'fatal: repository '../../modules/efs' does not exist'

../../../.conda/envs/ci/lib/python3.6/site-packages/git/cmd.py:820: GitCommandError

Given TF provisioner of the form (provisioners/foo/blah.tf)

module "efs" {
  source               = "../../modules/efs"
}

And repo structure:

├── environments
├── modules
│   ├── efs
│   └── ...
├── provisioners
│   ├── our_stuff
│   └── ...
└── test
    └── conftest.py

Relevant stack trace from pytest fixture (where terraform_path is subdirectory of provisioners/):

conftest.py:20: in validator
    v = terraform_validate.Validator(terraform_path)
../../../.conda/envs/ci/lib/python3.6/site-packages/terraform_validate/terraform_validate.py:442: in __init__
    self.terraform_config = self.parse_terraform_directory(path)
../../../.conda/envs/ci/lib/python3.6/site-packages/terraform_validate/terraform_validate.py:546: in parse_terraform_directory
    modules_to_process = self.check_terraform_for_modules(new_terraform)
../../../.conda/envs/ci/lib/python3.6/site-packages/terraform_validate/terraform_validate.py:528: in check_terraform_for_modules
    modules_to_process.append(self.get_git_module(value))
../../../.conda/envs/ci/lib/python3.6/site-packages/terraform_validate/terraform_validate.py:506: in get_git_module
    repo.clone(source, directory) ## by default, this checks out master.
../../../.conda/envs/ci/lib/python3.6/site-packages/git/cmd.py:550: in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
../../../.conda/envs/ci/lib/python3.6/site-packages/git/cmd.py:1009: in _call_process
    return self.execute(call, **exec_kwargs)

I believe there's logical issues with #28 - does it assume all modules are git repos?

Additionally, the fact that it's hard-coded to clone to /tmp/terraform_validate/name will fail when running concurrent tests with modules of the same name.

My recommendation is to revert #28 until it can be developed further.

[elmundio87]

Agreed @atward

@melbit-jonathanb I'm going to revert PR #28 just so the latest release is working for everyone. If the PR can be updated to work around the issues that @atward has uncovered, then I'm happy to bring it back into the master branch.

[elmundio87]

Latest release 2.8.0 should now match the functionality of the release prior to #28

[melbit-jonathanb]

Thanks for the feedback.

The change made only assumes that the repo is git compatible if the source url contains ".git" or "::git". It assumes those that start with "." or "/" are on the local filesystem. I don't personally use any other kinds of TF modules, so I haven't had the scope to add the functionality. I'll look further into it, but the logic should just add the path to the list of hcl to read if it begins with ".."

The first error @atward provided with 2.7.1 is a touch confusing for me because the "value" variable which is being assessed contains "../../modules/efs" which does not contain ".git" or "::git". I'll attempt to reproduce and correct.

I'll also look at using a unique name for each module - good catch.

[melbit-jonathanb]

@atward - are you using a mixture of git and local file system modules?

[atward]

@melbit-jonathanb one large git repo with directory layout outlined in #30 (comment). All references are to directories within the large repo.

Should https://github.com/elmundio87/terraform_validate/pull/28/files#diff-359c88e8fa0ba94f822694726b15bb08R527 be elif?

https://github.com/elmundio87/terraform_validate/pull/28/files#diff-359c88e8fa0ba94f822694726b15bb08R550-553 is also incorrect logic for terraform. Terraform is all *.tf files in the directory - it does not include all subdirectories. The linked diff lines could be replaced with a single for ... in glob.glob(os.path.join(module_directory, '*.tf')). This logical error applies to the main os.walk in the same function

https://www.terraform.io/docs/configuration/load.html
When invoking any command that loads the Terraform configuration, Terraform loads all configuration files within the directory specified in alphabetical order.

[melbit-jonathanb]

Thanks again.

I've been looking at it and had come up with a solution; however at my work we have repos which reference modules in other repos and they reference other repos, so my testing was not working. I've though about how to address it and will be making changes to address this.