Git Stub Downloader

[marcingrzejszczak]

Description from the docs

Do I need a Binary Storage? Can't I use Git?

In the polyglot world, there are languages that don't use binary storages like
Artifactory or Nexus. Starting from Spring Cloud Contract version 2.0.0 we provide
mechanisms to store contracts and stubs in a SCM repository. Currently the
only supported SCM is Git.

The repository would have to the following setup
(which you can checkout here https://github.com/spring-cloud-samples/spring-cloud-contract-samples/tree/2.0.x/contracts_git/):

.
└── META-INF
    └── com.example
        └── beer-api-producer-git
            └── 0.0.1-SNAPSHOT
                ├── contracts
                │   └── beer-api-consumer
                │       ├── messaging
                │       │   ├── shouldSendAcceptedVerification.groovy
                │       │   └── shouldSendRejectedVerification.groovy
                │       └── rest
                │           ├── shouldGrantABeerIfOldEnough.groovy
                │           └── shouldRejectABeerIfTooYoung.groovy
                └── mappings
                    └── beer-api-consumer
                        └── rest
                            ├── shouldGrantABeerIfOldEnough.json
                            └── shouldRejectABeerIfTooYoung.json

Under META-INF folder:

• we group applications via groupId (e.g. com.example)
• then each application is represented via the artifactId (e.g. beer-api-producer-git)
• next, the version of the application. The version is mandatory! (e.g. 0.0.1-SNAPSHOT)
• Finally, there are two folders:
  ** contracts - the good practice is to store the contracts required by each
  consumer in the folder with the consumer name (e.g. beer-api-consumer). That way you
  can use the stubs-per-consumer feature. Further directory structure is arbitrary.
  ** mappings - in this folder the Maven / Gradle Spring Cloud Contract plugins will push
  the stub server mappings. On the consumer side, Stub Runner will scan this folder
  to start stub servers with stub definitions. The folder structure will be a copy
  of the one created in the contracts subfolder.

Protocol convention

In order to control the type and location of the source of contracts (whether it's
a binary storage or an SCM repository), you can use the protocol in the URL of
the repository. Spring Cloud Contract iterates over registered protocol resolvers
and tries to fetch the contracts (via a plugin) or stubs (via Stub Runner).

For the SCM functionality, currently, we support the Git repository. To use it,
in the property, where the repository URL needs to be placed you just have to prefix
the connection URL with git://. Here you can find a couple of examples:

git://file:///foo/bar
git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git
git://git@github.com:spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git

Producer

For the producer, to use the SCM approach, we can reuse the same mechanism we use for external contracts. We route Spring Cloud Contract to use the SCM implementation via the URL that contains the git:// protocol.

IMPORTANT: You have to manually add the pushStubsToScm goal in Maven or execute (bind) the pushStubsToScm task in Gradle. We don't push stubs to origin of your git repository out of the box.

.Maven

<plugin>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-contract-maven-plugin</artifactId>
    <version>${spring-cloud-contract.version}</version>
    <extensions>true</extensions>
    <configuration>
        <!-- Base class mappings etc. -->

        <!-- We want to pick contracts from a Git repository -->
        <contractsRepositoryUrl>git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git</contractsRepositoryUrl>

        <!-- We reuse the contract dependency section to set up the path
        to the folder that contains the contract definitions. In our case the
        path will be /groupId/artifactId/version/contracts -->
        <contractDependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>${project.artifactId}</artifactId>
            <version>${project.version}</version>
        </contractDependency>

        <!-- The contracts mode can't be classpath -->
        <contractsMode>REMOTE</contractsMode>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <!-- By default we will not push the stubs back to SCM,
                you have to explicitly add it as a goal -->
                <goal>pushStubsToScm</goal>
            </goals>
        </execution>
    </executions>
</plugin>

.Gradle

contracts {
	// We want to pick contracts from a Git repository
	contractDependency {
		stringNotation = "${project.group}:${project.name}:${project.version}"
	}
	/*
	We reuse the contract dependency section to set up the path
	to the folder that contains the contract definitions. In our case the
	path will be /groupId/artifactId/version/contracts
	 */
	contractRepository {
		repositoryUrl = "git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git"
	}
	// The mode can't be classpath
	contractsMode = "REMOTE"
	// Base class mappings etc.
}

/*
In this scenario we want to publish stubs to SCM whenever
the `publish` task is executed
*/
publish.dependsOn("publishStubsToScm")

With such a setup:

• Git project will be cloned to a temporary directory
• The SCM stub downloader will go to META-INF/groupId/artifactId/version/contracts folder
  to find contracts. E.g. for com.example:foo:1.0.0 the path would be
  META-INF/com.example/foo/1.0.0/contracts
• Tests will be generated from the contracts
• Stubs will be created from the contracts
• Once the tests pass, the stubs will be committed in the cloned repository
• Finally, a push will be done to that repo's origin

Consumer

On the consumer side when passing the repositoryRoot parameter,
either from the @AutoConfigureStubRunner annotation, the
JUnit rule or properties, it's enough to pass the URL of the
SCM repository, prefixed with the protocol. For example

@AutoConfigureStubRunner(
    stubsMode="REMOTE",
    repositoryRoot="git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git",
    ids="com.example:bookstore:0.0.1.RELEASE"
)

With such a setup:

• Git project will be cloned to a temporary directory
• The SCM stub downloader will go to META-INF/groupId/artifactId/version/ folder
  to find stub definitions and contracts. E.g. for com.example:foo:1.0.0 the path would be
  META-INF/com.example/foo/1.0.0/
• Stub servers will be started and fed with mappings
• Messaging definitions will be read and used in the messaging tests

Using the SCM Stub Downloader

Whenever the repositoryRoot starts with a SCM protocol (currently we support only git://), the stub downloader will try to clone the repository and use it as a source of contracts to generate tests or stubs.

Either via environment variables, system properties, properties set inside the plugin or contracts repository configuration you can tweak the downloader's behaviour. Below you can find the list of properties

Technical issues

The approach

when stubsMode is set to LOCAL or REMOTE, and repositoryRoot starts with git:// we can clone the provided git repository, and search for the folder with stubs for the given artifact. So if the git repo has a folder structure of groupid/artifactid/version (where group id is either dot or slash separated), then we will provide a path to that repository for the stub runner to harvest the stubs

What's done?

• externalized versions
• added more debugging messages
• added ContractProjectUpdater that updates the project containing contracts from SCM. ATM supports only git
• added ResourceResolver that retrieves the ProtocolResolvers. It does it via spring.factories entries containing StubDownloaderBuilder. SDP extends ProtocolResovler.
• added StubRunner.properties map, that will contain any properties that will be later used by any StubDownloader implementations
• added PUBLISH_STUBS_TO_SCM env var for Docker, so that publishStubsToScm task gets called
• updated docs

Breaking:

• StubDownloaderBuilder extends ProtocolResovler. By default the ProtocolResolver methods return null.
• stubRunnerOptions.stubRepositoryRoot is a Resource not a String
• generateWireMockClientStubs Gradle task got removed
• if folder with contracts has a subfolder called contracts, we will pick contracts from the subfolder

fixes #580

[marcingrzejszczak]

parametrize

[codecov-io]

Codecov Report

Merging #596 into master will increase coverage by 0.09%.
The diff coverage is 62.12%.

@@             Coverage Diff              @@
##             master     #596      +/-   ##
============================================
+ Coverage     60.88%   60.98%   +0.09%     
- Complexity     1760     1826      +66     
============================================
  Files           203      210       +7     
  Lines          7033     7507     +474     
  Branches       1063     1121      +58     
============================================
+ Hits           4282     4578     +296     
- Misses         2154     2291     +137     
- Partials        597      638      +41

Impacted Files	Coverage Δ	Complexity Δ
.../stubrunner/spring/cloud/StubMapperProperties.java	89.47% <ø> (ø)	8 <0> (ø)	⬇️
...ork/cloud/contract/stubrunner/AetherFactories.java	83.78% <ø> (ø)	10 <0> (ø)	⬇️
...rk/cloud/contract/stubrunner/util/StubsParser.java	77.41% <ø> (ø)	5 <0> (ø)	⬇️
...pring/cloud/ribbon/StubRunnerRibbonServerList.java	63.63% <ø> (ø)	4 <0> (ø)	⬇️
...runner/spring/cloud/StubRunnerDiscoveryClient.java	16.98% <ø> (ø)	3 <0> (ø)	⬇️
...oud/contract/stubrunner/ClasspathStubProvider.java	5.43% <0%> (ø)	3 <0> (ø)	⬇️
...work/cloud/contract/stubrunner/StubRepository.java	61.42% <0%> (-2.86%)	17 <0> (ø)
.../contract/verifier/file/ContractFileScanner.groovy	64.19% <0%> (ø)	35 <0> (ø)	⬇️
...rifier/stubrunner/AetherStubDownloaderFactory.java	50% <0%> (-5.56%)	2 <0> (ø)
...loud/contract/stubrunner/junit/StubRunnerRule.java	48.27% <0%> (-1.14%)	16 <0> (ø)
... and 32 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6a8f49...488cbb0. Read the comment docs.

[jorahood]

I need to configure a CredentialsProvider to provide username and password for downloading from a private Git repo that requires https.

[marcingrzejszczak]

Have you read the docs? https://cloud.spring.io/spring-cloud-contract/single/spring-cloud-contract.html#scm-stub-downloader