Skip to content

Commit c7d43dc

Browse files
seebeesseebees
authored
feat: Adding test vector spec to MPL (#263)
Test vectors are important both for compatibility but also to help reason about correctness. Having empirical tests gives confidence to our proofs.
1 parent 83f5631 commit c7d43dc

19 files changed

+1572
-0
lines changed

‎framework/test-vectors/README.md

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
2+
[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
3+
4+
## Summary
5+
6+
These test vectors are intended to be used to validate interoperability
7+
across implementations of clients in a runtime,
8+
for the AWS Crypto Tools AWS Cryptographic Material Providers Library (MPL).
9+
10+
They are composed of JSON manifest files that define one or more test cases,
11+
including sufficient information to process each test case.
12+
These manifest files can also identify additional resources needed for a given test case.
13+
These resources will be identified with a URI.
14+
If identifying a local file, the URI will be a relative path from the manifest file's
15+
parent directory to the target file.
16+
17+
Every manifest type definition must include a unique type name that will be used by
18+
test vector handlers to identify the manifest type.
19+
20+
See https://github.com/awslabs/aws-crypto-tools-test-vector-framework for additional information.
21+
This framework is much of the motivation for this work.
22+
23+
The manifest is meant to produce encryption and decryption materials that every
24+
runtime implementation can process and produce the desired output for that test vector.
25+
26+
These test vectors are not meant to be processed by a top level client library like
27+
the AWS Encryption SDK or the AWS Database Encryption SDK. However; they may be able
28+
to use the manifest to see how to construct keyrings that they can use in their test
29+
vectors.
30+
31+
```mermaid
32+
flowchart LR
33+
subgraph MPL Encryption-Materials
34+
direction LR
35+
MPL-Manifest --> MPL-Java -->|MPL-TestVector|EncryptionMaterials
36+
MPL-Manifest --> MPL-NET -->|MPL-TestVector|EncryptionMaterials
37+
MPL-Manifest --> MPL-X -->|MPL-TestVector|EncryptionMaterials
38+
end
39+
```
40+
41+
```mermaid
42+
flowchart LR
43+
subgraph MPL Decryption-Materials
44+
direction LR
45+
MPL-Manifest+MPL-TestVector --> MPL-Java -->|MPL-TestVector|DecryptionMaterials
46+
MPL-Manifest+MPL-TestVector --> MPL-NET -->|MPL-TestVector|DecryptionMaterials
47+
MPL-Manifest+MPL-TestVector --> MPL-X -->|MPL-TestVector|DecryptionMaterials
48+
end
49+
```
50+
51+
```mermaid
52+
flowchart LR
53+
subgraph ESDK Usage of MPL Manifest
54+
direction LR
55+
MPL-Manifest-. Keyring-Construction .-> ESDK-Java --> ESDK-Manifest
56+
57+
MPL-Manifest-. Keyring-Construction .-> ESDK-NET --> ESDK-Manifest
58+
end
59+
```
60+
61+
## Glossary
62+
63+
- **Test Vector** : Information about a single test case. Used to either process existing data
64+
or create new data.
65+
- **Test Vector Manifest** : A document that describes one or more test vectors.
66+
67+
## Out of Scope
68+
69+
This file is not a definition or description of any specific type of test vector or manifest.
70+
71+
## Motivation
72+
73+
The AWS Crypto Tools team has built several tools that require multiple implementations.
74+
Interoperability between these implementations and runtimes is critical.
75+
To ensure that these implementations are actually interoperable,
76+
we needed to define test vectors that would allow validation
77+
of various aspects of these tools.
78+
As we built more tools and required more types of test vectors,
79+
it became evident that there would be value in defining an extensible framework
80+
for defining many different types of test vectors to avoid having to reinvent the wheel
81+
for every client.
82+
83+
We also need to be able to maintain backwards compatibility.
84+
By producing manifests that can be stored
85+
we can ensure that new clients can decrypt old messages.
86+
87+
The latest version of the MPL being written in Dafny
88+
and much of the design requirements are proven by Dafny specifications.
89+
These specifications are valuable but having empirical tests
90+
makes this proof even stronger.
91+
Because you can write and prove a specification
92+
but it may not be correct.
93+
This is especially true for backwards compatibility.
94+
But even for code reviews and understanding the impact of a change,
95+
looking at how test vectors are created helps reason about the edge cases of a change.
96+
97+
## Drawbacks
98+
99+
Not every configuration can be practically tested.
100+
See [test vector enumeration](test-vector-enumeration.md#selecting-a-representative-input-value) for details.
101+
102+
We will need to write minimal clients in every language with which we want to use these test
103+
vectors to understand the manifests described in subsequent features.
104+
105+
This should represent acceptable overhead: we would need to write some amount of code for each
106+
language to handle the test vectors anyway and this framework lets us define a consistent
107+
way of handling those test vectors while remaining simple to process.
108+
109+
Additionally, using Dafny can lower this overhead.
110+
The challenge now is to compose the proper level of reusability.
111+
The MPL is the base library for many encryption clients
112+
and there may be some work to be able to reuse parts of these test vectors
113+
in other libraries.

‎framework/test-vectors/complete-vectors/README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
2+
[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
3+
4+
## Summary
5+
6+
For each type of keyring or cmm description,
7+
there is a file that describes how this component is tested.
8+
This simplifies reasoning about each component.
9+
For example, to find out how the DefaultCMM is tested,
10+
look at the default-cmm.md file
11+
that describes how this component is tested
12+
and the various features that can be reasoned about.
13+
14+
The test vectors MUST aggregate all these individual tests together
15+
into larger manifest files.
16+
17+
Unless otherwise specified, all "Basic Tests" MUST use a `DefaultCMM`
18+
19+
## Motivation
20+
21+
By keeping every component separate it is easier to find
22+
and reason about the completeness of testing.
23+
This also allows top level clients like the ESDK or DBESDK to
24+
reuse the key or cmm description to avoid reinventing the wheel.

‎framework/test-vectors/complete-vectors/default-cmm.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
2+
[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
3+
4+
# Default CMM Vectors
5+
6+
## Version
7+
8+
1.0.0
9+
10+
## Summary
11+
12+
This describes the test cases for the [Default CMM](../../default-cmm.md)
13+
14+
## Reference-level Explanation
15+
16+
### Basic tests
17+
18+
A test MUST verify that on both encrypt and decrypt the correct
19+
plaintext data key is produced.
20+
21+
A test MUST verify that an encrypt keyring that returns
22+
an incorrect plaintext data key will fail.
23+
24+
A test MUST verify that a decrypt keyring that returns
25+
an incorrect plaintext data key will fail.
26+
27+
### Required and reproduced encryption context success cases
28+
29+
For a given [encryption context](../../structures.md#encryption-context),
30+
every subset of the keys for this encryption context
31+
MUST be attempted as the `requiredEncryptionContextKeys`.
32+
For example, `{ a: a, b: b }` produces the complete set of keys subsets: `{ {}, { a }, { b }, { a, b } }` called `requiredEncryptionContextKeys`.
33+
34+
For every `requiredEncryptionContextKeys` produced above
35+
`reproducedEncryptionContext` MUST be attempted
36+
for every subset of the encryption context
37+
where the `requiredEncryptionContextKeys` is a subset of the attempted subset.
38+
39+
For example:
40+
41+
- Given an empty `requiredEncryptionContextKeys`,
42+
every combination of the original encryption context
43+
will succeed as `reproducedEncryptionContext`.
44+
- Given a `requiredEncryptionContextKeys` consisting of `{a}` the
45+
`reproducedEncryptionContext` to try would be: - `{ a : a }` and `{ a : a, b : b}`
46+
47+
### Required encryption context keys failures on encrypt
48+
49+
For a given [encryption context](../../structures.md#encryption-context),
50+
every subset of the keys for this encryption context
51+
MUST be attempted as the `requiredEncryptionContextKeys`.
52+
The keys of the encryption context attempted however
53+
where the `requiredEncryptionContextKeys` MUST NOT be a subset of the attempted subset.
54+
55+
For example:
56+
57+
- Given an `encryptionContext`: `{a:a, b:b, c:c}` will produce the subset
58+
of `requiredEncryptionContextKeys`, `{{}, {a}, {b}, {c}, {a,b}, {a,c), {b,c}, {a,b,c}}`
59+
- Given a `requiredEncryptionContextKeys` consisting of `{a}` the
60+
`reproducedEncryptionContext` to try would be:
61+
- `{b:b, c:c}`
62+
- Given a `requiredEncryptionContextKeys` consisting of `{a,b}` the
63+
`reproducedEncryptionContext` to try would be:
64+
- `{}`, `{c:c}`, and `{a:a, c:c}`
65+
- Given a `requiredEncryptionContextKeys` consisting of `{a,b,c}` the
66+
`reproducedEncryptionContext` to try would be:
67+
- `{}`
68+
69+
### Reproduced encryption context failures on decrypt
70+
71+
For a given [encryption context](../../structures.md#encryption-context),
72+
every subset of the keys for this encryption context
73+
MUST be attempted as the `requiredEncryptionContextKeys`.
74+
An incorrect encryption context
75+
MUST be attempted that differs from the correct encryption context
76+
by both values and keys.
77+
78+
For example given the encryption context `{ a: a, b: b }`,
79+
all possible combinations that are not empty of `{ a: c, b: c, c: c}`.
80+
The test MUST attempt the union of every incorrect encryption context
81+
with every subset of the original encryption context.

‎framework/test-vectors/complete-vectors/encryption-context.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
2+
[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
3+
4+
# Encryption Context Vectors
5+
6+
## Version
7+
8+
1.0.0
9+
10+
## Summary
11+
12+
This is a description of the standard encryption contexts to test.
13+
14+
## Reference-level Explanation
15+
16+
### Standard Encryption Contexts Constraints
17+
18+
- MUST have an empty map.
19+
- The number of the items in the empty map MUST equal 0.
20+
- MUST have a small map.
21+
- The number of the items in the small map MUST be between 1 and 10.
22+
- MUST have a large map.
23+
- The number of the items in the large map MUST be greater than 10.
24+
- MUST have multibyte UTF8 characters in both the key and value.
25+
- MUST have multibyte non-BMP characters in both the key and value.

‎framework/test-vectors/complete-vectors/hierarchy.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
2+
[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
3+
4+
# Aws Kms Hierarchical Keyring Vectors
5+
6+
## Version
7+
8+
1.0.0
9+
10+
## Summary
11+
12+
This describes the test cases for the [Aws Kms Hierarchical Keyring](../../aws-kms/aws-kms-hierarchical-keyring.md)
13+
14+
## Reference-level Explanation
15+
16+
### Basic tests
17+
18+
For a given static branch key,
19+
a test MUST attempt to encrypt and decrypt
20+
with every available [algorithm suite](../../algorithm-suites.md#algorithm-suite-id)
21+
22+
For every available algorithm suite and static branch key,
23+
a test MUST attempt to encrypt and decrypt with every [standard encryption context](./encryption-context.md#standard-encryption-contexts).
24+
25+
### Input dimensions
26+
27+
#### Encrypt
28+
29+
- key description
30+
- key: range of [representative branch keys](#representative-branch-keys)
31+
32+
#### Decrypt
33+
34+
- key description
35+
- key: range of [representative branch keys](#representative-branch-keys)
36+
- logicalKeyStore:
37+
- "Default",
38+
- Represents the logical key store name on encrypt
39+
- "Other"
40+
- Represents any other logical key store name
41+
42+
### Representative branch keys
43+
44+
- `"static-branch-key-1"`
45+
- MUST be some valid branch key.
46+
- `"static-branch-key-2"`
47+
- MUST be some valid branch key other than `static-branch-key-1`.
48+
- `"branch-key-no-permissions"`
49+
- MUST be some valid branch key where the test vector runner does not have permissions for the KMS key.
50+
- `"branch-key-not-in-table"`
51+
- MUST be some branch key ID not present in the keystore table.,
52+
- `"branch-key-no-version"`
53+
- MUST be some branch key that is in the table, but the configured version is not in the table.
54+
- `"invalid-branch-key-material"`
55+
- MUST be some branch key with illegally mutated invalid branch key material.
56+
57+
### Evaluation rules
58+
59+
- If encrypting key type is anything other than `"aws-kms-hierarchy"`
60+
and decrypting key type is `"aws-kms-hierarchy"`,
61+
the result MUST be `"negative-decrypt"`.
62+
- If encrypting key type is `"aws-kms-hierarchy"`
63+
and decrypting key type is anything other than `"aws-kms-hierarchy"`,
64+
the result MUST be `"negative-decrypt"`.
65+
- If the logical key store is "Other",
66+
the result MUST be `"negative-decrypt"`.
67+
- If `"key"` is different on encrypt and decrypt,
68+
the result MUST be `"negative-decrypt"`.
69+
(i.e. no key specified here is interoperable with any other key.)
70+
- If `"key"` is any of:
71+
- `"branch-key-no-permissions"`
72+
- `"branch-key-not-in-table"`
73+
- `"branch-key-no-version"`
74+
- `"invalid-branch-key"`
75+
(i.e. an "invalid" key with a particular invalid condition)
76+
on either encrypt or decrypt,
77+
the result MUST be either `"negative-encrypt"` or `"negative-decrypt"`,
78+
depending on whether the invalid key was specified on encrypt or decrypt.
79+
- In all other cases, the result MUST be `"positive"`.

‎framework/test-vectors/complete-vectors/kms-mrk-aware-discovery.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
2+
[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
3+
4+
# Aws Kms Mrk Discovery Keyring Vectors
5+
6+
## Version
7+
8+
1.0.0
9+
10+
## Summary
11+
12+
This describes the test cases for the [Aws Kms Mrk Discovery Keyring](../../aws-kms/aws-kms-mrk-discovery-keyring.md)
13+
14+
## Reference-level Explanation
15+
16+
### Basic tests
17+
18+
A test MUST attempt to encrypt with a single region key.
19+
A test MUST attempt to encrypt with a multi region key.
20+
A test MUST attempt to decrypt with a discovery filter and without a filter.
21+
A test MUST attempt to encrypt and decrypt
22+
with every available [algorithm suite](../../algorithm-suites.md#algorithm-suite-id)
23+
A test MUST attempt every [standard encryption context](./encryption-context.md#standard-encryption-contexts).

‎framework/test-vectors/complete-vectors/kms-mrk-aware.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
2+
[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
3+
4+
# Aws Kms Mrk Keyring Vectors
5+
6+
## Version
7+
8+
1.0.0
9+
10+
## Summary
11+
12+
This describes the test cases for the [Aws Kms Mrk Keyring](../../aws-kms/aws-kms-mrk-keyring.md)
13+
14+
## Reference-level Explanation
15+
16+
### Basic tests
17+
18+
A test MUST attempt to encrypt and decrypt
19+
with every combination of 2 MRKs in different regions.
20+
A test MUST attempt to decrypt with only one of the MRKs used
21+
to encrypt.
22+
A test MUST attempt to encrypt and decrypt
23+
with every available [algorithm suite](../../algorithm-suites.md#algorithm-suite-id)
24+
A test MUST attempt every [standard encryption context](./encryption-context.md#standard-encryption-contexts).

0 commit comments

Comments
 (0)