Skip to content

tests: add retry conformance test cases and markdown #594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cojenco merged 8 commits into from
Sep 21, 2021
Merged

tests: add retry conformance test cases and markdown #594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f236c79
add scenario 5 to test schema
cojenco Sep 16, 2021
0d7324a
align json with conformance test repo
cojenco Sep 16, 2021
8e2d75d
add scenario 6 to test schema
cojenco Sep 16, 2021
d9084d8
sort import statements
cojenco Sep 16, 2021
8d0f2cc
add md file for retry conf tests
cojenco Sep 20, 2021
837006f
revise markdown
cojenco Sep 20, 2021
0d564ab
Merge branch 'main' into conftests5-16
cojenco Sep 20, 2021
a0873af
Merge branch 'main' into conftests5-16
cojenco Sep 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions tests/conformance/retry_conformance_testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Retry Strategy Conformance Testing

Calls fail for a host of transient reasons. In many cases the failures are ones that should be abstracted from customers, e.g. retryable issues. Retry strategies are included in the client library when it's safe to do so. These retry strategies are complex and need automated tests to ensure they work and continue to work in the future.

The Retry Strategy Conformance tests will ensure that retries are aligned across languages and operations are retried as specified.

## Test Suite Overview

The Retry Strategy Conformance tests leverage the conformance tests defined in [googleapis/conformance-tests](https://github.com/googleapis/conformance-tests/blob/master/storage/v1/retry_tests.json) to ensure adherence to expected behaviors.

The test suite uses the [storage-testbench](https://github.com/googleapis/storage-testbench)
to configure and generate tests cases which use fault injection to ensure conformance.

## Running the Conformance Test Suite

#### Prerequisites
1. Python 3.8
2. Nox
3. Docker

The Retry Strategy Conformance test suite is included in [`noxfile.py`](https://github.com/googleapis/python-storage/blob/main/noxfile.py) and run automatically as part of the Kokoro presubmits:
1. Running the testbench server via docker
2. Setup, validation, cleanup of individual test cases with the testbench
3. Test logs included in Kokoro build

To run the test suite locally:
```bash
nox -s conftest_retry-3.8
```
114 changes: 113 additions & 1 deletion tests/conformance/retry_strategy_test_data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"retryStrategyTests": [
"retryTests": [
{
"id": 1,
"description": "always_idempotent",
Expand Down Expand Up @@ -127,6 +127,118 @@
],
"preconditionProvided": false,
"expectSuccess": false
},
{
"id": 5,
"description": "non_retryable_errors",
"cases": [
{
"instructions": ["return-400"]
},
{
"instructions": ["return-401"]
}
],
"methods": [
{"name": "storage.bucket_acl.delete", "resources": ["BUCKET"]},
{"name": "storage.bucket_acl.get", "resources": ["BUCKET"]},
{"name": "storage.bucket_acl.insert", "resources": ["BUCKET"]},
{"name": "storage.bucket_acl.list", "resources": ["BUCKET"]},
{"name": "storage.bucket_acl.patch", "resources": ["BUCKET"]},
{"name": "storage.bucket_acl.update", "resources": ["BUCKET"]},
{"name": "storage.buckets.delete", "resources": ["BUCKET"]},
{"name": "storage.buckets.get", "resources": ["BUCKET"]},
{"name": "storage.buckets.getIamPolicy", "resources": ["BUCKET"]},
{"name": "storage.buckets.insert", "resources": ["BUCKET"]},
{"name": "storage.buckets.list", "resources": ["BUCKET"]},
{"name": "storage.buckets.lockRetentionPolicy", "resources": ["BUCKET"]},
{"name": "storage.buckets.patch", "resources": ["BUCKET"]},
{"name": "storage.buckets.setIamPolicy", "resources": ["BUCKET"]},
{"name": "storage.buckets.testIamPermissions", "resources": ["BUCKET"]},
{"name": "storage.buckets.update", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.delete", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.get", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.insert", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.list", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.patch", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.update", "resources": ["BUCKET"]},
{"name": "storage.hmacKey.create", "resources": []},
{"name": "storage.hmacKey.delete", "resources": ["HMAC_KEY"]},
{"name": "storage.hmacKey.get", "resources": ["HMAC_KEY"]},
{"name": "storage.hmacKey.list", "resources": ["HMAC_KEY"]},
{"name": "storage.hmacKey.update", "resources": ["HMAC_KEY"]},
{"name": "storage.notifications.delete", "resources": ["BUCKET", "NOTIFICATION"]},
{"name": "storage.notifications.get", "resources": ["BUCKET", "NOTIFICATION"]},
{"name": "storage.notifications.insert", "resources": ["BUCKET", "NOTIFICATION"]},
{"name": "storage.notifications.list", "resources": ["BUCKET", "NOTIFICATION"]},
{"name": "storage.object_acl.delete", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.object_acl.get", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.object_acl.insert", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.object_acl.list", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.object_acl.patch", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.object_acl.update", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.compose", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.copy", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.delete", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.get", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.insert", "resources": ["BUCKET"]},
{"name": "storage.objects.list", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.patch", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.rewrite", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.update", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.serviceaccount.get", "resources": []}
],
"preconditionProvided": false,
"expectSuccess": false
},
{
"id": 6,
"description": "mix_retryable_non_retryable_errors",
"cases": [
{
"instructions": ["return-503", "return-400"]
},
{
"instructions": ["return-reset-connection", "return-401"]
}
],
"methods": [
{"name": "storage.bucket_acl.get", "resources": ["BUCKET"]},
{"name": "storage.bucket_acl.list", "resources": ["BUCKET"]},
{"name": "storage.buckets.delete", "resources": ["BUCKET"]},
{"name": "storage.buckets.get", "resources": ["BUCKET"]},
{"name": "storage.buckets.getIamPolicy", "resources": ["BUCKET"]},
{"name": "storage.buckets.insert", "resources": []},
{"name": "storage.buckets.list", "resources": ["BUCKET"]},
{"name": "storage.buckets.lockRetentionPolicy", "resources": ["BUCKET"]},
{"name": "storage.buckets.patch", "resources": ["BUCKET"]},
{"name": "storage.buckets.setIamPolicy", "resources": ["BUCKET"]},
{"name": "storage.buckets.testIamPermissions", "resources": ["BUCKET"]},
{"name": "storage.buckets.update", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.get", "resources": ["BUCKET"]},
{"name": "storage.default_object_acl.list", "resources": ["BUCKET"]},
{"name": "storage.hmacKey.delete", "resources": ["HMAC_KEY"]},
{"name": "storage.hmacKey.get", "resources": ["HMAC_KEY"]},
{"name": "storage.hmacKey.list", "resources": ["HMAC_KEY"]},
{"name": "storage.hmacKey.update", "resources": ["HMAC_KEY"]},
{"name": "storage.notifications.delete", "resources": ["BUCKET", "NOTIFICATION"]},
{"name": "storage.notifications.get", "resources": ["BUCKET", "NOTIFICATION"]},
{"name": "storage.notifications.list", "resources": ["BUCKET", "NOTIFICATION"]},
{"name": "storage.object_acl.get", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.object_acl.list", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.compose", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.copy", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.delete", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.get", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.list", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.insert", "resources": ["BUCKET"]},
{"name": "storage.objects.patch", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.rewrite", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.objects.update", "resources": ["BUCKET", "OBJECT"]},
{"name": "storage.serviceaccount.get", "resources": []}
],
"preconditionProvided": true,
"expectSuccess": false
}
]
}
15 changes: 7 additions & 8 deletions tests/conformance/test_conformance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,16 @@

"""Conformance tests for retry. Verifies correct behavior around retryable errors, idempotency and preconditions."""

import functools
import logging
import os
import requests
import subprocess
import tempfile
import time
import uuid
import logging
import functools

import pytest
import subprocess
import time
import requests

from six.moves.urllib import parse as urlparse

Expand All @@ -34,9 +35,7 @@
from . import _read_local_json


_CONFORMANCE_TESTS = _read_local_json("retry_strategy_test_data.json")[
"retryStrategyTests"
]
_CONFORMANCE_TESTS = _read_local_json("retry_strategy_test_data.json")["retryTests"]

"""Environment variable or default host for Storage testbench emulator."""
_HOST = os.environ.get("STORAGE_EMULATOR_HOST", "http://localhost:9000")
Expand Down