Skip to content

Handle provisioning failures better? #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jml opened this issue Oct 18, 2016 · 1 comment
Closed

Handle provisioning failures better? #55

jml opened this issue Oct 18, 2016 · 1 comment

Comments

@jml
Copy link
Contributor

jml commented Oct 18, 2016

We sometimes get a bunch of messages like these:

time="2016-10-18T16:16:52Z" level=error msg="Failed to flush chunks for series: ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.\n\tstatus code: 400, request id: REDACTED" source="ingester.go:442"

Can we handle this case better?

  • back off?
  • instrument this with metrics?
  • increase our provisioning level?
@tomwilkie
Copy link
Contributor

Added backoff and much more instrumentation, so I think we can close this.

tomwilkie added a commit that referenced this issue Jan 12, 2017
@tomwilkie
Squashed 'tools/' changes from fd875e2..ed8e757
c0e571c 
ed8e757 Merge pull request #61 from weaveworks/test-defaults
57856e6 Merge pull request #56 from weaveworks/remove-wcloud
dd5f3e6 Add -p flag to test, run test in parallel
62f6f94 Make no-go-get the default, and don't assume -tags netgo
8946588 Merge pull request #60 from weaveworks/2647-gc-weave-net-tests
4085df9 Scheduler now also garbage-collects VMs from weave-net-tests.
4b7d5c6 Merge pull request #59 from weaveworks/57-fix-lint-properly
b7f0e69 Merge pull request #58 from weaveworks/fix-lint
794702c Pin version of shfmt
ab1b11d Fix lint
d1a5e46 Remove wcloud cli tool
81d80f3 Merge pull request #55 from weaveworks/lint-tf
05ad5f2 Review feedback
4c0d046 Use hclfmt to lint terraform.

git-subtree-dir: tools
git-subtree-split: ed8e757406d337c234468febfecdb8a51d6a5bab
tomwilkie added a commit that referenced this issue Jan 16, 2017
@tomwilkie
Squashed 'tools/' changes from fd875e2..2b3a1bb
c923cd3 
2b3a1bb Merge pull request #62 from weaveworks/revert-61-test-defaults
8c3883a Revert "Make no-go-get the default, and don't assume -tags netgo"
e75c226 Fix bug in GC of firewall rules.
e49754e Merge pull request #51 from weaveworks/gc-firewall-rules
191f487 Add flag to enale/disable firewall rules' GC.
567905c Add GC of firewall rules for weave-net-tests to scheduler.
03119e1 Fix typo in GC of firewall rules.
bbe3844 Fix regular expression for firewall rules.
c5c23ce Pre-change refactoring: splitted gc_project function into smaller methods for better readability.
ed5529f GC firewall rules
ed8e757 Merge pull request #61 from weaveworks/test-defaults
57856e6 Merge pull request #56 from weaveworks/remove-wcloud
dd5f3e6 Add -p flag to test, run test in parallel
62f6f94 Make no-go-get the default, and don't assume -tags netgo
8946588 Merge pull request #60 from weaveworks/2647-gc-weave-net-tests
4085df9 Scheduler now also garbage-collects VMs from weave-net-tests.
4b7d5c6 Merge pull request #59 from weaveworks/57-fix-lint-properly
b7f0e69 Merge pull request #58 from weaveworks/fix-lint
794702c Pin version of shfmt
ab1b11d Fix lint
d1a5e46 Remove wcloud cli tool
81d80f3 Merge pull request #55 from weaveworks/lint-tf
05ad5f2 Review feedback
4c0d046 Use hclfmt to lint terraform.

git-subtree-dir: tools
git-subtree-split: 2b3a1bbfd122056e51212ee2a53970c9390d2aa7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jml @tomwilkie