Insufficient visibility into distributor ring state #60
Comments
This was referenced Oct 20, 2016
5 tasks
|
"admin page" became #94 |
tomwilkie
added a commit
that referenced
this issue
Jan 12, 2017
ed8e757 Merge pull request #61 from weaveworks/test-defaults 57856e6 Merge pull request #56 from weaveworks/remove-wcloud dd5f3e6 Add -p flag to test, run test in parallel 62f6f94 Make no-go-get the default, and don't assume -tags netgo 8946588 Merge pull request #60 from weaveworks/2647-gc-weave-net-tests 4085df9 Scheduler now also garbage-collects VMs from weave-net-tests. 4b7d5c6 Merge pull request #59 from weaveworks/57-fix-lint-properly b7f0e69 Merge pull request #58 from weaveworks/fix-lint 794702c Pin version of shfmt ab1b11d Fix lint d1a5e46 Remove wcloud cli tool 81d80f3 Merge pull request #55 from weaveworks/lint-tf 05ad5f2 Review feedback 4c0d046 Use hclfmt to lint terraform. git-subtree-dir: tools git-subtree-split: ed8e757406d337c234468febfecdb8a51d6a5bab
tomwilkie
added a commit
that referenced
this issue
Jan 16, 2017
2b3a1bb Merge pull request #62 from weaveworks/revert-61-test-defaults 8c3883a Revert "Make no-go-get the default, and don't assume -tags netgo" e75c226 Fix bug in GC of firewall rules. e49754e Merge pull request #51 from weaveworks/gc-firewall-rules 191f487 Add flag to enale/disable firewall rules' GC. 567905c Add GC of firewall rules for weave-net-tests to scheduler. 03119e1 Fix typo in GC of firewall rules. bbe3844 Fix regular expression for firewall rules. c5c23ce Pre-change refactoring: splitted gc_project function into smaller methods for better readability. ed5529f GC firewall rules ed8e757 Merge pull request #61 from weaveworks/test-defaults 57856e6 Merge pull request #56 from weaveworks/remove-wcloud dd5f3e6 Add -p flag to test, run test in parallel 62f6f94 Make no-go-get the default, and don't assume -tags netgo 8946588 Merge pull request #60 from weaveworks/2647-gc-weave-net-tests 4085df9 Scheduler now also garbage-collects VMs from weave-net-tests. 4b7d5c6 Merge pull request #59 from weaveworks/57-fix-lint-properly b7f0e69 Merge pull request #58 from weaveworks/fix-lint 794702c Pin version of shfmt ab1b11d Fix lint d1a5e46 Remove wcloud cli tool 81d80f3 Merge pull request #55 from weaveworks/lint-tf 05ad5f2 Review feedback 4c0d046 Use hclfmt to lint terraform. git-subtree-dir: tools git-subtree-split: 2b3a1bbfd122056e51212ee2a53970c9390d2aa7
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When debugging the distributor, ring state is pretty important. We have some metrics like
prometheus_distributor_ingesters_total,prometheus_distributor_ingester_clients, andprometheus_distributor_ingester_ownership_percent, but these don't distinguish between ingesters that have had a recent heartbeat and those that haven't.It's possible that the "fix" for this is to just clean up old ingesters properly, but I think even then I'd like either:
The text was updated successfully, but these errors were encountered: