Skip to content

Ingesters get network errors when writing dynamo data during shutdown #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jml opened this issue Oct 10, 2016 · 2 comments
Closed

Ingesters get network errors when writing dynamo data during shutdown #45

jml opened this issue Oct 10, 2016 · 2 comments
Labels
type/bug
Milestone
KubeCon 2016

Comments

@jml
Copy link
Contributor

jml commented Oct 10, 2016 •
edited
Loading 

time="2016-10-10T13:17:29Z" level=error msg="Failed to flush chunks for series: RequestError: send request failed\ncaused by: Put https://weaveworks-prod-chunks.s3.amazonaws.com/2/17142991289174007630%3A1476104540889%3A1476105230889: dial tcp: lookup weaveworks-prod-chunks.s3.amazonaws.com on 10.0.0.10:53: dial udp 10.0.0.10:53: connect: network is unreachable" source="cortex.go:440"

This might be a cause of data loss.
@jml jml added this to the KubeCon 2016 milestone Oct 10, 2016
@jml jml added the type/bug label Oct 10, 2016
@tomwilkie
Copy link
Contributor

Waiting on loggly to come back before we close this.

@tomwilkie
Copy link
Contributor

K8s 1.4 upgrade seems to have fixed this (no occurrences in the log)

tomwilkie added a commit that referenced this issue Nov 18, 2016
@tomwilkie
Squashed 'tools/' changes from 4fe078a..fd875e2
6060f59 
fd875e2 Fix test wrt shellcheck
54ec2d9 Don't capitalise error messages
19d3b6e Merge pull request #49 from weaveworks/pin-shfmt
fea98f6 Go get from the vendor dir
1d867b0 Try and vendor a specific version of shfmt
76619c2 Merge pull request #48 from weaveworks/revert-41-user-tokens
4f96c51 Revert "Add experimental support for user tokens"
d00033f Merge pull request #41 from weaveworks/user-tokens
245ed26 Merge pull request #47 from weaveworks/46-shfmt
c1d7815 Fix shfmt error
cb39746 Don't overright lint_result with 0 when shellcheck succeeds
8ab80e8 Merge pull request #45 from weaveworks/lint
83d5bd1 getting integration/config and test shellcheck-compliant
cff9ec3 Fix some shellcheck errors
7a843d6 run shellcheck as part of lint if it is installed
31552a0 removing spurious space from test
6ca7c5f Merge pull request #44 from weaveworks/shfmt
952356d Allow lint to lint itself
b7ac59c Run shfmt on all shell files in this repo
5570b0e Add shfmt formatting of shell files in lint
0a67594 fix circle build by splatting gopath permissions
b990f48 Merge pull request #42 from kinvolk/lorenzo/fix-git-diff
224a145 Check if SHA1 exists before calling `git diff`
1c3000d Add auto_apply config for wcloud
0ebf5c0 Fix wcloud -serivice
354e083 Fixing lint
586060b Add experimental support for user tokens

git-subtree-dir: tools
git-subtree-split: fd875e27c5379d443574bcf20f24a52a594871ca
roystchiang added a commit to roystchiang/cortex that referenced this issue Apr 6, 2022
@roystchiang
Merge pull request cortexproject#45 from aws-observability/run-workfl…
1cfb591 
…ow-for-all

run github actions for all branches
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
KubeCon 2016
Development

No branches or pull requests
2 participants
@jml @tomwilkie