Fix races in tracing-appender unit tests #737
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hawkw
reviewed
May 27, 2020
hawkw
approved these changes
Jun 12, 2020
Comment on lines
+339
to
+340
| // Sleep a bit to prevent races. | ||
| thread::sleep(Duration::from_millis(200)); |
There was a problem hiding this comment.
I'm not a huge fan of tests that rely on load-bearing sleeps to work correctly. But, if there's no other solution, I'm fine with merging this.
There was a problem hiding this comment.
Let me know if this test is still being problematic. We can probably use some retry logic instead of sleeps here instead now that I think about it.
ericjheinz
pushed a commit
to ericjheinz/tracing
that referenced
this pull request
Jun 12, 2020
## Motivation I was able to reproduce the failures for `logs_dropped_if_lossy` on my machine when using loom. Realized that the Worker calls `recv` and the only time we actually will increment the error_counter is when we have a write blocking on the call to `write_all` (called by the worker) and a messege buffered in NonBlocking crossbeam sender. ## Solution I added some sleeps after writes to avoid some races I saw. For `multi_threaded_writes`, what likely is happening is that sometimes the last thread hasn't had a chance write to the queue, hence we now use `recv_timeout` instead of `try_recv`, to ensure there's more than enough time for the message to be visible in the channel. Co-authored-by: Zeki Sherif <[email protected]>
This was referenced Jun 12, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was able to reproduce the failures for
logs_dropped_if_lossyon my machine when usnig loom.Realized that the Worker calls
recvand the only time we actually will increment the error_counter is when we have a write blocking on the call towrite_all(called by the worker) and a messege buffered in NonBlocking crossbeam sender.I added some sleeps after writes to avoid some races I saw.
For
multi_threaded_writes, what likely is happening is that sometimes the last thread hasn't had a chance write to the queue, hence we now use recv_timeout instead of try_recv, to ensure there's more than enough time for the a message to be visible in the channel.