-
Notifications
You must be signed in to change notification settings - Fork 13.3k
Update debuginfo test runner to provide more useful output #113306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
(rustbot has picked a reviewer for you, use r? to override) |
This comment has been minimized.
This comment has been minimized.
@rustbot author |
This comment has been minimized.
This comment has been minimized.
3b612a4
to
35defa2
Compare
This comment has been minimized.
This comment has been minimized.
796dafb
to
a4698c2
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
533b46b
to
7871ac3
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
2726349
to
c485ca4
Compare
This comment has been minimized.
This comment has been minimized.
c485ca4
to
4e1dc23
Compare
Alright, the output looks good so I have reverted the intentional errors and CI changes. Ready for a look. @rustbot ready |
4e1dc23
to
e62dcf6
Compare
This change makes debuginfo tests more user friendly. Changes: - Print all lines that fail to match the patterns instead of just the first - Provide better error messages that also say what did match - Strip leading whitespace from directives so they are not skipped if indented - Improve documentation and improve nesting on some related items As an example, given the following debuginfo test with intentional fails: ```rust // from tests/debuginfo/rc_arc.rs // cdb-command:dx rc,d // cdb-check:rc,d : 111 [Type: alloc::rc::Rc<i32>] // cdb-check: [Reference count] : 11 [Type: core::cell FAIL::Cell<usize>] // cdb-check: [Weak reference count] : 2 [Type: core::cell FAIL::Cell<usize>] // ... ``` The current output (tested in rust-lang#113313) only shows the first mismatch: ``` 2023-07-04T08:10:00.1939267Z ---- [debuginfo-cdb] tests\debuginfo\rc_arc.rs stdout ---- 2023-07-04T08:10:00.1942182Z 2023-07-04T08:10:00.1957463Z error: line not found in debugger output: [Reference count] : 11 [Type: core::cell FAIL::Cell<usize>] 2023-07-04T08:10:00.1958272Z status: exit code: 0 ``` With this change, you are able to see all failures in that check group, as well as what parts were successful. The output is now: ``` 2023-07-04T09:45:57.2514224Z error: check directive(s) from `C:\a\rust\rust\tests\debuginfo\rc_arc.rs` not found in debugger output. errors: 2023-07-04T09:45:57.2514631Z (rc_arc.rs:31) ` [Reference count] : 11 [Type: core::cell FAIL::Cell<usize>]` 2023-07-04T09:45:57.2514908Z (rc_arc.rs:32) ` [Weak reference count] : 2 [Type: core::cell FAIL::Cell<usize>]` 2023-07-04T09:45:57.2515181Z (rc_arc.rs:41) ` [Reference count] : 21 [Type: core::sync::atomic FAIL::AtomicUsize]` 2023-07-04T09:45:57.2515452Z (rc_arc.rs:50) `dyn_rc,d [Type: alloc::rc::Rc<dyn$<core::fmt FAIL::Debug> >]` 2023-07-04T09:45:57.2515695Z the following subset of check directive(s) was found successfully:: 2023-07-04T09:45:57.2516080Z (rc_arc.rs:30) `rc,d : 111 [Type: alloc::rc::Rc<i32>]` 2023-07-04T09:45:57.2516312Z (rc_arc.rs:35) `weak_rc,d : 111 [Type: alloc::rc::Weak<i32>]` 2023-07-04T09:45:57.2516555Z (rc_arc.rs:36) ` [Reference count] : 11 [Type: core::cell::Cell<usize>]` 2023-07-04T09:45:57.2516881Z (rc_arc.rs:37) ` [Weak reference count] : 2 [Type: core::cell::Cell<usize>]` ... ``` Which makes it easier to see what did and didn't succeed without manual comparison against the source test file.
e62dcf6
to
b0a18cb
Compare
@bors r+ rollup=iffy |
☀️ Test successful - checks-actions |
Finished benchmarking commit (ba37a69): comparison URL. Overall result: ❌ regressions - ACTION NEEDEDNext Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression Instruction countThis is a highly reliable metric that was used to determine the overall result at the top of this comment.
Max RSS (memory usage)ResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
CyclesResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
Binary sizeThis benchmark run did not return any relevant results for this metric. Bootstrap: 658.609s -> 658.544s (-0.01%) |
I don’t think this PR can have anything to do with the regression since it’s only a test change |
This change makes debuginfo tests more user friendly. Changes:
As an example, given the following intentional fail (and a few not shown):
The current output (tested in #113313) will show:
With this change, you are able to see all failures in that check group, as well as what parts were successful. The output is now:
Which makes it easier to see what did and didn't succeed without manual comparison against the source test file.