Correctly display stdout and stderr in case a doctest is failing

[GuillaumeGomez]

Fixes #140289.

Since the doctest is actually running itself, we need to handle the output directly inside it.

cc @fmease
r? @notriddle

[GuillaumeGomez]

Ok I think this time I didn't forget some normalization. :')

[notriddle]

@bors r+

[bors]

📌 Commit c44068b has been approved by notriddle

It is now in the queue for this repository.

[bors]

⌛ Testing commit c44068b with merge 8947e16...

[bors]

☀️ Test successful - checks-actions
Approved by: notriddle
Pushing 8947e16 to master...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 496145b (parent) -> 8947e16 (this PR)

Test differences

Show 33070 test diffs

Stage 1

• ascii::is_ascii::unaligned_both_medium::case01_iter_all: pass -> [missing] (J1)
• iter::bench_peekable_ref_sum: pass -> [missing] (J1)
• path::bench_hash_path_long: pass -> [missing] (J1)
• sort::tests::stable::deterministic_i32_random: pass -> [missing] (J1)
• str::rfind_zzz_char::short_pile_of_poo: pass -> [missing] (J1)
• str::split_space_str::long_lorem_ipsum: pass -> [missing] (J1)
• string::bench_insert_str_short: pass -> [missing] (J1)
• tests::test_should_report_time: pass -> [missing] (J1)
• vec::bench_dedup_all_100000: pass -> [missing] (J1)
• collections::hash::map::tests::test_extend_ref: pass -> [missing] (J2)
• collections::hash::map::tests::test_find_mut: pass -> [missing] (J2)
• collections::linked_list::tests::test_iterator_double_end: pass -> [missing] (J2)
• collections::vec_deque::tests::test_try_reserve_exact: pass -> [missing] (J2)
• io::buffered::tests::test_buffered_reader_seek_underflow_discard_buffer_between_seeks: pass -> [missing] (J2)
• iter::traits::iterator::test_find_map: pass -> [missing] (J2)
• lazy::dropck: pass -> [missing] (J2)
• lazy_lock::lazy_default: pass -> [missing] (J2)
• mem::offset_of_alias: pass -> [missing] (J2)
• mem::offset_of_union: pass -> [missing] (J2)
• num::f64::max: pass -> [missing] (J2)
• num::i128::test_saturating_neg: pass -> [missing] (J2)
• num::i16::test_abs: pass -> [missing] (J2)
• num::i8::test_abs: pass -> [missing] (J2)
• num::test_try_i16i16: pass -> [missing] (J2)
• num::test_try_i16u64: pass -> [missing] (J2)
• ops::test_range: pass -> [missing] (J2)
• rc::test_rc_cyclic_with_one_ref: pass -> [missing] (J2)
• slice::slice_index::rangeinclusive_len_len::pass: pass -> [missing] (J2)
• slice::test_array_chunks_mut_zip: pass -> [missing] (J2)
• slice::test_iterator_advance_back_by: pass -> [missing] (J2)
• str::pattern::str_searcher_multibyte_haystack::fwd: pass -> [missing] (J2)
• str::slice_index::boundary::rangeinclusive_2::index_fail: pass -> [missing] (J2)
• str::test_char_indicesator: pass -> [missing] (J2)
• sys_common::wtf8::tests::wtf8buf_push_char: pass -> [missing] (J2)
• tests::smoke_no_dtor: pass -> [missing] (J2)
• thread::tests::test_spawn_sched_childs_on_default_sched: pass -> [missing] (J2)
• time::from_mins_overflow: pass -> [missing] (J2)
• clean::types::tests::should_not_ignore_first_line_indent_in_a_single_line_para: pass -> [missing] (J3)
• doctest::tests::make_test_named_wrapper: pass -> [missing] (J3)
• error::verify_query_system_reentrant_0: pass -> [missing] (J3)
• errors::ad_fallback::verify_builtin_macros_autodiff_not_build_35: pass -> [missing] (J3)
• errors::verify_ast_lowering_register_class_only_clobber_stable_20: pass -> [missing] (J3)
• errors::verify_ast_passes_optional_trait_supertrait_47: pass -> [missing] (J3)
• errors::verify_mir_build_lower_range_bound_must_be_less_than_upper_7: pass -> [missing] (J3)
• errors::verify_passes_used_static_76: pass -> [missing] (J3)
• errors::verify_session_invalid_float_literal_width_36: pass -> [missing] (J3)
• html::highlight::tests::test_keyword_highlight: pass -> [missing] (J3)
• html::markdown::tests::test_lang_string_tokenizer: pass -> [missing] (J3)
• lints::verify_lint_builtin_allow_internal_unsafe_6: pass -> [missing] (J3)
• markdown::parse::tests::test_newline_breaks: pass -> [missing] (J3)
• parser::tests::look_ahead_non_outermost_stream: pass -> [missing] (J3)
• parser::tests::non_overlapping: pass -> [missing] (J3)
• spec::tests::aarch64_be_unknown_linux_gnu_ilp32: pass -> [missing] (J3)
• spec::tests::sparc64_unknown_netbsd: pass -> [missing] (J3)
• spec::tests::x86_64_pc_cygwin: pass -> [missing] (J3)
• tests::test_switch_implies_cfg_test: pass -> [missing] (J3)
• tests::test_unterminated: pass -> [missing] (J3)
• [ui] tests/rustdoc-ui/doctest/stdout-and-stderr.rs: [missing] -> pass (J4)
• sys::process::unix::unix::unsupported_wait_status::tests::compare_with_linux: pass -> [missing] (J5)

Stage 2

• sort::tests::stable::correct_i32_random_s50: [missing] -> ignore (J0)
• sort::tests::stable::stability_i32_saw_mixed: [missing] -> ignore (J0)
• sort::tests::unstable::correct_1k_random_d20: [missing] -> ignore (J0)
• ascii::long::is_ascii_graphic: [missing] -> pass (J1)
• btree::map::clone_slim_10k_and_remove_half: [missing] -> pass (J1)
• hash::set_ops::set_is_subset: [missing] -> pass (J1)
• num::int_log::u128_log_random_small: [missing] -> pass (J1)
• num::int_sqrt::u64_sqrt_random_small: [missing] -> pass (J1)
• sort::tests::stable::correct_1k_random: [missing] -> pass (J1)
• sort::tests::stable::panic_retain_orig_set_cell_i32_ascending: [missing] -> pass (J1)
• sort::tests::unstable::correct_i32_random_s50: [missing] -> pass (J1)
• str::char_count::emoji_tiny::case00_libcore: [missing] -> pass (J1)
• str::split_a_str::short_ascii: [missing] -> pass (J1)
• str::split_slice: [missing] -> pass (J1)
• time::instant_contention_16_threads: [missing] -> pass (J1)
• collections::binary_heap::test_empty_peek: [missing] -> pass (J2)
• collections::btree::map::tests::test_bad_zst: [missing] -> pass (J2)
• f32::test_gamma: [missing] -> pass (J2)
• ffi::os_str::tests::test_os_str_default: [missing] -> pass (J2)
• fmt::float::test_format_f32_rounds_ties_to_even: [missing] -> pass (J2)
• io::buffered::tests::test_buffered_reader_read_to_end_consumes_buffer: [missing] -> pass (J2)
• nonzero::test_match_option_rc: [missing] -> pass (J2)
• num::test_i16f64: [missing] -> pass (J2)
• num::test_try_u8u8: [missing] -> pass (J2)
• num::u32::test_isolate_most_significant_one: [missing] -> pass (J2)
• slice::test_chunks_exact_mut_last: [missing] -> pass (J2)
• slice::test_copy_from_slice_dst_shorter: [missing] -> pass (J2)
• slice::test_rchunks_exact_iterator_0: [missing] -> pass (J2)
• slice::test_rchunks_mut_next: [missing] -> pass (J2)
• slice::test_rposition: [missing] -> pass (J2)
• str::pattern::char_searcher_short_haystack::bwd: [missing] -> pass (J2)
• str::slice_index::overflow::rangeinclusive::pass: [missing] -> pass (J2)
• sync::test_arc_get_mut: [missing] -> pass (J2)
• sync::test_from_str: [missing] -> pass (J2)
• vec::test_pop_if: [missing] -> pass (J2)
• errors::verify_expand_attribute_meta_item_16: [missing] -> pass (J3)
• errors::verify_hir_typeck_field_multiply_specified_in_initializer_0: [missing] -> pass (J3)
• graph::vec_graph::tests::predecessors: [missing] -> pass (J3)
• leb128::tests::test_usize_leb128: [missing] -> pass (J3)
• lints::verify_lint_identifier_non_ascii_char_73: [missing] -> pass (J3)
• parser::tests::parse_exprs: [missing] -> pass (J3)

(and 16452 additional test diffs)

Additionally, 16518 doctest diffs were found. These are ignored, as they are noisy.

Job group index

• J0: x86_64-gnu-aux
• J1: aarch64-apple, test-various, x86_64-apple-1
• J2: aarch64-apple, test-various, x86_64-apple-1, x86_64-gnu-aux
• J3: aarch64-apple, x86_64-apple-1
• J4: x86_64-gnu-llvm-19-3, x86_64-gnu-llvm-20-3
• J5: test-various

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 8947e167e58c5c4f23cae94585301f8eb90d93ea --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. x86_64-apple-2: 5108.6s -> 1675.4s (-67.2%)
2. test-various: 4140.8s -> 2401.9s (-42.0%)
3. x86_64-apple-1: 8455.0s -> 5498.7s (-35.0%)
4. dist-aarch64-linux: 7758.9s -> 5339.1s (-31.2%)
5. x86_64-gnu-aux: 5914.5s -> 4344.2s (-26.6%)
6. dist-apple-various: 6337.5s -> 7828.5s (23.5%)
7. aarch64-apple: 3916.6s -> 3111.3s (-20.6%)
8. dist-x86_64-apple: 7777.3s -> 8962.8s (15.2%)
9. x86_64-gnu-llvm-20-1: 6152.3s -> 5248.9s (-14.7%)
10. dist-aarch64-apple: 5640.5s -> 5014.3s (-11.1%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (8947e16): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary -0.8%, secondary -2.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.5%, 0.5%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.1%	[-3.6%, -0.6%]	2
Improvements ✅ (secondary)	-2.2%	[-2.2%, -2.2%]	1
All ❌✅ (primary)	-0.8%	[-3.6%, 0.5%]	4

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 764.349s -> 763.457s (-0.12%)
Artifact size: 365.22 MiB -> 365.17 MiB (-0.01%)