Add cargo lintcheck --recursive to check dependencies of crates

[Alexendoo]

r? @matthiaskrgr

This just adds the mode without changing how a regular run works

Takes a fair bit longer to run than a -j4 or regular run

cargo lintcheck -j4  468.33s user 43.78s system 290% cpu 2:56.42 total
cargo lintcheck  428.81s user 41.85s system 199% cpu 3:55.37 total
cargo lintcheck --recursive  679.83s user 45.01s system 210% cpu 5:43.72 total

But finds more results:

Stats

clippy::explicit_counter_loop 0 => 1
clippy::needless_question_mark 0 => 3
clippy::unnecessary_cast 0 => 2
clippy::to_string_in_format_args 0 => 4
clippy::deprecated_cfg_attr 0 => 23
clippy::redundant_closure 0 => 9
clippy::drop_copy 0 => 4
clippy::double_must_use 0 => 1
clippy::transmute_num_to_bytes 0 => 9
clippy::bind_instead_of_map 0 => 14
clippy::float_cmp 0 => 16
clippy::is_digit_ascii_radix 0 => 16
clippy::manual_swap 0 => 1
clippy::needless_match 0 => 2
clippy::vec_init_then_push 0 => 1
clippy::never_loop 0 => 1
clippy::option_map_or_none 0 => 4
clippy::tabs_in_doc_comments 0 => 1
clippy::naive_bytecount 0 => 1
clippy::collapsible_if 0 => 24
clippy::copy_iterator 0 => 5
clippy::unused_io_amount 0 => 2
clippy::result_large_err 0 => 141
clippy::useless_conversion 0 => 24
clippy::flat_map_option 0 => 8
clippy::useless_format 0 => 2
clippy::module_inception 0 => 1
clippy::drop_ref 0 => 2
clippy::unnecessary_fold 0 => 2
clippy::neg_multiply 0 => 1
clippy::while_let_loop 0 => 6
clippy::missing_inline_in_public_items 0 => 37
clippy::unnecessary_mut_passed 0 => 1
unknown_lints 0 => 15
clippy::wildcard_dependencies 0 => 3
clippy::same_item_push 0 => 2
clippy::useless_asref 0 => 1
clippy::unnecessary_unwrap 0 => 4
clippy::iter_not_returning_iterator 0 => 5
clippy::comparison_to_empty 0 => 10
clippy::ref_option_ref 0 => 4
clippy::unused_peekable 0 => 1
clippy::needless_range_loop 0 => 8
clippy::absurd_extreme_comparisons 0 => 2
clippy::unnecessary_operation 0 => 2
clippy::for_kv_map 0 => 5
clippy::unnecessary_owned_empty_strings 0 => 3
clippy::transmutes_expressible_as_ptr_casts 0 => 1
clippy::toplevel_ref_arg 0 => 2
clippy::uninit_vec 0 => 3
clippy::filter_next 0 => 1
clippy::wildcard_in_or_patterns 0 => 6
clippy::cast_ptr_alignment 0 => 48
clippy::manual_memcpy 0 => 1
clippy::assign_op_pattern 0 => 313
clippy::unnecessary_lazy_evaluations 0 => 14
clippy::println_empty_string 0 => 2
clippy::redundant_pattern 0 => 2
clippy::declare_interior_mutable_const 0 => 8
clippy::large_stack_arrays 0 => 4
clippy::match_bool 0 => 4
clippy::unicode_not_nfc 0 => 2075
clippy::inconsistent_digit_grouping 0 => 4
clippy::no_effect_underscore_binding 0 => 2
clippy::let_and_return 0 => 5
clippy::transmute_ptr_to_ref 0 => 12
clippy::op_ref 0 => 13
clippy::unnecessary_join 0 => 4
clippy::into_iter_on_ref 0 => 13
clippy::from_str_radix_10 0 => 7
clippy::ptr_offset_with_cast 0 => 48
clippy::erasing_op 0 => 1
clippy::swap_ptr_to_ref 0 => 3
clippy::needless_bitwise_bool 0 => 2
clippy::extend_with_drain 0 => 19
clippy::only_used_in_recursion 0 => 4
clippy::needless_late_init 0 => 8
clippy::excessive_precision 0 => 1959
clippy::match_ref_pats 0 => 10
clippy::unit_arg 0 => 20
clippy::bool_comparison 0 => 4
clippy::bool_assert_comparison 0 => 1
clippy::eq_op 0 => 6
clippy::cast_abs_to_unsigned 0 => 6
clippy::format_in_format_args 0 => 1
clippy::iter_cloned_collect 0 => 4
clippy::ptr_eq 0 => 3
clippy::needless_bool 0 => 5
clippy::transmute_ptr_to_ptr 0 => 16
clippy::needless_option_take 0 => 2
clippy::flat_map_identity 0 => 1
clippy::needless_splitn 0 => 2
clippy::blocks_in_if_conditions 0 => 1
clippy::write_literal 0 => 1
clippy::manual_split_once 0 => 1
clippy::result_unit_err 0 => 36
clippy::unused_unit 0 => 11
clippy::single_match 0 => 22
clippy::manual_find 0 => 3
clippy::derive_ord_xor_partial_ord 0 => 6
clippy::char_lit_as_u8 0 => 2
clippy::let_unit_value 0 => 2
clippy::needless_continue 0 => 19
clippy::zero_sized_map_values 0 => 4
clippy::needless_arbitrary_self_type 0 => 6
clippy::partialeq_to_none 0 => 11
clippy::partialeq_ne_impl 0 => 1
clippy::invalid_upcast_comparisons 0 => 1
clippy::mut_range_bound 0 => 4
clippy::match_result_ok 0 => 2
clippy::ptr_arg 0 => 8
clippy::iter_nth_zero 0 => 18
clippy::needless_for_each 0 => 1
clippy::manual_unwrap_or 0 => 1
clippy::transmute_int_to_float 0 => 6
clippy::cast_slice_from_raw_parts 0 => 1
clippy::match_wild_err_arm 0 => 2
clippy::match_like_matches_macro 4 => 116
clippy::enum_glob_use 50 => 380
clippy::get_first 3 => 33
clippy::needless_doctest_main 10 => 26
clippy::struct_excessive_bools 19 => 51
clippy::cast_possible_wrap 46 => 538
clippy::manual_string_new 10 => 27
clippy::match_same_arms 53 => 1039
clippy::manual_non_exhaustive 1 => 33
clippy::redundant_pattern_matching 2 => 13
clippy::new_without_default 5 => 73
clippy::option_as_ref_deref 2 => 9
clippy::unwrap_or_else_default 2 => 4
clippy::case_sensitive_file_extension_comparisons 6 => 9
clippy::cast_precision_loss 45 => 110
clippy::needless_pass_by_value 26 => 187
clippy::redundant_closure_for_method_calls 170 => 539
clippy::let_underscore_drop 33 => 133
clippy::single_match_else 51 => 138
clippy::needless_borrow 24 => 382
clippy::redundant_else 37 => 151
clippy::type_complexity 2 => 22
clippy::ptr_as_ptr 93 => 1135
clippy::needless_lifetimes 7 => 100
clippy::single_char_add_str 2 => 22
clippy::similar_names 99 => 352
clippy::cargo_common_metadata 25 => 276
clippy::int_plus_one 1 => 2
clippy::missing_safety_doc 9 => 152
clippy::redundant_slicing 2 => 13
clippy::mut_mut 2 => 17
clippy::derive_partial_eq_without_eq 8 => 141
clippy::derive_hash_xor_eq 2 => 20
clippy::from_iter_instead_of_collect 2 => 17
clippy::verbose_bit_mask 1 => 8
clippy::too_many_lines 58 => 162
clippy::module_name_repetitions 178 => 1104
clippy::explicit_into_iter_loop 12 => 32
clippy::cast_lossless 45 => 478
clippy::many_single_char_names 9 => 23
clippy::unnested_or_patterns 27 => 127
clippy::upper_case_acronyms 5 => 29
clippy::needless_return 5 => 97
clippy::precedence 1 => 11
clippy::len_zero 2 => 70
clippy::manual_strip 2 => 30
clippy::derivable_impls 2 => 12
clippy::unused_self 20 => 187
clippy::enum_variant_names 1 => 6
clippy::self_named_constructors 1 => 3
clippy::explicit_auto_deref 19 => 314
clippy::semicolon_if_nothing_returned 137 => 1861
clippy::should_implement_trait 1 => 7
clippy::expl_impl_clone_on_copy 159 => 1318
clippy::stable_sort_primitive 4 => 12
clippy::mem_replace_with_default 7 => 46
clippy::borrow_deref_ref 5 => 140
clippy::large_enum_variant 1 => 4
clippy::map_unwrap_or 30 => 203
clippy::zero_ptr 3 => 25
clippy::filter_map_next 2 => 6
clippy::identity_op 5 => 76
clippy::checked_conversions 1 => 8
clippy::len_without_is_empty 9 => 47
clippy::missing_errors_doc 372 => 2333
clippy::fn_params_excessive_bools 3 => 7
clippy::single_component_path_imports 6 => 28
clippy::unreadable_literal 366 => 9814
clippy::field_reassign_with_default 1 => 5
clippy::redundant_clone 1 => 8
clippy::cloned_instead_of_copied 36 => 78
clippy::too_many_arguments 4 => 22
clippy::option_map_unit_fn 7 => 9
clippy::extra_unused_lifetimes 1 => 24
clippy::unnecessary_wraps 26 => 128
clippy::used_underscore_binding 1 => 50
clippy::inconsistent_struct_constructor 2 => 7
clippy::manual_range_contains 9 => 120
clippy::map_clone 7 => 46
clippy::cast_slice_different_sizes 1 => 4
clippy::missing_panics_doc 114 => 603
renamed_and_removed_lints 3 => 9
clippy::items_after_statements 155 => 309
clippy::inefficient_to_string 5 => 6
clippy::comparison_chain 1 => 19
clippy::crate_in_macro_def 3 => 6
clippy::write_with_newline 2 => 36
clippy::manual_saturating_arithmetic 1 => 2
clippy::clone_on_copy 1 => 86
clippy::negative_feature_names 3 => 16
clippy::redundant_field_names 112 => 1013
clippy::from_over_into 2 => 28
clippy::wildcard_imports 178 => 376
clippy::unusual_byte_groupings 19 => 65
clippy::option_option 2 => 10
clippy::nonminimal_bool 1 => 17
clippy::borrow_as_ptr 2 => 172
clippy::redundant_static_lifetimes 24 => 1701
clippy::or_fun_call 1 => 63
clippy::single_char_pattern 3 => 79
clippy::explicit_iter_loop 72 => 148
clippy::collapsible_else_if 2 => 27
clippy::manual_str_repeat 1 => 6
clippy::if_same_then_else 3 => 31
clippy::while_let_on_iterator 4 => 28
clippy::multiple_crate_versions 5 => 19
clippy::cast_possible_truncation 115 => 1172
clippy::explicit_deref_methods 1 => 38
clippy::default_trait_access 48 => 130
clippy::question_mark 2 => 28
clippy::must_use_candidate 612 => 5369
clippy::manual_map 1 => 12
clippy::bool_to_int_with_if 2 => 15
clippy::doc_markdown 202 => 1709
clippy::cast_sign_loss 60 => 477
clippy::wrong_self_convention 11 => 45
clippy::transmute_float_to_int 6 => 18
clippy::return_self_not_must_use 66 => 736
clippy::range_plus_one 1 => 36
clippy::manual_assert 11 => 62
clippy::trivially_copy_pass_by_ref 40 => 189
clippy::match_on_vec_items 2 => 7
clippy::inline_always 59 => 1079
clippy::if_not_else 31 => 205
clippy::implicit_clone 10 => 32
clippy::match_wildcard_for_single_variants 16 => 101
clippy::doc_link_with_quotes 7 => 35
clippy::redundant_feature_names 4 => 41

changelog: none

[matthiaskrgr]

Can we avoid dependencies change between runs somehow?
I also vaguely remember we can run clippy on deps using cargo clippy -vvvv or something like that, this might allow much simpler code here

[Alexendoo]

The dependencies wouldn't change between runs on the same machine at least, as the sources that don't ship a Cargo.lock will have one added in target/lintcheck/sources. But if you wanted reproducible results between CI jobs or something like that you'd need to either select crates that come with Cargo.locks or preserve the generated ones somehow

You can get clippy to run on dependencies by doing RUSTC_WRAPPER=clippy-driver cargo check -vv, but it takes a lot longer that way: 1844.53s user 84.37s system 160% cpu 20:00.39 total, ~4mins of which are spent in Message::parse_stream...collect(). I guess just because it means a tonne more data is passing through cargo

[matthiaskrgr]

aren't these clippy args technically? Perhaps rename to clippy_args? 🤔

[Alexendoo]

Yeah, that's true

[matthiaskrgr]

What does this status do?
We just check that cargo check/clippy would run successful (no invalid args passed etc..)?
Would you mind adding a short comment at the top? 🙃

[matthiaskrgr]

neat! 👍

[matthiaskrgr]

this makes me wonder if we should actually offer a --release mode somehow 😅

[Alexendoo]

That could be cool, I wonder how much you need to lint before it makes sense to apply 😄

[matthiaskrgr]

I'm trying to wrap my head around this 😅

the lintcheck binary also acts as some kind of wrapper that pretends to be a rustc when called onto a dependency and runs lintcheck/clippy on the dependency it is called on?

[matthiaskrgr]

So the same binary acts as "worker" running clippy on a single crate, intercepting its output and sending it to a server, as well as server, (receiving the output from N workers at a time and summarizing them)". Do I understand this correctly? This seems a bit funky, but I've also never implemented something like this myself.

[Alexendoo]

Yeah that's right. It is pretty funky, I took the idea from cargo - https://github.com/rust-lang/cargo/blob/0.65.0/src/cargo/ops/fix.rs

The RUSTC_WRAPPER part could be made a separate binary that lintcheck cargo builds first if that would make it clearer

[matthiaskrgr]

this does set a different target dir for --recursive, right? did you run into some problems when not doing this?

[Alexendoo]

It shouldn't cause a problem, IIRC I was trying to avoid the regular lintcheck's target dir getting removed unnecessarily. But as it may get deleted anyway on account of clippy's timestamp maybe it's not really making a difference doing this

[matthiaskrgr]

Thanks!

[matthiaskrgr]

@bors r+
🚀

[bors]

📌 Commit fc77d91 has been approved by matthiaskrgr

It is now in the queue for this repository.

[bors]

⌛ Testing commit fc77d91 with merge 31b1741...

[bors]

☀️ Test successful - checks-action_dev_test, checks-action_remark_test, checks-action_test
Approved by: matthiaskrgr
Pushing 31b1741 to master...