Hand-rolling mini insta?

[matklad]

So far, we've been using insta for snapshot testing, and it is super useful, but not perfect. Some of the paper cuts I see:

• need to cargo install insta to fix snapshots
• the insta tool someimes fails with obscure errors in the workspace
• quite a few of dependencies
• snapshot format is somewhat obscure with respect to leading whitespace
• It's impossible to copy-paste the gold value from the test failure message

I am thinking that maybe less is more, and we should roll our own snapshot testing thing? I think getting rid of the coloring and using a simpler assert_eq_text style diff would be more convenient. The biggest problem is updating inline snapshots, but I think that shouldn't be too hard to implement.

[matklad]

I am warming up to this idea, here's the API I came up with:

use std::{
    env,
    path::{Path, PathBuf},
};

#[derive(Debug)]
pub struct Expectation {
    pub file: &'static str,
    pub line: u32,
    pub data: &'static str,
}

impl Expectation {
    pub fn assert_eq(&self, actual: &str) {
        if self.data != actual {
            let path = dbg!(project_root()).join(self.file);
            let text = std::fs::read_to_string(&path).unwrap();
            let expect_start = self.line as usize;
            let expect_len = text
                .lines()
                .skip(expect_start + 1)
                .take_while(|it| !it.contains(r##""#]"##))
                .count();
            for l in text.lines().skip(expect_start).take(expect_len) {
                println!("{}", l)
            }
        }
    }
}

fn project_root() -> PathBuf {
    Path::new(
        &env::var("CARGO_MANIFEST_DIR").unwrap_or_else(|_| env!("CARGO_MANIFEST_DIR").to_owned()),
    )
    .ancestors()
    .nth(2)
    .unwrap()
    .to_path_buf()
}

#[macro_export]
macro_rules! expect {
    [$(,)?$data:literal] => {
        $crate::Expectation {
            file: $crate::__rt::file!(), line: $crate::__rt::line!(), data: $data
        }
    };
}

#[test]
fn foo() {
    let expectation = expect![,r#"
        64..84 '{     ...1 }; }': ()
        70..81 'SS { x: 1 }': S<u32>
        78..79 '1': u32
    "#];

    eprintln!("{:?}", expectation);
}

#[doc(hidden)]
pub mod __rt {
    pub use std::{file, line};
}

#[test]
fn nested_module_resolution() {
    check(
        r"
//- /lib.rs
mod n1;

//- /n1.rs
mod n2;

//- /n1/n2.rs
struct X;
",
        expect![,r#"
crate
n1: t

crate::n1
n2: t

crate::n1::n2
X: t v
"#],
    );
}

[bjorn3]

That api wil never mark a test as failed.

[matklad]

Yeah, there should be assetion failure there as well :)

The bigger missing piece is that this doesn't (yet) show a 💡 in the editor which changes the data in-place (one of my main motivations for this).

Not sure how to approach that though -- my current plan is to dump expect failures to target/expect_db.json (intentionally limiting to only one test binary at a time) and than just read that file in our code-action handler.

It would be sweet if I can just print something to stdout which would be applicable in VS Code, but I don't thing that that is possible?

[bjorn3]

Can you use a problem matcher to match the test run output?

[matklad]

I think that's not possible unfortunately :-(

[matklad]

cc microsoft/vscode#99728

[flodiebold]

Wouldn't it be nicer to implement that support for insta, so other users can also profit from it?

In general, in this case I don't really understand the motivation to create a replacement; I think it'd be better to improve insta instead.

[matklad]

In general, in this case I don't really understand the motivation to create a replacement;

So, the super-big picture here is that we are going to have an infinite number of tests eventually, and so something tailored for our use-case would be better economically. If you are going to use a library for n use-cases, and n is large enough, it's less friction to write a specialized library than to use a generic one.

Medium level picture is that I think there's a different set of tradoffs we can try, in particular:

• focusing on snapshot!() macro which returns a data structure, and leaving the actual asserts to the caller
• cutting the number of features/dependencies (insta is a pretty big chunk in our Cargo.lock)
• experimenting with tighter integration with the editor
• not requiring an external tool

These feels sufficiently different from insta that I feel that it makes sense to try this out separately, and then do a synthesis, rather than to try to change insta in-place.

[matklad]

experimenting with tighter integration with the editor

In particular, I really want this, but I don't see how this can be implemented in a nice way without hard-coding it into rust-analyzer, and, for hard-coding, NIH thing seems easier :)

[bjorn3]

experimenting with tighter integration with the editor

That would be useful for other users of insta too.

[bjorn3]

not requiring an external tool

If the insta crate exported a function that would behave as if cargo insta was called, you could add an xtask that calls that function. Then using something like cargo xtask insta would work instead of having to install cargo-insta yourself.

[bjorn3]

cutting the number of features/dependencies (insta is a pretty big chunk in our Cargo.lock)

Feature flags? Could be useful for other users too.

[matklad]

If the insta crate exported a function that would behave as if cargo insta was called, you could add an xtask that calls that function.

Yeah, but cargo-insta is significantly more heavy-weight than insta library (which isn't a feather either), but xtask is one of the things which benefits from being lean.

[lnicola]

I think we could do without the diff and colors if we update the snapshots on disk (because you can run git diff instead).

[flodiebold]

I think it'd be pretty annoying to have to update snapshots to compare them, especially if the failure happens in CI.