QuickCheck port for Rust

[toddaaro]

One of the things Haskell is known for is Quick Check, a library for easily generating test cases that test invariants specified by the user. This testing approach has been ported to many languages, and it would be great if Rust ended up one of those languages.

http://en.wikipedia.org/wiki/QuickCheck
http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck

[kud1ing]

See also http://static.rust-lang.org/doc/std/rand.html which provides a Trait for generating random values.

[dbp]

I've started playing around with this. I have a very early implementation here: https://github.com/dbp/rust-quickcheck

I opted not to use the Rand trait for random values, but rather to introduce a different trait, Arbitrary. The reason for this is that I'm not sure if a uniform distribution is actually what is going to be the most useful for testing. For example, when testing vectors, I'd want to bias the empty one relatively high, because I think that's an edge case that is often forgotten. I'm not sure if changing the distribution is the right way to aim for common edge cases, but in case it is... not tying this to Rand seems useful.

The other main trait is Testable - which is a way to mimic lisp apply-like behavior for functions of different arity (this seems to be the approach in the Haskell library - though with a lot more layers of abstraction!)

Thoughts welcome!

[graydon]

I have many thoughts about this! Will try to find time to summarize. Very excited.

[glaebhoerl]

It might also be worth stealing ideas from SmallCheck: http://ro-che.info/articles/2013-02-19-smallcheck.html

[dbp]

Oh, this is relevant :) https://github.com/blake2-ppc/qc.rs

Should have searched more!

[metajack]

You'll want to be able to control the distribution as QuickCheck allows.

Also, we're going to want a Shrink trait, so that we can shrink failing cases to something reasonable.

And will probably want something like eqc_statem for doing model based testing with state.

We're extremely interested in this stuff for Servo.

[bluss]

The qc.rs Arbitrary trait passes down a size parameter. It's quite rudimentary, but it starts with small sizes and gradually increases the mean size. The Shrink thing is using a crazy lazy list thing (because it's interesting), maybe you can come up with something better there.

[killerswan]

So, how's Rust QuickCheck looking lately, @dbp, @mcandre, @blake2-ppc, and @jruderman? What's the state of the art?

[mcandre]

I'm not actively working on it, but I'm happy to accept any pull requests.

[pnkfelix]

Visiting for long delayed triage.

This strikes me as something that's nice to have, but need not be integrated into the central rust distribution.

[BurntSushi]

Just though I'd chime in and say that I've started fresh here: https://github.com/BurntSushi/quickcheck --- it's a pretty faithful port of Haskell's QuickCheck (shrinking, sized arbitrary, properties are polymorphic). Most of the shrinking is lazy with iterators, but not all of it.

[mcandre]

Excellent! I haven't kept up with rust, so I expect BurntSushi/quickcheck would be a lot better than my rust 0.6-based port. And I never added test case simplification, so that's amazing, too.

I'd be happy to deprecate mcandre/rustcheck in favor of this if the rust community likes it.

[sinistersnare]

If #14100 was closed because the Rust team was unsure of whether we actually want QuickCheck in-tree, should this be closed too?

[ghost]

In the same vein as #6167, I'm going to close this. @BurntSushi's library and all the alternatives are amazing but I don't think we need to track their progress in this issue. Please feel free to reopen if you strongly disagree.

[gburd]

@jakub I'd argue that Rust is redefining the baseline for many important aspects of software development and that testing should be one of those aspects. Property-based testing is in my experience (Erlang/QuickCheck+PULSE/Basho) a huge step forward (like lifetimes or other innovations in Rust) from unit-tests and tends to create more stable and well thought code. Including a mature quickcheck-like property-based testing framework would make a difference in the overall quality of software on this platform and that quality would translate to confidence and hopefully greater adoption.

Unit testing (or more generally, software tools) are hard to monetize (ask QuviQ) and generally require a business model that isn't inclusive and open and so the net result is that the community suffers until they re-invent the techniques (Proper, Triq, etc) in open source. This is really a stagnation of a good idea, not a full adoption. I'd argue that this is another good reason for Mozilla to invest in this feature of Rust.

Of course, the counter argument is "where do we draw the line, we can't be responsible for pushing all the limits." and I fully appreciate that, but...

I thought I'd provide some food for thought.

[ghost]

@gburd I agree with you on the superiority of property testing over unit testing.

However, the way the Rust team is going about deciding what should be part of the standard distribution is to lean on the side of moving out non-essential crates outside of it. This is possible because with Cargo and the future crate registry in place using an external crate will be a very easy thing to do and so there is no need at this point to pick winners even for important functionality like URL handling, M:N threading or, indeed, advanced property testing.

My understanding is that this could be revisited in the future.

[BurntSushi]

@gburd @jakub- For what it's worth, I think property based testing and unit testing are different approaches to testing. I see neither one as superior. I frequently use both.

With that said, QuickCheck was once considered for inclusion in the standard Rust distribution, but it was rejected. My understanding of why was that it hadn't generated enough evidence that it would be widely used. (I agree with their assessment.)

Moreover, I generally agree with the direction of "make the standard distribution more lean." Using QuickCheck is dead simple now that we have Cargo.

[gburd]

@jakub I fully understand and appreciate the trade-offs and balancing act you and the Rust Team must be facing and I think that you're doing an amazing job. Thank you all. I had to advocate for property based testing because I think it has the potential to have a transformative impact on programming. Cargo is a great tool and a good system to sponsor and encourage community contributions to the Rust ecosystem and I can see how QuickCheck-like tools could flourish within that framework.

I'm likely going to bet heavily on Rust in my future projects and I'm 100% sure I'll be incorporating software written in Rust developed by someone else in those projects. It's also very likely that I'll be building on top of other systems which use Rust in many other ways. It would be easier to bet on those dependencies (read: sleep comfortably through the night) were I to know that everyone had tuned into this new form of testing. Value judgments aside, doing the work to property test in addition to unit test would (IMO) dramatically improve software quality across the board. Unfortunately that's not going to happen unless a) the software is readily available to support it and b) the leaders in the space (you and the Rust Team and those first vocal/influential adopters in the Rust community) are moving in that direction as well.

The Rust Team has been breaking a lot of new ground and I can't thank you all enough for that hard work. Keep pushing, keep questioning and keep redefining the state of the art. I appreciate your thoughtful responses.