Skip to content
This repository was archived by the owner on Sep 16, 2021. It is now read-only.

Support additional test operators #13

Closed
mattt opened this issue Apr 20, 2020 · 2 comments · Fixed by #30
Closed

Support additional test operators #13

mattt opened this issue Apr 20, 2020 · 2 comments · Fixed by #30
Labels
help wanted Extra attention is needed question Further information is requested

Comments

@mattt
Copy link
Contributor

mattt commented Apr 20, 2020

For the initial proof of concept, I wanted to keep things simple with two operators:

  • => performs a string equality test on the standard output of the REPL
  • !! succeeds if the REPL writes anything to standard error

For a first take, this works pretty well. Assuming the REPL format remains relatively stable, this could work for a long time.

However, there are some drawbacks and limitations to the current approach:

  1. Symbolic operators are difficult to search for
  2. Literal string equality checks are fragile
  3. String equality checks are inconvenient for complex values
  4. String equality checks are incompatible with dynamic values (for example, relative date calculations, pointer addresses)

As an alternative, I'd like to propose a limited DSL / command syntax:

<swift code> // <command>[: <expression>]

Multiple expectations can be annotated for a single statement, either on subsequent lines or on the same line using /* block */ /* comments */

Here's a preliminary list of commands that I think we should support:

  • equals: <stdout content> - performs a string equality test on the content written to standard output by the REPL (equivalent to current =>)
  • matches: <regular expression pattern> - performs a regular expression test with the provided pattern
  • error - succeeds if the REPL writes anything to standard error (equivalent to current !!)
  • error: <stderr content> - performs a string equality test on the content written to standard error by the REPL
  • assert: <swift code> - runs the provided swift code as the next REPL statement; succeeds if true is written to standard output; fails otherwise.

Finally, some open questions:

  • assert can be used to implement any of the other checks (other than error). What's the cost/benefit for including them as well?
  • Should DocTest comments require a leading prefix to distinguish them from conventional comments (similar to how documentation comments use /** and ///)?
  • Should we deprecate the existing, symbolic operators, or designate them as aliases their new, corresponding commands? If we alias them, should we provide an alias for matches (~>)?
@mattt mattt added help wanted Extra attention is needed question Further information is requested labels Apr 20, 2020
@mattt mattt pinned this issue Apr 21, 2020
@orta
Copy link

orta commented Apr 24, 2020

For microsoft/TypeScript#35389 which is a similar-ish issue - we opted to not have different identifiers and internal logic for validations.

Instead we're keep track of the LSP response for hovering on all known identifiers and raised compiler errors in a JSON dump per code sample, that dump can then be compared across language releases and snapshots can be updated with it either manually (like will happen on the website) or automatically like the RFC on GH issues.

For an example click here then go to "Results" (its a WIP, so if there's no data, just press space in the text editor once it's loaded)

@schwa
Copy link

schwa commented May 2, 2020

I’d hesitate going too far here. The chief advantage of doctests are that they’re an almost frictionless way to add tests to simple functions. Having to learn a syntax of operators kind of defeats one of doctest’s advantages.

IMO Doctests are great for simple 1 liners of non-side effect producings functions where the output that is easy to express. Let XCTest or other systems do the heavy lifting for more complex tests.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants