Skip to content

create a way to run acceptance tests from folder trees directly in the source tree #2628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RonnyPfannschmidt opened this issue Jul 28, 2017 · 2 comments
Labels
type: infrastructure improvement to development/releases/CI structure

Comments

@RonnyPfannschmidt
Copy link
Member

our mechanism for acceptance testing is a disgrace, creating file trees in test scripts to run nested py.test instances against it is both wastefull, and hard to debug

instead we should have the file trees that show the issues inside of our source tree for direct use and examination

this also relates to #2207

@nicoddemus
Copy link
Member

I agree having the explicit files would make things easier at first glance.

We would have to figure out how to specify some things which are needed by acceptance tests:

  1. Which arguments to pass to pytest;
  2. Should tests fail or pass, if so how many of each?
  3. How to match against expected output.
  4. Sometimes we run a "test suite" more than once, changing things in between runs and then asserting either status code, expected output, etc.
  5. I'm sure I'm missing some other cases.

@nicoddemus nicoddemus added the type: infrastructure improvement to development/releases/CI structure label Jul 28, 2017
@RonnyPfannschmidt
Copy link
Member Author

  1. and 2. and 3. are part of handling this issue
  2. is clearly a case where creating a copy of the file tree is sensible, i do wonder if we can track changes to be inflicted on those files in a better way

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: infrastructure improvement to development/releases/CI structure
Projects
None yet
Development

No branches or pull requests

2 participants