Incorrect usage of Flask's app context during testing

[rsyring]

Our keg.testing:ContextManager is designed to cache the application instance as well as the app context. However, after reading the Flask docs on app context, it has become obvious that I'm using the app context wrong. Despite it's name, it's not intended to have the same life cycle as the app instance. It's intended to have a lifetime almost the same as the request context.

This is actually the underlying reason behind the bug I thought Flask-WTF had: pallets-eco/flask-wtf#301

It looks like some kind of pytest fixture support is what we really need in order to make the app instance available to our tests instead of relying on flask.current_app.

Some good ideas here:

https://github.com/pytest-dev/pytest-flask

[rsyring]

A lot of our apps assume an app context is going to be available in setup_class() and setup fixtures for things like entity usage. However, my attempt to fix Keg for this issue breaks those assumptions and would require a lot of work from the app developers to fix their tests. I've been hoping pytest would address this, but the issue hasn't received much attention. There is, however, a plugin now available:

pytest-dev/pytest#517 (comment)

[rsyring]

pytest 4.2 will have a fix for us:

pytest-dev/pytest#517 (comment)

[guruofgentoo]

This is now affecting both CSRF and how flask-login caches the logged-in user on flask.g. I took a look at this during this maintenance cycle to see if I could wrap this up, and there's a few things at play here that make it non-trivial to finish up. I've punted for this cycle of dependency updates.

Taking keg-auth's test suite as a sample, since it doesn't have the same type of custom app setup stuff going on during tests:

• it is possible to put in a session-scope autouse fixture in conftest.py, that pushes an app context to satisfy all of our setup_class usage
• a number of test decorators, like mock or skip marks, can be broken if there's a reference to something needing the app context
  • I believe all of those usages can get moved to inside the test method
• the main reason I punted was that there's still some weirdness around how the CSRF session token is applied for the webtest client. It will seem like tests are functioning normally, then if the session is cleared (like on logout), the token is lost, and there doesn't seem to be a way to get it back in there.
  • running more than one webtest client at a time seems to run into problems with this as well
• flask-webtest does not push an app context before running requests. If that's how it works in general (that we get a fresh app context and request context for each request), though, IMO it really should be doing that
  • we could make that update, as maintainers of flask-webtest now
  • testing with that app context push in place did resolve a number of issues around how these requests were working
  • the side effect is that ORM objects in views are no longer on the same session as the tests themselves. Which is not a bad thing, but breaks the assumptions in some of our tests.