Expectations test utils

[ivarflakstad]

This PR adds the concept of Expectations, which can be used to simplify handling different expected results depending on the environment the test is running in.

result = fn_being_tested()

expectations = Expectations({
    (None, None): "1",  // Default
    ("cuda, 8): "2"),
    ("cuda, 7): "3"),
    ("rocm, None): "4"),
)
expected = expectations.get_expectation()  # Automatically gets the correct expected result from detected environment.
assert result == expected

Addresses #36561

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

My first look has the impression that this is a bit too much. I have to think it a bit more and see what we can reduce but still get the improvements we want to have.

[ArthurZucker]

Agree a bit with @ydshieh (who is the owner 👀 ) a bit too bloated.
Happy to have a simple and explicit function to help with these issue + an updated test to show proper usage

[ivarflakstad]

Agree a bit with @ydshieh (who is the owner 👀 ) a bit too bloated. Happy to have a simple and explicit function to help with these issue + an updated test to show proper usage

Let me see if I can simplify it a bit. I'll fix a test with it as proper usage as well 👍

[ivarflakstad]

@ArthurZucker @ydshieh
I've simplified quite a bit. Added usage of it as well here ☺️

[ydshieh]

Hi @ivarflakstad I am wondering what's your thoughts about:

• Remove Expectation
• Make Expectations a subclass of dict
  • no need to have _inner and _default
  • key being (type, major), value being result
  • move _get_device_properties out to a place (as functioin) that could be reused
  • A new method named get_closed (get_best or whatever) with one argument named key
    • merge all the logic that needs to get the closet expectation inside its body and return it
  • to use it, just expection = my_expectation(_get_device_properties())

Overall, my suggestion is just to make it less abstract: just a simple class inherited a familar class (dict) with a new method get_best .

WDYK?

[ydshieh]

just a few nits 🙏

[ydshieh]

Just a final nits and we are good to go! Thank you

[ydshieh]

If you can think of another name other than type, it would be nice, as type is a built-in function.