Run prototype datasets test also on Windows and macOS?

[pmeier]

There are now three separate occasions of bugs in the prototype datasets that were not picked up by our CI.

• fix fromfile on windows #4980
• Fix DTD prototype dataset on Windows #5227
• fix eurosat prototype mock data setup #5549

We didn't notice them, because they happen on Windows on macOS, but we only run the prototype tests on Linux. Given that the prototypes work with files and the path handling is different on all platforms, this can be problematic.

The torchdata team brought the failures to our attention, since they run our dataset tests on the full matrix. I propose we also run the prototype datasets tests on Windows and macOS to avoid these post mortems. To limit the extra needed CI resources I would only run the tests only on Python3.7.

That means we are looking at 2 extra CI runs per PR with roughly 10 minutes runtime.

cc @pmeier @seemethere @bjuncek

[pmeier]

After #5618, the total runtime decreased to about 5 minutes, in which the largest chunk is taken up by building rather than testing.

[pmeier]

Another error not picked up by our CI: #5801 (comment).

[datumbox]

As discussed offline with @pmeier, this is the 4th time that platform specific issues creep in on datasets and need to be tested on each platform. Given that the prototype execution time is only 5 minutes, adding 2 jobs for Windows and macOS sounds like a reasonable tradeoff.

@seemethere @bigfootjon Just highlighting this case as another example of why often we need to run tests across different setups. This is something that was debated at #5479, so I thought to provide a bit more evidence. This of course doesn't mean that we shouldn't reduce the costs. I think the solution is to restructure the CI jobs to separate tests that can break on different platforms from those that don't. This way we can reduce significantly the costs by running across all configs only those that must be run everywhere. We are in a bit of a crunch resource-wise but we hope we could pick this up within Q2.