Food101 new dataset api #5584

yassineAlouini · 2022-03-10T14:01:53Z

In this PR, I migrate the Food101 dataset to the new datasets API. For the code, I have got some inspiration from the dtd.py new dataset particulary for the Demultiplexer usage.

The pre-commits and food101 tests seem to work locally, hope that's the case within the CI/CD.

Any reviews are welcome, many thanks in advance. 😺

facebook-github-bot · 2022-03-10T14:01:57Z

Hi @yassineAlouini!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

facebook-github-bot · 2022-03-10T14:02:00Z

💊 CI failures summary and remediations

As of commit facd139 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

…ision-1 into food101_new_dataset_api

pmeier

Hey @yassineAlouini thanks a lot for the PR. I did an initial pass and have some comments inline. In general this looks already quite good and we only need to iron out some wrinkles.

torchvision/datasets/food101.py

torchvision/prototype/datasets/_builtin/food101.py

yassineAlouini · 2022-03-14T10:26:03Z

Thanks @pmeier for the code review, will try to finish the fixes by Wednesday. I should have the CLA by then as well. 👌

Do you have a new PR to suggest? Should I work on another dataset migration since I am now familiar with the design or is there something more urgent to work on? Thanks for any pointer.

facebook-github-bot · 2022-03-17T10:26:57Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

yassineAlouini · 2022-03-17T10:31:40Z

@pmeier CLA is uploaded, let me know if I need to finish anything else. Thanks for your help and guidance.

pmeier

Hey @yassineAlouini, I have some suggestions inline. I have yet to review the mock data for the tests. Will do that in the next review round.

torchvision/prototype/datasets/_builtin/food101.py

…1_new_dataset_api

…ision-1 into food101_new_dataset_api

…1_new_dataset_api

pmeier · 2022-03-28T08:40:02Z

@yassineAlouini I see you are updating the PR from the current main branch. Please let me know if the PR is ready for another round of review.

yassineAlouini · 2022-03-28T08:53:12Z

@pmeier All is resolved as far as I know. The only part that needs reviewing is the test mock I guess? Let me know if anything else needs more work. 👌

pmeier

One comment about the meta data files. Mock data generation looks good in general. I'll have another look after the comment is resolved, since it affects the generation.

torchvision/prototype/datasets/_builtin/food101.py

pmeier · 2022-03-28T09:21:09Z

Also, don't worry about CI at the moment. There is a larger outage.

…nd simplify code + update mock data.

…ision-1 into food101_new_dataset_api

pmeier

Hey @yassineAlouini. I've added some suggestion and simplifications inline. Note that this doesn't mean that your version didn't work (it did!). Thanks a lot for the continued interest.

test/builtin_dataset_mocks.py

yassineAlouini · 2022-03-28T15:41:29Z

@pmeier Mock data should be good now. Ready for a final check I guess. 👌

pmeier

LGTM, thanks a ton @yassineAlouini!

NicolasHug

Stamping

Summary: * [FEAT] Start implementing Food101 using the new datasets API. WIP. * [FEAT] Generate Food101 categories and start the test mock. * [FEAT] food101 dataset code seems to work now. * [TEST] food101 mock update. * [FIX] Some fixes thanks to running food101 tests. * [FIX] Fix mypy checks for the food101 file. * [FIX] Remove unused numpy. * [FIX] Some changes thanks to code review. * [ENH] More idomatic dataset code thanks to code review. * [FIX] Remove unused cast. * [ENH] Set decompress and extract to True for some performance gains. * [FEAT] Use the preprocess=decompress keyword. * [ENH] Use the train and test.txt file instead of the .json variants and simplify code + update mock data. * [ENH] Better food101 mock data generation. * [FIX] Remove a useless print. Reviewed By: NicolasHug Differential Revision: D35393170 fbshipit-source-id: c7f51bdfb2e05913593cdcba9e30994557afaf87 Co-authored-by: Philip Meier <[email protected]>

Yassine Alouini added 5 commits March 4, 2022 18:47

[FEAT] Start implementing Food101 using the new datasets API. WIP.

ce4a471

[FEAT] Generate Food101 categories and start the test mock.

b6a6fef

[FEAT] food101 dataset code seems to work now.

536add0

[TEST] food101 mock update.

c0eb2cf

[FIX] Some fixes thanks to running food101 tests.

f02477b

pytorch-bot bot added the ciflow/default label Mar 10, 2022

Yassine Alouini and others added 4 commits March 10, 2022 18:14

[FIX] Fix mypy checks for the food101 file.

d039446

Merge branch 'main' into food101_new_dataset_api

8a9b3b8

Merge branch 'food101_new_dataset_api' of github.com:yassineAlouini/v…

b7af2ad

…ision-1 into food101_new_dataset_api

[FIX] Remove unused numpy.

bfbffee

pmeier self-requested a review March 11, 2022 15:35

pmeier added module: datasets prototype labels Mar 11, 2022

pmeier reviewed Mar 14, 2022

View reviewed changes

yassineAlouini and others added 3 commits March 15, 2022 10:27

Merge branch 'pytorch:main' into food101_new_dataset_api

0e6729f

[FIX] Some changes thanks to code review.

89b44b6

Merge branch 'main' into food101_new_dataset_api

7e7fa83

facebook-github-bot added the cla signed label Mar 17, 2022

Merge branch 'main' into food101_new_dataset_api

b34bed8

Merge branch 'main' into food101_new_dataset_api

3ff1858

pmeier reviewed Mar 17, 2022

View reviewed changes

Yassine Alouini added 4 commits March 17, 2022 15:47

Merge branch 'main' of github.com:yassineAlouini/vision-1 into food10…

0186699

…1_new_dataset_api

[ENH] More idomatic dataset code thanks to code review.

ec3e1ab

Merge branch 'food101_new_dataset_api' of github.com:yassineAlouini/v…

2546c2d

…ision-1 into food101_new_dataset_api

[FIX] Remove unused cast.

eff25ee

Merge branch 'main' into food101_new_dataset_api

1670434

pmeier mentioned this pull request Mar 23, 2022

Add sun397 prototype datapipe #5667

Open

Yassine Alouini added 2 commits March 23, 2022 15:56

[ENH] Set decompress and extract to True for some performance gains.

d9299ec

Merge branch 'food101_new_dataset_api' of github.com:yassineAlouini/v…

ebbb455

…ision-1 into food101_new_dataset_api

pmeier mentioned this pull request Mar 23, 2022

fix HttpResource.resolve() with preprocessing #5669

Merged

Yassine Alouini and others added 6 commits March 24, 2022 10:21

Merge branch 'main' of github.com:yassineAlouini/vision-1 into food10…

e1c1548

…1_new_dataset_api

[FEAT] Use the preprocess=decompress keyword.

2935593

Merge branch 'main' into food101_new_dataset_api

b1f592b

Merge branch 'main' into food101_new_dataset_api

e6c69af

Merge branch 'main' into food101_new_dataset_api

fda2f6a

Merge branch 'main' into food101_new_dataset_api

db04b1f

pmeier reviewed Mar 28, 2022

View reviewed changes

torchvision/prototype/datasets/_builtin/food101.py Outdated Show resolved Hide resolved

Yassine Alouini added 2 commits March 28, 2022 12:06

[ENH] Use the train and test.txt file instead of the .json variants a…

fc473f8

…nd simplify code + update mock data.

Merge branch 'food101_new_dataset_api' of github.com:yassineAlouini/v…

11a5a5d

…ision-1 into food101_new_dataset_api

pmeier reviewed Mar 28, 2022

View reviewed changes

Yassine Alouini added 2 commits March 28, 2022 16:36

[ENH] Better food101 mock data generation.

cee454d

[FIX] Remove a useless print.

c62e1d9

pmeier approved these changes Mar 28, 2022

View reviewed changes

pmeier requested a review from NicolasHug March 28, 2022 16:57

NicolasHug approved these changes Apr 4, 2022

View reviewed changes

pmeier added 2 commits April 4, 2022 10:45

Merge branch 'main' into food101_new_dataset_api

9c9c35e

Merge branch 'main' into food101_new_dataset_api

facd139

pmeier merged commit 31e503f into pytorch:main Apr 4, 2022

yassineAlouini deleted the food101_new_dataset_api branch April 4, 2022 12:01

pmeier mentioned this pull request Apr 6, 2022

Food101 #5342

Closed

Food101 new dataset api #5584

Food101 new dataset api #5584

Uh oh!

Conversation

yassineAlouini commented Mar 10, 2022

Uh oh!

facebook-github-bot commented Mar 10, 2022

Action Required

Process

Uh oh!

facebook-github-bot commented Mar 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

pmeier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yassineAlouini commented Mar 14, 2022

Uh oh!

facebook-github-bot commented Mar 17, 2022

Uh oh!

yassineAlouini commented Mar 17, 2022

Uh oh!

pmeier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmeier commented Mar 28, 2022

Uh oh!

yassineAlouini commented Mar 28, 2022

Uh oh!

pmeier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pmeier commented Mar 28, 2022

Uh oh!

pmeier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yassineAlouini commented Mar 28, 2022

Uh oh!

pmeier left a comment

Choose a reason for hiding this comment

Uh oh!

NicolasHug left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

facebook-github-bot commented Mar 10, 2022 •

edited

Loading