integrating results for comps #111
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kirahowe
reviewed
Jun 6, 2024
kirahowe
reviewed
Jun 6, 2024
polaris/hub/client.py
Outdated
| @@ -866,12 +869,15 @@ def evaluate_competition( | |||
| A `BenchmarkResults` object. | |||
There was a problem hiding this comment.
This doesn't return a BenchmarkResults anymore right? Just the JSON payload from the hub? We could turn it into one, but I don't think we need to. It's probably fine to just say this returns a 201 response on success or other error response from the hub.
cwognum
approved these changes
Jun 6, 2024
kirahowe
reviewed
Jun 6, 2024
| benchmark_owner=competition.owner, | ||
|
|
||
| # Inform the user about where to find their newly created artifact. | ||
| result_url = urljoin( |
There was a problem hiding this comment.
Can we grab this URL from the response Location header instead of constructing it manually?
There was a problem hiding this comment.
We could, I was just following the pattern from the Benchmark evaluate function
Co-authored-by: Cas Wognum <[email protected]>
Andrewq11
added a commit
that referenced
this pull request
Aug 19, 2024
* competition wip * wip * wip * adding methods for interfacing w/ competitions * Continuing to integrate polaris client with the Hub for comps * comp wip * updating date serializer * Competition evaluation (#103) * call hub evaluate endpoint from client evaluate_competitions method * add super basic test for evaluating competitions * be more specific in evaluate_benchmark signature * Update polaris/hub/client.py Co-authored-by: Andrew Quirke <[email protected]> * start refactoring object dependencies out of evaluation logic * refactor test subset object out of evaluation logic * clean up as much as possible for now * updating date serializer * call hub evaluate endpoint from client evaluate_competitions method * Update polaris/competition/_competition.py Co-authored-by: Andrew Quirke <[email protected]> * updating date serializer * call hub evaluate endpoint from client evaluate_competitions method * add super basic test for evaluating competitions * comp wip * updating date serializer * call hub evaluate endpoint from client evaluate_competitions method * fix bad merge resolution * only send competition artifact ID to hub --------- Co-authored-by: Andrew Quirke <[email protected]> Co-authored-by: Andrew Quirke <[email protected]> * Use evaluation logic directly in hub, no need for wrapper (#109) * use evaluation logic directly in hub, no need for wrapper * include evaluate_benchmark in package * remove unnecessary imports * read incoming scores sent as json * light formatting updates * updating fallback version for dev build * integrating results for comps (#111) * integrating results for comps * Update polaris/hub/client.py Co-authored-by: Cas Wognum <[email protected]> * addressing comments & adding CompetitionResults class * test competition evalution works for multi-column dataframes * add single column test to competition evaluation * fix multitask-single-test-set cases * fix bug with multi-test-set benchmarks * adding functions to serialize & deserialize pred objs for external eval * updating return for evaluate_competition method in client * updating evaluate_competition method to pass additional result info to hub --------- Co-authored-by: Cas Wognum <[email protected]> Co-authored-by: Kira McLean <[email protected]> * updates to enable fetching & interacting with comps * updating requirement for eval name * Feat/competition/eval (#114) * integrating results for comps * Update polaris/hub/client.py Co-authored-by: Cas Wognum <[email protected]> * addressing comments & adding CompetitionResults class * test competition evalution works for multi-column dataframes * add single column test to competition evaluation * fix multitask-single-test-set cases * fix bug with multi-test-set benchmarks * adding functions to serialize & deserialize pred objs for external eval * updating return for evaluate_competition method in client * updating evaluate_competition method to pass additional result info to hub * refuse early to upload a competition with a zarr-based dataset * removing merge conflicts --------- Co-authored-by: Andrew Quirke <[email protected]> Co-authored-by: Andrew Quirke <[email protected]> Co-authored-by: Cas Wognum <[email protected]> * test that all rows of a competition test set will have at least a value (#116) * update competition evaluation to support y_prob * run ruff on all files and fix issues * fix wrong url printout after upload * Clarifying typing for nested types * removing if_exists arg from comps * raising error for trying to make zarr comp * updating name of ArtifactType to ArtifactSubtype * updating comments & removing redundant class attributes * moving split validator logic from comp spec to benchmark spec * removing redundant checks from CompetitionDataset class * creating pydantic model for comp predictions * split validator logic, redundant pydantic checks, comp pred pydantic model * changes for comps wrap up * Adding CompetitionsPredictionsType * adding conversion validator for comp prediction type * setting predictions validator as class method * Using self instead of cls for field validators * removing model validation on fetch from hub * Creating HubOwner object in comp result eval method * Documentation & tutorials for competitions * Removing create comp method, fixing failing tests, updating benchmark label struct * Updating docs for create comp & benchmark pred structure * tiny wording change in competition tutorial * Addressing PR feedback * fixing tests & removing dataset redefinition from CompetitionDataset class * Commenting out line in tutorial to fix test * fixing formatting * small fixes & depending on tableContent for dataset storage info --------- Co-authored-by: Andrew Quirke <[email protected]> Co-authored-by: Andrew Quirke <[email protected]> Co-authored-by: Cas Wognum <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelogs
evalute_competitionmethod in client to handle result from Polaris Hub