-
Notifications
You must be signed in to change notification settings - Fork 9
Competitions feature #121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Competitions feature #121
Changes from 47 commits
Commits
Show all changes
48 commits
Select commit
Hold shift + click to select a range
1700882
competition wip
Andrewq11 1cdd330
wip
Andrewq11 cb60c04
wip
Andrewq11 252bae0
Merge branch 'main' into feat/competitions
Andrewq11 9e14e89
adding methods for interfacing w/ competitions
Andrewq11 3a55b27
Continuing to integrate polaris client with the Hub for comps
Andrewq11 b09fd08
comp wip
Andrewq11 26210b2
updating date serializer
Andrewq11 6704b6f
Competition evaluation (#103)
kiramclean 54c42e2
Use evaluation logic directly in hub, no need for wrapper (#109)
kiramclean 9edd693
light formatting updates
Andrewq11 818ff12
updating fallback version for dev build
Andrewq11 40c17f6
integrating results for comps (#111)
Andrewq11 8c2daae
updates to enable fetching & interacting with comps
Andrewq11 a4cfcbe
updating requirement for eval name
Andrewq11 44c5d7f
Feat/competition/eval (#114)
kiramclean 66c1913
test that all rows of a competition test set will have at least a val…
kiramclean 83a77a5
Merge branch 'main' into feat/competitions
kirahowe 36f04c0
update competition evaluation to support y_prob
kirahowe 173a8e3
Merge remote-tracking branch 'refs/remotes/origin/feat/competitions' …
kirahowe fd16b1b
Merge branch 'main' into feat/competitions
kirahowe d50a48d
run ruff on all files and fix issues
kirahowe efd739b
fix wrong url printout after upload
kirahowe c8c462e
Clarifying typing for nested types
Andrewq11 03ed6df
removing if_exists arg from comps
Andrewq11 352a6d5
raising error for trying to make zarr comp
Andrewq11 1169c15
updating name of ArtifactType to ArtifactSubtype
Andrewq11 09f5e7d
updating comments & removing redundant class attributes
Andrewq11 c7c5fb7
moving split validator logic from comp spec to benchmark spec
Andrewq11 7096a0f
removing redundant checks from CompetitionDataset class
Andrewq11 fd009f2
creating pydantic model for comp predictions
Andrewq11 250c612
split validator logic, redundant pydantic checks, comp pred pydantic …
Andrewq11 b774242
changes for comps wrap up
Andrewq11 ef94203
Adding CompetitionsPredictionsType
Andrewq11 7378d1f
adding conversion validator for comp prediction type
Andrewq11 d5179f3
setting predictions validator as class method
Andrewq11 ea4cb22
Using self instead of cls for field validators
Andrewq11 889f147
removing model validation on fetch from hub
Andrewq11 5686411
Creating HubOwner object in comp result eval method
Andrewq11 363938b
Documentation & tutorials for competitions
Andrewq11 ce7b4d5
Removing create comp method, fixing failing tests, updating benchmark…
Andrewq11 6b3bfe3
Updating docs for create comp & benchmark pred structure
Andrewq11 0005c78
tiny wording change in competition tutorial
Andrewq11 802d3bd
Addressing PR feedback
Andrewq11 54533da
fixing tests & removing dataset redefinition from CompetitionDataset …
Andrewq11 d6f2472
Commenting out line in tutorial to fix test
Andrewq11 fa39613
fixing formatting
Andrewq11 4bac9d2
small fixes & depending on tableContent for dataset storage info
Andrewq11 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
::: polaris.dataset.CompetitionDataset | ||
options: | ||
filters: ["!^_"] | ||
|
||
--- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
::: polaris.evaluate.CompetitionPredictions | ||
|
||
--- | ||
|
||
::: polaris.evaluate.CompetitionResults | ||
|
||
--- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
::: polaris.competition.CompetitionSpecification | ||
|
||
--- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,253 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"id": "40f99374-b47e-4f84-bdb9-148a11f9c07d", | ||
"metadata": { | ||
"editable": true, | ||
"slideshow": { | ||
"slide_type": "" | ||
}, | ||
"tags": [] | ||
}, | ||
"source": [ | ||
"# Participating in a Competition\n", | ||
"\n", | ||
"<div class=\"admonition abstract highlight\">\n", | ||
" <p class=\"admonition-title\">In short</p>\n", | ||
" <p>This tutorial walks you through how to fetch an active competition from Polaris, prepare your predictions and then submit them for secure evaluation by the Polaris Hub.</p>\n", | ||
"</div>\n", | ||
"\n", | ||
"Participating in a competition on Polaris is very similar to participating in a standard benchmark. The main difference lies in how predictions are prepared and how they are evaluated. We'll touch on each of these topics later in the tutorial. \n", | ||
"\n", | ||
"Before continuing, please ensure you are logged into Polaris." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 8, | ||
"id": "3d66f466", | ||
"metadata": { | ||
"editable": true, | ||
"slideshow": { | ||
"slide_type": "" | ||
}, | ||
"tags": [ | ||
"remove_cell" | ||
] | ||
}, | ||
"outputs": [], | ||
"source": [ | ||
"# Note: Cell is tagged to not show up in the mkdocs build\n", | ||
"%load_ext autoreload\n", | ||
"%autoreload 2" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 13, | ||
"id": "9b465ea4-7c71-443b-9908-3f9e567ee4c4", | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stderr", | ||
"output_type": "stream", | ||
"text": [ | ||
"\u001b[32m2024-08-09 18:05:23.205\u001b[0m | \u001b[32m\u001b[1mSUCCESS \u001b[0m | \u001b[36mpolaris.hub.client\u001b[0m:\u001b[36mlogin\u001b[0m:\u001b[36m267\u001b[0m - \u001b[32m\u001b[1mYou are successfully logged in to the Polaris Hub.\u001b[0m\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"import polaris as po\n", | ||
"from polaris.hub.client import PolarisHubClient\n", | ||
"\n", | ||
"# Don't forget to add your Polaris Hub username below!\n", | ||
"MY_POLARIS_USERNAME = \"\"\n", | ||
"\n", | ||
"client = PolarisHubClient()\n", | ||
"client.login()" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "5edee39f-ce29-4ae6-91ce-453d9190541b", | ||
"metadata": {}, | ||
"source": [ | ||
"## Fetching a Competition\n", | ||
"\n", | ||
"As with standard benchmarks, Polaris provides simple APIs that allow you to quickly fetch a competition from the Polaris Hub. All you need is the unique identifier for the competition which follows the format of `competition_owner`/`competition_name`." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 10, | ||
"id": "4e004589-6c48-4232-b353-b1700536dde6", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"competition_id = \"polaris/hello-world-competition\"\n", | ||
"competition = po.load_competition(competition_id)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "36f3e829", | ||
"metadata": {}, | ||
"source": [ | ||
"## Participate in the Competition\n", | ||
"The Polaris library is designed to make it easy to participate in a competition. In just a few lines of code, we can get the train and test partition, access the associated data in various ways and evaluate our predictions. There's two main API endpoints. \n", | ||
"\n", | ||
"- `get_train_test_split()`: For creating objects through which we can access the different dataset partitions.\n", | ||
"- `evaluate()`: For evaluating a set of predictions in accordance with the competition protocol." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "d8605928", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"train, test = competition.get_train_test_split()" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "e78bf878", | ||
"metadata": {}, | ||
"source": [ | ||
"The created test and train objects support various flavours to access the data." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "7b17bb31", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"# The objects are iterable\n", | ||
"for x, y in train:\n", | ||
" pass\n", | ||
"\n", | ||
"# The objects can be indexed\n", | ||
"for i in range(len(train)):\n", | ||
" x, y = train[i]\n", | ||
"\n", | ||
"# The objects have properties to access all data at once\n", | ||
"x = train.inputs\n", | ||
"y = train.targets" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "5ec12825", | ||
"metadata": {}, | ||
"source": [ | ||
"Now, let's create some predictions against the official Polaris `hello-world-competition`. We will train a simple random forest model on the ECFP representation through scikit-learn and datamol, and then we will submit our results for secure evaluation by the Polaris Hub." | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "902353bc", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"import datamol as dm\n", | ||
"from sklearn.ensemble import RandomForestRegressor\n", | ||
"\n", | ||
"# Load the competition (automatically loads the underlying dataset as well)\n", | ||
"competition = po.load_competition(\"polaris/hello-world-benchmark\")\n", | ||
"\n", | ||
"# Get the split and convert SMILES to ECFP fingerprints by specifying an featurize function.\n", | ||
"train, test = competition.get_train_test_split(featurization_fn=dm.to_fp)\n", | ||
"\n", | ||
"# Define a model and train\n", | ||
"model = RandomForestRegressor(max_depth=2, random_state=0)\n", | ||
"model.fit(train.X, train.y)\n", | ||
"\n", | ||
"predictions = model.predict(test.X)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "1a36e334", | ||
"metadata": {}, | ||
"source": [ | ||
"Now that we have created some predictions, we can construct a `CompetitionPredictions` object that will prepare our predictions for evaluation by the Polaris Hub. Here, you can also add metadata to your predictions to better describe your results and how you achieved them. " | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "2b36e09b", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"from polaris.evaluate import CompetitionPredictions\n", | ||
"\n", | ||
"competition_predictions = CompetitionPredictions(\n", | ||
" name=\"hello-world-result\",\n", | ||
" predictions=predictions,\n", | ||
" github_url=\"https://github.com/polaris-hub/polaris-hub\",\n", | ||
" paper_url=\"https://polarishub.io/\",\n", | ||
" description=\"Hello, World!\",\n", | ||
")" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "5ff06a9c", | ||
"metadata": {}, | ||
"source": [ | ||
"Once your `CompetitionPredictions` object is created, you're ready to submit them for evaluation! This will automatically save your result to the Polaris Hub, but it will be private. You can choose to make it public through the Polaris web application. " | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "e684c611", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"results = competition.evaluate(competition_predictions)\n", | ||
"\n", | ||
"client.close()" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "44973556", | ||
"metadata": {}, | ||
"source": [ | ||
"That's it! Just like that you have partaken in your first Polaris competition. Keep an eye on that leaderboard and best of luck in your future competitions!\n", | ||
"\n", | ||
"The End.\n", | ||
"\n", | ||
"---" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3 (ipykernel)", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.12.2" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 5 | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.