Skip to content

export data notebook and restructure fodlers #1129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 9, 2023
Merged

export data notebook and restructure fodlers #1129

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
146f55b
export data notebook and restructure fodlers
zeke-emerson Jun 8, 2023
50bb682
update readme and links
zeke-emerson Jun 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 54 additions & 63 deletions examples/README.md
View file
Open in desktop

Large diffs are not rendered by default.

343 changes: 343 additions & 0 deletions examples/basics/export_data.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,343 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {},
"cells": [
{
"metadata": {},
"source": [
"<td>\n",
" <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
"</td>"
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"<td>\n",
"<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/basics/export_data.ipynb\" target=\"_blank\"><img\n",
"src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
"</td>\n",
"\n",
"<td>\n",
"<a href=\"https://github.com/Labelbox/labelbox-python/tree/master/examples/basics/export_data.ipynb\" target=\"_blank\"><img\n",
"src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
"</td>"
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"# Export data\n",
"How to export data for projects, datasets, slices, and models, with examples for each type of v2 export along with details on optional parameters and filters."
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"!pip install -q \"labelbox[data]\""
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"import labelbox as lb"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"# API Key and Client\n",
"See the developer guide for [creating an API key](https://docs.labelbox.com/reference/create-api-key)."
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"API_KEY = \"\"\n",
"client = lb.Client(api_key=API_KEY)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"## Export data rows from a project\n",
"For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) developer guide.\n",
"\n",
"### Parameters\n",
"When you export data rows from a project, you may choose to include or exclude certain attributes, including:\n",
"- `attachments`\n",
"- `metadata_fields`\n",
"- `data_row_details`\n",
"- `project_details`\n",
"- `label_details`\n",
"- `performance_details`\n",
"- `interpolated_frames`\n",
" - Only applicable for video data rows.\n",
"\n",
"### Filters\n",
"When you export data rows from a project, you can specify the included data rows with the following filters:\n",
"- `last_activity_at`\n",
"- `label_created_at`\n",
"- `data_row_ids`\n",
"\n",
"#### Filter details\n",
"You can set the range for `last_activity_at` and `label_created_at` in the following formats: \n",
"- `YYYY-MM-DD`\n",
"- `YYYY-MM-DD hh:mm:ss`\n",
"- `YYYY-MM-DDThh:mm:ss\u00b1hhmm` (ISO 8601)\n",
"\n",
"The ISO 8061 format allows you to specify the timezone, while the other two formats assume timezone from the user's workspace settings.\n",
"\n",
"The `last_activity_at` filter captures the creation and modification of labels, metadata, workflow status, comments, and reviews.\n",
"\n",
"If you wish to specify data rows to export, uncomment the `data_row_ids` filter and provide a list of applicable IDs. The data rows must be part of a batch attached to the project in question. You can provide up to 2,000 data row IDs."
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"# Insert the project ID of the project from which you wish to export data rows.\n",
"PROJECT_ID = \"\"\n",
"project = client.get_project(PROJECT_ID)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"# Set the export params to include/exclude certain fields. \n",
"export_params= {\n",
" \"attachments\": True,\n",
" \"metadata_fields\": True,\n",
" \"data_row_details\": True,\n",
" \"project_details\": True,\n",
" \"label_details\": True,\n",
" \"performance_details\": True,\n",
" \"interpolated_frames\": True\n",
"}\n",
"\n",
"# Note: Filters follow AND logic, so typically using one filter is sufficient.\n",
"filters= {\n",
" \"last_activity_at\": [\"2000-01-01 00:00:00\", \"2050-01-01 00:00:00\"],\n",
" \"label_created_at\": [\"2000-01-01 00:00:00\", \"2050-01-01 00:00:00\"],\n",
" # \"data_row_ids\": [\"<data_row_id>\", \"<data_row_id>\"] \n",
"}\n",
"\n",
"export_task = project.export_v2(params=export_params, filters=filters)\n",
"export_task.wait_till_done()\n",
"\n",
"if export_task.errors:\n",
" print(export_task.errors)\n",
"\n",
"export_json = export_task.result\n",
"print(\"results: \", export_json)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"## Export data rows from a dataset\n",
"For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) developer guide.\n",
"\n",
"### Parameters\n",
"When you export data rows from a dataset, you may choose to include or exclude certain attributes, including:\n",
"- `attachments`\n",
"- `metadata_fields`\n",
"- `data_row_details`\n",
"- `project_details`\n",
"- `label_details`\n",
"- `performance_details`\n",
"- `interpolated_frames`\n",
" - Only applicable for video data rows.\n",
"- `project_ids`\n",
" - Accepts a list of project IDs. If provided, the labels created _in these projects_ on the exported data rows will be included. \n",
"- `model_run_ids`\n",
" - Accepts a list of model run IDs. If provided, the labels and predicitions created _in these model runs_ will be included. \n",
"\n",
"### Filters\n",
"When you export data rows from a project, you can specify the included data rows with the following filters:\n",
"- `last_activity_at`\n",
"- `label_created_at`\n",
"- `data_row_ids`\n",
"\n",
"See the _Export data rows from a project_ section above for additional details on each filter. "
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"# Insert the dataset ID of the dataset from which you wish to export data rows.\n",
"DATASET_ID = \"\"\n",
"dataset = client.get_dataset(DATASET_ID)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"# Set the export params to include/exclude certain fields.\n",
"export_params= {\n",
" \"attachments\": True,\n",
" \"metadata_fields\": True,\n",
" \"data_row_details\": True,\n",
" \"project_details\": True,\n",
" \"label_details\": True,\n",
" \"performance_details\": True,\n",
" \"interpolated_frames\": True,\n",
" # \"project_ids\": [\"<project_id>\", \"<project_id>\"],\n",
" # \"model_run_ids\": [\"<model_run_id>\", \"<model_run_id>\"] \n",
"}\n",
"\n",
"# Note: Filters follow AND logic, so typically using one filter is sufficient.\n",
"filters= {\n",
" \"last_activity_at\": [\"2000-01-01 00:00:00\", \"2050-01-01 00:00:00\"]\n",
"}\n",
"\n",
"export_task = dataset.export_v2(params=export_params, filters=filters)\n",
"export_task.wait_till_done()\n",
"\n",
"if export_task.errors:\n",
" print(export_task.errors)\n",
"\n",
"export_json = export_task.result\n",
"print(\"results: \", export_json)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"## Export data rows from a slice\n",
"For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) developer guide.\n",
"\n",
"### Parameters\n",
"When exporting from a slice, you can apply the same parameters as exporting from a dataset.\n",
"\n",
"### Filters\n",
"No filters are applicable to exports from a slice. All the data rows of the slice must be exported."
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"# Insert the Catalog slice ID of the slice from which you wish to export data rows.\n",
"CATALOG_SLICE_ID = \"\"\n",
"catalog_slice = client.get_catalog_slice(CATALOG_SLICE_ID)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"# Set the export params to include/exclude certain fields.\n",
"export_params = {\n",
" \"attachments\": True,\n",
" \"metadata_fields\": True,\n",
" \"data_row_details\": True,\n",
" \"project_details\": True,\n",
" \"label_details\": True,\n",
" \"performance_details\": True,\n",
" \"interpolated_frames\": True,\n",
" # \"project_ids\": [\"<project_id>\", \"<project_id>\"],\n",
" # \"model_run_ids\": [\"<model_run_id>\", \"<model_run_id>\"]\n",
"}\n",
"\n",
"export_task = catalog_slice.export_v2(params=export_params)\n",
"export_task.wait_till_done()\n",
"\n",
"if export_task.errors:\n",
" print(export_task.errors)\n",
"\n",
"export_json = export_task.result\n",
"print(\"results: \", export_json)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"## Export data rows from a model run\n",
"For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) developer guide.\n",
"\n",
"### Parameters\n",
"- `attachments`\n",
"- `metadata_fields`\n",
"- `data_row_details`\n",
"- `interpolated_frames`\n",
" - Only applicable for video data rows.\n",
"- `predictions`\n",
" - If true, all predictions made in the model run will be included for each data row in the export.\n",
"\n",
"### Filters\n",
"No filters are applicable to exports from a model run. All the data rows of the model run must be exported.\n"
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"# Insert the model run ID of the model run from which you wish to export data rows.\n",
"MODEL_RUN_ID = \"\"\n",
"model_run = client.get_model_run(MODEL_RUN_ID)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"# Set the export params to include/exclude certain fields.\n",
"export_params = {\n",
" \"attachments\": True,\n",
" \"metadata_fields\": True,\n",
" \"data_row_details\": True,\n",
" \"interpolated_frames\": True,\n",
" \"predictions\": True\n",
"}\n",
"\n",
"export_task = model_run.export_v2(params=export_params)\n",
"export_task.wait_till_done()\n",
"\n",
"if export_task.errors:\n",
" print(export_task.errors)\n",
"\n",
"export_json = export_task.result\n",
"print(\"results: \", export_json)"
],
"cell_type": "code",
"outputs": [],
"execution_count": null
}
]
}
2 changes: 1 addition & 1 deletion examples/basics/labels.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
"metadata": {},
"source": [
"#### *** This section explains how to use the Label object but It is reccomended that you use bulk export for exporting labels *** \n",
"* [Bulk export examples](https://github.com/Labelbox/labelbox-python/tree/master/examples/label_export)\n",
"* [Export data](https://github.com/Labelbox/labelbox-python/tree/master/examples/basics/export_data.ipynb)\n",
"* [Label format documentation](https://docs.labelbox.com/data-model/en/index-en#label)"
],
"cell_type": "markdown"
Expand Down
Loading