Skip to content

Commit 401bd14

Browse files
brandon-wadaAuto-format Bot
brandon-wada
and
Auto-format Bot
authored
Expose evaluations and metrics (#319)
Makes metrics and evaluations available --------- Co-authored-by: Auto-format Bot <[email protected]>
1 parent 0fa6244 commit 401bd14

30 files changed

+2256
-24
lines changed

‎generated/.openapi-generator/FILES

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,11 @@ docs/ImageQueriesApi.md
2929
docs/ImageQuery.md
3030
docs/ImageQueryTypeEnum.md
3131
docs/InlineResponse200.md
32+
docs/InlineResponse2001.md
33+
docs/InlineResponse2001EvaluationResults.md
34+
docs/InlineResponse2002.md
35+
docs/InlineResponse200Summary.md
36+
docs/InlineResponse200SummaryClassCounts.md
3237
docs/LabelValue.md
3338
docs/LabelValueRequest.md
3439
docs/LabelsApi.md
@@ -98,6 +103,11 @@ groundlight_openapi_client/model/escalation_type_enum.py
98103
groundlight_openapi_client/model/image_query.py
99104
groundlight_openapi_client/model/image_query_type_enum.py
100105
groundlight_openapi_client/model/inline_response200.py
106+
groundlight_openapi_client/model/inline_response2001.py
107+
groundlight_openapi_client/model/inline_response2001_evaluation_results.py
108+
groundlight_openapi_client/model/inline_response2002.py
109+
groundlight_openapi_client/model/inline_response200_summary.py
110+
groundlight_openapi_client/model/inline_response200_summary_class_counts.py
101111
groundlight_openapi_client/model/label_value.py
102112
groundlight_openapi_client/model/label_value_request.py
103113
groundlight_openapi_client/model/mode_enum.py
@@ -132,4 +142,6 @@ setup.cfg
132142
setup.py
133143
test-requirements.txt
134144
test/__init__.py
145+
test/test_payload_template.py
146+
test/test_payload_template_request.py
135147
tox.ini

‎generated/README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,8 @@ Class | Method | HTTP request | Description
123123
*DetectorsApi* | [**create_detector**](docs/DetectorsApi.md#create_detector) | **POST** /v1/detectors |
124124
*DetectorsApi* | [**delete_detector**](docs/DetectorsApi.md#delete_detector) | **DELETE** /v1/detectors/{id} |
125125
*DetectorsApi* | [**get_detector**](docs/DetectorsApi.md#get_detector) | **GET** /v1/detectors/{id} |
126+
*DetectorsApi* | [**get_detector_evaluation**](docs/DetectorsApi.md#get_detector_evaluation) | **GET** /v1/detectors/{id}/evaluation |
127+
*DetectorsApi* | [**get_detector_metrics**](docs/DetectorsApi.md#get_detector_metrics) | **GET** /v1/detectors/{detector_id}/metrics |
126128
*DetectorsApi* | [**list_detectors**](docs/DetectorsApi.md#list_detectors) | **GET** /v1/detectors |
127129
*DetectorsApi* | [**update_detector**](docs/DetectorsApi.md#update_detector) | **PATCH** /v1/detectors/{id} |
128130
*EdgeApi* | [**get_model_urls**](docs/EdgeApi.md#get_model_urls) | **GET** /v1/edge/fetch-model-urls/{detector_id}/ |
@@ -161,6 +163,11 @@ Class | Method | HTTP request | Description
161163
- [ImageQuery](docs/ImageQuery.md)
162164
- [ImageQueryTypeEnum](docs/ImageQueryTypeEnum.md)
163165
- [InlineResponse200](docs/InlineResponse200.md)
166+
- [InlineResponse2001](docs/InlineResponse2001.md)
167+
- [InlineResponse2001EvaluationResults](docs/InlineResponse2001EvaluationResults.md)
168+
- [InlineResponse2002](docs/InlineResponse2002.md)
169+
- [InlineResponse200Summary](docs/InlineResponse200Summary.md)
170+
- [InlineResponse200SummaryClassCounts](docs/InlineResponse200SummaryClassCounts.md)
164171
- [LabelValue](docs/LabelValue.md)
165172
- [LabelValueRequest](docs/LabelValueRequest.md)
166173
- [ModeEnum](docs/ModeEnum.md)

‎generated/docs/DetectorsApi.md

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ Method | HTTP request | Description
77
[**create_detector**](DetectorsApi.md#create_detector) | **POST** /v1/detectors |
88
[**delete_detector**](DetectorsApi.md#delete_detector) | **DELETE** /v1/detectors/{id} |
99
[**get_detector**](DetectorsApi.md#get_detector) | **GET** /v1/detectors/{id} |
10+
[**get_detector_evaluation**](DetectorsApi.md#get_detector_evaluation) | **GET** /v1/detectors/{id}/evaluation |
11+
[**get_detector_metrics**](DetectorsApi.md#get_detector_metrics) | **GET** /v1/detectors/{detector_id}/metrics |
1012
[**list_detectors**](DetectorsApi.md#list_detectors) | **GET** /v1/detectors |
1113
[**update_detector**](DetectorsApi.md#update_detector) | **PATCH** /v1/detectors/{id} |
1214

@@ -243,6 +245,160 @@ Name | Type | Description | Notes
243245
- **Accept**: application/json
244246

245247

248+
### HTTP response details
249+
250+
| Status code | Description | Response headers |
251+
|-------------|-------------|------------------|
252+
**200** | | - |
253+
254+
[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
255+
256+
# **get_detector_evaluation**
257+
> InlineResponse2001 get_detector_evaluation(id)
258+
259+
260+
261+
Get Detector evaluation results. The result is null if there isn't enough ground truth data to evaluate the detector. Returns the time of the evaulation, total ground truth labels, the ml based kfold accuracies, and the system accuracies at different confidence thresholds
262+
263+
### Example
264+
265+
* Api Key Authentication (ApiToken):
266+
267+
```python
268+
import time
269+
import groundlight_openapi_client
270+
from groundlight_openapi_client.api import detectors_api
271+
from groundlight_openapi_client.model.inline_response2001 import InlineResponse2001
272+
from pprint import pprint
273+
# Defining the host is optional and defaults to https://api.groundlight.ai/device-api
274+
# See configuration.py for a list of all supported configuration parameters.
275+
configuration = groundlight_openapi_client.Configuration(
276+
host = "https://api.groundlight.ai/device-api"
277+
)
278+
279+
# The client must configure the authentication and authorization parameters
280+
# in accordance with the API server security policy.
281+
# Examples for each auth method are provided below, use the example that
282+
# satisfies your auth use case.
283+
284+
# Configure API key authorization: ApiToken
285+
configuration.api_key['ApiToken'] = 'YOUR_API_KEY'
286+
287+
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
288+
# configuration.api_key_prefix['ApiToken'] = 'Bearer'
289+
290+
# Enter a context with an instance of the API client
291+
with groundlight_openapi_client.ApiClient(configuration) as api_client:
292+
# Create an instance of the API class
293+
api_instance = detectors_api.DetectorsApi(api_client)
294+
id = "id_example" # str |
295+
296+
# example passing only required values which don't have defaults set
297+
try:
298+
api_response = api_instance.get_detector_evaluation(id)
299+
pprint(api_response)
300+
except groundlight_openapi_client.ApiException as e:
301+
print("Exception when calling DetectorsApi->get_detector_evaluation: %s\n" % e)
302+
```
303+
304+
305+
### Parameters
306+
307+
Name | Type | Description | Notes
308+
------------- | ------------- | ------------- | -------------
309+
**id** | **str**| |
310+
311+
### Return type
312+
313+
[**InlineResponse2001**](InlineResponse2001.md)
314+
315+
### Authorization
316+
317+
[ApiToken](../README.md#ApiToken)
318+
319+
### HTTP request headers
320+
321+
- **Content-Type**: Not defined
322+
- **Accept**: application/json
323+
324+
325+
### HTTP response details
326+
327+
| Status code | Description | Response headers |
328+
|-------------|-------------|------------------|
329+
**200** | | - |
330+
331+
[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
332+
333+
# **get_detector_metrics**
334+
> InlineResponse200 get_detector_metrics(detector_id)
335+
336+
337+
338+
Get Detector metrics, primarily the counts of different types of labels
339+
340+
### Example
341+
342+
* Api Key Authentication (ApiToken):
343+
344+
```python
345+
import time
346+
import groundlight_openapi_client
347+
from groundlight_openapi_client.api import detectors_api
348+
from groundlight_openapi_client.model.inline_response200 import InlineResponse200
349+
from pprint import pprint
350+
# Defining the host is optional and defaults to https://api.groundlight.ai/device-api
351+
# See configuration.py for a list of all supported configuration parameters.
352+
configuration = groundlight_openapi_client.Configuration(
353+
host = "https://api.groundlight.ai/device-api"
354+
)
355+
356+
# The client must configure the authentication and authorization parameters
357+
# in accordance with the API server security policy.
358+
# Examples for each auth method are provided below, use the example that
359+
# satisfies your auth use case.
360+
361+
# Configure API key authorization: ApiToken
362+
configuration.api_key['ApiToken'] = 'YOUR_API_KEY'
363+
364+
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
365+
# configuration.api_key_prefix['ApiToken'] = 'Bearer'
366+
367+
# Enter a context with an instance of the API client
368+
with groundlight_openapi_client.ApiClient(configuration) as api_client:
369+
# Create an instance of the API class
370+
api_instance = detectors_api.DetectorsApi(api_client)
371+
detector_id = "detector_id_example" # str |
372+
373+
# example passing only required values which don't have defaults set
374+
try:
375+
api_response = api_instance.get_detector_metrics(detector_id)
376+
pprint(api_response)
377+
except groundlight_openapi_client.ApiException as e:
378+
print("Exception when calling DetectorsApi->get_detector_metrics: %s\n" % e)
379+
```
380+
381+
382+
### Parameters
383+
384+
Name | Type | Description | Notes
385+
------------- | ------------- | ------------- | -------------
386+
**detector_id** | **str**| |
387+
388+
### Return type
389+
390+
[**InlineResponse200**](InlineResponse200.md)
391+
392+
### Authorization
393+
394+
[ApiToken](../README.md#ApiToken)
395+
396+
### HTTP request headers
397+
398+
- **Content-Type**: Not defined
399+
- **Accept**: application/json
400+
401+
246402
### HTTP response details
247403

248404
| Status code | Description | Response headers |

‎generated/docs/InlineResponse200.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
## Properties
55
Name | Type | Description | Notes
66
------------ | ------------- | ------------- | -------------
7-
**username** | **str** | The user&#39;s username | [optional]
7+
**summary** | [**InlineResponse200Summary**](InlineResponse200Summary.md) | | [optional]
88
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
99

1010
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)

‎generated/docs/InlineResponse2001.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# InlineResponse2001
2+
3+
4+
## Properties
5+
Name | Type | Description | Notes
6+
------------ | ------------- | ------------- | -------------
7+
**evaluation_results** | [**InlineResponse2001EvaluationResults**](InlineResponse2001EvaluationResults.md) | | [optional]
8+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
9+
10+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
11+
12+

‎generated/docs/InlineResponse2001EvaluationResults.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# InlineResponse2001EvaluationResults
2+
3+
4+
## Properties
5+
Name | Type | Description | Notes
6+
------------ | ------------- | ------------- | -------------
7+
**eval_timestamp** | **datetime** | | [optional]
8+
**total_ground_truth_examples** | **int, none_type** | | [optional]
9+
**kfold_pooled__balanced_accuracy** | **float** | | [optional]
10+
**kfold_pooled__positive_accuracy** | **float** | | [optional]
11+
**kfold_pooled__negative_accuracy** | **float** | | [optional]
12+
**balanced_system_accuracies** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
13+
**positive_system_accuracies** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
14+
**negative_system_accuracies** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
15+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
16+
17+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
18+
19+

‎generated/docs/InlineResponse2002.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# InlineResponse2002
2+
3+
4+
## Properties
5+
Name | Type | Description | Notes
6+
------------ | ------------- | ------------- | -------------
7+
**username** | **str** | The user&#39;s username | [optional]
8+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
9+
10+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
11+
12+

‎generated/docs/InlineResponse200Summary.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# InlineResponse200Summary
2+
3+
4+
## Properties
5+
Name | Type | Description | Notes
6+
------------ | ------------- | ------------- | -------------
7+
**num_ground_truth** | **int** | | [optional]
8+
**num_current_source_human** | **int** | | [optional]
9+
**class_counts** | [**InlineResponse200SummaryClassCounts**](InlineResponse200SummaryClassCounts.md) | | [optional]
10+
**unconfident_counts** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
11+
**total_iqs** | **int** | | [optional]
12+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
13+
14+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
15+
16+

‎generated/docs/InlineResponse200SummaryClassCounts.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# InlineResponse200SummaryClassCounts
2+
3+
4+
## Properties
5+
Name | Type | Description | Notes
6+
------------ | ------------- | ------------- | -------------
7+
**source_ml** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
8+
**source_human** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
9+
**cloud_labeler** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
10+
**cloud** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
11+
**total** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional]
12+
**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
13+
14+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
15+
16+

‎generated/docs/TextRecognitionResult.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
## Properties
55
Name | Type | Description | Notes
66
------------ | ------------- | ------------- | -------------
7-
**text** | **str, none_type** | |
7+
**text** | **str** | |
88
**truncated** | **bool** | |
99
**confidence** | **float, none_type** | | [optional]
1010
**source** | **str** | | [optional]

‎generated/docs/UserApi.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Method | HTTP request | Description
88

99

1010
# **who_am_i**
11-
> InlineResponse200 who_am_i()
11+
> InlineResponse2002 who_am_i()
1212
1313

1414

@@ -22,7 +22,7 @@ Retrieve the current user.
2222
import time
2323
import groundlight_openapi_client
2424
from groundlight_openapi_client.api import user_api
25-
from groundlight_openapi_client.model.inline_response200 import InlineResponse200
25+
from groundlight_openapi_client.model.inline_response2002 import InlineResponse2002
2626
from pprint import pprint
2727
# Defining the host is optional and defaults to https://api.groundlight.ai/device-api
2828
# See configuration.py for a list of all supported configuration parameters.
@@ -60,7 +60,7 @@ This endpoint does not need any parameter.
6060

6161
### Return type
6262

63-
[**InlineResponse200**](InlineResponse200.md)
63+
[**InlineResponse2002**](InlineResponse2002.md)
6464

6565
### Authorization
6666

0 commit comments

Comments
 (0)