updated shadow_variants with Alwin's work (aws#225)

* initial commit

* minor fix

* minor fixes

* add inference experiment notebook

add inference experiment notebook

* Add shadow endpoint notebook (aws#215)

* initial commit

* add inference experiment notebook

add inference experiment notebook

Co-authored-by: Qingwei Li<[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Qingwei Li <[email protected]>

* Revert "Add shadow endpoint notebook (aws#215)" (aws#218)

This reverts commit b6d2fd203f7f85670478556e902ad2bb86a1a882.

* reformat

* reviewer's comments addressed

* clear output

* fix and reformat nb

* reformat nb

* remove notebook

* markdown change

* Alwin's edit

add edits from Alwin

* reformat

* change folder name

Co-authored-by: EC2 Default User <[email protected]>
Co-authored-by: Shreya Pandit <[email protected]>
Co-authored-by: Qingwei Li <[email protected]>
Co-authored-by: EC2 Default User <[email protected]>
Co-authored-by: atqy <[email protected]>
Co-authored-by: atqy <[email protected]>

Author: 6 people

sagemaker-shadow-variant/Shadow_variant.ipynb

@@ -273,13 +273,13 @@
     "            \"InitialVariantWeight\": 1,\n",
     "        }\n",
     "    ],\n",
-    "    ShadowProductionVariants=[  # Type: Array of ProductionVariant (https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html) objects\n",
+    "    ShadowProductionVariants=[\n",
     "        {\n",
     "            \"VariantName\": shadow_variant_name,\n",
     "            \"ModelName\": model_name2,\n",
-    "            \"InitialInstanceCount\": 3,\n",
-    "            \"InitialVariantWeight\": 0.5,\n",
     "            \"InstanceType\": \"ml.m5.xlarge\",\n",
+    "            \"InitialInstanceCount\": 1,\n",
+    "            \"InitialVariantWeight\": 0.5,\n",
     "        }\n",
     "    ],\n",
     ")\n",
@@ -536,7 +536,7 @@
     "tags": []
    },
    "source": [
-    "Finally, let us review the 4xx and 5xx returned by the model. "
+    "Finally, let us review the 4xx, 5xx and total model errors returned by the model serving container. "
    ]
   },
   {
@@ -546,7 +546,10 @@
    "outputs": [],
    "source": [
     "Invocation4xxErrors = plot_endpoint_invocation_metrics(endpoint_name, \"Invocation4XXErrors\", \"Sum\")\n",
-    "Invocation5xxErrors = plot_endpoint_invocation_metrics(endpoint_name, \"Invocation5XXErrors\", \"Sum\")"
+    "Invocation5xxErrors = plot_endpoint_invocation_metrics(endpoint_name, \"Invocation5XXErrors\", \"Sum\")\n",
+    "Invocation5xxErrors = plot_endpoint_invocation_metrics(\n",
+    "    endpoint_name, \"InvocationModelErrors\", \"Sum\"\n",
+    ")"
    ]
   },
   {
@@ -557,10 +560,30 @@
    "source": [
     "We can consider promoting the shadow model if we do not see any differences in 4xx and 5xx errors between the production shadow variants. \n",
     "\n",
-    "To promote the shadow model to production, create a new endpoint configuration with current ShadowProductionVariant as the new ProductionVariant and removing the ShadowProductionVariant. This will remove the current ProductionVariant and promote the shadow variant to become the new production variant. As always, all SageMaker updates are orchestrated as blue/green deployments under the hood and there is no loss of availability while performing the update. Optionally, you can leverage [Deployment Guardrails](https://docs.aws.amazon.com/sagemaker/latest/dg/deployment-guardrails.html) if you want to use linear and canary traffic shifting modes and auto rollbacks during your update\n",
-    "\n",
+    "To promote the shadow model to production, create a new endpoint configuration with current ShadowProductionVariant as the new ProductionVariant and removing the ShadowProductionVariant. This will remove the current ProductionVariant and promote the shadow variant to become the new production variant. As always, all SageMaker updates are orchestrated as blue/green deployments under the hood and there is no loss of availability while performing the update. Optionally, you can leverage [Deployment Guardrails](https://docs.aws.amazon.com/sagemaker/latest/dg/deployment-guardrails.html) if you want to use all-at-once traffic shifting and auto rollbacks during your update."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "promote_ep_config_name = f\"PromoteShadow-EpConfig-{datetime.now():%Y-%m-%d-%H-%M-%S}\"\n",
     "\n",
-    "If you do not want to create multiple endpoint configurations and want SageMaker to manage the end to end workflow of creating, managing, and acting on the results of the shadow tests, consider using the SageMaker Inference Experiement APIs/Console experience. As stated earlier, they enable you to setup shadow tests for a predefined duration of time, monitor the progress through a live dashboard, presents clean up options upon completion, and act on the results. To get started, please navigate to the 'Shadow Tests' section of the SageMaker Inference console. "
+    "create_endpoint_config_response = sm.create_endpoint_config(\n",
+    "    EndpointConfigName=promote_ep_config_name,\n",
+    "    ProductionVariants=[\n",
+    "        {\n",
+    "            \"VariantName\": shadow_variant_name,\n",
+    "            \"ModelName\": model_name2,\n",
+    "            \"InstanceType\": \"ml.m5.xlarge\",\n",
+    "            \"InitialInstanceCount\": 2,\n",
+    "            \"InitialVariantWeight\": 1.0,\n",
+    "        }\n",
+    "    ],\n",
+    ")\n",
+    "print(f\"Created EndpointConfig: {create_endpoint_config_response['EndpointConfigArn']}\")"
    ]
   },
   {
@@ -569,7 +592,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "plot_endpoint_invocation_metrics(endpoint_name, \"CPUUtilization\", \"Average\")"
+    "update_endpoint_api_response = sm.update_endpoint(\n",
+    "    EndpointName=endpoint_name,\n",
+    "    EndpointConfigName=promote_ep_config_name,\n",
+    ")\n",
+    "\n",
+    "wait_for_endpoint_in_service(endpoint_name)\n",
+    "\n",
+    "sm.describe_endpoint(EndpointName=endpoint_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you do not want to create multiple endpoint configurations and want SageMaker to manage the end to end workflow of creating, managing, and acting on the results of the shadow tests, consider using the SageMaker Inference Experiement APIs/Console experience. As stated earlier, they enable you to setup shadow tests for a predefined duration of time, monitor the progress through a live dashboard, presents clean up options upon completion, and act on the results. To get started, please navigate to the 'Shadow Tests' section of the SageMaker Inference console. "
    ]
   },
   {
@@ -593,7 +630,10 @@
    "outputs": [],
    "source": [
     "sm.delete_endpoint(EndpointName=endpoint_name)\n",
-    "sm.delete_endpoint_config(EndpointConfigName=ep_config_name)"
+    "sm.delete_endpoint_config(EndpointConfigName=ep_config_name)\n",
+    "sm.delete_endpoint_config(EndpointConfigName=promote_ep_config_name)\n",
+    "sm.delete_model(ModelName=model_name)\n",
+    "sm.delete_model(ModelName=model_name2)"
    ]
   }
  ],