diff --git a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb new file mode 100644 index 000000000..fc69c01f9 --- /dev/null +++ b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb @@ -0,0 +1,611 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cell-00", + "metadata": {}, + "source": [ + "# Serve embeddings on demand with SageMaker serverless inference\n", + "\n", + "## What serverless inference is\n", + "\n", + "A real-time SageMaker endpoint runs on instances you pick and pay for by the\n", + "hour, whether or not any requests arrive. You choose the instance type and\n", + "count, attach an autoscaling policy, and keep at least one instance warm so the\n", + "endpoint can answer immediately. That is the right shape for steady, predictable\n", + "traffic, and wasteful for everything else.\n", + "\n", + "**Serverless inference** takes the instances out of the picture. You give\n", + "SageMaker two numbers (how much memory a copy of the model needs and how many\n", + "requests it may handle at once) and it provisions compute on demand, scales it\n", + "with traffic, and scales it to **zero** when the endpoint is idle. You pay per\n", + "millisecond of request processing and for the data processed, and nothing while\n", + "no requests are in flight, so an idle on-demand endpoint is free. (The exception\n", + "is provisioned concurrency, described below: reserving warm workers bills around\n", + "the clock, so with it set the endpoint keeps costing money even while completely\n", + "idle.)\n", + "\n", + "![SageMaker serverless inference workflow](https://docs.aws.amazon.com/images/sagemaker/latest/dg/images/serverless-endpoints-how-it-works.png)\n", + "\n", + "The trade-off is the **cold start**. When a request arrives and no compute is\n", + "warm (right after deployment, or after an idle stretch), SageMaker has to start\n", + "a worker, pull the container, and load the model before it can answer. That\n", + "first request is slow; the ones behind it, while the worker stays warm, are\n", + "fast. If you cannot tolerate the occasional slow request, *provisioned\n", + "concurrency* keeps a set number of workers warm at all times, at the cost of\n", + "paying for them whether or not they are used.\n", + "\n", + "## When it fits, and when it doesn't\n", + "\n", + "Serverless is the cost-effective choice when traffic is **intermittent or\n", + "unpredictable** and the workload can absorb an occasional cold start:\n", + "\n", + "- Internal tools, dashboards, and low-QPS APIs that sit idle most of the day.\n", + "- Dev, test, and staging endpoints you do not want billed around the clock.\n", + "- New models whose traffic you cannot forecast yet.\n", + "\n", + "Reach for a provisioned real-time endpoint instead when traffic is **steady and\n", + "high** (a busy always-on instance is cheaper per request and has no cold starts),\n", + "or when a strict latency SLA leaves no room for one. And mind the hard limits:\n", + "serverless is **CPU-only** (no GPUs), memory is capped at **6 GB**, and a single\n", + "endpoint handles at most **200 concurrent requests**. Large or GPU-bound models\n", + "do not belong here, and long-running or large-payload jobs belong on\n", + "[asynchronous inference](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html)\n", + "instead.\n", + "\n", + "## The use case we picked\n", + "\n", + "This tutorial serves an **embedding model for a low-traffic semantic-search\n", + "box**. Embedding a user's live query is small, synchronous, and latency-shaped,\n", + "but on an internal search tool the queries arrive in bursts with long idle gaps\n", + "between them, so paying for an always-on instance is hard to justify. That is\n", + "exactly the shape serverless was built for. (The offline other half, embedding\n", + "a whole corpus in one large batch, is a job for\n", + "[asynchronous inference](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html);\n", + "this notebook is only the online query side.)\n", + "\n", + "The model is `BAAI/bge-small-en-v1.5`: 384-dimensional vectors and a ~130 MB\n", + "download that loads comfortably inside the smallest serverless memory tier. It\n", + "is served with\n", + "[Text Embeddings Inference (TEI)](https://huggingface.co/docs/text-embeddings-inference),\n", + "Hugging Face's container for embedding models.\n", + "\n", + "References:\n", + "\n", + "- [SageMaker serverless inference](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html)\n", + "- [Serverless endpoint operations](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-create-invoke-update-delete.html)\n", + "- [Minimizing cold starts with provisioned concurrency](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-autoscale.html)\n" + ] + }, + { + "cell_type": "markdown", + "id": "cell-01", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "Run the next cell before importing the SDK. It installs the SageMaker Python\n", + "SDK into the active kernel.\n", + "\n", + "You also need an existing SageMaker execution role with access to SageMaker,\n", + "S3, CloudWatch, and the ECR repository that hosts the TEI container. Serverless\n", + "inference is available in a\n", + "[subset of AWS regions](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html).\n", + "Run this in one of them.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-02", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install \"sagemaker>=3.0.0\" matplotlib --upgrade --quiet" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-03", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime as dt\n", + "import json\n", + "import math\n", + "import os\n", + "import time\n", + "\n", + "import boto3\n", + "from botocore.exceptions import ClientError\n", + "\n", + "from sagemaker.core import image_uris\n", + "from sagemaker.core.helper.session_helper import Session, get_execution_role\n", + "from sagemaker.core.inference_config import ServerlessInferenceConfig\n", + "from sagemaker.serve import ModelBuilder, ModelServer\n", + "from sagemaker.serve.builder.schema_builder import SchemaBuilder" + ] + }, + { + "cell_type": "markdown", + "id": "cell-04", + "metadata": {}, + "source": [ + "## The model and how serverless is sized\n", + "\n", + "A serverless endpoint is configured by two numbers instead of an instance type:\n", + "\n", + "- **Memory** (`MEMORY_SIZE_IN_MB`) must be at least as large as one copy of the\n", + " model plus its container. It has to be one of 1024, 2048, 3072, 4096, 5120, or\n", + " 6144 MB, and SageMaker gives the container more vCPUs as you go up.\n", + " `BAAI/bge-small-en-v1.5` is tiny, so 2 GB is ample; a larger embedding model\n", + " would need a larger tier.\n", + "- **Max concurrency** (`MAX_CONCURRENCY`) caps how many requests the endpoint\n", + " processes at once, up to 200. Invocations beyond it are throttled rather than\n", + " queued, which keeps one endpoint from consuming your whole account quota.\n", + "\n", + "Leaving `PROVISIONED_CONCURRENCY` unset keeps the endpoint fully on-demand: it\n", + "scales to zero when idle and you accept cold starts. Set it to a small integer\n", + "(no greater than `MAX_CONCURRENCY`) to keep that many workers permanently warm:\n", + "no cold starts, but you pay for the reserved capacity around the clock.\n", + "\n", + "To serve a different model, set `HF_MODEL_ID`, set `EMBEDDING_DIM` to its output\n", + "dimension, and raise `MEMORY_SIZE_IN_MB` if the model is larger.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-05", + "metadata": {}, + "outputs": [], + "source": [ + "PROJECT = \"hf-serverless-embed\"\n", + "RUN_ID = dt.datetime.now(dt.timezone.utc).strftime(\"%Y%m%d%H%M%S\")\n", + "\n", + "MODEL_ID = os.getenv(\"HF_MODEL_ID\", \"BAAI/bge-small-en-v1.5\")\n", + "EMBEDDING_DIM = int(os.getenv(\"EMBEDDING_DIM\", \"384\"))\n", + "TEI_VERSION = os.getenv(\"TEI_VERSION\", \"1.8.2\")\n", + "ENDPOINT_NAME = os.getenv(\"SAGEMAKER_ENDPOINT_NAME\", f\"{PROJECT}-{RUN_ID}\")\n", + "\n", + "# Serverless sizing. Memory must be one of 1024/2048/3072/4096/5120/6144 MB and\n", + "# at least the size of one model copy; max concurrency is capped at 200.\n", + "MEMORY_SIZE_IN_MB = int(os.getenv(\"MEMORY_SIZE_IN_MB\", \"2048\"))\n", + "MAX_CONCURRENCY = int(os.getenv(\"MAX_CONCURRENCY\", \"5\"))\n", + "\n", + "# Leave unset for a pure scale-to-zero endpoint; set it to keep workers warm.\n", + "_provisioned = os.getenv(\"PROVISIONED_CONCURRENCY\")\n", + "PROVISIONED_CONCURRENCY = int(_provisioned) if _provisioned and int(_provisioned) > 0 else None\n", + "\n", + "# Set CLEANUP=false to inspect the endpoint after the tutorial finishes.\n", + "CLEANUP = os.getenv(\"CLEANUP\", \"true\").lower() not in {\"0\", \"false\", \"no\"}\n", + "\n", + "assert MEMORY_SIZE_IN_MB in {1024, 2048, 3072, 4096, 5120, 6144}, MEMORY_SIZE_IN_MB\n", + "assert 1 <= MAX_CONCURRENCY <= 200, MAX_CONCURRENCY\n", + "assert PROVISIONED_CONCURRENCY is None or PROVISIONED_CONCURRENCY <= MAX_CONCURRENCY" + ] + }, + { + "cell_type": "markdown", + "id": "cell-06", + "metadata": {}, + "source": [ + "## Set up the SageMaker session\n", + "\n", + "The endpoint runs under a SageMaker execution role: an IAM role that grants\n", + "access to S3, ECR, and CloudWatch. Set `SAGEMAKER_EXECUTION_ROLE_ARN` to the\n", + "role you want to use, or `SAGEMAKER_EXECUTION_ROLE_NAME` if you only have its\n", + "name. Inside SageMaker Studio or a notebook instance you can leave both unset\n", + "and the role is detected automatically.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-07", + "metadata": {}, + "outputs": [], + "source": [ + "requested_region = os.getenv(\"AWS_REGION\") or os.getenv(\"AWS_DEFAULT_REGION\")\n", + "boto_session = boto3.Session(region_name=requested_region) if requested_region else boto3.Session()\n", + "sess = Session(boto_session=boto_session)\n", + "region = sess.boto_region_name\n", + "\n", + "sm = boto_session.client(\"sagemaker\")\n", + "\n", + "\n", + "def resolve_role(session, sagemaker_session):\n", + " role_arn = os.getenv(\"SAGEMAKER_EXECUTION_ROLE_ARN\")\n", + " if role_arn:\n", + " return role_arn\n", + "\n", + " role_name = os.getenv(\"SAGEMAKER_EXECUTION_ROLE_NAME\")\n", + " if role_name:\n", + " iam = session.client(\"iam\")\n", + " return iam.get_role(RoleName=role_name)[\"Role\"][\"Arn\"]\n", + "\n", + " return get_execution_role(sagemaker_session=sagemaker_session)\n", + "\n", + "\n", + "role = resolve_role(boto_session, sess)\n", + "\n", + "print(f\"region: {region}\")\n", + "print(f\"role: {role}\")\n", + "print(f\"endpoint: {ENDPOINT_NAME}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-08", + "metadata": {}, + "source": [ + "## Select the TEI serving container\n", + "\n", + "Serverless inference is CPU-only, so we always use the CPU build of Text\n", + "Embeddings Inference, `huggingface-tei-cpu`. `image_uris.retrieve` resolves the\n", + "image URI for the current region. No instance type is involved. Serverless has\n", + "no instances to size.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-09", + "metadata": {}, + "outputs": [], + "source": [ + "image_uri = image_uris.retrieve(\n", + " framework=\"huggingface-tei-cpu\",\n", + " region=region,\n", + " version=TEI_VERSION,\n", + " image_scope=\"inference\",\n", + ")\n", + "print(image_uri)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-10", + "metadata": {}, + "source": [ + "## Build the model\n", + "\n", + "`ModelBuilder` describes the SageMaker model: the Hub model ID, the TEI\n", + "container, the model server, and a small input/output example. The container\n", + "downloads the model from the Hub when the worker starts. We do not pass an\n", + "instance type (serverless provisions its own compute), and the explicit\n", + "`image_uri` above is what pins the container. For gated or private models, set\n", + "`HF_TOKEN` (or `HUGGING_FACE_HUB_TOKEN`) before running the notebook.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-11", + "metadata": {}, + "outputs": [], + "source": [ + "hf_token = os.getenv(\"HF_TOKEN\") or os.getenv(\"HUGGING_FACE_HUB_TOKEN\")\n", + "\n", + "env_vars = {\n", + " \"HF_MODEL_ID\": MODEL_ID,\n", + " \"MAX_BATCH_TOKENS\": os.getenv(\"MAX_BATCH_TOKENS\", \"16384\"),\n", + " \"MAX_CLIENT_BATCH_SIZE\": os.getenv(\"MAX_CLIENT_BATCH_SIZE\", \"32\"),\n", + "}\n", + "\n", + "if hf_token:\n", + " env_vars[\"HF_TOKEN\"] = hf_token\n", + " env_vars[\"HUGGING_FACE_HUB_TOKEN\"] = hf_token\n", + "\n", + "resource_tags = [\n", + " {\"Key\": \"Project\", \"Value\": PROJECT},\n", + " {\"Key\": \"ModelId\", \"Value\": MODEL_ID},\n", + " {\"Key\": \"CreatedBy\", \"Value\": \"hf-sagemaker-docs\"},\n", + "]\n", + "\n", + "model_builder = ModelBuilder(\n", + " model=MODEL_ID,\n", + " role_arn=role,\n", + " sagemaker_session=sess,\n", + " image_uri=image_uri,\n", + " model_server=ModelServer.TEI,\n", + " env_vars=env_vars,\n", + " schema_builder=SchemaBuilder(\n", + " sample_input={\"inputs\": [\"who wrote the origin of species\"]},\n", + " sample_output=[[0.0] * EMBEDDING_DIM],\n", + " ),\n", + ")\n", + "\n", + "tei_model = model_builder.build(model_name=f\"{PROJECT}-model-{RUN_ID}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-12", + "metadata": {}, + "source": [ + "## Deploy the serverless endpoint\n", + "\n", + "`ServerlessInferenceConfig` carries the two sizing numbers and, optionally,\n", + "provisioned concurrency. Passing it to `deploy` as the `inference_config` is\n", + "what makes the endpoint serverless; there is no `instance_type` or\n", + "`initial_instance_count` to set. Deployment still takes a few minutes while\n", + "SageMaker registers the endpoint; the first *invocation* is where the cold start\n", + "shows up.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-13", + "metadata": {}, + "outputs": [], + "source": [ + "serverless_config = ServerlessInferenceConfig(\n", + " memory_size_in_mb=MEMORY_SIZE_IN_MB,\n", + " max_concurrency=MAX_CONCURRENCY,\n", + " provisioned_concurrency=PROVISIONED_CONCURRENCY,\n", + ")\n", + "\n", + "endpoint = model_builder.deploy(\n", + " endpoint_name=ENDPOINT_NAME,\n", + " inference_config=serverless_config,\n", + " container_timeout_in_seconds=900,\n", + " tags=resource_tags,\n", + " wait=True,\n", + ")\n", + "\n", + "endpoint_description = sm.describe_endpoint(EndpointName=ENDPOINT_NAME)\n", + "endpoint_config_name = endpoint_description[\"EndpointConfigName\"]\n", + "model_name = tei_model.model_name\n", + "\n", + "print(f\"endpoint status: {endpoint_description['EndpointStatus']}\")\n", + "print(f\"endpoint config: {endpoint_config_name}\")\n", + "print(f\"model: {model_name}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-14", + "metadata": {}, + "source": [ + "## Invoke the endpoint and watch the cold start\n", + "\n", + "TEI takes `{\"inputs\": [...]}` and returns one vector per input. The first call\n", + "below lands on a cold endpoint: SageMaker starts a worker and loads the model,\n", + "so it is slow. The calls after it reuse the warm worker and return in a fraction\n", + "of the time. The gap between them *is* the cold start: the cost you accept in\n", + "exchange for scaling to zero.\n", + "\n", + "The numbers vary from run to run, and CloudWatch reports the cold-start portion\n", + "separately as the `OverheadLatency` metric. To force another cold start, leave\n", + "the endpoint idle for a while and invoke again.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-15", + "metadata": {}, + "outputs": [], + "source": [ + "def embed(texts):\n", + " response = endpoint.invoke(\n", + " body=json.dumps({\"inputs\": texts}),\n", + " content_type=\"application/json\",\n", + " accept=\"application/json\",\n", + " )\n", + " return json.loads(response.body.read())\n", + "\n", + "\n", + "def timed_embed(texts):\n", + " start = time.perf_counter()\n", + " vectors = embed(texts)\n", + " return vectors, time.perf_counter() - start\n", + "\n", + "\n", + "cold_vectors, cold_latency = timed_embed([\"who wrote the origin of species\"])\n", + "assert len(cold_vectors) == 1 and len(cold_vectors[0]) == EMBEDDING_DIM\n", + "\n", + "warm_latencies = []\n", + "for _ in range(3):\n", + " _, warm_latency = timed_embed([\"a warm follow-up request\"])\n", + " warm_latencies.append(warm_latency)\n", + "\n", + "print(f\"cold-start invocation: {cold_latency:.2f} s\")\n", + "print(f\"warm invocations: {', '.join(f'{value:.2f} s' for value in warm_latencies)}\")\n", + "print(f\"embedding dimensions: {len(cold_vectors[0])}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-15a", + "metadata": {}, + "source": [ + "### Visualize the cold-start overhead\n", + "\n", + "The gap between the first (cold) call and the warm calls that follow is the\n", + "cold-start overhead: the time SageMaker spends starting a worker and loading\n", + "the model before it can serve a response. Plotting the latencies measured\n", + "above makes that gap concrete. After an idle stretch the endpoint scales back\n", + "to zero, so a later call pays the cold start again." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-15b", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "os.environ.setdefault(\"MPLCONFIGDIR\", \"/tmp/matplotlib\")\n", + "import matplotlib.pyplot as plt\n", + "\n", + "warm_mean = sum(warm_latencies) / len(warm_latencies)\n", + "overhead = cold_latency - warm_mean\n", + "\n", + "labels = [\"cold start\\n(worker load + inference)\", \"warm\\n(inference only)\"]\n", + "values = [cold_latency, warm_mean]\n", + "\n", + "fig, ax = plt.subplots(figsize=(5, 4))\n", + "bars = ax.bar(labels, values, color=[\"#e07a5f\", \"#3d9a8b\"])\n", + "ax.set_ylabel(\"latency (seconds)\")\n", + "ax.set_title(f\"Cold start adds ~{overhead:.2f}s of overhead\")\n", + "ax.margins(y=0.15)\n", + "for bar, value in zip(bars, values):\n", + " ax.text(\n", + " bar.get_x() + bar.get_width() / 2,\n", + " value,\n", + " f\"{value:.2f}s\",\n", + " ha=\"center\",\n", + " va=\"bottom\",\n", + " )\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "cell-16", + "metadata": {}, + "source": [ + "## Check the embeddings are usable\n", + "\n", + "Latency is only half the story: the vectors also have to be good enough to rank\n", + "text by meaning. We embed a query together with a handful of candidate sentences\n", + "in one call, score each candidate against the query with cosine similarity, and\n", + "confirm the passage that actually answers the query comes out on top.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-17", + "metadata": {}, + "outputs": [], + "source": [ + "def cosine(a, b):\n", + " dot = sum(x * y for x, y in zip(a, b))\n", + " norm_a = math.sqrt(sum(x * x for x in a))\n", + " norm_b = math.sqrt(sum(y * y for y in b))\n", + " if norm_a == 0 or norm_b == 0:\n", + " raise ValueError(\"cosine similarity is undefined for a zero vector\")\n", + " return dot / (norm_a * norm_b)\n", + "\n", + "\n", + "assert cosine([1.0, 0.0], [0.0, 1.0]) == 0.0\n", + "assert abs(cosine([1.0, 2.0, 3.0], [1.0, 2.0, 3.0]) - 1.0) < 1e-9\n", + "\n", + "query = \"who wrote the origin of species\"\n", + "candidates = [\n", + " \"Charles Darwin published On the Origin of Species in 1859, laying out his theory of evolution by natural selection.\",\n", + " \"The Great Barrier Reef is the world's largest coral reef system, off the coast of Queensland, Australia.\",\n", + " \"Python is a high-level programming language known for its readable syntax.\",\n", + " \"Mount Kilimanjaro is the highest mountain in Africa.\",\n", + "]\n", + "\n", + "vectors = embed([query] + candidates)\n", + "query_vector, candidate_vectors = vectors[0], vectors[1:]\n", + "\n", + "ranked = sorted(\n", + " (\n", + " {\"score\": cosine(query_vector, vector), \"text\": text}\n", + " for text, vector in zip(candidates, candidate_vectors)\n", + " ),\n", + " key=lambda item: item[\"score\"],\n", + " reverse=True,\n", + ")\n", + "\n", + "for hit in ranked:\n", + " print(f\"{hit['score']:.3f} {hit['text']}\")\n", + "\n", + "assert ranked[0][\"text\"] == candidates[0]\n", + "print(\"\\nretrieval smoke test passed: the Darwin passage ranks first\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-18", + "metadata": {}, + "source": [ + "## Clean up\n", + "\n", + "Serverless endpoints cost nothing while idle, but the endpoint, its\n", + "configuration, and the model resource linger until you delete them, and a\n", + "provisioned-concurrency endpoint keeps billing for the reserved workers.\n", + "Deleting the endpoint also returns its share of your account's serverless\n", + "concurrency quota.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-19", + "metadata": {}, + "outputs": [], + "source": [ + "def ignore_not_found(error):\n", + " code = error.response.get(\"Error\", {}).get(\"Code\", \"\")\n", + " message = error.response.get(\"Error\", {}).get(\"Message\", \"\")\n", + " return code in {\"ResourceNotFound\", \"ResourceNotFoundException\"} or \"not exist\" in message\n", + "\n", + "\n", + "def cleanup_resources():\n", + " print(\"deleting endpoint\")\n", + " try:\n", + " sm.delete_endpoint(EndpointName=ENDPOINT_NAME)\n", + " sm.get_waiter(\"endpoint_deleted\").wait(\n", + " EndpointName=ENDPOINT_NAME,\n", + " WaiterConfig={\"Delay\": 15, \"MaxAttempts\": 60},\n", + " )\n", + " except ClientError as error:\n", + " if not ignore_not_found(error):\n", + " raise\n", + "\n", + " print(\"deleting endpoint config\")\n", + " try:\n", + " sm.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n", + " except ClientError as error:\n", + " if not ignore_not_found(error):\n", + " raise\n", + "\n", + " print(\"deleting model\")\n", + " try:\n", + " sm.delete_model(ModelName=model_name)\n", + " except ClientError as error:\n", + " if not ignore_not_found(error):\n", + " raise\n", + "\n", + "\n", + "if CLEANUP:\n", + " cleanup_resources()\n", + "else:\n", + " print(f\"left endpoint running: {ENDPOINT_NAME}\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "SageMaker v3 (.venv-sm-v3)", + "language": "python", + "name": "sagemaker-v3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}