diff --git a/docs/sagemaker/notebooks/sagemaker-sdk/trip-planner-agent-vllm/assets/cover.png b/docs/sagemaker/notebooks/sagemaker-sdk/trip-planner-agent-vllm/assets/cover.png new file mode 100644 index 000000000..e7dd6d181 Binary files /dev/null and b/docs/sagemaker/notebooks/sagemaker-sdk/trip-planner-agent-vllm/assets/cover.png differ diff --git a/docs/sagemaker/notebooks/sagemaker-sdk/trip-planner-agent-vllm/sagemaker-notebook.ipynb b/docs/sagemaker/notebooks/sagemaker-sdk/trip-planner-agent-vllm/sagemaker-notebook.ipynb new file mode 100644 index 000000000..d15987030 --- /dev/null +++ b/docs/sagemaker/notebooks/sagemaker-sdk/trip-planner-agent-vllm/sagemaker-notebook.ipynb @@ -0,0 +1,703 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "74a1429c", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "id": "03be25ec", + "metadata": {}, + "source": [ + "# Build a reasoning trip-planning agent on Amazon SageMaker AI with Hugging Face vLLM\n", + "\n", + "![](./assets/cover.png)\n", + "\n", + "In this notebook, we'll deploy [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) with the Hugging Face **vLLM** Deep Learning Container (DLC) on Amazon SageMaker, and use it to build a **trip-planning agent**: an assistant that reasons through multi-constraint travel requests and calls tools to check the weather, convert currencies, and find places to visit.\n", + "\n", + "We'll walk through the following steps:\n", + "\n", + "- Select a reasoning and tool-calling model and a SageMaker DLC for your use case\n", + "- Deploy the model to SageMaker with the SageMaker Python SDK `ModelBuilder`\n", + "- Configure vLLM for reasoning and tool calling\n", + "- Build an agent with [Strands Agents](https://strandsagents.com/) and give it custom tools\n", + "- Hold a multi-turn conversation where the agent reasons, calls tools, and remembers context\n", + "- Optionally connect a Hugging Face MCP server and launch a Gradio app\n", + "- Clean up the endpoint resources to avoid ongoing charges\n", + "\n", + "For this example, you'll need AWS credentials and a SageMaker execution role. Qwen3-8B is a public model, so a Hugging Face token is optional here, but recommended for reliable, authenticated downloads (and required if you switch to a gated model)." + ] + }, + { + "cell_type": "markdown", + "id": "7a5fc25b", + "metadata": {}, + "source": [ + "## How an agent works\n", + "\n", + "An agent is a loop around a language model. The model does not just answer in one shot; it decides, step by step, what information it still needs and how to get it.\n", + "\n", + "Two capabilities make this possible:\n", + "\n", + "- **Reasoning**: the model thinks through the problem before answering, which helps with multi-step requests such as planning a trip under several constraints.\n", + "- **Tool calling**: instead of guessing a fact, the model can call a function you provide (check the weather, convert a currency, look up places) and continue with the real result.\n", + "\n", + "Put together, the loop looks like this: the user asks a question, the model reasons about it and requests a tool, your code runs the tool and returns the result, and the model reasons again with that new information. This repeats until the model has everything it needs to give a final answer.\n", + "\n", + "[`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) supports both a thinking mode and native tool calling, so it can drive this loop on its own. We'll deploy it, then let [Strands Agents](https://strandsagents.com/) run the loop for us." + ] + }, + { + "cell_type": "markdown", + "id": "5c8f3f9e", + "metadata": {}, + "source": [ + "## Hugging Face vLLM and Strands Agents\n", + "\n", + "[vLLM](https://github.com/vllm-project/vllm) is a high-throughput inference engine for large language models. The Hugging Face **vLLM** DLC packages it for SageMaker with current `transformers` and `huggingface_hub`, and exposes an OpenAI-compatible API (including tool calling and reasoning parsing) through simple environment variables. Its multimodal sibling, vLLM-Omni, serves tasks like text-to-speech or image generation; here we serve a text model, so we use the plain vLLM image.\n", + "\n", + "[Strands Agents](https://strandsagents.com/) is a lightweight agent framework with a native SageMaker integration. Instead of hand-writing the tool-calling loop, we point Strands at our endpoint and it orchestrates the reasoning, tool calls, and results for us." + ] + }, + { + "cell_type": "markdown", + "id": "f9393c0e", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "To run this example, we'll install the SageMaker Python SDK for model deployment, `huggingface_hub` for authentication, `strands-agents` (with its SageMaker integration) for the agent, and `gradio` for the optional interactive app." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "743252a7", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q \"sagemaker>=3\" huggingface_hub \"strands-agents[sagemaker]>=1.48.0\" \"gradio>=5\"" + ] + }, + { + "cell_type": "markdown", + "id": "3931c873", + "metadata": {}, + "source": [ + "We are going to need:\n", + "\n", + "- An `HF_TOKEN`: used to download the model from Hugging Face. Optional for the public Qwen3-8B, but recommended for authenticated downloads.\n", + "- A SageMaker execution role: used to pull the DLC from ECR and deploy the model to SageMaker.\n", + "\n", + "Let's start by setting up the token and the execution role." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e7205ff1", + "metadata": {}, + "outputs": [], + "source": [ + "from huggingface_hub import get_token, notebook_login\n", + "\n", + "if not (HF_TOKEN := get_token()):\n", + " notebook_login()\n", + " HF_TOKEN = get_token()\n", + "\n", + "print(\"HF_TOKEN_LOADED\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6a3586a9", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "import boto3\n", + "from sagemaker.core.helper.session_helper import Session, get_execution_role\n", + "\n", + "REGION = boto3.Session().region_name or os.environ.get(\"AWS_REGION\", \"us-east-1\")\n", + "boto_sess = boto3.Session(region_name=REGION)\n", + "sess = Session(boto_session=boto_sess)\n", + "\n", + "try:\n", + " role = get_execution_role(sagemaker_session=sess)\n", + " print(f\"Role extracted from execution role: {role}\")\n", + "except Exception:\n", + " role_name = \"sagemaker_execution_role\"\n", + " iam_client = boto_sess.client(\"iam\")\n", + " role = iam_client.get_role(RoleName=role_name)[\"Role\"][\"Arn\"]\n", + " print(f\"Role extracted from iam client: {role}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e67976ca", + "metadata": {}, + "source": [ + "## Choosing a model and a DLC\n", + "\n", + "There are many open LLMs on the Hugging Face Hub. For an agent we want strong instruction-following, reasoning, and tool-calling, while staying small enough to run on a single GPU.\n", + "\n", + "We'll use [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) and serve it with the Hugging Face **vLLM** DLC. `ModelBuilder` can auto-select this container for a text-generation model, but here we pin an explicit image tag so the tutorial always uses the exact build we tested. You can browse the available tags on the [AWS Available Images](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-dg.pdf#available-images) page.\n", + "\n", + "We'll target an `ml.g5.xlarge` instance (a single 24 GB GPU). For higher concurrency or larger models, consider larger GPU instance types such as `ml.g5.2xlarge` or `ml.g6e.*`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f1274c9e", + "metadata": {}, + "outputs": [], + "source": [ + "from time import strftime\n", + "\n", + "MODEL_ID = \"Qwen/Qwen3-8B\"\n", + "INSTANCE_TYPE = \"ml.g5.xlarge\"\n", + "IMAGE_URI = (\n", + " f\"763104351884.dkr.ecr.{REGION}.amazonaws.com/\"\n", + " \"huggingface-vllm:0.22.1-gpu-py312-cu130-ubuntu22.04\"\n", + ")\n", + "\n", + "RESOURCE_SUFFIX = strftime(\"%Y%m%d-%H%M%S\")\n", + "MODEL_NAME = f\"trip-planner-agent-model-{RESOURCE_SUFFIX}\"\n", + "ENDPOINT_NAME = f\"trip-planner-agent-endpoint-{RESOURCE_SUFFIX}\"" + ] + }, + { + "cell_type": "markdown", + "id": "7b2b5d01", + "metadata": {}, + "source": [ + "## Configuring vLLM for reasoning and tool calling\n", + "\n", + "The vLLM container is configured entirely through environment variables. Any vLLM server flag can be passed by uppercasing it, replacing dashes with underscores, and prefixing it with `SM_VLLM_` (for example `--max-model-len` becomes `SM_VLLM_MAX_MODEL_LEN`).\n", + "\n", + "Three of these variables are what turn a plain text model into an agent backend:\n", + "\n", + "- `SM_VLLM_ENABLE_AUTO_TOOL_CHOICE=true` lets the model decide when to call a tool.\n", + "- `SM_VLLM_TOOL_CALL_PARSER=hermes` parses Qwen3's tool calls into the OpenAI format.\n", + "- `SM_VLLM_REASONING_PARSER=qwen3` separates the model's thinking from its final answer.\n", + "\n", + "We also set `SM_VLLM_HOST=0.0.0.0` (required so the container passes the SageMaker health check) and `SM_VLLM_TRUST_REMOTE_CODE=true` (needed for Qwen architectures). `ModelBuilder` sets `HF_MODEL_ID` for us, so we don't repeat the model id here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f470a252", + "metadata": {}, + "outputs": [], + "source": [ + "env_vars = {\n", + " \"HF_TOKEN\": HF_TOKEN, # Token to download the model from Hugging Face\n", + " \"SM_VLLM_HOST\": \"0.0.0.0\", # Bind to all interfaces so the health check passes\n", + " \"SM_VLLM_TRUST_REMOTE_CODE\": \"true\", # Required for Qwen architectures\n", + " \"SM_VLLM_MAX_MODEL_LEN\": \"16384\", # Context length; bounds KV cache on a 24 GB GPU\n", + " \"SM_VLLM_GPU_MEMORY_UTILIZATION\": \"0.9\",\n", + " \"SM_VLLM_ENABLE_AUTO_TOOL_CHOICE\": \"true\", # Let the model call tools\n", + " \"SM_VLLM_TOOL_CALL_PARSER\": \"hermes\", # Parse Qwen3 tool calls\n", + " \"SM_VLLM_REASONING_PARSER\": \"qwen3\", # Separate reasoning from the answer\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "11126805", + "metadata": {}, + "source": [ + "## Deploy the model\n", + "\n", + "The SageMaker Python SDK v3 deploys models with `ModelBuilder`: we describe the model, the serving container, and the instance, then call `build` and `deploy`. There is no need to assemble model, endpoint-config, and endpoint resources by hand.\n", + "\n", + "One detail worth calling out: this image is built on CUDA 13 (`cu130` in the tag). vLLM images with CUDA 13 or newer require the `al2-ami-sagemaker-inference-gpu-3-1` inference AMI, which we pass to `deploy`. Without it the container fails to start before any logs are produced." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "76b099b3", + "metadata": {}, + "outputs": [], + "source": [ + "from sagemaker.serve import ModelBuilder, ModelServer\n", + "\n", + "model_builder = ModelBuilder(\n", + " model=MODEL_ID,\n", + " role_arn=role,\n", + " sagemaker_session=sess,\n", + " instance_type=INSTANCE_TYPE,\n", + " image_uri=IMAGE_URI,\n", + " model_server=ModelServer.VLLM,\n", + " env_vars=env_vars,\n", + ")\n", + "\n", + "built_model = model_builder.build(model_name=MODEL_NAME)\n", + "\n", + "model_builder.deploy(\n", + " endpoint_name=ENDPOINT_NAME,\n", + " initial_instance_count=1,\n", + " instance_type=INSTANCE_TYPE,\n", + " inference_ami_version=\"al2-ami-sagemaker-inference-gpu-3-1\",\n", + " container_timeout_in_seconds=900,\n", + " wait=True,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "2a82c67f", + "metadata": {}, + "source": [ + "## A first look: reasoning and tool calls\n", + "\n", + "Before bringing in the agent framework, let's send a single request straight to the endpoint to see what the model returns. SageMaker requests go through the `/invocations` route; we use `CustomAttributes` to forward them to vLLM's OpenAI-compatible Chat Completions API.\n", + "\n", + "We pass a `get_weather` tool definition and let the model decide whether to call it (`tool_choice=\"auto\"`). The response separates the model's reasoning (`reasoning_content`) from any tool calls it wants to make (`tool_calls`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f7e95bfd", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "\n", + "runtime = boto_sess.client(\"sagemaker-runtime\")\n", + "\n", + "weather_tool = {\n", + " \"type\": \"function\",\n", + " \"function\": {\n", + " \"name\": \"get_weather\",\n", + " \"description\": \"Get the weather forecast for a city on a given date.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"city\": {\"type\": \"string\", \"description\": \"City name.\"},\n", + " \"date\": {\"type\": \"string\", \"description\": \"Date in YYYY-MM-DD format.\"},\n", + " },\n", + " \"required\": [\"city\", \"date\"],\n", + " },\n", + " },\n", + "}\n", + "\n", + "response = runtime.invoke_endpoint(\n", + " EndpointName=ENDPOINT_NAME,\n", + " ContentType=\"application/json\",\n", + " Body=json.dumps({\n", + " \"model\": MODEL_ID,\n", + " \"messages\": [\n", + " {\"role\": \"user\", \"content\": \"What should I pack for Lisbon on 2026-10-12?\"}\n", + " ],\n", + " \"tools\": [weather_tool],\n", + " \"tool_choice\": \"auto\",\n", + " }),\n", + " CustomAttributes=\"route=/v1/chat/completions\",\n", + ")\n", + "\n", + "message = json.loads(response[\"Body\"].read())[\"choices\"][0][\"message\"]\n", + "print(\"Reasoning:\\n\", message.get(\"reasoning_content\"), \"\\n\")\n", + "print(\"Tool calls:\\n\", json.dumps(message.get(\"tool_calls\"), indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "2f2ff783", + "metadata": {}, + "source": [ + "## Building the agent\n", + "\n", + "We describe the endpoint once with `SageMakerAIModel`, hand the agent our tools, and from then on a single call runs the full reason-and-act loop until the model produces an answer.\n", + "\n", + "Two `payload_config` choices matter here:\n", + "\n", + "- `stream=True`: a reasoning model can generate for a while. Streaming returns tokens as they are produced instead of waiting for the whole answer, which keeps the response flowing and avoids the real-time endpoint timing out on longer generations.\n", + "- `chat_template_kwargs={\"enable_thinking\": True}`: keeps Qwen3 in thinking mode.\n", + "\n", + "The provider logs the full request and response (and every streamed chunk) at `INFO`, so we first raise the log level to `WARNING` to keep the agent's output readable." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c80cace7", + "metadata": {}, + "outputs": [], + "source": [ + "import logging\n", + "\n", + "# The SageMaker model provider logs the full request, every streamed chunk, and\n", + "# the final response at INFO level. Raise the level to keep the output readable.\n", + "logging.getLogger(\"strands\").setLevel(logging.WARNING)\n", + "\n", + "from strands.models.sagemaker import SageMakerAIModel\n", + "\n", + "sagemaker_model = SageMakerAIModel(\n", + " endpoint_config={\"endpoint_name\": ENDPOINT_NAME, \"region_name\": REGION},\n", + " payload_config={\n", + " \"max_tokens\": 8192,\n", + " \"temperature\": 0.7,\n", + " \"top_p\": 0.95,\n", + " \"stream\": True,\n", + " \"additional_args\": {\"chat_template_kwargs\": {\"enable_thinking\": True}},\n", + " },\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "333e6937", + "metadata": {}, + "source": [ + "### Giving the agent tools\n", + "\n", + "Strands turns any Python function into a tool with the `@tool` decorator: it reads the type hints and docstring to build the schema the model sees. Our concierge gets three tools it can combine to plan a trip.\n", + "\n", + "To keep the notebook self-contained and reproducible, these are deterministic mocks. Swap in real APIs (or a built-in like `strands_tools.http_request`) when you adapt this to production; the agent code does not change." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6453ba66", + "metadata": {}, + "outputs": [], + "source": [ + "from strands import tool\n", + "\n", + "\n", + "@tool\n", + "def get_weather(city: str, date: str) -> str:\n", + " \"\"\"Get the weather forecast for a city on a given date.\n", + "\n", + " Args:\n", + " city: City to look up.\n", + " date: Date in YYYY-MM-DD format.\n", + " \"\"\"\n", + " return f\"{city} on {date}: sunny, 22°C during the day and 14°C at night.\"\n", + "\n", + "\n", + "@tool\n", + "def convert_currency(amount: float, from_currency: str, to_currency: str) -> str:\n", + " \"\"\"Convert an amount between two currencies.\n", + "\n", + " Args:\n", + " amount: Amount to convert.\n", + " from_currency: ISO code to convert from, e.g. EUR.\n", + " to_currency: ISO code to convert to, e.g. USD.\n", + " \"\"\"\n", + " per_eur = {\"EUR\": 1.0, \"USD\": 1.08, \"GBP\": 0.85, \"JPY\": 170.0}\n", + " in_eur = amount / per_eur[from_currency.upper()]\n", + " converted = in_eur * per_eur[to_currency.upper()]\n", + " return f\"{amount:.2f} {from_currency.upper()} = {converted:.2f} {to_currency.upper()}\"\n", + "\n", + "\n", + "@tool\n", + "def search_places(city: str, category: str) -> str:\n", + " \"\"\"Find points of interest in a city by category.\n", + "\n", + " Args:\n", + " city: City to search in.\n", + " category: Kind of place, e.g. \"museum\", \"restaurant\", or \"park\".\n", + " \"\"\"\n", + " catalog = {\n", + " \"museum\": [\"National Tile Museum\", \"Berardo Collection\", \"MAAT\"],\n", + " \"restaurant\": [\"Time Out Market\", \"Cervejaria Ramiro\", \"A Cevicheria\"],\n", + " \"park\": [\"Eduardo VII Park\", \"Monsanto Forest Park\", \"Jardim da Estrela\"],\n", + " }\n", + " found = catalog.get(category.lower(), [\"City-center walking tour\"])\n", + " return f\"{category.title()} options in {city}: \" + \", \".join(found)" + ] + }, + { + "cell_type": "markdown", + "id": "6914b2df", + "metadata": {}, + "source": [ + "### Running the agent\n", + "\n", + "Now we send a request that no single tool can answer on its own. The agent has to reason about what it needs, call several tools, and combine the results into an itinerary.\n", + "\n", + "The `run_agent` helper below sends a prompt and then prints two things: the tool calls the agent chose to make (so we can see it acting), followed by the final answer. It reads them from `agent.messages`, the running conversation history, so the printout stays clean while the verbose provider logs remain off." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "29c388fd", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "\n", + "from strands import Agent\n", + "\n", + "agent = Agent(\n", + " model=sagemaker_model,\n", + " tools=[get_weather, convert_currency, search_places],\n", + " callback_handler=None,\n", + ")\n", + "\n", + "\n", + "def run_agent(prompt: str):\n", + " \"\"\"Run the agent, print the tool calls it made, then the final answer.\"\"\"\n", + " start = len(agent.messages)\n", + " result = agent(prompt)\n", + "\n", + " for message in agent.messages[start:]:\n", + " for block in message.get(\"content\", []):\n", + " if \"toolUse\" in block:\n", + " call = block[\"toolUse\"]\n", + " print(f\"[tool call] {call['name']}({json.dumps(call['input'])})\")\n", + "\n", + " print()\n", + " print(result)\n", + " return result\n", + "\n", + "\n", + "result = run_agent(\n", + " \"Plan a 3-day trip to Lisbon starting 2026-10-12. My budget is 800 EUR, \"\n", + " \"so tell me what that is in USD. Suggest a few museums and restaurants, \"\n", + " \"and tell me what to pack based on the weather.\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "365182a7", + "metadata": {}, + "source": [ + "### Following up in the same conversation\n", + "\n", + "A Strands `Agent` keeps the conversation history, so it behaves like a real assistant across turns. We can ask a follow-up and it reuses everything it already worked out, calling tools again only where the change requires it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ec261184", + "metadata": {}, + "outputs": [], + "source": [ + "follow_up = run_agent(\n", + " \"Actually, change the destination to Porto but keep the same budget and dates. \"\n", + " \"Update the packing tips if the weather is different.\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "e2b83dad", + "metadata": {}, + "source": [ + "## Optional: connect the Hugging Face MCP server\n", + "\n", + "[Model Context Protocol (MCP)](https://huggingface.co/docs/hub/en/hf-mcp-server) servers expose ready-made tools an agent can call. Strands connects to them with `MCPClient`, so you can, for example, give the concierge access to the Hugging Face Hub without writing any new tools.\n", + "\n", + "This section is optional. Set `ENABLE_MCP = True` to try it. MCP tools must be used inside the client's context manager." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ee06bcfd", + "metadata": {}, + "outputs": [], + "source": [ + "ENABLE_MCP = False\n", + "\n", + "if ENABLE_MCP:\n", + " from mcp.client.streamable_http import streamablehttp_client\n", + " from strands.tools.mcp import MCPClient\n", + "\n", + " hf_mcp = MCPClient(\n", + " lambda: streamablehttp_client(\n", + " \"https://huggingface.co/mcp\",\n", + " headers={\"Authorization\": f\"Bearer {HF_TOKEN}\"},\n", + " )\n", + " )\n", + " with hf_mcp:\n", + " mcp_agent = Agent(\n", + " model=sagemaker_model,\n", + " tools=[get_weather, convert_currency, search_places, *hf_mcp.list_tools_sync()],\n", + " callback_handler=None,\n", + " )\n", + " print(mcp_agent(\"Find a small Qwen model with tool-calling support and include the Hub links.\"))" + ] + }, + { + "cell_type": "markdown", + "id": "c0ddbacf", + "metadata": {}, + "source": [ + "## Optional: an interactive chat app\n", + "\n", + "For a nicer experience than a notebook cell, we can wrap the agent in a small [Gradio](https://www.gradio.app/) chat app. It streams the agent's events as they happen, rendering the reasoning, each tool call, its result, and the final answer as separate, collapsible blocks.\n", + "\n", + "This section is optional. Set `ENABLE_GRADIO = True` to launch the app.\n", + "\n", + "> **Security and cost warning:** `share=True` creates a public Gradio URL. Anyone with that URL can chat with the app, which invokes the already-deployed billable SageMaker endpoint using your AWS credentials. Share the URL only with trusted users, and disable the app or delete the endpoint when you are finished." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ff46181", + "metadata": {}, + "outputs": [], + "source": [ + "import gradio as gr\n", + "\n", + "# A dedicated agent for the app, so the chat starts fresh instead of inheriting\n", + "# the conversation from the demo cells above.\n", + "chat_agent = Agent(\n", + " model=sagemaker_model,\n", + " tools=[get_weather, convert_currency, search_places],\n", + " callback_handler=None,\n", + ")\n", + "\n", + "\n", + "async def chat(message, history):\n", + " # Gradio passes an empty history at the start of a conversation (including\n", + " # after the \"Clear\" button), which is our cue to reset the agent's memory.\n", + " if not history:\n", + " chat_agent.messages.clear()\n", + "\n", + " turn = []\n", + " kind = None # \"reasoning\" | \"tool\" | \"text\" for the current block\n", + " tool_name = \"tool\"\n", + "\n", + " async for event in chat_agent.stream_async(message):\n", + " if \"event\" in event:\n", + " raw = event[\"event\"]\n", + "\n", + " # A tool block announces its name at the start.\n", + " tool_use = raw.get(\"contentBlockStart\", {}).get(\"start\", {}).get(\"toolUse\")\n", + " if tool_use:\n", + " tool_name = tool_use.get(\"name\", \"tool\")\n", + " kind = None # force a fresh block for this tool call\n", + "\n", + " delta = raw.get(\"contentBlockDelta\", {}).get(\"delta\")\n", + " if not delta:\n", + " continue\n", + "\n", + " # Classify the chunk: reasoning, tool arguments, or final answer.\n", + " if \"reasoningContent\" in delta:\n", + " piece, this, meta = delta[\"reasoningContent\"].get(\"text\", \"\"), \"reasoning\", {\"title\": \"💭 Thinking\"}\n", + " elif \"toolUse\" in delta:\n", + " piece, this, meta = delta[\"toolUse\"].get(\"input\", \"\"), \"tool\", {\"title\": f\"🛠️ Using tool `{tool_name}`\"}\n", + " else:\n", + " piece, this, meta = delta.get(\"text\", \"\"), \"text\", None\n", + "\n", + " if this != kind:\n", + " kind = this\n", + " turn.append(gr.ChatMessage(role=\"assistant\", content=\"\", metadata=meta))\n", + " turn[-1].content += piece\n", + " yield turn\n", + "\n", + " elif \"message\" in event:\n", + " msg = event[\"message\"]\n", + " if msg.get(\"role\") != \"user\":\n", + " continue\n", + " for block in msg.get(\"content\", []):\n", + " result = block.get(\"toolResult\")\n", + " if not result:\n", + " continue\n", + " text = \"\\n\".join(\n", + " item.get(\"text\", \"\") for item in result.get(\"content\", []) if \"text\" in item\n", + " ).strip()\n", + " if text:\n", + " status = result.get(\"status\", \"success\").upper()\n", + " turn.append(gr.ChatMessage(\n", + " role=\"assistant\",\n", + " content=text,\n", + " metadata={\"title\": f\"✅ Tool result [{status}]\"},\n", + " ))\n", + " kind = None\n", + " yield turn\n", + "\n", + " yield turn\n", + "\n", + "\n", + "ENABLE_GRADIO = False\n", + "\n", + "if ENABLE_GRADIO:\n", + " demo = gr.ChatInterface(\n", + " fn=chat,\n", + " title=\"Trip-planning concierge\",\n", + " description=\"Ask for help planning a trip. The agent reasons, calls tools, and answers.\",\n", + " examples=[\n", + " \"Plan a 2-day trip to Lisbon on 2026-10-12 with a 500 EUR budget in USD.\",\n", + " \"What museums should I visit in Lisbon, and what's the weather on 2026-10-13?\",\n", + " ],\n", + " )\n", + " demo.launch(share=True, server_name=\"0.0.0.0\", server_port=7860, show_error=True)" + ] + }, + { + "cell_type": "markdown", + "id": "b95e5881", + "metadata": {}, + "source": [ + "## Cleanup\n", + "\n", + "SageMaker endpoints are billed while they are `InService`, so delete the endpoint and its associated resources when you're done." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "23018758", + "metadata": {}, + "outputs": [], + "source": [ + "sm = boto_sess.client(\"sagemaker\")\n", + "endpoint_config_name = sm.describe_endpoint(EndpointName=ENDPOINT_NAME)[\"EndpointConfigName\"]\n", + "\n", + "# delete_endpoint is asynchronous; wait for it to finish before removing the\n", + "# config and model, which stay in use while the endpoint is still deleting.\n", + "sm.delete_endpoint(EndpointName=ENDPOINT_NAME)\n", + "sm.get_waiter(\"endpoint_deleted\").wait(EndpointName=ENDPOINT_NAME)\n", + "\n", + "sm.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n", + "sm.delete_model(ModelName=built_model.model_name)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}