{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/torchmetrics/utilities/imports.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n", " from pkg_resources import DistributionNotFound, get_distribution\n", "Installing hub:\u001b[35m/\u001b[0m\u001b[35m/brainlogic/\u001b[0m\u001b[95mhigh_quality_translation...\u001b[0m\n", "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/torchmetrics/utilities/imports.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n", " from pkg_resources import DistributionNotFound, get_distribution\n", "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/transformers/utils/generic.py:441: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n", " _torch_pytree._register_pytree_node(\n", "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n", " _torch_pytree._register_pytree_node(\n", "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n", " _torch_pytree._register_pytree_node(\n", "Fetching 5 files: 100%|████████████████████████| 5/5 [00:00<00:00, 42711.85it/s]\n", "Lightning automatically upgraded your loaded checkpoint from v1.8.2 to v2.5.5. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../.cache/huggingface/hub/models--Unbabel--wmt22-cometkiwi-da/snapshots/1ad785194e391eebc6c53e2d0776cada8f83179a/checkpoints/model.ckpt`\n", "Encoder model frozen.\n", "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/pytorch_lightning/core/saving.py:195: Found keys that are not in the model state dict but in the checkpoint: ['encoder.model.embeddings.position_ids']\n", "Installation complete\n", "✅Successfully installed brainlogic/high_quality_translation version \u001b[1;36m0.0\u001b[0m.\u001b[1;36m0\u001b[0m!\n", "\n", "\n" ] } ], "source": [ "!guardrails hub install hub://brainlogic/high_quality_translation -q" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Translate text with quality checks\n", "\n", "**Note:**\n", "To download this example as a Jupyter notebook, click [here](https://github.com/guardrails-ai/guardrails/blob/main/docs/examples/translation_with_quality_check.ipynb).\n", "\n", "In this example, we will use Guardrails during the translation of a statement from another language to English. We will check whether the translated statement is likely of high quality.\n", "\n", "## Objective\n", "\n", "We want to translate a statement from different languages to English and ensure that the translated statement accurately reflects the original content.\n", "\n", "### Setup\n", "\n", "- Install the `unbabel-comet` from source:\n", " `pip install git+https://github.com/Unbabel/COMET`\n", "- Please accept the model license from:\n", " https://huggingface.co/Unbabel/wmt22-cometkiwi-da\n", "- Login into Huggingface Hub using:\n", " huggingface-cli login --token $HUGGINGFACE_TOKEN\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "! pip install git+https://github.com/Unbabel/COMET -q" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from guardrails import Guard\n", "from rich import print\n", "from guardrails.hub import HighQualityTranslation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Define a Guard that uses the validator\n", "\n", "This guard will use the HighQualityTranslation validator to validate some string outputs.\n", "\n", "\n", "We define the prompt and the guard." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading the model Unbabel/wmt22-cometkiwi-da...\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "59f51fc27c0847298112f07de8ffae52", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Fetching 5 files: 0%| | 0/5 [00:00Raw LLM Output: I have no idea what I’m supposed to write here.\n", "\n" ], "text/plain": [ "Raw LLM Output: I have no idea what I’m supposed to write here.\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Validated Output: I have no idea what I’m supposed to write here.\n",
       "
\n" ], "text/plain": [ "Validated Output: I have no idea what I’m supposed to write here.\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Set your OPENAI_API_KEY as an environment variable\n", "# import os\n", "# os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n", "\n", "statement = \"Ich habe keine Ahnung, was ich hier schreiben soll.\"\n", "\n", "res = guard(\n", " messages=[{\"role\": \"user\", \"content\": prompt}],\n", " prompt_params={\"statement_to_be_translated\": statement},\n", " metadata={\"translation_source\": statement},\n", " model=\"gpt-5-nano\",\n", " max_tokens=1024,\n", " temperature=1,\n", ")\n", "\n", "print(f\"Raw LLM Output: {res.raw_llm_output}\")\n", "print(f\"Validated Output: {res.validated_output}\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "We can look at the logs to see the quality check results:\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Logs\n",
       "└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n",
       "    │ ╭─────────────────────────────────────────────── Messages ────────────────────────────────────────────────╮ │\n",
       "    │ │ ┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                                          │ │\n",
       "    │ │ ┃ Role ┃ Content                                             ┃                                          │ │\n",
       "    │ │ ┡━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩                                          │ │\n",
       "    │ │ │ user │                                                     │                                          │ │\n",
       "    │ │ │      │ Translate the given statement into English:         │                                          │ │\n",
       "    │ │ │      │                                                     │                                          │ │\n",
       "    │ │ │      │ Ich habe keine Ahnung, was ich hier schreiben soll. │                                          │ │\n",
       "    │ │ │      │                                                     │                                          │ │\n",
       "    │ │ └──────┴─────────────────────────────────────────────────────┘                                          │ │\n",
       "    │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │\n",
       "    │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │\n",
       "    │ │ I have no idea what I’m supposed to write here.                                                         │ │\n",
       "    │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │\n",
       "    │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │\n",
       "    │ │ I have no idea what I’m supposed to write here.                                                         │ │\n",
       "    │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │\n",
       "    ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
       "
\n" ], "text/plain": [ "Logs\n", "└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n", " │ \u001b[48;2;231;223;235m╭─\u001b[0m\u001b[48;2;231;223;235m──────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m Messages \u001b[0m\u001b[48;2;231;223;235m───────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m─╮\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┃\u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[1;48;2;231;223;235mRole\u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┃\u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[1;48;2;231;223;235mContent \u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┃\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┡━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235muser\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235mTranslate the given statement into English: \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235mIch habe keine Ahnung, was ich hier schreiben soll.\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m└──────┴─────────────────────────────────────────────────────┘\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n", " │ \u001b[48;2;245;245;220m╭─\u001b[0m\u001b[48;2;245;245;220m───────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m Raw LLM Output \u001b[0m\u001b[48;2;245;245;220m────────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m─╮\u001b[0m │\n", " │ \u001b[48;2;245;245;220m│\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220mI have no idea what I’m supposed to write here.\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m│\u001b[0m │\n", " │ \u001b[48;2;245;245;220m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n", " │ \u001b[48;2;240;255;240m╭─\u001b[0m\u001b[48;2;240;255;240m──────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m Validated Output \u001b[0m\u001b[48;2;240;255;240m───────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m─╮\u001b[0m │\n", " │ \u001b[48;2;240;255;240m│\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240mI have no idea what I’m supposed to write here.\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m│\u001b[0m │\n", " │ \u001b[48;2;240;255;240m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n", " ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(guard.history.last.tree)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The `guard` wrapper returns the raw LLM response, which is the translated statement and also the validated output. In this case, the translated statement was of a good quality (above the threshold of 0.5), so the validated output is the same as the raw LLM response.\n", "\n", "#### Now, let's test with a really low quality translation, and see how Guardrails handles it.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/guardrails/validator_service/__init__.py:84: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n", " warnings.warn(\n", "💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.\n", "GPU available: True (mps), used: False\n", "TPU available: False, using: 0 TPU cores\n", "HPU available: False, using: 0 HPUs\n", "/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/setup.py:177: GPU available but not used. You can set it by doing `Trainer(accelerator='gpu')`.\n", "Predicting DataLoader 0: 100%|██████████| 1/1 [00:00<00:00, 12.05it/s]\n" ] }, { "data": { "text/html": [ "
Raw LLM Output: It's such a beautiful day, I'm going to the beach.\n",
       "
\n" ], "text/plain": [ "Raw LLM Output: It's such a beautiful day, I'm going to the beach.\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Validated Output: \n",
       "
\n" ], "text/plain": [ "Validated Output: \n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Parse the code snippet\n", "statement = \"अरे भाऊ, आज रात्री जोरदार पार्टी मारूया, जमून टाकूया आणि धमाल करूया!\"\n", "\n", "## Ideal translation from Marathi -> English:\n", "# \"Hey bro, let's have a great party tonight and have fun!\"\n", "\n", "output = guard.parse(\n", " llm_output=\"It's such a beautiful day, I'm going to the beach.\", ## here, providing a really bad translation\n", " metadata={\"translation_source\": statement},\n", ")\n", "\n", "# Print the output\n", "print(f\"Raw LLM Output: {output.raw_llm_output}\")\n", "print(f\"Validated Output: {output.validated_output}\")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Logs\n",
       "└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n",
       "    │ ╭─────────────────────────────────────────────── Messages ────────────────────────────────────────────────╮ │\n",
       "    │ │ No messages.                                                                                            │ │\n",
       "    │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │\n",
       "    │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │\n",
       "    │ │ It's such a beautiful day, I'm going to the beach.                                                      │ │\n",
       "    │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │\n",
       "    │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │\n",
       "    │ │ ''                                                                                                      │ │\n",
       "    │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │\n",
       "    ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
       "
\n" ], "text/plain": [ "Logs\n", "└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n", " │ \u001b[48;2;231;223;235m╭─\u001b[0m\u001b[48;2;231;223;235m──────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m Messages \u001b[0m\u001b[48;2;231;223;235m───────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m─╮\u001b[0m │\n", " │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235mNo messages.\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n", " │ \u001b[48;2;231;223;235m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n", " │ \u001b[48;2;245;245;220m╭─\u001b[0m\u001b[48;2;245;245;220m───────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m Raw LLM Output \u001b[0m\u001b[48;2;245;245;220m────────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m─╮\u001b[0m │\n", " │ \u001b[48;2;245;245;220m│\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220mIt's such a beautiful day, I'm going to the beach.\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m│\u001b[0m │\n", " │ \u001b[48;2;245;245;220m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n", " │ \u001b[48;2;240;255;240m╭─\u001b[0m\u001b[48;2;240;255;240m──────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m Validated Output \u001b[0m\u001b[48;2;240;255;240m───────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m─╮\u001b[0m │\n", " │ \u001b[48;2;240;255;240m│\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m''\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m│\u001b[0m │\n", " │ \u001b[48;2;240;255;240m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n", " ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(guard.history.last.tree)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the translation quality is really bad, and the `HighQualityTranslation` check failed as the translation quality was below the threshold. The validated response is an empty string.\n", "\n", "## In this way, you can use Guardrails to ensure that the output of your LLM is of high quality.\n" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.11" } }, "nbformat": 4, "nbformat_minor": 4 }