456 lines
37 KiB
Plaintext
456 lines
37 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/torchmetrics/utilities/imports.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n",
|
||
" from pkg_resources import DistributionNotFound, get_distribution\n",
|
||
"Installing hub:\u001b[35m/\u001b[0m\u001b[35m/brainlogic/\u001b[0m\u001b[95mhigh_quality_translation...\u001b[0m\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/torchmetrics/utilities/imports.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n",
|
||
" from pkg_resources import DistributionNotFound, get_distribution\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/transformers/utils/generic.py:441: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
|
||
" _torch_pytree._register_pytree_node(\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
|
||
" _torch_pytree._register_pytree_node(\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
|
||
" _torch_pytree._register_pytree_node(\n",
|
||
"Fetching 5 files: 100%|████████████████████████| 5/5 [00:00<00:00, 42711.85it/s]\n",
|
||
"Lightning automatically upgraded your loaded checkpoint from v1.8.2 to v2.5.5. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../.cache/huggingface/hub/models--Unbabel--wmt22-cometkiwi-da/snapshots/1ad785194e391eebc6c53e2d0776cada8f83179a/checkpoints/model.ckpt`\n",
|
||
"Encoder model frozen.\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/pytorch_lightning/core/saving.py:195: Found keys that are not in the model state dict but in the checkpoint: ['encoder.model.embeddings.position_ids']\n",
|
||
"Installation complete\n",
|
||
"✅Successfully installed brainlogic/high_quality_translation version \u001b[1;36m0.0\u001b[0m.\u001b[1;36m0\u001b[0m!\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!guardrails hub install hub://brainlogic/high_quality_translation -q"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Translate text with quality checks\n",
|
||
"\n",
|
||
"**Note:**\n",
|
||
"To download this example as a Jupyter notebook, click [here](https://github.com/guardrails-ai/guardrails/blob/main/docs/examples/translation_with_quality_check.ipynb).\n",
|
||
"\n",
|
||
"In this example, we will use Guardrails during the translation of a statement from another language to English. We will check whether the translated statement is likely of high quality.\n",
|
||
"\n",
|
||
"## Objective\n",
|
||
"\n",
|
||
"We want to translate a statement from different languages to English and ensure that the translated statement accurately reflects the original content.\n",
|
||
"\n",
|
||
"### Setup\n",
|
||
"\n",
|
||
"- Install the `unbabel-comet` from source:\n",
|
||
" `pip install git+https://github.com/Unbabel/COMET`\n",
|
||
"- Please accept the model license from:\n",
|
||
" https://huggingface.co/Unbabel/wmt22-cometkiwi-da\n",
|
||
"- Login into Huggingface Hub using:\n",
|
||
" huggingface-cli login --token $HUGGINGFACE_TOKEN\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"! pip install git+https://github.com/Unbabel/COMET -q"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from guardrails import Guard\n",
|
||
"from rich import print\n",
|
||
"from guardrails.hub import HighQualityTranslation"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 1: Define a Guard that uses the validator\n",
|
||
"\n",
|
||
"This guard will use the HighQualityTranslation validator to validate some string outputs.\n",
|
||
"\n",
|
||
"\n",
|
||
"We define the prompt and the guard."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Loading the model Unbabel/wmt22-cometkiwi-da...\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"application/vnd.jupyter.widget-view+json": {
|
||
"model_id": "59f51fc27c0847298112f07de8ffae52",
|
||
"version_major": 2,
|
||
"version_minor": 0
|
||
},
|
||
"text/plain": [
|
||
"Fetching 5 files: 0%| | 0/5 [00:00<?, ?it/s]"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Lightning automatically upgraded your loaded checkpoint from v1.8.2 to v2.5.5. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../.cache/huggingface/hub/models--Unbabel--wmt22-cometkiwi-da/snapshots/1ad785194e391eebc6c53e2d0776cada8f83179a/checkpoints/model.ckpt`\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/huggingface_hub/file_download.py:942: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
|
||
" warnings.warn(\n",
|
||
"Encoder model frozen.\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/pytorch_lightning/core/saving.py:195: Found keys that are not in the model state dict but in the checkpoint: ['encoder.model.embeddings.position_ids']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"prompt = \"\"\"\n",
|
||
"Translate the given statement into English:\n",
|
||
"\n",
|
||
"${statement_to_be_translated}\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"guard = Guard().use(HighQualityTranslation(on_fail=\"fix\"))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 2: Wrap the LLM API call with `Guard`\n"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"First, let's try translating a statement that is relatively easy to translate.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\u001b[92m15:57:09 - LiteLLM:INFO\u001b[0m: utils.py:3389 - \n",
|
||
"LiteLLM completion() model= gpt-5-nano; provider = openai\n",
|
||
"\n",
|
||
"LiteLLM completion() model= gpt-5-nano; provider = openai\n",
|
||
"\u001b[92m15:57:15 - LiteLLM:INFO\u001b[0m: utils.py:1282 - Wrapper: Completed Call, calling success_handler\n",
|
||
"Wrapper: Completed Call, calling success_handler\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/guardrails/validator_service/__init__.py:84: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
|
||
" warnings.warn(\n",
|
||
"💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.\n",
|
||
"GPU available: True (mps), used: False\n",
|
||
"TPU available: False, using: 0 TPU cores\n",
|
||
"HPU available: False, using: 0 HPUs\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/setup.py:177: GPU available but not used. You can set it by doing `Trainer(accelerator='gpu')`.\n",
|
||
"Predicting: 0it [00:00, ?it/s]huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
|
||
"To disable this warning, you can either:\n",
|
||
"\t- Avoid using `tokenizers` before the fork if possible\n",
|
||
"\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
|
||
"Predicting DataLoader 0: 100%|██████████| 1/1 [00:00<00:00, 15.67it/s]\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Raw LLM Output: I have no idea what I’m supposed to write here.\n",
|
||
"</pre>\n"
|
||
],
|
||
"text/plain": [
|
||
"Raw LLM Output: I have no idea what I’m supposed to write here.\n"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Validated Output: I have no idea what I’m supposed to write here.\n",
|
||
"</pre>\n"
|
||
],
|
||
"text/plain": [
|
||
"Validated Output: I have no idea what I’m supposed to write here.\n"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Set your OPENAI_API_KEY as an environment variable\n",
|
||
"# import os\n",
|
||
"# os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n",
|
||
"\n",
|
||
"statement = \"Ich habe keine Ahnung, was ich hier schreiben soll.\"\n",
|
||
"\n",
|
||
"res = guard(\n",
|
||
" messages=[{\"role\": \"user\", \"content\": prompt}],\n",
|
||
" prompt_params={\"statement_to_be_translated\": statement},\n",
|
||
" metadata={\"translation_source\": statement},\n",
|
||
" model=\"gpt-5-nano\",\n",
|
||
" max_tokens=1024,\n",
|
||
" temperature=1,\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Raw LLM Output: {res.raw_llm_output}\")\n",
|
||
"print(f\"Validated Output: {res.validated_output}\")"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can look at the logs to see the quality check results:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Logs\n",
|
||
"└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">╭─────────────────────────────────────────────── Messages ────────────────────────────────────────────────╮</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ ┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ ┃</span><span style=\"background-color: #e7dfeb; font-weight: bold\"> Role </span><span style=\"background-color: #e7dfeb\">┃</span><span style=\"background-color: #e7dfeb; font-weight: bold\"> Content </span><span style=\"background-color: #e7dfeb\">┃ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ ┡━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ │ user │ │ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ │ │ Translate the given statement into English: │ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ │ │ │ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ │ │ Ich habe keine Ahnung, was ich hier schreiben soll. │ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ │ │ │ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ └──────┴─────────────────────────────────────────────────────┘ │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> │\n",
|
||
" │ <span style=\"background-color: #f5f5dc\">╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮</span> │\n",
|
||
" │ <span style=\"background-color: #f5f5dc\">│ I have no idea what I’m supposed to write here. │</span> │\n",
|
||
" │ <span style=\"background-color: #f5f5dc\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> │\n",
|
||
" │ <span style=\"background-color: #f0fff0\">╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮</span> │\n",
|
||
" │ <span style=\"background-color: #f0fff0\">│ I have no idea what I’m supposed to write here. │</span> │\n",
|
||
" │ <span style=\"background-color: #f0fff0\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> │\n",
|
||
" ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
|
||
"</pre>\n"
|
||
],
|
||
"text/plain": [
|
||
"Logs\n",
|
||
"└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n",
|
||
" │ \u001b[48;2;231;223;235m╭─\u001b[0m\u001b[48;2;231;223;235m──────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m Messages \u001b[0m\u001b[48;2;231;223;235m───────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m─╮\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┏━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┃\u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[1;48;2;231;223;235mRole\u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┃\u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[1;48;2;231;223;235mContent \u001b[0m\u001b[1;48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┃\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m┡━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235muser\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235mTranslate the given statement into English: \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235mIch habe keine Ahnung, was ich hier schreiben soll.\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m└──────┴─────────────────────────────────────────────────────┘\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n",
|
||
" │ \u001b[48;2;245;245;220m╭─\u001b[0m\u001b[48;2;245;245;220m───────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m Raw LLM Output \u001b[0m\u001b[48;2;245;245;220m────────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m─╮\u001b[0m │\n",
|
||
" │ \u001b[48;2;245;245;220m│\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220mI have no idea what I’m supposed to write here.\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;245;245;220m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n",
|
||
" │ \u001b[48;2;240;255;240m╭─\u001b[0m\u001b[48;2;240;255;240m──────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m Validated Output \u001b[0m\u001b[48;2;240;255;240m───────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m─╮\u001b[0m │\n",
|
||
" │ \u001b[48;2;240;255;240m│\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240mI have no idea what I’m supposed to write here.\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;240;255;240m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n",
|
||
" ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"print(guard.history.last.tree)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The `guard` wrapper returns the raw LLM response, which is the translated statement and also the validated output. In this case, the translated statement was of a good quality (above the threshold of 0.5), so the validated output is the same as the raw LLM response.\n",
|
||
"\n",
|
||
"#### Now, let's test with a really low quality translation, and see how Guardrails handles it.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/guardrails/validator_service/__init__.py:84: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
|
||
" warnings.warn(\n",
|
||
"💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.\n",
|
||
"GPU available: True (mps), used: False\n",
|
||
"TPU available: False, using: 0 TPU cores\n",
|
||
"HPU available: False, using: 0 HPUs\n",
|
||
"/Users/calebcourier/Projects/guardrails/docs/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/setup.py:177: GPU available but not used. You can set it by doing `Trainer(accelerator='gpu')`.\n",
|
||
"Predicting DataLoader 0: 100%|██████████| 1/1 [00:00<00:00, 12.05it/s]\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Raw LLM Output: It's such a beautiful day, I'm going to the beach.\n",
|
||
"</pre>\n"
|
||
],
|
||
"text/plain": [
|
||
"Raw LLM Output: It's such a beautiful day, I'm going to the beach.\n"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Validated Output: \n",
|
||
"</pre>\n"
|
||
],
|
||
"text/plain": [
|
||
"Validated Output: \n"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Parse the code snippet\n",
|
||
"statement = \"अरे भाऊ, आज रात्री जोरदार पार्टी मारूया, जमून टाकूया आणि धमाल करूया!\"\n",
|
||
"\n",
|
||
"## Ideal translation from Marathi -> English:\n",
|
||
"# \"Hey bro, let's have a great party tonight and have fun!\"\n",
|
||
"\n",
|
||
"output = guard.parse(\n",
|
||
" llm_output=\"It's such a beautiful day, I'm going to the beach.\", ## here, providing a really bad translation\n",
|
||
" metadata={\"translation_source\": statement},\n",
|
||
")\n",
|
||
"\n",
|
||
"# Print the output\n",
|
||
"print(f\"Raw LLM Output: {output.raw_llm_output}\")\n",
|
||
"print(f\"Validated Output: {output.validated_output}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Logs\n",
|
||
"└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">╭─────────────────────────────────────────────── Messages ────────────────────────────────────────────────╮</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">│ No messages. │</span> │\n",
|
||
" │ <span style=\"background-color: #e7dfeb\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> │\n",
|
||
" │ <span style=\"background-color: #f5f5dc\">╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮</span> │\n",
|
||
" │ <span style=\"background-color: #f5f5dc\">│ It's such a beautiful day, I'm going to the beach. │</span> │\n",
|
||
" │ <span style=\"background-color: #f5f5dc\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> │\n",
|
||
" │ <span style=\"background-color: #f0fff0\">╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮</span> │\n",
|
||
" │ <span style=\"background-color: #f0fff0\">│ '' │</span> │\n",
|
||
" │ <span style=\"background-color: #f0fff0\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span> │\n",
|
||
" ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
|
||
"</pre>\n"
|
||
],
|
||
"text/plain": [
|
||
"Logs\n",
|
||
"└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮\n",
|
||
" │ \u001b[48;2;231;223;235m╭─\u001b[0m\u001b[48;2;231;223;235m──────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m Messages \u001b[0m\u001b[48;2;231;223;235m───────────────────────────────────────────────\u001b[0m\u001b[48;2;231;223;235m─╮\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m│\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235mNo messages.\u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m \u001b[0m\u001b[48;2;231;223;235m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;231;223;235m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n",
|
||
" │ \u001b[48;2;245;245;220m╭─\u001b[0m\u001b[48;2;245;245;220m───────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m Raw LLM Output \u001b[0m\u001b[48;2;245;245;220m────────────────────────────────────────────\u001b[0m\u001b[48;2;245;245;220m─╮\u001b[0m │\n",
|
||
" │ \u001b[48;2;245;245;220m│\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220mIt's such a beautiful day, I'm going to the beach.\u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m \u001b[0m\u001b[48;2;245;245;220m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;245;245;220m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n",
|
||
" │ \u001b[48;2;240;255;240m╭─\u001b[0m\u001b[48;2;240;255;240m──────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m Validated Output \u001b[0m\u001b[48;2;240;255;240m───────────────────────────────────────────\u001b[0m\u001b[48;2;240;255;240m─╮\u001b[0m │\n",
|
||
" │ \u001b[48;2;240;255;240m│\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m''\u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m \u001b[0m\u001b[48;2;240;255;240m│\u001b[0m │\n",
|
||
" │ \u001b[48;2;240;255;240m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m │\n",
|
||
" ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"print(guard.history.last.tree)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"As you can see, the translation quality is really bad, and the `HighQualityTranslation` check failed as the translation quality was below the threshold. The validated response is an empty string.\n",
|
||
"\n",
|
||
"## In this way, you can use Guardrails to ensure that the output of your LLM is of high quality.\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.12.11"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
}
|