{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Playing Valid Chess Moves\n", "\n", "!!! note\n", " To download this example as a Jupyter notebook, click [here](https://github.com/guardrails-ai/guardrails/blob/main/docs/examples/valid_chess_moves.ipynb).\n", "\n", "!!! warning\n", " This example is currently under development (it cannot be used to play a full chess game yet).\n", "\n", "In this example, we will use Guardrails to play chess with an LLM and ensure that it makes valid moves.\n", "\n", "## Objective\n", "\n", "We want to generate a valid chess moves for a given board state." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/dtam/.pyenv/versions/3.12.3/envs/litellm/lib/python3.12/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:13: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n", " from tqdm.autonotebook import tqdm, trange\n" ] } ], "source": [ "import guardrails as gd\n", "from rich import print" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "! pip install chess --quiet" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Create the RAIL Spec\n", "\n", "Ordinarily, we would create an RAIL spec in a separate file. For the purposes of this example, we will create the spec in this notebook as a string following the RAIL syntax. For more information on RAIL, see the [RAIL documentation](/docs/how_to_guides/rail). We will also show the same RAIL spec in a code-first format using a Pydantic model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First we define a custom Validator:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from guardrails.validators import (\n", " Validator,\n", " register_validator,\n", " ValidationResult,\n", " PassResult,\n", " FailResult,\n", ")\n", "\n", "from typing import Dict, Any\n", "\n", "import chess\n", "\n", "BOARD = chess.Board()\n", "\n", "\n", "@register_validator(name=\"is-valid-chess-move\", data_type=\"string\")\n", "class IsValidChessMove(Validator):\n", " board = BOARD\n", "\n", " def validate(self, value: Any, metadata: Dict) -> ValidationResult:\n", " global BOARD\n", " try:\n", " # Push the move onto the board.\n", " BOARD.push_san(value)\n", " except Exception as e:\n", " # If the move is invalid, raise an error.\n", " return FailResult(\n", " error_message=f\"Value {value} is not a valid chess move. {e}\"\n", " )\n", "\n", " return PassResult()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then we can define our RAIL spec either as XML:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "rail_str = \"\"\"\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", "\n", "Generate a move for the chess board. Do not repeat any moves in the following state. The board is currently in the following state:\n", "${board_state}\n", "${gr.complete_xml_suffix}\n", "\n", "\n", "\n", "\n", "\"\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or as a Pydantic model:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from pydantic import BaseModel, Field\n", "\n", "prompt = \"\"\"\n", "Generate a move for the chess board. Do not repeat any moves in the following state. The board is currently in the following state:\n", "${board_state}\n", "${gr.complete_xml_suffix}\n", "\"\"\"\n", "\n", "\n", "class ChessMove(BaseModel):\n", " move: str = Field(\n", " description=\"A move in standard algebraic notation.\",\n", " validators=[IsValidChessMove(on_fail=\"reask\")],\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 2: Create a `Guard` object with the RAIL Spec\n", "\n", "We create a `gd.Guard` object that will check, validate and correct the output of the LLM. This object:\n", "\n", "1. Enforces the quality criteria specified in the RAIL spec.\n", "2. Takes corrective action when the quality criteria are not met.\n", "3. Compiles the schema and type info from the RAIL spec and adds it to the prompt." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From XML:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "guard = gd.Guard.for_rail_string(rail_str)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From a Pydantic model:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "guard = gd.Guard.for_pydantic(output_class=ChessMove)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Let's get the reference to the board." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "
r n b q k b n r\n",
       "p p p p p p p p\n",
       ". . . . . . . .\n",
       ". . . . . . . .\n",
       ". . . . . . . .\n",
       ". . . . . . . .\n",
       "P P P P P P P P\n",
       "R N B Q K B N R
" ], "text/plain": [ "Board('rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1')" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "board = guard._validator_map.get(\"$.move\")[0].board\n", "board" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3: Wrap the LLM API call with `Guard`" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/dtam/dev/guardrails/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n", " warnings.warn(\n" ] } ], "source": [ "# Set your OPENAI_API_KEY as an environment variable\n", "# import os\n", "# os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n", "\n", "raw_llm_response, validated_response, *rest = guard(\n", " messages=[{\"role\": \"user\", \"content\": prompt}],\n", " prompt_params={\n", " \"board_state\": str(board.move_stack)\n", " if board.move_stack\n", " else \"Starting position.\"\n", " },\n", " model=\"gpt-4o-mini\",\n", " max_tokens=2048,\n", " temperature=0.3,\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "We can see in the prompt that was sent to the LLM, the `{board_state}` parameter is substituted with the current state of the board." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n",
       "Generate a move for the chess board. Do not repeat any moves in the following state. The board is currently in the \n",
       "following state:\n",
       "Starting position.\n",
       "\n",
       "Given below is XML that describes the information to extract from this document and the tags to extract it into.\n",
       "\n",
       "<output>\n",
       "  <string description=\"A move in standard algebraic notation.\" format=\"is-valid-chess-move\" name=\"move\" \n",
       "required=\"true\"></string>\n",
       "</output>\n",
       "\n",
       "ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` \n",
       "attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON\n",
       "MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and \n",
       "specific types. Be correct and concise. If you are unsure anywhere, enter `null`.\n",
       "\n",
       "Here are examples of simple (XML, JSON) pairs that show the expected behavior:\n",
       "- `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}`\n",
       "- `<list name='bar'><string format='upper-case' /></list>` => `{\"bar\": ['STRING ONE', 'STRING TWO', etc.]}`\n",
       "- `<object name='baz'><string name=\"foo\" format=\"capitalize two-words\" /><integer name=\"index\" format=\"1-indexed\" \n",
       "/></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}`\n",
       "\n",
       "\n",
       "
\n" ], "text/plain": [ "\n", "Generate a move for the chess board. Do not repeat any moves in the following state. The board is currently in the \n", "following state:\n", "Starting position.\n", "\n", "Given below is XML that describes the information to extract from this document and the tags to extract it into.\n", "\n", "\u001b[1m<\u001b[0m\u001b[1;95moutput\u001b[0m\u001b[39m>\u001b[0m\n", "\u001b[39m <\u001b[0m\u001b[35m/\u001b[0m\u001b[95mstring\u001b[0m\u001b[39m>\u001b[0m\n", "\u001b[39m<\u001b[0m\u001b[35m/\u001b[0m\u001b[95moutput\u001b[0m\u001b[39m>\u001b[0m\n", "\n", "\u001b[39mONLY return a valid JSON object \u001b[0m\u001b[1;39m(\u001b[0m\u001b[39mno other text is necessary\u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m, where the key of the field in JSON is the `name` \u001b[0m\n", "\u001b[39mattribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON\u001b[0m\n", "\u001b[39mMUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and \u001b[0m\n", "\u001b[39mspecific types. Be correct and concise. If you are unsure anywhere, enter `null`.\u001b[0m\n", "\n", "\u001b[39mHere are examples of simple \u001b[0m\u001b[1;39m(\u001b[0m\u001b[39mXML, JSON\u001b[0m\u001b[1;39m)\u001b[0m\u001b[39m pairs that show the expected behavior:\u001b[0m\n", "\u001b[39m- `` => `\u001b[0m\u001b[1;39m{\u001b[0m\u001b[32m'foo'\u001b[0m\u001b[39m: \u001b[0m\u001b[32m'example one'\u001b[0m\u001b[1;39m}\u001b[0m\u001b[39m`\u001b[0m\n", "\u001b[39m- `<\u001b[0m\u001b[35m/\u001b[0m\u001b[95mlist\u001b[0m\u001b[39m>` => `\u001b[0m\u001b[1;39m{\u001b[0m\u001b[32m\"bar\"\u001b[0m\u001b[39m: \u001b[0m\u001b[1;39m[\u001b[0m\u001b[32m'STRING ONE'\u001b[0m\u001b[39m, \u001b[0m\u001b[32m'STRING TWO'\u001b[0m\u001b[39m, etc.\u001b[0m\u001b[1;39m]\u001b[0m\u001b[1;39m}\u001b[0m\u001b[39m`\u001b[0m\n", "\u001b[39m- `<\u001b[0m\u001b[35m/\u001b[0m\u001b[95mobject\u001b[0m\u001b[39m>` =\u001b[0m\u001b[1m>\u001b[0m `\u001b[1m{\u001b[0m\u001b[32m'baz'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'foo'\u001b[0m: \u001b[32m'Some String'\u001b[0m, \u001b[32m'index'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m`\n", "\n", "\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(guard.history.last.iterations.last.inputs.messages[0][\"content\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `guard` wrapper returns the raw_llm_respose (which is a simple string), and the validated and corrected output (which is a dictionary).\n", "\n", "We can see that the output is a dictionary with the correct schema and types." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{'move': 'e4'}\n",
       "
\n" ], "text/plain": [ "\u001b[1m{\u001b[0m\u001b[32m'move'\u001b[0m: \u001b[32m'e4'\u001b[0m\u001b[1m}\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(validated_response)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "
r n b q k b n r\n",
       "p p p p p p p p\n",
       ". . . . . . . .\n",
       ". . . . . . . .\n",
       ". . . . P . . .\n",
       ". . . . . . . .\n",
       "P P P P . P P P\n",
       "R N B Q K B N R
" ], "text/plain": [ "Board('rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq - 0 1')" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "board" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Let's make a move." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "
r n b q k b n r\n",
       "p p p p . p p p\n",
       ". . . . . . . .\n",
       ". . . . p . . .\n",
       ". . . . P . . .\n",
       ". . . . . . . .\n",
       "P P P P . P P P\n",
       "R N B Q K B N R
" ], "text/plain": [ "Board('rnbqkbnr/pppp1ppp/8/4p3/4P3/8/PPPP1PPP/RNBQKBNR w KQkq - 0 2')" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "board.push_san(\"e5\")\n", "board" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Ask for another move from the model." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/dtam/dev/guardrails/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n", " warnings.warn(\n" ] } ], "source": [ "raw_llm_response, validated_response, *rest = guard(\n", " messages=[{\"role\": \"user\", \"content\": prompt}],\n", " prompt_params={\"board_state\": str(board.move_stack)},\n", " model=\"gpt-4o-mini\",\n", " max_tokens=2048,\n", " temperature=0.0,\n", ")" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "
r n b q k b n r\n",
       "p p p p . p p p\n",
       ". . . . . . . .\n",
       ". . . . p . . .\n",
       ". . . . P . . .\n",
       ". . . . . N . .\n",
       "P P P P . P P P\n",
       "R N B Q K B . R
" ], "text/plain": [ "Board('rnbqkbnr/pppp1ppp/8/4p3/4P3/5N2/PPPP1PPP/RNBQKB1R b KQkq - 1 2')" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "board" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "
r . b q k b n r\n",
       "p p p p . p p p\n",
       ". . n . . . . .\n",
       ". . . . p . . .\n",
       ". . . . P . . .\n",
       ". . . . . N . .\n",
       "P P P P . P P P\n",
       "R N B Q K B . R
" ], "text/plain": [ "Board('r1bqkbnr/pppp1ppp/2n5/4p3/4P3/5N2/PPPP1PPP/RNBQKB1R w KQkq - 2 3')" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "board.push_san(\"Nc6\")\n", "board" ] } ], "metadata": { "kernelspec": { "display_name": "litellm", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }