lasta/AI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

참고소스 수정본

Browse Source
This commit is contained in:
LASTA_DEV01\lasta
2026-05-12 19:40:31 +09:00
parent 0f34a451fc
commit 2e9204243d
8708 changed files with 3259488 additions and 869 deletions
Download Patch File Download Diff File Expand all files Collapse all files

139
참고/instructor-main/docs/prompting/decomposition/decomp.md Normal file
View File

@@ -0,0 +1,139 @@
---
description: "DECOMP involves using a LLM to break down a complicated task into sub tasks that it has been provided with"
---
Decomposed Prompting<sup><a href="https://arxiv.org/pdf/2210.02406">1</a></sup> leverages a Language Model (LLM) to deconstruct a complex task into a series of manageable sub-tasks. Each sub-task is then processed by specific functions, enabling the LLM to handle intricate problems more effectively and systematically.
In the code snippet below, we define a series of data models and functions to implement this approach.
The `derive_action_plan` function generates an action plan using the LLM, which is then executed step-by-step. Each action can be
1. InitialInput: Which represents the chunk of the original prompt we need to process
2. Split : An operation to split strings using a given separator
3. StrPos: An operation to help extract a string given an index
4. Merge: An operation to join a list of strings together using a given character
We can implement this using `instructor` as seen below.
```python hl_lines="57-58"
import instructor
from pydantic import BaseModel, Field
from typing import Union
client = instructor.from_provider("openai/gpt-5-nano")
class Split(BaseModel):
split_char: str = Field(
description="""This is the character to split
the string with"""
)
def split_chars(self, s: str, c: str):
return s.split(c)
class StrPos(BaseModel):
index: int = Field(
description="""This is the index of the character
we wish to return"""
)
def get_char(self, s: list[str], i: int):
return [c[i] for c in s]
class Merge(BaseModel):
merge_char: str = Field(
description="""This is the character to merge the
inputs we plan to pass to this function with"""
)
def merge_string(self, s: list[str]):
return self.merge_char.join(s)
class Action(BaseModel):
id: int = Field(
description="""Unique Incremental id to identify
this action with"""
)
action: Union[Split, StrPos, Merge]
class ActionPlan(BaseModel):
initial_data: str
plan: list[Action]
def derive_action_plan(task_description: str) -> ActionPlan:
return client.create(
messages=[
{
"role": "system",
"content": """Generate an action plan to help you complete
the task outlined by the user""",
},
{"role": "user", "content": task_description},
],
response_model=ActionPlan,
max_retries=3,
model="gpt-4o",
)
if __name__ == "__main__":
task = """Concatenate the second letter of every word in Jack
Ryan together"""
plan = derive_action_plan(task)
print(plan.model_dump_json(indent=2))
"""
{
"initial_data": "Jack Ryan",
"plan": [
{
"id": 1,
"action": {
"split_char": " "
}
},
{
"id": 2,
"action": {
"index": 1
}
},
{
"id": 3,
"action": {
"merge_char": ""
}
}
]
}
"""
curr = plan.initial_data
cache = {}
for action in plan.plan:
if isinstance(action.action, Split) and isinstance(curr, str):
curr = action.action.split_chars(curr, action.action.split_char)
elif isinstance(action.action, StrPos) and isinstance(curr, list):
curr = action.action.get_char(curr, action.action.index)
elif isinstance(action.action, Merge) and isinstance(curr, list):
curr = action.action.merge_string(curr)
else:
raise ValueError("Unsupported Operation")
print(action, curr)
#> id=1 action=Split(split_char=' ') ['Jack', 'Ryan']
#> id=2 action=StrPos(index=1) ['a', 'y']
#> id=3 action=Merge(merge_char='') ay
print(curr)
#> ay
```
### References
<sup id="ref-1">1</sup>: [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://arxiv.org/pdf/2210.02406)

101
참고/instructor-main/docs/prompting/decomposition/faithful_cot.md Normal file
View File

@@ -0,0 +1,101 @@
---
description: "Faithful Chain of Thought aims to use multiple reasoning steps to improve the quality of the final outputs"
---
Faithful Chain of Thought<sup><a href="https://arxiv.org/pdf/2301.13379">1</a></sup> improves the faithfulness of reasoning chains generated by Language Models by breaking it up into two stages
1. **Translation** : We first translate a user query into a series of reasoning steps. These are a task specific set of steps that we can execute deterministically.
2. **Problem Solving**: We execute our steps and arrive at a final answer that we can derive. This ensures that our Chain Of Thought is able to derive a answer that is consistent with the reasoning steps.
They list a few examples in the paper of what these task-specific steps could be
1. **Math Word Problems** : Python Code that can be executed by an interpreter to derive a final answer
2. **Multi-Hop QA** : This is a multi-step reasoning process. To solve this, they use a mix of python and Datalog ( which is a relation and log programming language ) to arrive at a final answer
3. **Planning** : When trying to generate a plan to solve a user query, they generate a list of symbolic goals in a Programming Language and then call a PDDL Planner to obtain a plan to solve the user's query
![](../../img/faithful_cot_example.png)
In the example below, we show how you can use a LLM to generate python code that can be executed by an Interpreter to arrive at a final answer.
We can implement it in `instructor` as seen below
```python hl_lines="30-45"
import instructor
from pydantic import BaseModel, Field
client = instructor.from_provider("openai/gpt-5-nano")
class ReasoningStep(BaseModel):
id: int = Field(description="Unique ID")
rationale: list[str] = Field(
description="""Specific sections from prior reasoning
steps or the context that ground this reasoning step"""
)
dependencies: list[int] = Field(
description="""IDs of prior reasoning steps that this
reasoning step depends on"""
)
eval_string: str = Field(
description="""Python Code to execute to generate the
final evaluation"""
)
def generate_reasoning_steps(query: str) -> list[ReasoningStep]:
return client.create(
messages=[
{
"role": "system",
"content": """
You are a world class AI who excels at
generating reasoning steps to answer a
question. You will be given a question
and you will generate a list of reasoning
steps that are needed to answer the
question.
At each point you should either
- declare a variable to be referenced
later on
- combine multiple variables together to
generate a new result that you should
store in another variable
The final answer should be stored in a
variable called `answer`.
""",
},
{"role": "user", "content": query},
],
model="gpt-4o",
response_model=list[ReasoningStep],
)
if __name__ == "__main__":
steps = generate_reasoning_steps(
"""If there are 3 cars in the parking lot and 2 more
cars arrive, how many cars are in the parking lot
after another 2 more arrive?"""
)
code = "\n".join([step.eval_string for step in steps])
print(code)
"""
initial_cars = 3
arriving_cars = 2
cars_after_first_arrival = initial_cars + arriving_cars
final_car_count = cars_after_first_arrival + 2
answer = final_car_count
"""
exec(code)
local_vars = {}
exec(code, {}, local_vars)
print(local_vars.get("answer"))
#> 7
```
### References
<sup id="ref-1">1</sup>: [Faithful Chain-of-Thought Reasoning](https://arxiv.org/pdf/2301.13379)

103
참고/instructor-main/docs/prompting/decomposition/least_to_most.md Normal file
View File

@@ -0,0 +1,103 @@
---
title: "Solve simpler subproblems"
description: "Least-to-Most is a prompting technique that breaks a complex problem down into a series of increasingly complex subproblems."
---
Given a complex problem, how can we encourage an LLM to solve simpler subproblems?
Least-to-Most is a prompting technique that breaks a complex problem down into a series of increasingly complex subproblems.
!!! example "Subproblems Example"
**original problem**: Adam is twice as old as Mary. Adam will be 11 in 1 year. How old is Mary?
**subproblems**: (1) How old is Adam now? (2) What is half of Adam's current age?
These subproblems are solved sequentially, allowing the answers from earlier (simpler) subproblems to inform the LLM while solving later (more complex) subproblems.
```python
import instructor
from pydantic import BaseModel
from typing import Iterable
class Subquestion(BaseModel):
question: str
class Answer(BaseModel):
answer: int
class SubquestionWithAnswers(BaseModel):
question: str
answer: int
client = instructor.from_provider("openai/gpt-5-nano")
def decompose(question):
return client.create(
model="gpt-4o",
response_model=Iterable[Subquestion],
messages=[
{
"role": "user",
"content": f"Break this question down into subquestions to solve sequentially: {question}",
}
],
)
def solve(question, solved_questions, original_question):
return client.create(
model="gpt-4o",
response_model=Answer,
messages=[
{
"role": "user",
"content": f"""
<original_question>
{original_question}
</original_question>
<solved_subquestions>
{solved_questions}
</solved_subquestions>
Solve this next subquestion: {question}
""",
}
],
).answer
if __name__ == "__main__":
question = "Four years ago, Kody was only half as old as Mohamed. If Mohamed is currently twice 30 years old, how old is Kody?"
# Stage 1: Decompose Question into Subquestions
subquestions = decompose(question)
# Stage 2: Sequentially Solve Subquestions
solved_questions = []
for subquestion in subquestions:
solved_questions.append(
SubquestionWithAnswers(
question=subquestion.question,
answer=solve(subquestion, solved_questions, question),
)
)
# Print
for item in solved_questions:
print(f"{item.question} {item.answer}")
#> How old is Mohamed currently? 60
#> How old was Mohamed four years ago? 56
#> How old was Kody four years ago if he was half as old as Mohamed? 28
#> How old is Kody currently? 32
```
### References
<sup id="ref-1">1</sup>: [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://arxiv.org/abs/2205.10625)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

151
참고/instructor-main/docs/prompting/decomposition/plan_and_solve.md Normal file
View File

@@ -0,0 +1,151 @@
---
description: "Plan and Solve involves the use of an improved zero-shot CoT prompt. This generates more robust reasoning processes than standard Zero-Shot CoT on multiple reasoning datasets"
---
Plan and Solve<sup><a href="https://arxiv.org/pdf/2305.04091">1</a></sup> improves the use of an improved Zero-Shot Chain Of Thought (CoT) prompt which adds more detailed instructions to the prompt given to these large language models.
!!! example "Plan and Solve Prompt"
[User Prompt]
**Let’s first understand the problem, extract relevant variables and their corresponding numerals, and make a complete plan.Then, let’s carry out the plan, calculate intermediate variables (pay attention to correct numerical calculation and commonsense), solve the problem step by step, and show the answer.**
[Model Response]
**Therefore the answer(arabic numerals) is**
This is a two step process which guides the LLM to pay more attention to calculation and intermediate results to ensure that they are correctly performed as much as possible.
1. **Generate Reasoning**: In the first step we prompt the model with the user's query and prime the model using plan and solve prompting to explicitly devise a plan for solving a problem before generating an intermediate reasoning process
2. **Extract Answer** : Once we've obtained the model's reasoning, we then extract the answer from a new prompt which includes the model's chain of thought.
![](../../img/plan_and_solve.png)
We can implement this using `instructor` as seen below.
```python hl_lines="26-34 67"
import instructor
from pydantic import BaseModel
client = instructor.from_provider("openai/gpt-5-nano")
class Reasoning(BaseModel):
chain_of_thought: str
class Response(BaseModel):
correct_answer: str
def generate_reasoning(query: str):
return client.create(
messages=[
{
"role": "user",
"content": f"""
<user query>
{query}
</user query>
Let's first understand the problem,
extract relevant variables and their
corresponding numerals, and make a
complete plan. Then, let's carry out
the plan, calculate intermediate
variables (pay attention to correct
numerical calculation and commonsense),
solve the problem step by step, and
show the answer.
""",
},
],
response_model=Reasoning,
model="gpt-4o",
)
def extract_answer(query: str, reasoning: Reasoning):
return client.create(
messages=[
{
"role": "user",
"content": f"""
<user query>
{query}
</user query>
Let's first understand the problem,
extract relevant variables and their
corresponding numerals, and make a
complete plan. Then, let's carry out
the plan, calculate intermediate
variables (pay attention to correct
numerical calculation and commonsense),
solve the problem step by step, and
show the answer.
<reasoning>
{reasoning.chain_of_thought}
</reasoning>
Therefore the answer (arabic numerals) is
""",
}
],
model="gpt-4o",
response_model=Response,
)
if __name__ == "__main__":
query = (
"In a dance class of 20 students, 20% enrolled "
"in contemporary dance, 25% of the remaining "
"enrolled in jazz dance and the rest enrolled "
"in hip-hop dance. What percentage of the entire "
"students enrolled in hip-hop dance?"
)
reasoning = generate_reasoning(query)
print(reasoning.model_dump_json(indent=2))
"""
{
"chain_of_thought": "Let's first break down the
problem:\n\n1. Total number of students = 20\n2.
Percentage enrolled in contemporary dance = 20%\n\n
Step-by-Step Plan:\n1. Calculate the number of
students enrolled in contemporary dance.\n2.
Calculate the remaining students after contemporary
dance enrollment.\n3. Calculate the percentage and
number of students from the remaining who enrolled in
jazz dance.\n4. Determine the remaining students who
enrolled in hip-hop dance.\n5. Finally, calculate the
percentage of the entire students who enrolled in
hip-hop dance.\n\nLet's carry out the plan:\n\n1.
Number of students enrolled in contemporary dance =
20% of 20 = (20/100) * 20 = 4\n2. Remaining students
after contemporary = 20 - 4 = 16\n3. Percentage of
remaining students enrolled in jazz dance = 25%\n
Number of students enrolled in jazz dance = 25% of 16
= (25/100) * 16 = 4\n4. Remaining students after
contemporary and jazz = 16 - 4 = 12\n5. The number of
students enrolled in hip-hop dance = 12\n6.
Percentage of entire students enrolled in hip-hop =
(Number of hip-hop students / Total students) *
100\n Percentage = (12 / 20) * 100 = 60%\n\nThus,
60% of the entire students enrolled in hip-hop dance."
}
"""
response = extract_answer(query, reasoning)
print(response.model_dump_json(indent=2))
"""
{
"correct_answer": "60"
}
"""
```
### References
<sup id="ref-1">1</sup>: [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://arxiv.org/pdf/2305.04091)

163
참고/instructor-main/docs/prompting/decomposition/program_of_thought.md Normal file
View File

@@ -0,0 +1,163 @@
---
description: "Program Of Thought"
---
Program of Thought aims to leverage an external python interpreter in order to generate intermediate reasoning steps. This helps us to achieve a greater degree of performance in mathematical and programming-related tasks by grounding our final response in deterministic code.
![](../../img/pot.jpeg)
We can implement it in `instructor` as seen below
```python hl_lines="120-125"
from pydantic import BaseModel, Field, field_validator
import instructor
from textwrap import dedent
from typing import Literal
client = instructor.from_provider("openai/gpt-5-nano")
prefix = """
# Answer this question by implementing a solver()
# function, use for loop if necessary.
def solver():
# Let's write a Python program step by step,
# and then return the answer
# Firstly, we need to define the following
# variable:
""".strip()
def execute_program(code: str):
code = code.strip() + "\nans = solver()"
print(code)
"""
# Answer this question by implementing a
# solver() function, use for loop if necessary.
def solver():
# Let's write a Python program step by step,
# and then return the answer
# Firstly, we need to define the following
# variable:
selling_price = 360
profit_percentage = 20
# To find the cost price, use the formula:
# cost_price = selling_price / (1 + profit_percentage / 100)
cost_price = selling_price / (1 + profit_percentage / 100)
return cost_price
# Running the solver function to get the cost price
result = solver()
print(result)
ans = solver()
"""
exec(code)
locals_ = locals()
return locals_.get("ans")
class Prediction(BaseModel):
choice: Literal["A", "B", "C", "D", "E"]
class ProgramExecution(BaseModel):
program_code: str = Field(
description="""Program Code that
once executed contains the final answer"""
)
@field_validator("program_code")
@classmethod
def ensure_valid_code(cls, v: str) -> str:
if not v.startswith(prefix):
raise ValueError(
f"""Program Code must begin with the desired
prefix of {prefix}"""
)
answer = execute_program(v)
if not answer:
raise ValueError(
f"""Make sure to return the answer to the
question within the solver function"""
)
return str(answer)
def generate_intermediate_reasoning(query: str):
return client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": dedent(
f"""
You are a world class AI system that excels
at answering user queries in a systematic
and detailed manner. You are about to be
passed a user query to respond to. Make sure
to generate a valid program that can be
executed to answer the user query.
Make sure to begin your generated program
with the following prefix
{prefix}
"""
),
},
{
"role": "user",
"content": query,
},
],
response_model=ProgramExecution,
)
def generate_prediction(
predicted_answer: str, options: list[str], query: str
) -> Prediction:
formatted_options = ",".join(options)
return client.create(
model="gpt-4o",
response_model=Prediction,
messages=[
{
"role": "system",
"content": dedent(
f"""
Find the closest options based on the
question and prediction.
Question: {query}
Prediction: {predicted_answer}
Options: [{formatted_options}]
"""
),
}
],
)
if __name__ == "__main__":
query = """A trader sold an article at a profit of 20%
for Rs.360. What is the cost price of the article?"""
reasoning = generate_intermediate_reasoning(query)
options = ["A)270", "B)300", "C)280", "D)320", "E)315"]
print(reasoning.model_dump_json(indent=2))
"""
{
"program_code": "300.0"
}
"""
prediction = generate_prediction(reasoning.program_code, options, query)
print(prediction.model_dump_json(indent=2))
"""
{
"choice": "B"
}
"""
```

7
참고/instructor-main/docs/prompting/decomposition/recurs_of_thought.md Normal file
View File

@@ -0,0 +1,7 @@
---
title: ""
description: ""
keywords: ""
---
[wip]

152
참고/instructor-main/docs/prompting/decomposition/skeleton_of_thought.md Normal file
View File

@@ -0,0 +1,152 @@
---
title: "Generate in Parallel"
description: "Skelelton-of-Thought is a technique which prompts an LLM to generate a skeleton outline of the response, then completes each point in the skeleton in parallel."
---
How do we decrease the latency of an LLM pipeline?
Skelelton-of-Thought is a technique which prompts an LLM to generate a skeleton outline of the response, then completes each point in the skeleton in parallel. The parallelism can be achieved by parallel API calls or batched decoding.
Below is an example of an implementation using parallel API calls with `instructor`:
```python
import instructor
from pydantic import BaseModel
import asyncio
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Point(BaseModel):
index: int
description: str
class Skeleton(BaseModel):
points: list[Point]
class Response(BaseModel):
response: str
async def get_skeleton(question):
return await client.create(
model="gpt-4o",
response_model=Skeleton,
messages=[
{
"role": "user",
"content": f"""
You’re an organizer responsible for only giving the skeleton (not the full content) for answering the question.
Provide the skeleton in a list of points (numbered 1., 2., 3., etc.) to answer the question.
Instead of writing a full sentence, each skeleton point should be very short with only 3∼5 words.
Generally, the skeleton should have 3∼10 points.
Now, please provide the skeleton for the following question.
<question>
{question}
</question>
Skeleton:
""",
}
],
)
async def expand_point(question, skeleton, point_index):
return await client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "user",
"content": f"""
You’re responsible for continuing the writing of one and only one point in the overall answer to the following question.
<question>
{question}
</question>
The skeleton of the answer is:
<skeleton>
{skeleton}
</skeleton>
Continue and only continue the writing of point {point_index}.
Write it **very shortly** in 1∼2 sentence and do not continue with other points!
""",
}
],
)
async def main():
query = "Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions."
# Step 1: Get the skeleton
skeleton = await get_skeleton(query)
for point in skeleton.points:
print(point)
#> index=1 description='Introduction to Hawaii trip'
#> index=2 description='Arrival and first impressions'
#> index=3 description='Traditional Hawaiian cuisine'
#> index=4 description='Exploring local markets'
#> index=5 description='Visit to historic sites'
#> index=6 description='Experience a Hawaiian luau'
#> index=7 description='Day at the beach'
#> index=8 description='Hiking adventures'
#> index=9 description='Scenic viewpoints'
#> index=10 description='Closing remarks and tips'
# Step 2: Expand on each point in parallel
tasks = [expand_point(query, skeleton, point.index) for point in skeleton.points]
responses = await asyncio.gather(*tasks)
for response in responses:
print(response.response)
"""
Hawaii-a paradise of golden beaches, lush landscapes, and vibrant culture-beckoned us with the promise of adventure and unforgettable experiences. Our journey began the moment we landed on this magical archipelago, ready to explore its unique blend of natural beauty and rich traditions.
"""
"""
The moment we landed in Hawaii, we were greeted with warm aloha spirit, lush tropical landscapes, and the gentle aroma of hibiscus flowers in the air.
"""
"""
The traditional Hawaiian cuisine was an exotic delight; from savoring the rich flavors of poke bowls to indulging in the sweet taste of haupia, every bite was a unique cultural experience.
"""
"""
Exploring local markets was a vibrant and delightful experience, where the air was filled with the scent of exotic fruits, freshly-made poke, and sounds of local musicians. We discovered unique handicrafts and interacted with friendly vendors eager to share their stories and traditions.
"""
"""
A visit to Pearl Harbor is a poignant reminder of the past, offering a chance to pay respects and learn about the events that shaped history. Walking through the USS Arizona Memorial and exploring the interactive exhibits was both humbling and enlightening.
"""
"""
Point 6: Experience a Hawaiian luau - Attending a traditional Hawaiian luau was unforgettable, filled with vibrant dances, soulful music, and a feast of mouthwatering dishes cooked in an imu (underground oven). It was a magical evening that immersed us in the heart of Hawaiian culture.
"""
"""
A day at the beach in Hawaii was pure bliss. The crystal-clear waters and soft sands were the perfect backdrop for both relaxation and adventure, from sunbathing to snorkeling.
"""
"""
Hiking adventures in Hawaii offer a unique chance to connect with nature, with trails leading to stunning waterfalls and lush rainforests. Don’t miss out on the Na Pali Coast's breathtaking hikes!
"""
"""
One of the highlights of my trip was visiting the scenic viewpoints such as the Na Pali Coast and Haleakalā National Park, offering breathtaking panoramic views that are perfect for photography aficionados and nature lovers alike.
"""
"""
As you plan your trip, don't forget to pack plenty of sunscreen and a camera to capture every magical moment. Hawaii offers a unique blend of relaxation and adventure that's sure to leave you with unforgettable memories.
"""
if __name__ == "__main__":
asyncio.run(main())
```
### References
<sup id="ref-1">1</sup>: [Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation](https://arxiv.org/abs/2307.15337)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

7
참고/instructor-main/docs/prompting/decomposition/tree-of-thought.md Normal file
View File

@@ -0,0 +1,7 @@
---
title: ""
description: ""
keywords: ""
---
[wip]

270
참고/instructor-main/docs/prompting/ensembling/cosp.md Normal file
View File

@@ -0,0 +1,270 @@
---
description: "Consistency Based Self Adaptive Prompting (COSP) is a ensembling technique that aims to combine multiple Chain Of Thought reasoning calls"
---
Consistency Based Self Adaptive Prompting (COSP)<sup><a href="https://arxiv.org/pdf/2305.14106">1</a></sup> aims to improve LLM output quality by generating high quality few shot examples to be included in the final prompt. These are examples without labelled ground truth so they use self-consistency and a metric known as normalized entropy to select the best examples.
Once they've selected the examples, they then append them to the prompt and generate multiple reasoning chains before selecting the final result using [Self-Consistency](self_consistency.md).
## COSP process
![](../../img/cosp.png)
How does this look in practice? Let's dive into greater detail.
### Step 1 - Selecting Examples
In the first step, we try to generate high quality examples from questions that don't have ground truth labels. This is challenging because we want to find a way to automatically determine answer quality when sampling our model multiple times.
In this case, we have `n` questions which we want to generate `m` possible reasoning chains for each question. This gives a total of `nm` examples. We then want to filter out `k` final few shot examples from these `nm` examples to be included inside our final prompt.
1. Using chain of thought, we first generate `m` responses for each question. These responses contain a final answer and a rationale behind that answer.
2. We compute a score for each response using a weighted sum of two values - normalized entropy and repetitiveness ( How many times this rationale appears for this amswer )
3. We rank all of our `nm` responses using this score and choose the `k` examples with the lowest scores as our final few shot examples.
#### Normalized Entropy
> In the paper, the authors write that normalized entropy is a good proxy over a number of different tasks where low entropy is positively correlated with correctness. Entropy is also supposed to range from 0 to 1.
>
> Therefore in order to do so, we introduce a `-` term in our implementation so that the calculated values range from 0 to 1.
![](../../img/cosp_entropy.png)
Assuming that for a specific question $x^{(i)}$, we have generated $m$ final answers of which $u$ are unique. ( Note that this only cares about the answer itself and not the rationale )
$$
\mathcal{H}\left(x^{(i)} \mid \left\{\hat{y}_j^{(i)}\right\}_{j=1}^m\right) = \frac{\sum_{\alpha=1}^u \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right) \log \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)}{\log m},
$$
We can measure the entropy of the generated responses using the formula above where
- $x_i$ is the original question that we prompted the model with
- $y_j^{i}$ represents the $i$-th sampled response from the $m$ that we generated
- $\hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)$ is the frequency of the unique answer in all the $m$ generated answers. (Eg. if we generate 8 responses and 4 of them return the value 10, then $\hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)$ is just going to be 0.5)
#### Repetitiveness
$$
R_r(r_j^{(i)}) = \frac{2}{Q(Q-1)} \sum_{a=1}^{Q} \sum_{b=a+1}^{Q} W_{ab}
$$
In the formula above, $Q$ refers to the number of phrases in the sentence and $W_{ab}$ refers to the cosine similarity of two phrases $a$ and $b$.
Repetitiveness aims to measure how often the language model repeats itself. To do so, the paper sums up the cosine similarity between each sentence inside the generated chain of thought rationale before normalizing it.
The intuition behind this is that high repetitiveness indicates redundancy, which can lead to poorer performance. Therefore responses with a high number of similar sentences will have a larger score for repetitiveness ( since cosine similarity will be larger for each sentence ).
### Step 2 - Self Consistency
We now take our `k` responses and append them to our prompt. We then sample our model multiple times using this new prompt and take the majority vote as the answer.
## Implementation
Now that we understand what COSP is, let's see how we can implement it in instructor. Note that here we'll measure repetitiveness using cosine similarity between sentence embeddings.
```python
import instructor
from pydantic import BaseModel
from openai import AsyncOpenAI, OpenAI
from collections import defaultdict, Counter
import asyncio
from textwrap import dedent
import math
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Response(BaseModel):
chain_of_thought: list[str]
answer: int
class ResponseScore(BaseModel):
query: str
response: Response
score: float
def format_response(self):
return dedent(
f"""
Q: {self.query}
A: {''.join(self.response.chain_of_thought)}. Therefore the answer is {self.response.answer}.
"""
)
def cosine_similarity(vec1: list[float], vec2: list[float]):
dot_product = sum(a * b for a, b in zip(vec1, vec2))
magnitude1 = math.sqrt(sum(a * a for a in vec1))
magnitude2 = math.sqrt(sum(b * b for b in vec2))
if magnitude1 * magnitude2 == 0:
return 0 # Handle the case of zero vectors
return dot_product / (magnitude1 * magnitude2)
def score_repetitiveness(prediction: Response):
if len(prediction.chain_of_thought) == 1:
return 0
embedding = OpenAI().embeddings.create(
input=prediction.chain_of_thought, model="text-embedding-3-small"
)
embedding = [item.embedding for item in embedding.data]
ttl = 0
num_comparisons = 0
for idx in range(len(embedding)):
for idx2 in range(idx + 1, len(embedding)):
ttl += cosine_similarity(embedding[idx], embedding[idx2])
num_comparisons += 1
return ttl / num_comparisons if num_comparisons > 0 else 0
async def generate_cot_response(query: str) -> tuple[Response, str]:
return (
await client.create(
model="gpt-4o",
messages=[{"role": "user", "content": query}],
response_model=Response,
temperature=0.4,
),
query,
)
async def generate_batch_cot_responses(
queries: list[str], m: int
) -> list[tuple[Response, str]]:
coros = [generate_cot_response(query) for query in queries for _ in range(m)]
return await asyncio.gather(*coros)
def score_entropy(predictions: list[Response]):
counter = Counter([prediction.answer for prediction in predictions])
prob = [counter[i] / len(predictions) for i in counter]
numer = -sum([p * math.log(p) for p in prob])
denom = math.log(len(predictions))
return numer / denom
def score_responses(
predictions: list[tuple[Response, str]], trade_off_param: float
) -> list[ResponseScore]:
query_to_responses: dict[str, list[Response]] = defaultdict(list)
for prediction, query in predictions:
query_to_responses[query].append(prediction)
query_to_entropy = {
query: score_entropy(predictions)
for query, predictions in query_to_responses.items()
}
return [
ResponseScore(
query=query,
response=prediction,
score=query_to_entropy[query]
+ trade_off_param * score_repetitiveness(prediction),
)
for prediction, query in predictions
]
def get_top_k_examples(queries: list[ResponseScore], k: int):
"""
This gets the top k responses that have the minimum possible score
"""
sorted_responses = sorted(queries, key=lambda x: x.score)
return sorted_responses[:k]
async def generate_answer_with_examples(query: str, examples: list[ResponseScore]):
formatted_examples = "\n".join([example.format_response() for example in examples])
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": dedent(
f"""
You are a world class AI system that excels at answering user queries
<query>
{query}
</query>
<examples>
{formatted_examples}
</examples>
"""
),
}
],
response_model=Response,
)
async def generate_final_answers(
query: str, examples: list[ResponseScore], number_samples: int
):
coros = [
generate_answer_with_examples(query, examples) for _ in range(number_samples)
]
return await asyncio.gather(*coros)
if __name__ == "__main__":
query = (
"The schools debate team had 5 boys and 40 girls on it. "
"If they were split into groups of 9 how many groups "
"could they make?"
)
example_questions = [
(
"Debby's class is going on a field trip to the zoo. "
"If each van can hold 4 people and there are 2 students "
"and 6 adults going, how many vans will they need?"
),
(
"Nancy had 80 files on her computer. She deleted 31 of "
"them and put the rest into folders with 7 files in each "
"one. How many folders did Nancy end up with?"
),
(
"At the arcade, Tom won 32 tickets playing 'whack a mole' "
"and 25 tickets playing 'skee ball'. If he spent 7 of his "
"tickets on a hat, how many tickets does Tom have left?"
),
]
m = 2 # Number of Reasoning Chains per example ( Step 1 )
k = 3 # Number of Examples to include in final prompt (Step 2)
n = 2 # Number of Reasoning Chains For Self-Consistency ( Step 2 )
# Step 1 : Generate the examples
responses = asyncio.run(generate_batch_cot_responses(example_questions, m))
scored_responses = score_responses(responses, 0.2)
chosen_examples = get_top_k_examples(scored_responses, k)
# Step 2 : Run Self-Consistency
final_responses = asyncio.run(generate_final_answers(query, chosen_examples, n))
c = Counter([response.answer for response in final_responses])
answer = c.most_common(1)[0][0]
print(answer)
#> 5
```
### References
<sup id="ref-1">1</sup>: [Better Zero-Shot Reasoning with Self-Adaptive Prompting](https://arxiv.org/pdf/2305.14106)

103
참고/instructor-main/docs/prompting/ensembling/dense.md Normal file
View File

@@ -0,0 +1,103 @@
---
description: "Demonstration Ensembling(DENSE) creates multiple few-shot prompts, each containing a distinct subset of examples from the training set. We then use that to generate a final response"
---
We can maximise the use of our examples by prompting our model multiple times, each time using a different subset of examples. We can then take these multiple outputs and aggregate over them to generate a final response. This is known as Demonstration Ensembling ( DENSE ) <sup><a href="https://arxiv.org/pdf/2308.08780">1</a></sup>.
> For simplicity in this example, we simply iterate over the examples and partition them equally to get equally sized clusters. However, depending on your use-case you might also want to consider sampling these using some form of embedding clusering.
We can implement this using `instructor` as seen below.
```python hl_lines="26-41"
import instructor
from pydantic import BaseModel
import asyncio
from collections import Counter
from typing import Literal
from textwrap import dedent
class DemonstrationResponse(BaseModel):
correct_answer: Literal["Positive", "Negative", "Neutral"]
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
async def generate_self_consistent_response(prompt: str, examples: list[str]):
concetenated_examples = "\n".join(examples)
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": dedent(
f"""
You are an intelligent AI System that excels
at classifying user queries into three
possible labels:
- Positive
- Negative
- Neutral
You are about to be given a user query and
asked to classify it into one of the three
categories. Make sure to refer closely to
the examples provided to you, examining each
individual example before coming up with the
final answer.
Here are the examples:
{concetenated_examples}
"""
),
},
{"role": "user", "content": prompt},
],
response_model=DemonstrationResponse,
temperature=0,
)
async def generate_self_consistent_responses(
prompt: str, num_responses: int, examples: list[str]
):
assert (
len(examples) % num_responses == 0
), "The number of examples must be evenly divisible by num_responses"
# Batch the examples into num_responses batches
batch_size = len(examples) // num_responses
coros = [
generate_self_consistent_response(prompt, examples[i : i + batch_size])
for i in range(0, len(examples), batch_size)
]
responses = await asyncio.gather(*coros)
return responses
if __name__ == "__main__":
user_query = "What is the weather like today?"
examples = [
"I love this product! [Positive]",
"This is the worst service ever. [Negative]",
"The movie was okay, not great but not terrible. [Neutral]",
"I'm so happy with my new phone! [Positive]",
"The food was terrible and the service was slow. [Negative]",
"It's an average day, nothing special. [Neutral]",
"Fantastic experience, will come again! [Positive]",
"I wouldn't recommend this to anyone. [Negative]",
"The book was neither good nor bad. [Neutral]",
"Absolutely thrilled with the results! [Positive]",
]
responses = asyncio.run(generate_self_consistent_responses(user_query, 5, examples))
answer_counts = Counter([response.correct_answer for response in responses])
most_common_answer, _ = answer_counts.most_common(1)[0]
print(most_common_answer)
#> Neutral
```
### References
<sup id="ref-1">1</sup>: [Exploring Demonstration Ensembling for In Context Learning](https://arxiv.org/pdf/2308.08780)

185
참고/instructor-main/docs/prompting/ensembling/diverse.md Normal file
View File

@@ -0,0 +1,185 @@
---
description: "Diverse creates multiple prompts for a given problem before performing self-consistency for each. It then generates multiple reaosning paths before choosing the best final response"
---
Diverse Verifier On Reasoning Step (DiVeRSe)<sup><a href="https://aclanthology.org/2023.acl-long.291/">1</a></sup> is a prompting technique which provides two main improvements
1. **Diverse Prompts** : They generate multiple variations of the same prompt by varying the examples used in each prompt
2. **Verification** : They use a finetuned `Deberta-V3-Large` to determine the quality of a generated response. Instead of using majority voting, they use their model to score each generated response from 0 to 1. They then aggregate these scores for each unique answer to determine the best generated solution.
In the paper itself, they also train a step-wise verifier that is able to score each individual reasoning step. This enables much more fine-grained predictions but is challenging to obtain training data for.
We can implement this in `instructor`. However, instead of using a `deberta-v3-large` model, we'll be using gpt-4o to score its own outputs and generate a quality score.
```python
import instructor
from pydantic import BaseModel
from typing import Literal
from textwrap import dedent
import asyncio
from collections import defaultdict
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Response(BaseModel):
chain_of_thought: str
answer: int
class Grading(BaseModel):
grade: Literal["Poor", "Average", "Good", "Excellent"]
def get_score(self):
mapping = {
"Poor": 0.25,
"Average": 0.5,
"Good": 0.75,
"Excellent": 1,
}
return mapping[self.grade]
async def generate_response(query: str, examples: list[str]):
formatted_examples = "\n".join(examples)
return await client.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": dedent(
f"""
You are a world class AI that excels at answering
user queries in a succint and accurate manner.
<query>
{query}
</query>
<examples>
{formatted_examples}
</examples>
"""
),
}
],
response_model=Response,
)
async def score_response(query: str, response: Response) -> tuple[Response, Grading]:
return (
response,
await client.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": dedent(
f"""
You are a world class AI that excels at grading
responses to a user query in a succint and clear
manner.
<query>
{query}
</query>
<response>
{response}
</response>
"""
),
}
],
response_model=Grading,
),
)
async def generate_response_batch(
query: str, examples: list[str], n_examples_per_batch: int
):
batches: list[list[str]] = []
for i in range(0, len(examples), n_examples_per_batch):
batches.append(examples[i : i + n_examples_per_batch])
coros = [generate_response(query, example_batch) for example_batch in batches]
return await asyncio.gather(*coros)
async def score_responses(
query: str, responses: list[Response]
) -> list[tuple[Response, Grading]]:
coros = [score_response(query, response) for response in responses]
return await asyncio.gather(*coros)
if __name__ == "__main__":
examples = [
"""
Q: James decides to run 3 sprints 3 times a week.
He runs 60 meters each sprint. How many total
meters does he run a week?
A: James decides to run 3 sprints 3 times a week.
He runs 60 meters each sprint. So he runs 60 meters
x 3 sprints x 3 times a week. That is 60 meters x 9.
The answer is 540.
""",
"""
Q: Brandon's iPhone is four times as old as Ben's
iPhone. Ben's iPhone is two times older than Suzy's
iPhone. If Suzy's iPhone is 1 year old, how old is
Brandon's iPhone?
A: Brandon's iPhone is 4 times as old as Ben's
iPhone. Ben's iPhone is 2 times older than Suzy's
iPhone. So Brandon's iPhone is 4 x 2 = 8 times older
than Suzy's iPhone. Suzy's iPhone is 1 year old. So
Brandon's iPhone is 8 x 1 = 8 years old. The answer
is 8.
""",
"""
Q: Jean has 30 lollipops. Jean eats 2 of the
lollipops. With the remaining lollipops, Jean wants
to package 2 lollipops in one bag. How many bags can
Jean fill?
A: Jean started with 30 lollipops. She ate 2 of
them. So she has 28 lollipops left. She wants to
package 2 lollipops in one bag. So she can package
28 / 2 = 14 bags. The answer is 14.
""",
"""
Q: Weng earns $12 an hour for babysitting.
Yesterday, she just did 50 minutes of babysitting.
How much did she earn?
A: Weng earns 12/60 = $<<12/60=0.2>>0.2 per minute.
Working 50 minutes, she earned 0.2 x 50 =
$<<0.2*50=10>>10. The answer is 10
""",
]
query = """Betty is saving money for a new wallet which
costs $100. Betty has only half of the money she needs.
Her parents decided to give her $15 for that purpose,
and her grandparents twice as much as her parents. How
much more money does Betty need to buy the wallet?"""
generated_responses = asyncio.run(generate_response_batch(query, examples, 1))
scored_responses = asyncio.run(score_responses(query, generated_responses))
scores: dict[int, float] = defaultdict(int)
for response, grade in scored_responses:
scores[response.answer] += grade.get_score()
print(scores)
#> defaultdict(<class 'int'>, {5: 3.5})
answer = max(scores, key=scores.get)
print(answer)
#> 5
```
### References
<sup id="ref-1">1</sup>: [Making Language Models Better Reasoners with Step-Aware Verifier](https://aclanthology.org/2023.acl-long.291/)

259
참고/instructor-main/docs/prompting/ensembling/max_mutual_information.md Normal file
View File

@@ -0,0 +1,259 @@
---
description: "Max Mutual Information creates multiple prompt templates and then selects the optimal template as the one which maximises mutual information between the prompt and the LLM's outputs"
---
## What's Max Mutual Information?
Max Mutual Information Method is a method of prompting that aims to find the best prompt to elicit the desired response from a LLM. We do so by maximising a metric called Mutual Information - which indicates the reduction in a model's uncertainty as a result of the prompt.
### Entropy
When a language model recieves a prompt as input, it outputs a series of token probabilities sequentially until it reaches the `<EOS>` token. In the paper, they take the final probability distribution as $P(Y|X)$ where $Y$ is the final prediction of the model and $X$ the prompt.
When we have a probability distribution, we can calculate a probability known as entropy. The lower this value is, the better. This is because a lower entropy value means that the model is more confident in its prediction.
We can calculate entropy with the following formula where $P(T_i)$ represents the probability of the $i$-th token in the final output distribution.
$$
H(P(Y|X)) = \sum_{i=0}^n P(T_i) log (P(T_i))
$$
### Mutual Information
![](../../img/mutual_information.png)
We can apply this to the calculation of Mutual Information as seen above.
We'll indicate the calculate of entropy of a probability distribution as $H(X)$ where $X$ here represents a final probability distribution. We also assume you have a train dataset of $n$ examples to use.
1. First, we choose a set of tokens that are likely to be part of the final answer. This could be words that appear inside the choices we have provided.
2. Once we've chosen these tokens, we extract out the log probs for each token from our final distribution. We then normalise it so that these new log probs now sum up to 1.
3. We do this for the $n$ example inside our train set, this gives us a new distribution $P(Y_i|X_i)$ for each $i$-th example.
4. We then take the average of these $n$ distributions to get $H_{marginal}$
5. We then calculate the average of the entropy of each distribution to get $H_{conditional}$
6. We then derive the Mutual Information by taking $H_{marginal} - H_{conditional}$, the higher this metric the better.
??? info "Unsure how to calculate $H_{marginal}$ and $H\_{conditional}$"
$$
H_{marginal} = H(\frac{1}{n} \sum_{i=0}^n P(Y_i | X_i) )
$$
$$
H_{conditional} = \frac{1}{n} \sum_{i=0}^n H(P(Y_i|X_i))
$$
We can then use this new mutual information metric to compare the effectiveness of different prompts at eliciting a desired response from our train dataset.
## Implementation
Since we don't have access to the raw log probabilites of specific tokens we want in the OpenAI API, we'll instead get the language model to generate a final score from 1 - 10 of its confidence in it's prediction.
We'll then convert this to a probability distribution with two outcomes and calculate a value for the entropy off of that.
Next we'll compare the Mutual Information value for different prompts before choosing what the best prompt is. For this example, we'll be using values from the Story Cloze set.
```python
import instructor
from pydantic import BaseModel
from typing import Callable, Literal
from textwrap import dedent
import math
import asyncio
class Response(BaseModel):
chain_of_thought: str
response: Literal["A", "B"]
confidence: Literal[
"Very High Confidence",
"High Confidence",
"Moderate Confidence",
"Low Confidence",
"Very Low Confidence",
]
def generate_score(self) -> float:
confidence_scores = {
"Very High Confidence": 1,
"High Confidence": 0.8,
"Moderate Confidence": 0.6,
"Low Confidence": 0.4,
"Very Low Confidence": 0.2,
}
return confidence_scores[self.confidence]
client = instructor.from_provider("openai/gpt-4o-mini", async_client=True)
def prompt_template_1(question: str, options: list[str]):
assert len(options) == 2
a, b = options
return dedent(
f"""
You are a world class AI System which excels at understanding complex user stories and generating responses. Output your prediction and also quantify your confidence in your prediction with the following scale.
- Very High Confidence: The model is highly confident in its prediction, displaying deep understanding, flawless execution, and no noticeable errors.
- High Confidence: The model is confident in its prediction, with strong relevance and minor errors that do not detract from overall quality.
- Moderate Confidence: The model has moderate confidence in its prediction, which is generally relevant with some inaccuracies, and meets minimum requirements.
- Low Confidence: The model has low confidence in its prediction, with limited relevance and several inaccuracies.
- Very Low Confidence: The model has very low confidence in its prediction, which is largely irrelevant, inaccurate, or incomplete, needing significant improvement
Context
{question}
Options
A. {a}
B. {b}
"""
)
def prompt_template_2(question: str, options: list[str]):
assert len(options) == 2
a, b = options
return dedent(
f"""
<prompt>
<Task>
You are about to be passed a story. You are to select the correct response from the options provided.
<confidence-levels>
<level>
<name>Very High Confidence</name>
<description>The model is highly confident in its prediction, displaying deep understanding, flawless execution, and no noticeable errors.</description>
</level>
<level>
<name>High Confidence</name>
<description>The model is confident in its prediction, with strong relevance and minor errors that do not detract from overall quality.</description>
</level>
<level>
<name>Moderate Confidence</name>
<description>The model has moderate confidence in its prediction, which is generally relevant with some inaccuracies, and meets minimum requirements.</description>
</level>
<level>
<name>Low Confidence</name>
<description>The model has low confidence in its prediction, with limited relevance and several inaccuracies.</description>
</level>
<level>
<name>Very Low Confidence</name>
<description>The model has very low confidence in its prediction, which is largely irrelevant, inaccurate, or incomplete, needing significant improvement</description>
</level>
</confidence-levels>
</Task>
<Question>
{question}
</Question>
<Options>
<option>A: {a}</option>
<option>B: {b}</option>
</Options>
</prompt>
"""
)
async def generate_response(
question: str, options: list[str], prompt_template: Callable[[str, list[str]], str]
):
return await client.create(
model="gpt-3.5-turbo",
messages=[
{
"role": "system",
"content": prompt_template(question, options),
}
],
response_model=Response,
)
async def generate_responses(
questions: list[str], prompt_template: Callable[[str, list[str]], str]
):
return await asyncio.gather(
*[
generate_response(
question=question["question"],
options=question["options"],
prompt_template=prompt_template,
)
for question in questions
]
)
def calculate_entropy(probs: list[float]) -> float:
return sum([p * math.log(p) if p != 0 else 0 for p in probs])
def calculate_mutual_information(predictions: list[Response]) -> float:
probs = [
[prediction.generate_score(), 1 - prediction.generate_score()]
for prediction in predictions
]
avg_probs = [0, 0]
for p1, p2 in probs:
avg_probs[0] += p1
avg_probs[1] += p2
h_marginal = calculate_entropy([i / len(probs) for i in avg_probs])
h_conditional = sum([calculate_entropy(prob) for prob in probs]) / len(probs)
return h_marginal - h_conditional
if __name__ == "__main__":
queries = [
{
"question": "Karen was assigned a roommate her first year of college. Her roommate asked her to go to a nearby city for a concert. Karen agreed happily. The show was absolutely exhilarating.",
"options": [
"Karen became good friends with her roommate.",
"Karen hated her roommate.",
],
},
{
"question": "Jim got his first credit card in college. He didn’t have a job so he bought everything on his card. After he graduated he amounted a $10,000 debt. Jim realized that he was foolish to spend so much money. ",
"options": [
"Jim decided to devise a plan for repayment.",
"Jim decided to open another credit card.",
],
},
{
"question": "Gina misplaced her phone at her grandparents. It wasn’t anywhere in the living room. She realized she was in the car before. She grabbed her dad’s keys and ran outside.",
"options": [
"She found her phone in the car.",
"She didn’t want her phone anymore.",
],
},
]
best_mi_score = float("-inf")
best_template = None
for prompt_template in [prompt_template_1, prompt_template_2]:
responses = asyncio.run(generate_responses(queries, prompt_template))
mi_score = calculate_mutual_information(responses)
print(f"{prompt_template.__name__}: {mi_score}")
#> prompt_template_1: -0.0781292189485728
#> prompt_template_2: -0.05907285153542691
if mi_score > best_mi_score:
best_mi_score = mi_score
best_template = prompt_template.__name__
print(best_template, best_mi_score)
#> prompt_template_2 -0.05907285153542691
```

199
참고/instructor-main/docs/prompting/ensembling/meta_cot.md Normal file
View File

@@ -0,0 +1,199 @@
---
description: "Meta Chain Of Thought involves decomposing an initial query into multiple sub questions. We then aggregate the response from each of these chains as context before prompting another LLM to generate a response"
---
Meta Chain Of Thought (Meta COT) <sup><a href="https://arxiv.org/pdf/2304.13007">1</a></sup>. involves the use of multiple reasoning chains to generate a response to a given query. This helps our model evaluate multiple potential reasoning paths and from there, determine a more accurate answer.
We can implement this using `instructor` as seen below.
```python hl_lines="41-42 57-61 96-99"
import instructor
from pydantic import BaseModel, Field
import asyncio
from typing import Optional
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class ReasoningAndResponse(BaseModel):
intermediate_reasoning: str = Field(
description="""
Intermediate reasoning steps"""
)
correct_answer: str
class MaybeResponse(BaseModel):
result: Optional[ReasoningAndResponse]
error: Optional[bool]
error_message: Optional[str] = Field(
description="""Informative explanation of why
the reasoning chain was unable to generate
a result"""
)
class QueryDecomposition(BaseModel):
queries: list[str] = Field(
description="""A list of queries that need to be
answered in order to derive the final answer"""
)
async def generate_queries(query: str):
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """You are a helpful assistant that
decomposes a query into multiple sub-queries.""",
},
{"role": "user", "content": query},
],
response_model=QueryDecomposition,
)
async def generate_reasoning_chain(query: str) -> MaybeResponse:
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """
Given a question and a context,
answer the question step-by-step.
Indicate the intermediate reasoning
steps.
""",
},
{"role": "user", "content": query},
],
response_model=MaybeResponse,
)
async def batch_reasoning_chains(
queries: list[str],
) -> list[MaybeResponse]:
coros = [generate_reasoning_chain(query) for query in queries]
results = await asyncio.gather(*coros)
return results
async def generate_response(query: str, context: list[MaybeResponse]):
formatted_context = "\n".join(
[
f"""
{item.result.intermediate_reasoning}
{item.result.correct_answer}
"""
for item in context
if not item.error and item.result
]
)
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """
Given a question and a context,
answer the question step-by-step.
If you are unsure, answer Unknown.
""",
},
{
"role": "user",
"content": f"""
<question>
{query}
</question>
<context>
{formatted_context}
</context>
""",
},
],
response_model=ReasoningAndResponse,
)
if __name__ == "__main__":
query = """Would Arnold Schwarzenegger have been
able to deadlift an adult Black rhinoceros at his
peak strength?"""
decomposed_queries = asyncio.run(generate_queries(query))
for generated_query in decomposed_queries.queries:
print(generated_query)
#> How much weight could Arnold Schwarzenegger
#> deadlift at his peak strength?
#> What is the average weight of an adult Black
#> rhinoceros?
chains = asyncio.run(batch_reasoning_chains(decomposed_queries.queries))
for chain in chains:
print(chain.model_dump_json(indent=2))
"""
{
"result": {
"intermediate_reasoning": "Determining Arnold
Schwarzenegger's peak deadlift involves
researching historical records, interviews,
and Arnold’s competitive powerlifting
results.",
"correct_answer": "Arnold Schwarzenegger's
peak deadlift was reportedly 710 lbs (322
kg)."
},
"error": false,
"error_message": null
}
"""
"""
{
"result": {
"intermediate_reasoning": "To determine the
average weight of an adult Black rhinoceros,
I need to consult reliable sources such as
wildlife encyclopedias, zoological databases,
or scientific articles. Commonly, the average
weight of adult Black rhinoceros ranges
between 800 to 1,400 kg.",
"correct_answer": "The average weight of an
adult Black rhinoceros ranges between 800 to
1,400 kg."
},
"error": false,
"error_message": null
}
"""
response = asyncio.run(generate_response(query, chains))
print(response.model_dump_json(indent=2))
"""
{
"intermediate_reasoning": "Arnold Schwarzenegger's
peak deadlift was 710 lbs (322 kg). The average
weight of an adult Black rhinoceros ranges between
800 to 1,400 kg (1764 to 3086 lbs). Even at the
lower end of the rhinoceros weight range (800 kg
or 1764 lbs), it exceeds Arnold Schwarzenegger's
peak deadlift capacity of 710 lbs (322 kg).
Therefore, Arnold Schwarzenegger would not have
been able to deadlift an adult Black rhinoceros at
his peak strength.",
"correct_answer": "No"
}
"""
```
### References
<sup id="ref-1">1</sup>: [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://arxiv.org/pdf/2304.13007)

161
참고/instructor-main/docs/prompting/ensembling/more.md Normal file
View File

@@ -0,0 +1,161 @@
---
description: "MoRE creates a set of diverse reasoning experts by using different specialized prompts for different reasoning types. THe best answer from all experts is then selected using an agreement score"
---
Language Models struggle to generalize across question types that require distinct reasoning abilities. By combining a variety of different specialized language models, we can improve the quality of our responses. This is done through a technique called Mixture Of Reasoning Experts (MoRE).
In the original paper, they utilise four different experts
1. Factual Expert : This is a model that is augmented by a RAG prompting pipeline. WHen it recieves a query, it retrieves the top 10 most relevant passages from Wikipedia and appends them to the prompt right before the question.
2. Multihop Expert : This is an expert that has manually written rationales after each demo to elicit multi-step reasoning processes for the questions
3. Math Expert : This is an expert that has manually written explanations for the GSM8k Dataset to bias the model towards different reasoning steps
4. Commonsense expert: This is an expert that is provided with 10 different facts that are generated by a Codex model which are appended to the prompt right before the question
![](../../img/more.png)
Once each expert has genearted a response, they then use a random forest classifier to score it from 0 to 1. This is then used for selecting the final answer and determining if we've generated a sufficiently good answer ( Since we have the option to abstain at each point )
We can implement a simplified version of MoRE with `instructor` with a few modifications.
```python
from pydantic import BaseModel, Field
import instructor
from textwrap import dedent
client = instructor.from_provider("openai/gpt-5-nano")
class MultihopExpert(BaseModel):
chain_of_thought: str
answer: str
class FactualExpert(BaseModel):
answer: str
class ModelScore(BaseModel):
score: float = Field(ge=0, lt=1)
def query_factual_expert(query: str, evidence: list[str]):
formatted_evidence = "\n-".join(evidence)
return client.create(
model="gpt-4o",
response_model=FactualExpert,
messages=[
{
"role": "system",
"content": dedent(
f"""
<query>
{query}
</query>
<evidences>
{formatted_evidence}
</evidences>
"""
),
}
],
)
def query_multihop_expert(query: str):
return client.create(
model="gpt-4o",
response_model=MultihopExpert,
messages=[
{
"role": "system",
"content": dedent(
f"""
<query>
{query}
</query>
"""
),
}
],
)
def score_answer(query: str, answer: str):
return client.create(
model="gpt-4o",
response_model=ModelScore,
messages=[
{
"role": "system",
"content": """You are a helpful assistant that scores
answers based on well they are able to answer a
specific user query""",
},
{
"role": "user",
"content": f"""
<user query>
{query}
</user query>
<response>
{answer}
</response>
""",
},
],
)
if __name__ == "__main__":
query = """Who's the original singer of Help Me Make It
Through The Night?"""
evidences = [
"""Help Me Make It Through The Night is a country
music ballad written and composed by Kris Kristofferson
and released on his 1970 album 'Kristofferson'"""
]
threshold = 0.8
factual_expert_output = query_factual_expert(query, evidences)
print(factual_expert_output.model_dump_json(indent=2))
"""
{
"answer": "The original singer of 'Help Me Make It Through the
Night' is Kris Kristofferson, who released it on his 1970 album
'Kristofferson'."
}
"""
multihop_expert_output = query_multihop_expert(query)
print(multihop_expert_output.model_dump_json(indent=2))
"""
{
"chain_of_thought": "To identify the original singer of 'Help Me
Make It Through The Night,' I need to look for the person who
first recorded and released the song.",
"answer": "The original singer of 'Help Me Make It Through
The Night' is Kris Kristofferson."
}
"""
factual_expert_score = score_answer(query, factual_expert_output.answer)
multihop_expert_score = score_answer(query, multihop_expert_output.answer)
if max(factual_expert_score.score, multihop_expert_score.score) < threshold:
answer = "Abstaining from responding"
elif factual_expert_score.score > multihop_expert_score.score:
answer = factual_expert_output.answer
else:
answer = multihop_expert_output.answer
print(answer)
"""
The original singer of 'Help Me Make It Through the Night' is Kris
Kristofferson, who released it on his 1970 album 'Kristofferson'.
"""
```

124
참고/instructor-main/docs/prompting/ensembling/prompt_paraphrasing.md Normal file
View File

@@ -0,0 +1,124 @@
---
description: "Use Large Language Models to perform back translation in order to improve prompt performance"
---
Large Language Models are sensitive to the way that they are prompted. When prompted incorrectly, they might perform much worse despite having the information or capability to respond to the prompt. We can help find semantically similar prompts by performing back translation - where we translate our prompts to another language and back to encourage more diversity in the rephrased prompts.
Prompt paraphrasing <sup><a href="https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00324/96460/How-Can-We-Know-What-Language-Models-Know">1</a></sup>. provides some ways for us to improve on the phrasing of our prompts to do so.
We can implement this using `instructor` as seen below.
```python hl_lines="20-25"
import instructor
from pydantic import BaseModel
import random
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class TranslatedPrompt(BaseModel):
translation: str
async def translate_prompt(prompt: str, from_language: str, to_language: str):
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": f"""
You are an expert translation assistant.
You are going to be given a prompt and
asked to translate it from {from_language}
to {to_language}. Paraphrase and use
synonyms where possible, especially for
the examples.
""",
},
{"role": "user", "content": f"Prompt: {prompt}"},
],
response_model=TranslatedPrompt,
)
async def generate_permutation(prompt: str, language: str) -> str:
tranlated_prompt = await translate_prompt(prompt, "english", language)
backtranslated_prompt = await translate_prompt(
tranlated_prompt.translation, language, "english"
)
return backtranslated_prompt.translation
async def generate_prompts(
prompt: str, languages: list[str], permutations: int
) -> list[str]:
coros = [
generate_permutation(prompt, random.choice(languages))
for _ in range(permutations)
]
return await asyncio.gather(*coros)
if __name__ == "__main__":
import asyncio
prompt = """
You are an expert system that excels at Sentiment
Analysis of User Reviews.
Here are a few examples to refer to:
1. That was a fantastic experience I had! I'm
definitely recommending this to all my friends
// Positive
2. I think it was a passable evening. I don't think
there was anything remarkable or off-putting for me.
// Negative
3. I'm horrified at the state of affairs in this new
restaurant // Negative
Sentence: This was a fantastic experience!
"""
languages = ["french", "spanish", "chinese"]
permutations = 2
generated_prompts = asyncio.run(generate_prompts(prompt, languages, permutations))
for prompt in generated_prompts:
print(prompt)
"""
You are an expert system specializing in user review sentiment analysis. Here are a few examples to guide you: 1. It was an exceptional experience! I will definitely recommend it to all my friends // Positive 2. I think it was a mediocre evening. There wasn't anything outstanding or particularly bad for me // Negative 3. I am horrified by the condition of things in this new restaurant // Negative Sentence: It was an amazing experience!
"""
"""
You are an expert system that excels in User Review Sentiment Analysis.
Here are some reference examples:
1. I had an amazing experience! I will definitely recommend it to all my friends.
// Positive
2. I think it was an average evening. I don’t believe there was anything remarkable or unpleasant about it for me.
// Negative
3. I am horrified by the situation at this new restaurant.
// Negative
Sentence: This was a fantastic experience!
"""
"""
You are an expert system skilled in conducting user
review sentiment analysis.
Here are some examples for reference:
1. That was an awesome experience! I'll definitely
recommend it to all my friends // Positive
2. I think it was an okay evening. I don't find
anything particularly outstanding or unpleasant.
// Neutral
3. I am very shocked by the condition of this new
restaurant // Negative
Sentence: This was a wonderful experience!
"""
```
### References
<sup id="ref-1">1</sup>: [How Can We Know What Language Models Know? ](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00324/96460/How-Can-We-Know-What-Language-Models-Know)

73
참고/instructor-main/docs/prompting/ensembling/self_consistency.md Normal file
View File

@@ -0,0 +1,73 @@
---
description: "Self Consistency aims to help maximise llm performance by sampling multiple potential calls. We then take a majority vote on the final response to derive the answer"
---
By generating multiple candidate responses in parallel and choosing the most common answer among them, we can get a more accurate answer. This is known as Self-Consistency <sup><a href="https://arxiv.org/pdf/2203.11171">1</a></sup>
We can implement this using `instructor` as seen below.
```python hl_lines="25-29"
import instructor
from pydantic import BaseModel, Field
import asyncio
from collections import Counter
from textwrap import dedent
class SelfConsistencyResponse(BaseModel):
chain_of_thought: str = Field(
description="reasoning behind the final correct answer"
)
correct_answer: int
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
async def generate_self_consistent_response(prompt: str):
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """You are an intelligent question
answering AI system that excels at answering
user queries. Make sure to generate a
comprehensive explanation of your thought
process before providing the final answer""",
},
{"role": "user", "content": prompt},
],
response_model=SelfConsistencyResponse,
temperature=0.5,
)
async def generate_self_consistent_responses(prompt: str, num_responses: int):
coros = [generate_self_consistent_response(prompt) for _ in range(num_responses)]
responses = await asyncio.gather(*coros)
return responses
if __name__ == "__main__":
prompt = dedent(
"""
Janet's ducks lay 16 eggs per day.
She eats three for breakfast every
morning and bakes muffins for her
friends every day with four. She sells
the remainder for $2 per egg. How
much does she make every day?
"""
)
responses = asyncio.run(generate_self_consistent_responses(prompt, 5))
answer_counts = Counter([response.correct_answer for response in responses])
most_common_answer, _ = answer_counts.most_common(1)[0]
print(most_common_answer)
#> 18
```
### References
<sup id="ref-1">1</sup>: [Self-Consistency Improves Chain Of Thought
Reasoning In Language Models](https://arxiv.org/pdf/2210.03350)

180
참고/instructor-main/docs/prompting/ensembling/universal_self_consistency.md Normal file
View File

@@ -0,0 +1,180 @@
---
description: "Universal Self Consistency aims to extend Self-Consistency by using Large Language Models themselves to select the most consistent answer among multiple candidates"
---
Universal Self Consistency<sup><a href="https://arxiv.org/pdf/2311.17311">1</a></sup> aims to extend self-consistency by using a second LLM model to judge the quality of individual responses. Therefore instead of choosing the final answer based on the most frequently occuring value among each reasoning chain, we instead prompt the model to choose the most consistent answer for us relative to the prompt.
![](../../img/universal_self_consistency.png)
This enables us to support a greater variety of different response formats and answer, leading to greater diversity of outputs and hence higher accuracy.
We can implement this in `instructor` as seen below.
```python hl_lines="71-73"
from pydantic import BaseModel, Field, ValidationInfo, field_validator
import instructor
from textwrap import dedent
import asyncio
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Response(BaseModel):
chain_of_thought: str
answer: str
class SelectedResponse(BaseModel):
most_consistent_response_id: int = Field(
description="""The ID of the most consistent response that
was provided"""
)
@field_validator("most_consistent_response_id")
@classmethod
def validate_id(cls, v: int, info: ValidationInfo):
context = info.context
number_responses = context.get("number_responses", float("inf"))
if v > number_responses:
raise ValueError(
f"""Most consistent response ID {v} is greater than the
number of responses {number_responses}. Please return a
valid id between 0 and {number_responses-1}"""
)
return v
async def generate_response(query: str) -> Response:
return await client.create(
model="gpt-4o",
response_model=Response,
messages=[{"role": "user", "content": query}],
)
async def generate_batch_responses(query: str, no_responses: int):
coros = [generate_response(query) for _ in range(no_responses)]
return await asyncio.gather(*coros)
async def select_consistent_response(responses: list[Response], query: str):
formatted_responses = "\n".join(
[
f"Response {idx}: {response.chain_of_thought}. {response.answer}"
for idx, response in enumerate(responses)
]
)
return await client.create(
model="gpt-4o",
response_model=SelectedResponse,
messages=[
{
"role": "user",
"content": dedent(
f"""
<user query>
{query}
</user query>
{formatted_responses}
Evaluate these responses.
Select the most consistent response based on majority
consensus
"""
),
}
],
context={"number_responses": len(responses)},
)
if __name__ == "__main__":
query = """The three-digit number 'ab5' is divisible by 3. How many different
three-digit numbers can 'ab5' represent?"""
responses = asyncio.run(generate_batch_responses(query, 3))
for response in responses:
print(response.model_dump_json(indent=2))
"""
{
"chain_of_thought": "A number is divisible by 3 if
the sum of its digits is divisible by 3. Given the
number 'ab5', we need to check how many different
values of 'a' and 'b', where both are digits (0-9)
can make the sum divisible by 3.\n\nThe sum of the
digits is a + b + 5.\n\nWe need to find pairs (a, b)
such that (a + b + 5) % 3 == 0.",
"answer": "30"
}
"""
"""
{
"chain_of_thought": "A number is divisible by 3 if
the sum of its digits is divisible by 3. Let's
denote the digits a and b. The number 'ab5' has
digits a, b, and 5. Therefore, the sum of the
digits is a + b + 5. Since the number is divisible
by 3, a + b + 5 must be divisible by 3.\n\nNow,
since a and b are single digits (0-9), we need to
find pairs (a, b) such that a + b + 5 is divisible
by 3. We will evaluate all possible combinations of
values for a and b to count how many valid pairs
(a, b) exist.\n\nLet's start by considering b's
values:\n1. If b = 0, then a + 5 must be divisible
by 3.\n2. If b = 1, then a + 6 must be divisible by
3.\n3. If b = 2, then a + 7 must be divisible by
3.\n4. If b = 3, then a + 8 must be divisible by
3.\n5. If b = 4, then a + 9 must be divisible by
3.\n6. If b = 5, then a + 10 must be divisible by
3.\n7. If b = 6, then a + 11 must be divisible by
3.\n8. If b = 7, then a + 12 must be divisible by
3.\n9. If b = 8, then a + 13 must be divisible by
3.\n10. If b = 9, then a + 14 must be divisible by
3.\n\nWe will find all corresponding a values for
each b and count the valid combinations.\n",
"answer": "There are 30 different three-digit
numbers that 'ab5' can represent."
}
"""
"""
{
"chain_of_thought": "A number is divisible by 3 if
the sum of its digits is divisible by 3. The given
number is in the form 'ab5', where 'a' and 'b' are
digits from 0 to 9. To find the total number of
different three-digit numbers that 'ab5' can
represent, we need to determine all possible digit
combinations for 'a' and 'b' such that 'a + b + 5'
is divisible by 3.",
"answer": "30"
}
"""
selected_response = asyncio.run(select_consistent_response(responses, query))
print(selected_response.model_dump_json(indent=2))
"""
{
"most_consistent_response_id": 0
}
"""
print(
responses[selected_response.most_consistent_response_id].model_dump_json(
indent=2
)
)
"""
{
"chain_of_thought": "A number is divisible by 3 if the sum of its digits is divisible by 3. Given the number 'ab5', we need to
check how many different values of 'a' and 'b', where both are digits (0-9) can make the sum divisible by 3.\n\nThe sum of the
digits is a + b + 5.\n\nWe need to find pairs (a, b) such that (a + b + 5) % 3 == 0.",
"answer": "30"
}
"""
```
### References
<sup id="ref-1">1</sup>: [Universal Self-Consistency For Large Language Model Generation](https://arxiv.org/pdf/2311.17311)

236
참고/instructor-main/docs/prompting/ensembling/usp.md Normal file
View File

@@ -0,0 +1,236 @@
---
description: "Universal Self Prompting is a technique that aims to use unlabeled data to generate exemplars and a more complicated scoring function to select them."
---
Universal Self Prompting is a two stage process similar to [Consistency Based Self Adaptive Prompting (COSP)](../few_shot/cosp.md). Here is a breakdown of the two stages.
1. **Generate Examples** : LLMs are prompted to generate a collection of candidate responses using a test dataset
2. **Answer Query** : We then select a few of these model-generated responses as examples to prompt the LLM to obtain a final prediction.
Note here that the final answer is obtained using a single forward pass with greedy decoding.
## USP Process
![](../../img/universal_self_adaptive_prompting.png)
Let's see how this works in greater detail.
### Generate Few Shot Examples
We first prompt our model to generate responses for a given set of prompts. Instead of measuring the entropy and repetitiveness as in COSP, we use one of three possible methods to measure the quality of the generated responses. These methods are decided based on the three categories supported.
This category has to be specified by a user ahead of time.
Note that for Short Form and Long Form generation, we generate $m$ different samples. This is not the case for classification tasks.
- **Classification** : Classification Tasks are evaluated using the normalized probability of each label using the raw logits from the LLM.
$$
F_{CLS}(p^{(j)}|d^{(j)}) := -\sum_{c \in C} P(c|d^{(j)}) \log P(c|d^{(j)})
$$
In short, we take the raw logit for each token corresponding to the label, use a softmax to normalize each of them and then sum across the individual probabilities and their log probs. We also try to sample enough queries such that we have a balanced number of predictions across each class ( so that our model doesn't have a bias towards specific classes )
- **Short Form Generation**: This is done by using a similar formula to COSP but without the normalizing term
$$
\mathcal{H}\left(x^{(i)} \mid \left\{\hat{y}_j^{(i)}\right\}_{j=1}^m\right) = \frac{\sum_{\alpha=1}^u \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right) \log \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)}{\log m},
$$
- **Long Form Generation**: This is done by using the average pairwise ROUGE score between all pairs of the $m$ responses.
What is key here is that depending on the task specified by the user, we have a task-specific form of evaluation. This eventually allows us to better evaluate our individual generated examples. Samples of tasks for each category include
1. **Classification**: Natural Language Inference, Topic Classification and Sentiment Analysis
2. **Short Form Generation** : Question Answering and Sentence Completion
3. **Long Form Generation** : Text Summarization and Machine Translation
This helps to ultimately improve the performance of these large language models across different types of tasks.
### Generate Single Response
Once we've selected our examples, the second step is relatively simple. We just need to append a few of our chosen examples that score best on our chosen metric to append to our solution.
## Implementation
We've implemented a classification example below that tries to sample across different classes in a balanced manner before generating a response using a single inference call.
We bias this sampling towards samples that the model is more confident towards by using a confidence label.
```python
from pydantic import BaseModel
from typing import Literal
import instructor
import asyncio
from collections import defaultdict
class Classification(BaseModel):
chain_of_thought: str
label: Literal["Happy", "Angry", "Sadness"]
confidence: Literal[
"Uncertain", "Somewhat Confident", "Confident", "Highly Confident"
]
def confidence_score(self) -> int:
confidence_order = {
"Highly Confident": 4,
"Confident": 3,
"Somewhat Confident": 2,
"Uncertain": 1,
}
return confidence_order[self.confidence]
client = instructor.from_provider("openai/gpt-4o-mini", async_client=True)
async def generate_prediction(query: str):
return (
await client.create(
model="gpt-3.5-turbo",
messages=[
{
"role": "user",
"content": f"""Classify the following query {query} into
one of the following categories: Happy, Angry, Sadness""",
}
],
response_model=Classification,
),
query,
)
async def generate_predictions(queries: list[str]) -> list[tuple[Classification, str]]:
return await asyncio.gather(*[generate_prediction(query) for query in queries])
def get_balanced_sample(predictions: list[tuple[Classification, str]], k: int):
label_to_queries: dict[str, list[tuple[Classification, str]]] = defaultdict(list)
for prediction in predictions:
label_to_queries[prediction[0].label].append(prediction)
num_classes = len(label_to_queries)
num_samples_per_class = k // num_classes
res: list[str] = []
for label, label_queries in label_to_queries.items():
label_queries = sorted(
label_queries, key=lambda x: x[0].confidence_score(), reverse=True
)
label_queries = [
label_queries[1] for label_queries in label_queries[:num_samples_per_class]
]
res.extend([f"{query} ({label})" for query in label_queries])
return res
async def generate_response_with_examples(query: str, examples: list[str]):
formatted_examples = "\n".join(examples)
return await client.create(
model="gpt-4o",
response_model=Classification,
messages=[
{
"role": "system",
"content": f"""
You are a helpful assistant that classifies queries into one of the following categories: Happy, Angry, Sadness.
Here are some samples of queries and their categories:
<examples>
{formatted_examples}
</examples>
Here is a user query to classify
<query>
{query}
</query>
""",
},
],
)
if __name__ == "__main__":
examples = [
"""
i do feel that running is a divine experience and
that i can expect to have some type of spiritual
encounter
""",
"""
i get giddy over feeling elegant in a perfectly
fitted pencil skirt
""",
"""
i plan to share my everyday life stories traveling
adventures inspirations and handmade creations with
you and hope you will also feel inspired
""",
"""
i need to feel the dough to make sure its just
perfect
""",
"""
i found myself feeling a little discouraged that
morning
""",
"i didnt really feel that embarrassed",
"i feel like a miserable piece of garbage",
"""
i feel like throwing away the shitty piece of shit
paper
""",
"""
i feel irritated and rejected without anyone doing
anything or saying anything
""",
"i feel angered and firey",
"""
im feeling bitter today my mood has been strange the
entire day so i guess its that
""",
"i just feel really violent right now",
"i know there are days in which you feel distracted",
]
labels = asyncio.run(generate_predictions(examples))
balanced_sample = get_balanced_sample(labels, 3)
for sample in balanced_sample:
print(sample)
"""
i do feel that running is a divine experience and that i can
expect to have some type of spiritual encounter (Happy)
"""
#> i feel like a miserable piece of garbage (Sadness)
#> i feel like throwing away the shitty piece of shit paper (Angry)
response = asyncio.run(
generate_response_with_examples(
"""
i feel furious that right to life advocates can
and do tell me how to live and die through
lobbying and supporting those politicians
sympathic to their views
""",
balanced_sample,
)
)
print(response.model_dump_json(indent=2))
"""
{
"chain_of_thought": "The user expresses feelings of
anger and frustration specifically directed at right
to life advocates. The language used, such as
'furious,' indicates a high level of emotion
associated with anger.",
"label": "Angry",
"confidence": "Highly Confident"
}
"""
```

193
참고/instructor-main/docs/prompting/few_shot/cosp.md Normal file
View File

@@ -0,0 +1,193 @@
---
description: "Consistency Based Self Adaptive Prompting (COSP) is a technique that uses entropy and repetitiveness to select high-quality examples for few-shot learning."
---
# Consistency Based Self Adaptive Prompting (COSP)
COSP is a technique that aims to improve few-shot learning by selecting high-quality examples based on the consistency and confidence of model responses. This approach helps create more effective prompts by identifying examples that the model can process reliably.
## Overview
The COSP process involves two main stages:
1. **Example Generation**: Generate multiple responses for potential examples
- Run each example through the model multiple times
- Collect responses and confidence scores
2. **Example Selection**: Select the best examples based on entropy and repetitiveness
- Calculate entropy of responses to measure consistency
- Evaluate repetitiveness to ensure reliability
## How COSP Works
### Stage 1: Example Generation
For each potential example in your dataset:
1. Generate multiple responses (typically 3-5)
2. Calculate the entropy of these responses
3. Measure the repetitiveness across responses
```python
from typing import List
from pydantic import BaseModel, Field
import instructor
from openai import OpenAI
class Response(BaseModel):
content: str = Field(description="The model's response to the prompt")
confidence: float = Field(description="Confidence score between 0 and 1")
client = instructor.from_provider("openai/gpt-5-nano")
def generate_responses(prompt: str, n: int = 3) -> List[Response]:
responses = []
for _ in range(n):
response = client.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
response_model=Response
)
responses.append(response)
return responses
```
### Stage 2: Example Selection
Calculate metrics for each example:
1. **Entropy**: Measure response variability
2. **Repetitiveness**: Check response consistency
```python
import numpy as np
from scipy.stats import entropy
def calculate_metrics(responses: List[Response]) -> tuple[float, float]:
# Calculate entropy
confidences = [r.confidence for r in responses]
entropy_score = entropy(confidences)
# Calculate repetitiveness
unique_responses = len(set(r.content for r in responses))
repetitiveness = 1 - (unique_responses / len(responses))
return entropy_score, repetitiveness
```
## Implementation Example
Here's a complete example of COSP implementation:
```python
from typing import List, Tuple
from pydantic import BaseModel, Field
import instructor
from openai import OpenAI
import numpy as np
from scipy.stats import entropy
class Example(BaseModel):
text: str
score: float = Field(description="Combined quality score")
entropy: float = Field(description="Entropy of responses")
repetitiveness: float = Field(description="Repetitiveness of responses")
class COSPSelector:
def __init__(self, client: OpenAI, n_samples: int = 3):
self.client = instructor.from_provider("openai/gpt-4o")
self.n_samples = n_samples
def generate_responses(self, prompt: str) -> List[Response]:
return [
self.client.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
response_model=Response
)
for _ in range(self.n_samples)
]
def calculate_metrics(self, responses: List[Response]) -> Tuple[float, float]:
confidences = [r.confidence for r in responses]
entropy_score = entropy(confidences)
unique_responses = len(set(r.content for r in responses))
repetitiveness = 1 - (unique_responses / len(responses))
return entropy_score, repetitiveness
def select_examples(self, candidates: List[str], k: int) -> List[Example]:
examples = []
for text in candidates:
responses = self.generate_responses(text)
entropy_score, repetitiveness = self.calculate_metrics(responses)
# Combined score (lower is better)
score = entropy_score - repetitiveness
examples.append(Example(
text=text,
score=score,
entropy=entropy_score,
repetitiveness=repetitiveness
))
# Sort by score (lower is better) and select top k
return sorted(examples, key=lambda x: x.score)[:k]
```
## Usage Example
```python
# Initialize COSP selector
client = OpenAI()
selector = COSPSelector(client)
# Candidate examples
candidates = [
"The quick brown fox jumps over the lazy dog",
"Machine learning is a subset of artificial intelligence",
"Python is a high-level programming language",
# ... more examples
]
# Select best examples
best_examples = selector.select_examples(candidates, k=3)
# Use selected examples in your prompt
selected_texts = [ex.text for ex in best_examples]
prompt = f"""Use these examples to guide your response:
Examples:
{chr(10).join(f'- {text}' for text in selected_texts)}
Now, please respond to: [your query here]
"""
```
## Benefits of COSP
1. **Improved Consistency**: By selecting examples with low entropy and high repetitiveness
2. **Better Performance**: More reliable few-shot learning
3. **Automated Selection**: No manual example curation needed
4. **Quality Metrics**: Quantifiable measure of example quality
## Limitations
1. **Computational Cost**: Requires multiple API calls per example
2. **Time Overhead**: Selection process can be slow for large candidate sets
3. **Model Dependency**: Performance may vary across different models
## Related Techniques
- [Universal Self Prompting (USP)](../ensembling/usp.md)
- Chain of Thought Prompting
- Self-Consistency
## References
1. Original COSP Paper: [arXiv:2305.14121](https://arxiv.org/abs/2305.14121)
2. Related Work: [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171)

134
참고/instructor-main/docs/prompting/few_shot/example_generation/sg_icl.md Normal file
View File

@@ -0,0 +1,134 @@
---
title: "Generate In-Context Examples"
description: ""
---
How can we generate examples for our prompt?
Self-Generated In-Context Learning (SG-ICL) is a technique which uses an LLM to generate examples to be used during the task. This allows for in-context learning, where examples of the task are provided in the prompt.
We can implement SG-ICL using `instructor` as seen below.
```python
import instructor
from pydantic import BaseModel
from typing import Literal
n = 4 # num examples to generate per class
class GeneratedReview(BaseModel):
review: str
sentiment: Literal["positive", "negative"]
class SentimentPrediction(BaseModel):
sentiment: Literal["positive", "negative"]
client = instructor.from_provider("openai/gpt-5-nano")
def generate_sample(input_review, sentiment):
return client.create(
model="gpt-4o",
response_model=GeneratedReview,
messages=[
{
"role": "user",
"content": f"""
Generate a '{sentiment}' review similar to: {input_review}
Generated review:
""",
}
],
)
def predict_sentiment(input_review, in_context_samples):
return client.create(
model="gpt-4o",
response_model=SentimentPrediction,
messages=[
{
"role": "user",
"content": "".join(
[
f"Review: {sample.review}\nSentiment: {sample.sentiment}\n\n"
for sample in in_context_samples
]
)
+ f"Review: {input_review}\nSentiment:",
}
],
).sentiment
if __name__ == "__main__":
input_review = (
"This movie was a rollercoaster of emotions, keeping me engaged throughout."
)
# Generate in-context samples
samples = [
generate_sample(input_review, sentiment)
for sentiment in ('positive', 'negative')
for _ in range(n)
]
for sample in samples:
print(sample)
"""
review='This film was an enthralling experience from start to finish, leaving me captivated every moment.' sentiment='positive'
"""
"""
review='This film was an emotional journey that captivated me from start to finish.' sentiment='positive'
"""
"""
review='The film took me on an unforgettable journey, capturing my attention at every moment.' sentiment='positive'
"""
"""
review='This book was a riveting journey, capturing my attention from start to finish.' sentiment='positive'
"""
"""
review='The movie was a total letdown, failing to hold my interest from start to finish.' sentiment='negative'
"""
"""
review='This movie was a disjointed mess of emotions, leaving me confused throughout.' sentiment='negative'
"""
"""
review='The movie was an emotional rollercoaster, but it left me feeling more confused than engaged.' sentiment='negative'
"""
"""
review='This movie was a monotonous ride, failing to engage me at any point.' sentiment='negative'
"""
"""
review='This film was an emotional journey, captivating me from start to finish.' sentiment='positive'
"""
"""
review='This film captivated me from start to finish with its thrilling plot and emotional depth.' sentiment='positive'
"""
"""
review='This movie was a breathtaking journey, capturing my attention from start to finish.' sentiment='positive'
"""
"""
review='This movie was a chaotic mess of emotions, losing me at every turn.' sentiment='negative'
"""
"""
review='This movie was a confusing mess, leaving me disengaged throughout.' sentiment='negative'
"""
"""
review='This movie was a chore to sit through, leaving me bored most of the time.' sentiment='negative'
"""
"""
review='This movie was a mishmash of confusing scenes, leaving me frustrated throughout.' sentiment='negative'
"""
# Predict sentiment
print(predict_sentiment(input_review, samples))
#> positive
```
### References
<sup id="ref-1">1</sup>: [Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator](https://arxiv.org/abs/2206.08082)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

41
참고/instructor-main/docs/prompting/few_shot/example_ordering.md Normal file
View File

@@ -0,0 +1,41 @@
---
title: "Example Ordering"
description: "LLM outputs are heavily impacted by ordering of few shot examples"
---
# Example Ordering
The order of few-shot examples in the prompt can affect LLM outputs <sup><a href="https://arxiv.org/abs/2104.08786">1</a><a href="https://arxiv.org/abs/2106.01751">2</a><a href="https://arxiv.org/abs/2101.06804">3</a><a href="https://aclanthology.org/2022.naacl-main.191/">4</a></sup><sup><a href="https://arxiv.org/abs/2406.06608">\*</a></sup>. Consider permutating the order of these examples in your prompt to achieve better results.
## Choosing Your Examples
Depending on your use-case, here are a few different methods that you can consider using to improve the quality of your examples.
### Combinatorics
One of the easiest methods is for us to manually iterate over each of the examples that we have and try all possible combinations we could create. This will in turn allow us to find the best combination that we can find.
### KATE
KATE (k-Nearest Example Tuning) is a method designed to enhance GPT-3's performance by selecting the most relevant in-context examples. The method involves:
For each example in the test set, K nearest neighbors (examples) are retrieved based on semantic similarity.
Among these K examples, those that appear most frequently across different queries are selected as the best in-context examples.
### Using a Unsupervised Retriever
![Retriever Image](../../img/retriever.png)
We can use a large LLM to compute a single score for each example with respect to a given prompt. This allows us to create a training set that scores an example's relevance when compared against a prompt. Using this training set, we can train a model that mimics this functionality. This allows us to determine the top `k` most relevant and most irrelevant examples when a user makes a query so that we can include this in our final prompt.
### References
<sup id="ref-1">1</sup>: [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://arxiv.org/abs/2104.08786)
<sup id="ref-2">2</sup>: [Reordering Examples Helps during Priming-based Few-Shot Learning](https://arxiv.org/abs/2106.01751)
<sup id="ref-2">3</sup>: [What Makes Good In-Context Examples for GPT-3?](https://arxiv.org/abs/2101.06804)
<sup id="ref-3">4</sup>: [Learning To Retrieve Prompts for In-Context Learning](https://aclanthology.org/2022.naacl-main.191/)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

147
참고/instructor-main/docs/prompting/few_shot/exemplar_selection/knn.md Normal file
View File

@@ -0,0 +1,147 @@
---
title: "Select Effective Examples"
description: "KNN can be leveraged to choose the most effective examples to use for a given query."
---
We can select effective in-context examples by choosing those that are semantically closer to the query using `KNN`.
In the below implementation using `instructor`, we follow these steps:
1. Embed the query examples
2. Embed the query that we want to answer
3. Find the _k_ query examples closest to the query
4. Use the chosen examples and their as the context for the LLM
```python
import instructor
from pydantic import BaseModel
from openai import OpenAI
import math
from textwrap import dedent
class Example(BaseModel):
question: str
answer: str
class Response(BaseModel):
answer: str
oai = OpenAI()
client = instructor.from_provider("openai/gpt-4o")
def distance(a: list[float], b: list[float]):
return 1 - sum(ai * bi for ai, bi in zip(a, b)) / (
math.sqrt(sum(ai**2 for ai in a)) * math.sqrt(sum(bi**2 for bi in b))
)
def embed_queries(queries: list[str]) -> list[tuple[list[float], str]]:
return [
(embedding_item.embedding, query)
for embedding_item, query in zip(
oai.embeddings.create(input=queries, model="text-embedding-3-large").data,
queries,
)
]
def knn(
embedded_examples: list[tuple[list[float], str]],
query_embedding: list[float],
k: int,
):
distances = [
(distance(embedding, query_embedding), example)
for embedding, example in embedded_examples
]
distances.sort(key=lambda x: x[0])
return distances[:k]
def generate_response(examples: list[str], query: str):
formatted_examples = "\n".join(examples)
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "user",
"content": dedent(
f"""
Respond to the following query with the most accurate
and concise answer possible.
<examples>
{formatted_examples}
</examples>
<query>
{query}
</query>
"""
),
}
],
)
def generate_question_and_answer_pair(
questions: list[str], question_and_answers: list[dict[str, str]]
) -> list[str]:
question_to_answer = {}
for question in question_and_answers:
question_to_answer[question["question"]] = question["answer"]
return [
dedent(
f"""
<example>
<question>{question}</question>
<answer>{question_to_answer[question]}</answer>
</example>
"""
)
for question in questions
]
if __name__ == "__main__":
examples = [
{"question": "What is the capital of France?", "answer": "Paris"},
{"question": "Who wrote Romeo and Juliet", "answer": "Shakespeare"},
{"question": "What is the capital of Germany?", "answer": "Berlin"},
]
query = "What is the capital of Italy?"
# Step 1 : Embed the Examples
embeddings = embed_queries([example["question"] for example in examples] + [query])
embedded_examples = embeddings[:-1]
embedded_query = embeddings[-1]
# # Step 3: Find the k closest examples to the query
k_closest_examples = knn(embedded_examples, embedded_query[0], 2)
for example in k_closest_examples:
print(example)
#> (0.4013468481736857, 'What is the capital of France?')
#> (0.4471368596136872, 'What is the capital of Germany?')
# Step 4: Use these examples as in-context examples
formatted_examples = generate_question_and_answer_pair(
[example[1] for example in k_closest_examples], examples
)
response = generate_response(formatted_examples, query)
print(response.answer)
#> Rome
```
### References
<sup id="ref-1">1</sup>: [What Makes Good In-Context Examples for GPT-3?](https://arxiv.org/abs/2101.06804)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

7
참고/instructor-main/docs/prompting/few_shot/exemplar_selection/vote_k.md Normal file
View File

@@ -0,0 +1,7 @@
---
title: ""
description: ""
keywords: ""
---
[wip]

240
참고/instructor-main/docs/prompting/index.md Normal file
View File

@@ -0,0 +1,240 @@
---
title: Advanced Prompting Techniques Guide
description: Research-backed prompting techniques to improve LLM performance with Instructor
---
# Advanced Prompting Techniques
<div class="grid cards" markdown>
- :material-lightbulb: **Basic Approaches**
Zero-shot and few-shot techniques for immediate improvements
[:octicons-arrow-right-16: Zero-Shot](#zero-shot) · [:octicons-arrow-right-16: Few-Shot](#few-shot)
- :material-brain: **Reasoning Methods**
Techniques to improve model reasoning and problem-solving
[:octicons-arrow-right-16: Thought Generation](#thought-generation) · [:octicons-arrow-right-16: Decomposition](#decomposition)
- :material-check-all: **Verification**
Methods for self-assessment and correction
[:octicons-arrow-right-16: Self-Criticism](#self-criticism)
- :material-group: **Collaboration**
Ensemble techniques for aggregating multiple model outputs
[:octicons-arrow-right-16: Ensembling](#ensembling)
</div>
This guide presents 58 research-backed prompting techniques mapped to Instructor implementations. Based on [The Prompt Report](https://trigaten.github.io/Prompt_Survey_Site) by [Learn Prompting](https://learnprompting.org) which analyzed over 1,500 academic papers on prompting.
## Prompting Technique Map
The following diagram shows how different prompting techniques relate to each other and when to use them:
```mermaid
flowchart TD
A[Choose Prompting Technique] --> B{Have Examples?}
B -->|No| C[Zero-Shot Techniques]
B -->|Yes| D[Few-Shot Techniques]
C --> C1[Role Prompting]
C --> C2[Emotional Language]
C --> C3[Style Definition]
C --> C4[Follow-Up Generation]
D --> D1[Example Ordering]
D --> D2[Example Selection]
D --> D3[Example Generation]
A --> E{Need Reasoning?}
E -->|Yes| F[Thought Generation]
F --> F1[Chain of Thought]
F --> F2[Step-Back Prompting]
F --> F3[Thread of Thought]
A --> G{Complex Problem?}
G -->|Yes| H[Decomposition]
H --> H1[Least-to-Most]
H --> H2[Tree of Thought]
H --> H3[Plan and Solve]
A --> I{Need Verification?}
I -->|Yes| J[Self-Criticism]
J --> J1[Self-Verification]
J --> J2[Chain of Verification]
J --> J3[Self-Refinement]
A --> K{Want Multiple Perspectives?}
K -->|Yes| L[Ensembling]
L --> L1[Self-Consistency]
L --> L2[Meta-CoT]
L --> L3[Specialized Experts]
classDef category fill:#e2f0fb,stroke:#b8daff,color:#004085;
classDef technique fill:#d4edda,stroke:#c3e6cb,color:#155724;
classDef decision fill:#fff3cd,stroke:#ffeeba,color:#856404;
class A,C,D,F,H,J,L category
class C1,C2,C3,C4,D1,D2,D3,F1,F2,F3,H1,H2,H3,J1,J2,J3,L1,L2,L3 technique
class B,E,G,I,K decision
```
## When to Use Each Technique
| Goal | Recommended Techniques |
|------|------------------------|
| Improve accuracy | Chain of Thought, Self-Verification, Self-Consistency |
| Handle complex problems | Decomposition, Tree of Thought, Least-to-Most |
| Generate creative content | Role Prompting, Emotional Language, Style Definition |
| Verify factual correctness | Chain of Verification, Self-Calibration |
| Optimize with few examples | KNN Example Selection, Active Prompting |
| Handle uncertainty | Uncertainty-Routed CoT, Self-Consistency |
## Zero-Shot {#zero-shot}
These techniques improve model performance without examples:
| Technique | Description | Use Case |
|-----------|-------------|----------|
| [Emotional Language](zero_shot/emotion_prompting.md) | Add emotional tone to prompts | Creative writing, empathetic responses |
| [Role Assignment](zero_shot/role_prompting.md) | Give the model a specific role | Expert knowledge, specialized perspectives |
| [Style Definition](zero_shot/style_prompting.md) | Specify writing style | Content with particular tone or format |
| [Prompt Refinement](zero_shot/s2a.md) | Automatic prompt optimization | Iterative improvement of results |
| [Perspective Simulation](zero_shot/simtom.md) | Have the model adopt viewpoints | Multiple stakeholder analysis |
| [Ambiguity Clarification](zero_shot/rar.md) | Identify and resolve unclear aspects | Improving precision of responses |
| [Query Repetition](zero_shot/re2.md) | Ask model to restate the task | Better task understanding |
| [Follow-Up Generation](zero_shot/self_ask.md) | Generate clarifying questions | Deep exploration of topics |
## Few-Shot {#few-shot}
Techniques for effectively using examples in prompts:
| Technique | Description | Use Case |
|-----------|-------------|----------|
| [Example Generation](few_shot/example_generation/sg_icl.md) | Automatically create examples | Domains with limited example data |
| [Example Ordering](few_shot/example_ordering.md) | Optimal sequencing of examples | Improved pattern recognition |
| [KNN Example Selection](few_shot/exemplar_selection/knn.md) | Choose examples similar to query | Domain-specific accuracy |
| [Vote-K Selection](few_shot/exemplar_selection/vote_k.md) | Advanced similarity-based selection | Complex pattern matching |
## Thought Generation {#thought-generation}
Methods to encourage human-like reasoning in models:
### Zero-Shot Reasoning
| Technique | Description | Use Case |
|-----------|-------------|----------|
| [Analogical CoT](thought_generation/chain_of_thought_zero_shot/analogical_prompting.md) | Generate reasoning using analogies | Complex problem-solving |
| [Step-Back Prompting](thought_generation/chain_of_thought_zero_shot/step_back_prompting.md) | Consider higher-level questions first | Scientific and abstract reasoning |
| [Thread of Thought](thought_generation/chain_of_thought_zero_shot/thread_of_thought.md) | Encourage step-by-step analysis | Detailed explanation generation |
| [Tabular CoT](thought_generation/chain_of_thought_zero_shot/tab_cot.md) | Structure reasoning in table format | Multi-factor analysis |
### Few-Shot Reasoning
| Technique | Description | Use Case |
|-----------|-------------|----------|
| [Active Prompting](thought_generation/chain_of_thought_few_shot/active_prompt.md) | Annotate uncertain examples | Improved accuracy on edge cases |
| [Auto-CoT](thought_generation/chain_of_thought_few_shot/auto_cot.md) | Choose diverse examples | Broad domain coverage |
| [Complexity-Based CoT](thought_generation/chain_of_thought_few_shot/complexity_based.md) | Use complex examples | Challenging problem types |
| [Contrastive CoT](thought_generation/chain_of_thought_few_shot/contrastive.md) | Include correct and incorrect cases | Error detection and avoidance |
| [Memory of Thought](thought_generation/chain_of_thought_few_shot/memory_of_thought.md) | Use high-certainty examples | Reliability in critical applications |
| [Uncertainty-Routed CoT](thought_generation/chain_of_thought_few_shot/uncertainty_routed_cot.md) | Select the most certain reasoning path | Decision-making under uncertainty |
| [Prompt Mining](thought_generation/chain_of_thought_few_shot/prompt_mining.md) | Generate templated prompts | Efficient prompt engineering |
## Ensembling {#ensembling}
Techniques for combining multiple prompts or responses:
| Technique | Description | Use Case |
|-----------|-------------|----------|
| [Consistent, Diverse Sets](ensembling/cosp.md) | Build consistent example sets | Stable performance |
| [Batched In-Context Examples](ensembling/dense.md) | Efficient example batching | Performance optimization |
| [Step Verification](ensembling/diverse.md) | Validate individual steps | Complex workflows |
| [Maximizing Mutual Information](ensembling/max_mutual_information.md) | Information theory optimization | Information-dense outputs |
| [Meta-CoT](ensembling/meta_cot.md) | Merge multiple reasoning chains | Complex problem-solving |
| [Specialized Experts](ensembling/more.md) | Use different "expert" prompts | Multi-domain tasks |
| [Self-Consistency](ensembling/self_consistency.md) | Choose most consistent reasoning | Logical accuracy |
| [Universal Self-Consistency](ensembling/universal_self_consistency.md) | Domain-agnostic consistency | General knowledge tasks |
| [Task-Specific Selection](ensembling/usp.md) | Choose examples per task | Specialized domain tasks |
| [Prompt Paraphrasing](ensembling/prompt_paraphrasing.md) | Use variations of the same prompt | Robust outputs |
## Self-Criticism {#self-criticism}
Methods for models to verify or improve their own responses:
| Technique | Description | Use Case |
|-----------|-------------|----------|
| [Chain of Verification](self_criticism/chain_of_verification.md) | Generate verification questions | Fact-checking, accuracy |
| [Self-Calibration](self_criticism/self_calibration.md) | Ask if answer is correct | Confidence estimation |
| [Self-Refinement](self_criticism/self_refine.md) | Auto-generate feedback and improve | Iterative improvement |
| [Self-Verification](self_criticism/self_verification.md) | Score multiple solutions | Quality assessment |
| [Reverse CoT](self_criticism/reversecot.md) | Reconstruct the problem | Complex reasoning verification |
| [Cumulative Reasoning](self_criticism/cumulative_reason.md) | Generate possible steps | Thorough analysis |
## Decomposition {#decomposition}
Techniques for breaking down complex problems:
| Technique | Description | Use Case |
|-----------|-------------|----------|
| [Functional Decomposition](decomposition/decomp.md) | Implement subproblems as functions | Modular problem-solving |
| [Faithful CoT](decomposition/faithful_cot.md) | Use natural and symbolic language | Mathematical reasoning |
| [Least-to-Most](decomposition/least_to_most.md) | Solve increasingly complex subproblems | Educational applications |
| [Plan and Solve](decomposition/plan_and_solve.md) | Generate a structured plan | Project planning |
| [Program of Thought](decomposition/program_of_thought.md) | Use code for reasoning | Algorithmic problems |
| [Recursive Thought](decomposition/recurs_of_thought.md) | Recursively solve subproblems | Hierarchical problems |
| [Skeleton of Thought](decomposition/skeleton_of_thought.md) | Generate outline structure | Writing, planning |
| [Tree of Thought](decomposition/tree-of-thought.md) | Search through possible paths | Decision trees, exploration |
## Implementation with Instructor
All these prompting techniques can be implemented with Instructor by:
1. Defining appropriate Pydantic models that capture the expected structure
2. Incorporating the prompting technique in your model docstrings or field descriptions
3. Using the patched LLM client with your response model
```python
import instructor
from pydantic import BaseModel, Field
# Example implementing Chain of Thought with a field
class ReasonedAnswer(BaseModel):
"""Answer the following question with detailed reasoning."""
chain_of_thought: str = Field(
description="Step-by-step reasoning process to solve the problem"
)
final_answer: str = Field(
description="The final conclusion after reasoning"
)
client = instructor.from_provider("openai/gpt-5-nano")
response = client.create(
model="gpt-4",
response_model=ReasonedAnswer,
messages=[
{"role": "user", "content": "What is the cube root of 27?"}
]
)
print(f"Reasoning: {response.chain_of_thought}")
print(f"Answer: {response.final_answer}")
```
## References
<sup>\*</sup> Based on [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

171
참고/instructor-main/docs/prompting/self_criticism/chain_of_verification.md Normal file
View File

@@ -0,0 +1,171 @@
---
description: "We get a model to output a baseline response. Next, we independently verify the response by using a model to generate questions and to verify these questions. Lastly, we use a final API call to verify the baseline response with the generated data"
---
Chain Of Verification ( CoVe )<sup><a href="https://arxiv.org/pdf/2309.11495">1</a></sup> is a method that allows us to be able to verify our LLM's generated responses. We can do so using the following steps
1. First we get our LLM to generate a response to a query
2. Then we generate a set of follow up questions that need to be answered to validate the response
3. We then independently generate a set of responses to these questions
4. Lastly, we use a final LLM call to verify the response in light of these new question and answer pairs that we've generated
```python hl_lines="49-52 95-100"
import instructor
from pydantic import BaseModel, Field
import asyncio
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class QueryResponse(BaseModel):
correct_answer: str
class ValidationQuestions(BaseModel):
question: list[str] = Field(
description="""A list of questions that need to be
answered to validate the response"""
)
class ValidationAnswer(BaseModel):
answer: str
class FinalResponse(BaseModel):
correct_answer: str
async def generate_initial_response(query: str):
return await client.create(
model="gpt-4o",
response_model=QueryResponse,
messages=[
{
"role": "system",
"content": "You are an expert question answering system",
},
{"role": "user", "content": query},
],
)
async def generate_verification_questions(llm_response: str):
return await client.create(
model="gpt-4o",
response_model=ValidationQuestions,
messages=[
{
"role": "system",
"content": """You are an expert AI system that excels at
generating follow up questions to validate a response.
These questions should validate key assumptions, facts
and other important portions of the generated response""",
},
{"role": "user", "content": llm_response},
],
)
async def generate_verification_response(questions: list[str]):
async def verify_question(question: str) -> tuple[ValidationAnswer, str]:
return (
await client.create(
model="gpt-4o",
response_model=ValidationAnswer,
messages=[
{
"role": "system",
"content": """You are an expert AI system that
excels at answering validation questions.""",
},
{"role": "user", "content": question},
],
),
question,
)
coros = [verify_question(question) for question in questions]
return await asyncio.gather(*coros)
async def generate_final_response(
answers: list[tuple[ValidationAnswer, str]],
initial_response: QueryResponse,
original_query: str,
):
formatted_answers = "\n".join(
[f"Q: {question}\nA: {answer.answer}" for answer, question in answers]
)
return await client.create(
model="gpt-4o",
response_model=FinalResponse,
messages=[
{
"role": "system",
"content": """You are an expert AI system that excels at
validating and verifying if an initial answer answers an
initial query based off some Verification Questions and
Answers provided. Return the original answer if it is
valid else generate a new response off the verification
questions and answers provided.""",
},
{
"role": "user",
"content": f"""
Initial query: {original_query}
Initial Answer : {initial_response.correct_answer}
Verification Questions and Answers:
{formatted_answers}
""",
},
],
)
if __name__ == "__main__":
query = "What was the primary cause of the Mexican-American war and how long did it last?"
initial_response = asyncio.run(generate_initial_response(query))
print(initial_response.model_dump_json())
"""
{"correct_answer":"The primary cause of the Mexican-American War was
the annexation of Texas by the United States and the dispute over
whether Texas ended at the Nueces River (as the Mexicans claimed) or
the Rio Grande (as the U.S. claimed). The war lasted from April 25,
1846, to February 2, 1848, totaling nearly two years."}
"""
verification_questions = asyncio.run(
generate_verification_questions(initial_response.correct_answer)
)
print(verification_questions.model_dump_json())
"""
{"question":["Is it accurate that the primary cause of the
Mexican-American War was the annexation of Texas by the United
States?","Was there a dispute over whether Texas ended at the Nueces
River or the Rio Grande?","Did the Mexican-American War last from
April 25, 1846, to February 2, 1848?","Is it correct to state that
the disagreement over the Texas border was between the Nueces River
and the Rio Grande?","Was the Mexican claim that Texas ended at the
Nueces River while the U.S. claimed it was at the Rio Grande?"]}
"""
responses = asyncio.run(
generate_verification_response(verification_questions.question)
)
final_answer = asyncio.run(
generate_final_response(responses, initial_response, query)
)
print(final_answer.model_dump_json())
"""
{"correct_answer":"The primary cause of the Mexican-American War was
the annexation of Texas by the United States and the dispute over
whether Texas ended at the Nueces River (as the Mexicans claimed) or
the Rio Grande (as the U.S. claimed). The war lasted from April 25,
1846, to February 2, 1848, totaling nearly two years."}
"""
```
### References
<sup id="ref-1">1</sup>: [Chain-Of-Verification Reduces Hallucination In Large Language Models](https://arxiv.org/pdf/2309.11495)

231
참고/instructor-main/docs/prompting/self_criticism/cumulative_reason.md Normal file
View File

@@ -0,0 +1,231 @@
---
description: "Cumulative Reasoning breaks the reasoning process into three separate steps so that our model has enough room to reason and filter out the reasoning steps at each point, thus improving model performance"
---
Cumulative Reasoning<sup><a href="https://arxiv.org/pdf/2308.04371">1</a></sup> aims to generate better outputs by dividing the reasoning process into three separate steps
1. **Propose** : A LLM first suggests potential steps based on the current context, initiating the reasoning cycle
2. **Verify** : We then assess the proposer's suggestions for accuracy, incorporating valid steps into the ongoing context
3. **Report** : We then determine the appropriate moment to conclude the reasoning process
By first generating potential steps and separating out each portions of the reasoning process, we are able to obtain significant improvements in logical inference tasks and mathematical problems.
We can implement this using `instructor` as seen below
```python hl_lines="46-61 94-100 138-148"
import instructor
from pydantic import BaseModel, Field
from textwrap import dedent
from typing import Literal
import asyncio
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Proposition(BaseModel):
premise1: str
premise2: str
reasoning: str
proposition: str
class ProposerOutput(BaseModel):
reasoning: str
valid_propositions: list[Proposition] = Field(
description="Concise list of Propositions that are derived from the premises that are relevant to the hypothesis. Note that each Proposition is derived from two given premises at most",
min_length=4,
)
prediction: Literal["False", "True", "Unknown"]
class VerifiedProposition(BaseModel):
proposition: str
reasoning: str
is_valid: bool
class ReporterOutput(BaseModel):
reasoning: str
is_valid_hypothesis: bool
async def generate_propositions(premises: list[str], hypothesis: str) -> ProposerOutput:
formatted_premises = "\n- ".join(premises)
return await client.create(
messages=[
{
"role": "system",
"content": dedent(
"""
Suppose you are one of the greatest AI
scientists, logicians, and mathematicians.
Let us think step by step. Please use
First-Order Logic (FOL) to deduce a list
of Propositions. Each Proposition is
derived from two given Premises and
should be logically correct. Most
importantly, each Proposition should
not duplicate the two premises that it
is derived from. Please make sure your
reasoning is directly deduced from the
Premises and Propositions rather than
introducing unsourced common knowledge
and unsourced information by common
sense reasoning.
"""
),
},
{
"role": "user",
"content": dedent(
f"""
Premises:
{formatted_premises}
We want to deduce more Propositions to
determine the correctness of the following
Hypothesis:
Hypothesis: {hypothesis}
"""
),
},
],
response_model=ProposerOutput,
model="gpt-4o",
)
async def verify_propositions(
premise_evaluation: ProposerOutput,
) -> list[VerifiedProposition]:
async def create_verification_task(proposition: Proposition) -> VerifiedProposition:
return await client.create(
messages=[
{
"role": "system",
"content": """
Suppose you are one of the greatest AI
scientists, logicians, and mathematicians.
Let us think step by step. Please use
First-Order Logic (FOL) to determine
whether the deduction of two given
Premises to a Proposition is valid or not,
and reply with True or False.
""",
},
{
"role": "user",
"content": f"""
Premises:
{proposition.premise1}
{proposition.premise2}
Proposition:
{proposition.proposition}
""",
},
],
response_model=VerifiedProposition,
model="gpt-4o",
)
tasks = [
create_verification_task(proposition)
for proposition in premise_evaluation.valid_propositions
]
return await asyncio.gather(*tasks)
async def final_evaluation(
verification_result: list[str], hypothesis: str, premises: list[str]
) -> ReporterOutput:
formatted_premises = "\n- ".join(premises)
formatted_propositions = "\n- ".join(verification_result)
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """
Suppose you are one of the greatest AI
scientists, logicians, and mathematicians.
Let us think step by step. Read and analyze
the “Premises” first, then use First-Order
Logic (FOL) to judge whether the “Hypothesis”
is True, False, or Unknown. Please make sure
your reasoning is directly deduced from the
"Premises" and "Propositions" rather than
introducing unsourced common knowledge and
unsourced information by common sense
reasoning.
""",
},
{
"role": "user",
"content": f"""
Premises:
{formatted_premises}
Hypothesis: {hypothesis}
""",
},
{
"role": "assistant",
"content": f"""
Let's think step by step. From the premises,
we can deduce the following propositions:
{formatted_propositions}
Recall the Hypothesis: {hypothesis}
""",
},
],
response_model=ReporterOutput,
)
if __name__ == "__main__":
hypothesis = "Hyraxes lay eggs"
premises = [
"The only types of mammals that lay eggs are platypuses and echidnas",
"Platypuses are not hyrax",
"Echidnas are not hyrax",
"No mammals are invertebrates",
"All animals are either vertebrates or invertebrates",
"Mammals are animals",
"Hyraxes are mammals",
"Grebes lay eggs",
"Grebes are not platypuses and also not echidnas",
]
premise_evaluation = asyncio.run(generate_propositions(premises, hypothesis))
verification_result = asyncio.run(verify_propositions(premise_evaluation))
filtered_propositions = [
proposition.proposition
for proposition in verification_result
if proposition.is_valid
]
reporter_output = asyncio.run(
final_evaluation(filtered_propositions, hypothesis, premises)
)
print(reporter_output.model_dump_json(indent=2))
"""
{
"reasoning": "Based on the premises provided, the
only mammals that lay eggs are platypuses and
echidnas. Hyraxes are mammals but are explicitly
stated as not being platypuses or echidnas. Hence,
there is no basis in the premises to conclude that
hyraxes lay eggs. \n\nTherefore, the hypothesis that
hyraxes lay eggs is False.",
"is_valid_hypothesis": false
}
"""
```
### References
<sup id="ref-1">1</sup>: [Cumulative Reasoning with Large Language Models](https://arxiv.org/pdf/2308.04371)

267
참고/instructor-main/docs/prompting/self_criticism/reversecot.md Normal file
View File

@@ -0,0 +1,267 @@
---
description: "Reverse Chain Of Thought is a method to help identify logical inconsistencies in the reasoning steps of a large language model's response"
---
We can use a method called Reverse Chain Of Thought<sup><a href="https://arxiv.org/pdf/2305.11499">1</a></sup> to reverse engineer a problem given a solution. This helps us to find specific inconsistencies in the reasoning steps taken by our model and to give targetted feedback which can improve the quality of the solution.
This is done through a 3 step process
1. **Reconstruct The Question** : We first attempt to reconstruct the original problem given the solution and reasoning steps generated
2. **Identify Inconsistencies** : Identify the inconsistencies between the original problem and the reconstructed problem
3. **Generate Feedback** : Give fine-grained fedback to guide the LLM in revising its solution
We can implement this using `instructor` as seen below.
```python hl_lines="54-59 76-83 98-107 127-140 155-167"
import instructor
from pydantic import BaseModel, Field
client = instructor.from_provider("openai/gpt-5-nano")
class ReconstructedPrompt(BaseModel):
chain_of_thought: str
reconstructed_prompt: str = Field(
description="""Reconstruction of a potential prompt
that could have been used to generate the reasoning
and final solution provided by the user"""
)
class ConditionList(BaseModel):
conditions: list[str] = Field(
description="""Key information and conditions present
in the reasoning steps which are relevant to answering
the question"""
)
class ModelFeedback(BaseModel):
detected_inconsistencies: list[str] = Field(
description="""Inconsistencies that were detected between
the original condition list and the reconstructed condition
list"""
)
feedback: str = Field(
description="""Feedback on how to fix the inconsistencies
detected in the original condition list and the reconstructed
condition list"""
)
is_equal: bool
class ModelResponse(BaseModel):
chain_of_thought: str = Field(
description="""Logical Steps that were taken to derive
the final concluding statement"""
)
correct_answer: str
def generate_response(query: str):
return client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """
You are a helpful AI Question Answerer. You are
about to be passed a query by a User.
Make sure to generate a series of logical steps
and reason about the problem before generating
a solution.
""",
},
{"role": "user", "content": query},
],
response_model=ModelResponse,
)
def reconstruct_prompt(model_response: ModelResponse):
return client.create(
model="gpt-4o",
response_model=ReconstructedPrompt,
messages=[
{
"role": "system",
"content": f"""
Give the concrete prompt (problem) that can
generate this answer. The problem should
contain all basic and necessary information
and correspond to the answer. The problem
can only ask for one result
Reasoning: {model_response.chain_of_thought}
Response: {model_response.correct_answer}
""",
}
],
)
def deconstruct_prompt_into_condition_list(prompt: str):
return client.create(
model="gpt-4o",
response_model=ConditionList,
messages=[
{
"role": "system",
"content": """
You are an expert AI system that excels at
analyzing and decomposing questions into their
constituent parts.
Please list the conditions of the problem given
below. There might be multiple conditions in the
problem so make sure to navigate through the
prompt incrementally, indentifying and extracting
the conditions necessary to answer the question
in your final response.
""",
},
{"role": "user", "content": prompt},
],
)
def generate_feedback(
original_condition_list: list[str], final_condition_list: list[str]
):
formatted_original_conditions = "\n- ".join(original_condition_list)
formatted_final_conditions = "\n- ".join(final_condition_list)
return client.create(
model="gpt-4o",
response_model=ModelFeedback,
messages=[
{
"role": "system",
"content": f"""
You are an expert AI system that excels at
analyzing and comparing two lists of conditions.
Original Condition List:
{formatted_original_conditions}
Reconstructed Condition List:
{formatted_final_conditions}
Determine if the two condition lists are roughly
equivalent. If they are not, give targetted
feedback on what is missing from the reconstructed
condition list as compared to the original condition
list and how it can be fixed.
""",
}
],
)
def revise_response(response: ModelResponse, feedback: ModelFeedback):
formatted_inconsistencies = "\n- ".join(feedback.detected_inconsistencies)
return client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": f"""
Here are the mistakes and reasons in your answer
to the prompt
Original Response: {response.correct_answer}
You have overlooked some real conditions:
{formatted_inconsistencies}
Here are detailed reasons:
{feedback.feedback}
Generate a revised response that takes into account
the detailed feedback and includes the ignored
conditions
""",
}
],
response_model=ModelResponse,
)
if __name__ == "__main__":
query = """
Mary is an avid gardener. Yesterday, she received 18 new
potted plants from her favorite plant nursery. She already
has 2 potted plants on each of the 40 window ledges of her
large backyard. How many potted plants will Mary remain
with?
"""
response = generate_response(query)
reconstructed_prompt = reconstruct_prompt(response)
print(reconstructed_prompt.reconstructed_prompt)
"""
Mary received 18 new potted plants. She already has 2 potted plants on each
of the 40 window ledges in her backyard. How many potted plants does she have now?
"""
original_condition_list = deconstruct_prompt_into_condition_list(query)
new_condition_list = deconstruct_prompt_into_condition_list(
reconstructed_prompt.reconstructed_prompt
)
print(original_condition_list.model_dump_json(indent=2))
"""
{
"conditions": [
"Mary received 18 new potted plants.",
"Mary has 2 potted plants on each of the 40 window ledges in her backyard.",
"We are required to find the total number of potted plants Mary will have."
]
}
"""
print(new_condition_list.model_dump_json(indent=2))
"""
{
"conditions": [
"Mary received 18 new potted plants.",
"She already has 2 potted plants on each of the 40 window ledges in her backyard."
]
}
"""
feedback = generate_feedback(
original_condition_list.conditions, new_condition_list.conditions
)
print(feedback.model_dump_json(indent=2))
"""
{
"detected_inconsistencies": [
"The reconstructed list is missing the requirement
to find the total number of potted plants Mary will
have."
],
"feedback": "Add the requirement of finding the total
number of potted plants Mary will have to the
reconstructed condition list to match the original
condition list.",
"is_equal": false
}
"""
if not feedback.is_equal:
response = revise_response(response, feedback)
print(response.model_dump_json(indent=2))
"""
{
"chain_of_thought": "First, we note that Mary starts
with 18 potted plants. According to the problem, she
bought 2 packs of 40 new potted plants. So, to find
the total number of plants she will have, we add the
number of plants she initially has to the number she
bought. This gives us 18 (initial) + 2 * 40 (new) =
18 + 80 = 98 potted plants.",
"correct_answer": "98 potted plants"
}
"""
```
### References
<sup id="ref-1">1</sup>: [RCoT: Detecting And Rectifying Factual Inconsistency In Reasoning By Reversing Chain-Ofthought](https://arxiv.org/pdf/2305.11499)

89
참고/instructor-main/docs/prompting/self_criticism/self_calibration.md Normal file
View File

@@ -0,0 +1,89 @@
---
description: "Self Calibration aims to get language models to determine what they know and do not know"
---
We want our language models to be able to output the extent of their confidence in predictions. To do so, we can get language models to evaluate their responses to a given prompt using a technique called Self Calibration <sup><a href="https://arxiv.org/pdf/2207.05221">1</a></sup>
> The original paper used a fine-tuned regression head over the language model's final output. However, since we don't have access to the model's final hidden states, we can substitute it for a function call instead to achieve a similar result.
We can ask language models to evaluate their outputs by using the following template
We can implement this using `instructor` as seen below
```python hl_lines="23-27"
import instructor
from pydantic import BaseModel, Field
client = instructor.from_provider("openai/gpt-5-nano")
class SelfCalibration(BaseModel):
chain_of_thought: str
is_valid_answer: bool = Field(description="Whether the answer is correct or not")
def evaluate_model_output(original_prompt: str, model_response: str):
return client.create(
messages=[
{
"role": "user",
"content": f"""
Question: {original_prompt}
{model_response}
Is this a valid answer to the question?
Make sure to examine the question
thoroughly and generate a complete
reasoning for why the answer is correct
or not before responding.
""",
}
],
response_model=SelfCalibration,
model="gpt-4o",
)
if __name__ == "__main__":
original_prompt = """
Question: Who was the third president of the
United States?
"""
model_response = """
Here are some brainstormed ideas: James Monroe
Thomas Jefferson
Jefferson
Thomas Jefferson
George Washington
"""
response = evaluate_model_output(original_prompt, model_response)
print(response.model_dump_json(indent=2))
"""
{
"chain_of_thought": "Let's examine the question
carefully: 'Who was the third president of the
United States?'\n\nThe brainstormed ideas are:
\n1. James Monroe\n2. Thomas Jefferson\n3.
Jefferson\n4. Thomas Jefferson\n5. George
Washington.\n\nTo determine the validity of these
answers, I'll cross-check with historical
records.\n\n1. James Monroe was not the third
president; he was the fifth president.\n2. Thomas
Jefferson was indeed the third president of the
United States.\n3. 'Jefferson' is a correct but
incomplete answer; it lacks the first name, though
it is commonly understood.\n4. 'Thomas Jefferson'
is the full name and correct answer.\n5. George
Washington was the first president, not the
third.\n\nTherefore, the correct, valid answer to
the question 'Who was the third president of the
United States?' is 'Thomas Jefferson,' and this
answer is correct.",
"is_valid_answer": true
}
"""
```
### References
<sup id="ref-1">1</sup>: [Language Models (Mostly) Know What They Know](https://arxiv.org/pdf/2207.05221)

210
참고/instructor-main/docs/prompting/self_criticism/self_refine.md Normal file
View File

@@ -0,0 +1,210 @@
---
title: "Improve With Feedback"
description: "Self-refine is an approach that uses an LLM to generate an output, provide feedback on the output, and improve the output based on the provided feedback."
---
How can we provide feedback for an LLM to improve its responses?
Self-refine is an approach that uses an LLM to generate an output, provide feedback on the output, and improve the output based on the provided feedback. This processes repeats until a stopping condition is achieved. The same LLM is used for all three steps.
```mermaid
graph TD
A[Generate initial response]:::blue --> B[Generate feedback]:::orange
B --> C{Stopping<br>condition<br>met?}:::orange
C -->|No| D[Refine response]:::orange
C -->|Yes| E[Final output]:::green
D --> B
classDef blue fill:#E3F2FD,stroke:#90CAF9,color:#1565C0
classDef orange fill:#FFF3E0,stroke:#FFE0B2,color:#E65100
classDef green fill:#E8F5E9,stroke:#A5D6A7,color:#2E7D32
linkStyle default stroke:#90A4AE,stroke-width:2px;
linkStyle 1,2,4 stroke:#FFB74D,stroke-width:2px;
```
```python hl_lines="102-106"
import instructor
from pydantic import BaseModel, Field
from typing import Optional
class Response(BaseModel):
code: str
class Feedback(BaseModel):
feedback: list[str] = Field(
description="A list of actions to take to improve the code."
)
done: bool
class Timestep(BaseModel):
response: str
feedback: Optional[list[str]] = Field(default_factory=list)
refined_response: Optional[str] = Field(default="")
class History(BaseModel):
history: list[Timestep] = Field(default_factory=list)
def add(self, code, feedback, refined_code):
self.history.append(
Timestep(response=code, feedback=feedback, refined_response=refined_code)
)
client = instructor.from_provider("openai/gpt-5-nano")
def generate_feedback(response):
return client.create(
model="gpt-4o",
response_model=Feedback,
messages=[
{
"role": "user",
"content": f"""
You are an expert Python coder.
Provide feedback on this code.
How can we make it (1) faster and (2) more readable?
<code>
{response.code}
</code>
If the code does not need to be improved, then indicate by setting "done" to True.
""",
}
],
)
def refine(response, feedback):
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "user",
"content": f"""
You are an expert Python coder.
<response>
{response.code}
</response>
<feedback>
{feedback.feedback}
</feedback>
Refine your response.
""",
}
],
)
def stop_condition(feedback, history):
return feedback.done or len(history.history) >= 3
if __name__ == "__main__":
response = client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "user",
"content": "Write Python code to calculate the fibonacci sequence.",
}
],
)
history = History()
while True:
feedback = generate_feedback(response)
if stop_condition(feedback, history):
break
refined_response = refine(response, feedback)
# Save to history
history.add(response.code, feedback.feedback, refined_response.code)
response = refined_response
print(history.history[0].response)
"""
def fibonacci(n):
sequence = [0, 1]
while len(sequence) < n:
sequence.append(sequence[-1] + sequence[-2])
return sequence[:n]
# Example usage:
n = 10
print(fibonacci(n))
"""
print(history.history[0].feedback)
"""
[
'Use a generator to reduce memory consumption for large `n` values and improve speed.',
'Enhance readability by adding type hints for input and output.',
"Add docstrings to explain the function's purpose and parameters.",
"Avoid slicing the list at the end if it's not necessary; instead, ensure the loop condition is precise.",
]
"""
print(history.history[0].refined_response)
"""
def fibonacci(n: int) -> list[int]:
"""Generate a Fibonacci sequence of length n.
Args:
n (int): The length of the Fibonacci sequence to generate.
Returns:
list[int]: A list containing the Fibonacci sequence of length n.
"""
def fibonacci_generator():
a, b = 0, 1
for _ in range(n):
yield a
a, b = b, a + b
return list(fibonacci_generator())
# Example usage:
n = 10
print(fibonacci(n))
"""
print(f"...process repeated {len(history.history)} times...")
#> ...process repeated 3 times...
print(response.code)
"""
def fibonacci(n: int) -> list[int]:
"""Generate a Fibonacci sequence of length n.
Args:
n (int): The length of the Fibonacci sequence to generate.
Returns:
list[int]: A list containing the Fibonacci sequence of length n.
"""
if n <= 0:
return []
sequence = [0] * n
if n > 1:
sequence[1] = 1
for i in range(2, n):
sequence[i] = sequence[i-1] + sequence[i-2]
return sequence
# Example usage:
n = 10
print(fibonacci(n))
"""
```
### References
<sup id="ref-1">1</sup>: [Self-Refine: Iterative Refinement with Self-Feedback](https://arxiv.org/abs/2303.17651)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

144
참고/instructor-main/docs/prompting/self_criticism/self_verification.md Normal file
View File

@@ -0,0 +1,144 @@
---
title: "Self-Verify LLM Responses"
description: "The self-verification framework generates multiple response candidates, then uses an LLM to verify these candidates."
---
We want to verify that an LLM response is correct. How can we automate this?
The self-verification framework generates multiple response candidates, then uses an LLM to verify these candidates. The process follows two stages:
1. Forward Reasoning
2. Backward Verification
## Forward Reasoning
In forward reasoning, we leaverage CoT to generate multiple candidate solutions.
## Backward Verification
Backward verification involves three steps.
### Rewrite As Declarative
Rewrite the original question and its solution as a declarative.
!!! example "Rewritten Declaritive Example"
**original question**: Jackie has 10 apples. Adam has 8 apples. How many more apples does Jackie have than Adam?
**response candidate**: Jackie has 10 apples. so Jackie has 10-8=2 more apples than Adam, and the answer is 2.
**rewritten declarative**: Jackie has 10 apples. Adam has 8 apples. Jackie has 2 more apples than Adam.
### Construct New Question
Construct a new question and prompt the LLM to verify it. Two possible methods are:
1. True-False Item Verification (TFV)
2. Condition Mask Verification (CMV)
TFV asks the LLM if the rewritten declarative is correct. CMV filters out conditions provided in the original question and asks an LLM to predict the filtered condition.
!!! example "TFV Example Prompt"
Jackie has 10 apples. Adam has 8 apples. Jackie has 2 more apples than Adam. Is this correct?
!!! example "CMV Example Prompt"
Jackie has X apples. Adam has 8 apples. Jackie has 2 more apples than Adam. What is X?
### Compute Verification Score
The LLM is then queried with the new question for each candidate *k* times. If TFV is used, the verification score is simply the number of times the LLM outputs "True". If CMV is used, the verification score is the number of times the masked value and the real value match.
The candidate with the highest verification score is then chosen as the final answer.
## Implementation
The full pipeline with forward reasoning and backward verification can be implemented using `instructor` as seen below:
```python
import instructor
from pydantic import BaseModel
from typing import Literal
client = instructor.from_provider("openai/gpt-5-nano")
n = 3 # Number of candidates to generate
k = 5 # Number of times to verify
class Date(BaseModel):
month: int
day: int
class Candidate(BaseModel):
reasoning_steps: list[str]
month: str
class Rewritten(BaseModel):
declarative: str
class Verification(BaseModel):
correct: Literal["True", "False"]
def query_llm(query, model):
return client.create(
model="gpt-4o",
response_model=model,
messages=[
{
"role": "user",
"content": f"Think step by step: {query}",
}
],
)
def rewrite(query, candidate):
return client.create(
model="gpt-4o",
response_model=Rewritten,
messages=[
{
"role": "user",
"content": f"""
Please change the questions and answers into complete declarative sentences
{query}
The answer is {candidate.month}.
""",
}
],
)
def verify(question):
return client.create(
model="gpt-4o",
response_model=Verification,
messages=[{"role": "user", "content": question}],
)
if __name__ == "__main__":
query = "What month is it now if it has been 3 weeks, 10 days, and 2 hours since May 1, 2024 6pm?"
# Step 1: Forward Reasoning
candidates = [query_llm(query, Candidate) for _ in range(n)]
# Step 2: Backwards Verification
for candidate in candidates:
# 2.a Rewrite
rewritten = rewrite(query, candidate)
# 2.b Construct new questions
question = f"{rewritten.declarative} Do it is correct (True or False)?"
# 2.c Compute verification score
scores = [verify(question).correct for _ in range(k)]
verification_score = sum(1 for s in scores if s == "True")
print(f"Candidate: {candidate.month}, Verification Score: {verification_score}")
#> Candidate: May, Verification Score: 0
#> Candidate: June, Verification Score: 2
#> Candidate: May, Verification Score: 1
```
### References
<sup id="ref-1">1</sup>: [Large Language Models are Better Reasoners with Self-Verification](https://arxiv.org/abs/2212.09561)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

99
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_few_shot/active_prompt.md Normal file
View File

@@ -0,0 +1,99 @@
---
description: "Active prompting is a method used to identify the most effective examples for human annotation. "
---
When we have a large pool of unlabeled examples that could be used in a prompt, how should we decide which examples to manually label?
Active prompting is a method used to identify the most effective examples for human annotation. The process involves four key steps:
1. **Uncertainty Estimation**: Assess the uncertainty of the LLM's predictions on each possible example
2. **Selection**: Choose the most uncertain examples for human annotation
3. **Annotation**: Have humans label the selected examples
4. **Inference**: Use the newly labeled data to improve the LLM's performance
## Uncertainty Estimation
In this step, we define an unsupervised method to measure the uncertainty of an LLM in answering a given example.
!!! example "Uncertainty Estimation Example"
Let's say we ask an LLM the following query:
>query = "Classify the sentiment of this sentence as positive or negative: I am very excited today."
and the LLM returns:
>response = "positive"
The goal of uncertainty estimation is to answer: **How sure is the LLM in this response?**
In order to do this, we query the LLM with the same example _k_ times. Then, we use the _k_ responses to determine how dissimmilar these responses are. Three possible metrics<sup><a href="https://arxiv.org/abs/2302.12246">1</a></sup> are:
1. **Disagreement**: Ratio of unique responses to total responses.
2. **Entropy**: Measurement based on frequency of each response.
3. **Variance**: Calculation of the spread of numerical responses.
Below is an example of uncertainty estimation for a single input example using the disagreement uncertainty metric.
```python
import instructor
from pydantic import BaseModel
class Response(BaseModel):
height: int
client = instructor.from_provider("openai/gpt-5-nano")
def query_llm():
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "user",
"content": "How tall is the Empire State Building in meters?",
}
],
)
def calculate_disagreement(responses):
unique_responses = set(responses)
h = len(unique_responses)
return h / k
if __name__ == "__main__":
k = 5 # (1)!
responses = [query_llm() for _ in range(k)] # Query the LLM k times
for response in responses:
print(response)
#> height=443
#> height=443
#> height=443
#> height=443
#> height=381
print(
calculate_disagreement([response.height for response in responses])
) # Calculate the uncertainty metric
#> 0.4
```
1. _k_ is the number of times to query the LLM with a single unlabeled example
This process will then be repeated for all unlabeled examples.
## Selection & Annotation
Once we have a set of examples and their uncertainties, we can select _n_ of them to be annotated by humans. Here, we choose the examples with the highest uncertainties.
## Inference
Now, each time the LLM is prompted, we can include the newly-annotated examples.
## References
<sup id="ref-1">1</sup>: [Active Prompting with Chain-of-Thought for Large Language Models](https://arxiv.org/abs/2302.12246)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

150
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_few_shot/auto_cot.md Normal file
View File

@@ -0,0 +1,150 @@
---
description: "Automate few-shot chain of thought to choose diverse examples"
---
How can we improve the performance of few-shot CoT?
While few-shot CoT reasoning is effective, its effectiveness relies on manually crafted examples. Further, choosing diverse examples has shown effective in reducing reasoning errors from CoT.
Here, we automate CoT to choose diverse examples. Given a list of potential examples:
1. **Cluster**: Cluster potential examples
2. **Sample**: For each cluster,
1. Sort examples by distance from cluster center
2. Select the first example that meets a predefined selection criteria
3. **Prompt**: Incorporate the chosen questions from each cluster as examples in the LLM prompt
!!! info
A sample selection criteria could be limiting the number of reasoning steps to a maximum of 5 steps to encourage sampling examples with simpler rationales.
```python hl_lines="72 75 106"
import instructor
import numpy as np
from openai import OpenAI
from pydantic import BaseModel
from sklearn.cluster import KMeans
from sentence_transformers import SentenceTransformer
client = instructor.from_provider("openai/gpt-4o")
NUM_CLUSTERS = 2
class Example(BaseModel):
question: str
reasoning_steps: list[str]
class FinalAnswer(BaseModel):
reasoning_steps: list[str]
answer: int
def cluster_and_sort(questions, n_clusters=NUM_CLUSTERS):
# Cluster
embeddings = SentenceTransformer('all-MiniLM-L6-v2').encode(questions)
kmeans = KMeans(n_clusters=n_clusters, random_state=42, n_init=10).fit(embeddings)
# Sort
sorted_clusters = [[] for _ in range(kmeans.n_clusters)]
for question, embedding, label in zip(questions, embeddings, kmeans.labels_):
center = kmeans.cluster_centers_[label]
distance = np.linalg.norm(embedding - center)
sorted_clusters[label].append((distance, question))
for cluster in sorted_clusters:
cluster.sort() # Sort by distance
return sorted_clusters
def sample(cluster):
for question in cluster:
response = client.create(
model="gpt-4o",
response_model=Example,
messages=[
{
"role": "system",
"content": "You are an AI assistant that generates step-by-step reasoning for mathematical questions.",
},
{
"role": "user",
"content": f"Q: {question}\nA: Let's think step by step.",
},
],
)
if (
len(response.reasoning_steps) <= 5
): # If we satisfy the selection criteria, we've found our question for this cluster
return response
if __name__ == "__main__":
questions = [
"How many apples are left if you have 10 apples and eat 3?",
"What's the sum of 5 and 7?",
"If you have 15 candies and give 6 to your friend, how many do you have left?",
"What's 8 plus 4?",
"You start with 20 stickers and use 8. How many stickers remain?",
"Calculate 6 added to 9.",
]
# Cluster and sort the questions
sorted_clusters = cluster_and_sort(questions)
# Sample questions that match selection criteria for each cluster
selected_examples = [sample(cluster) for cluster in sorted_clusters]
print(selected_examples)
"""
[
Example(
question='If you have 15 candies and give 6 to your friend, how many do you have left?',
reasoning_steps=[
'Start with the total number of candies you have, which is 15.',
'Subtract the number of candies you give to your friend, which is 6, from the total candies.',
'15 - 6 = 9, so you are left with 9 candies.',
],
),
Example(
question="What's the sum of 5 and 7?",
reasoning_steps=[
'Identify the numbers to be added: 5 and 7.',
'Perform the addition: 5 + 7.',
'The sum is 12.',
],
),
]
"""
# Use selected questions as examples for the LLM
response = client.create(
model="gpt-4o",
response_model=FinalAnswer,
messages=[
{
"role": "user",
"content": f"""
{selected_examples}
If there are 10 books in my bad and I read 8 of them, how many books do I have left? Let's think step by step.
""",
}
],
)
print(response.reasoning_steps)
"""
[
'Start with the total number of books in the bag, which is 10.',
"Subtract the number of books you've read, which is 8, from the total books.",
'10 - 8 = 2, so you have 2 books left.',
]
"""
print(response.answer)
#> 2
```
### References
<sup id="ref-1">1</sup>: [Automatic Chain of Thought Prompting in Large Language Models](https://arxiv.org/abs/2210.03493)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

111
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_few_shot/complexity_based.md Normal file
View File

@@ -0,0 +1,111 @@
---
description: "Complexity Based Prompting involves choosing examples based on their reasoning steps. If reasoning length isn't available, then we can use proxies such as response length"
---
We can improve the performance of our language models by choosing more complex examples. This refers to examples that have either more reasoning steps or a longer response ( when reasoning steps are not available ).
In the event that no examples are available, we can sample multiple responses and generate an answer based off the top few most complex examples. We can determine the complexity based on the length of their reasoning step in a process known as Complexity Based Consistency
<sup><a href="https://arxiv.org/pdf/2210.00720">1</a></sup> .
We can implement Complexity Based Consistency using `instructor` as seen below.
```python hl_lines="40-42"
import instructor
from pydantic import BaseModel, Field
from textwrap import dedent
import asyncio
from collections import Counter
import random
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class ReasoningStep(BaseModel):
step: int = Field(..., description="The step number")
subquestion: str = Field(..., description="Subquestion to solve")
procedure: str = Field(
description="""Any intermediate computation
that was done in the reasoning process. Leave
empty if no computation is needed""",
)
result: str
class Response(BaseModel):
reasoning: list[ReasoningStep] = Field(
description="reasoning steps to derive answer",
)
correct_answer: int
async def generate_single_response(query: str, context: str) -> Response:
return await client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "system",
"content": dedent(
f"""
You are an expert Question Answering system. Make sure
to output your reasoning in structured reasoning steps
before generating a response to the user's query.
Context:
{context}
Query:
{query}
"""
),
},
],
)
async def complexity_based_consistency(
query: str, context: str, samples: int, top_k: int
):
generated_responses = [
generate_single_response(query, context) for _ in range(samples)
]
responses = await asyncio.gather(*generated_responses)
sorted_responses = sorted(responses, key=lambda x: len(x.reasoning), reverse=True)
top_responses = sorted_responses[:top_k]
return top_responses
if __name__ == "__main__":
query = "How many loaves of bread did they have left?"
context = """
The bakers at the Beverly Hills Bakery baked
200 loaves of bread on Monday morning. They
sold 93 loaves in the morning and 39 loaves
in the afternoon. A grocery store returned 6
unsold loaves.
"""
number_of_reasoning_chains = 5
top_k_to_sample = 3
response = asyncio.run(
complexity_based_consistency(
query, context, number_of_reasoning_chains, top_k_to_sample
)
)
answer_counts = Counter([res.correct_answer for res in response])
most_common_count = answer_counts.most_common(len(answer_counts))[0][1]
max_answers = [
answer for answer, count in answer_counts.items() if count == most_common_count
]
final_answer = random.choice(max_answers)
print(final_answer)
#> 74
```
### References
<sup id="ref-1">1</sup>: [Complexity-based prompting for multi-step reasoning](https://arxiv.org/pdf/2210.00720)

147
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_few_shot/contrastive.md Normal file
View File

@@ -0,0 +1,147 @@
---
description: "We can improve model performance by deliberating including incorrect examples of reasoning for our model to see"
---
We can get better performance from our model when using chain-of-thought by including examples of incorrect reasoning. This helps our language model to learn what mistakes to avoid when generating a response. This is known as Contrastive Chain Of Thought<sup><a href="https://arxiv.org/pdf/2311.09277">1</a></sup> and can be done using the following template.
!!! example "Contrastive Chain Of Thought template"
<context>sample question</context>
<question>sample question</question>
<Explanations>
<Explanation>correct reasoning</Explanation>
<WrongExplanation>incorrect reasoning example</WrongExplanation>
<Explanations>
<context>sample question</context>
<question>sample question</question>
We can implement Contrastive Chain Of Thought using `instructor` as seen below.
```python hl_lines="35-40"
import instructor
from pydantic import BaseModel, Field
from textwrap import dedent
client = instructor.from_provider("openai/gpt-5-nano")
class ChainOfThought(BaseModel):
chain_of_thought: str = Field(description="Incorrect reasoning for the answer")
correct_answer: str
def contrastive_chain_of_thought(
query: str,
context: str,
example_prompt: str,
correct_examples: list[str],
incorrect_examples: list[str],
):
correct_example_prompt = "\n".join(
[f"<Explanation>{example}</Explanation>" for example in correct_examples]
)
incorrect_example_prompt = "\n".join(
[
f"<WrongExplanation>{example}</WrongExplanation>"
for example in incorrect_examples
]
)
""
return client.create(
model="gpt-4o",
response_model=ChainOfThought,
messages=[
{
"role": "system",
"content": dedent(
f"""
<prompt>
<role>system</role>
<context>
You are an expert question answering AI System.
You are about to be given some examples of incorrect
and correct reasoning for a question. You will then
be asked to correctly reason through another question
to generate a valid response.
</context>
<question>{example_prompt}</question>
<Explanations>
{correct_example_prompt}
{incorrect_example_prompt}
</Explanations>
<context>{context}</context>
<question>{query}</question>
</prompt>
"""
),
}
],
)
if __name__ == "__main__":
context = """
James writes a 3-page letter to 2
different friends twice a week.
"""
query = "How many pages does James write in a year?"
sample_question = """
James has 30 teeth. His dentist drills 4
of them and caps 7 more teeth than he drills.
What percentage of James' teeth does the dentist fix?
"""
incorrect_examples = [
"""James has 30 teeth. The dentist drills and caps some
teeth. Since drills are normally used on cars and not
teeth, it's safe to say none of the teeth were actually
fixed.""",
"""The dentist drills 4 teeth and caps 11 of them, which
means that he fixes 15 teeth. So we take 15 and multiply
it by the number of petals on a daisy, and the result is
30%, which is the percentage of teeth he fixes.""",
]
correct_examples = [
"""The dentist drills 4 teeth, so there are 30 - 4 = 26
teeth left. The dentist caps 7 more teeth than he drills,
so he caps 4 + 7 = 11 teeth. Therefore, the dentist fixes
a total of 4 + 11 = 15 teeth. To find the percentage of
teeth the dentist fixes, we divide the number of teeth
fixed by the total number of teeth and multiply by 100:
15/30 x 100 = 50%"""
]
response = contrastive_chain_of_thought(
query=query,
context=context,
example_prompt=sample_question,
correct_examples=correct_examples,
incorrect_examples=incorrect_examples,
)
print(response.model_dump_json(indent=2))
"""
{
"chain_of_thought": "First, let's determine how many pages James writes per week.
He writes a 3-page letter to 2 different friends, so for one writing session, he
writes 3 pages x 2 friends = 6 pages. He does this twice a week, so the total number
of pages written per week is 6 pages/session x 2 sessions/week = 12 pages/week. \n\n
Next, we need to find out how many weeks are in a year. There are 52 weeks in a year,
so we multiply the number of pages James writes per week by the number of weeks in a year:
12 pages/week x 52 weeks/year = 624 pages/year.\n\nTherefore, James writes 624 pages in a year.",
"correct_answer": "624"
}
"""
```
### References
<sup id="ref-1">1</sup>: [Contrastive Chain-of-Thought Prompting](https://arxiv.org/pdf/2311.09277)

7
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_few_shot/memory_of_thought.md Normal file
View File

@@ -0,0 +1,7 @@
---
title: ""
description: ""
keywords: ""
---
[wip]

76
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_few_shot/prompt_mining.md Normal file
View File

@@ -0,0 +1,76 @@
---
description: "We get a LLM to generate prompts"
---
Large Language Models are sensitive to the way that they are prompted. When prompted incorrectly, they might perform much worse despite having the information or capability to respond to the prompt. Prompt Mining aims to help us discover better formats that occur more frequently in the corpus.
Here are some examples of mined completions that were provided in the paper.
| Manual Prompts | Mined Prompts |
| ----------------------------------- | ----------------------- |
| x is affiliated with the y religion | x who converted to y |
| The headquarter of x is in y | x is based in y |
| x died in y | x died at his home in y |
| x is represented by music label y | x recorded for y |
| x is a subclass of y | x is a type of y |
> The original paper uses a large wikipedia corpus to automatically extract prompt templates by looking at middle words of the prompts and parsing the dependencies within the sentence. We present a more lightweight approach to help achieve a similar result with `instructor`.
We can implement Prompt Mining using `instructor` as seen below.
```python hl_lines="29-33"
from pydantic import BaseModel, Field
import instructor
class PromptTemplate(BaseModel):
prompt_template: str = Field(
description=(
"""
A template that has the subject and object that we
want to extract from the prompt replaced with a
single placeholder of {subject} and {object}.
Rephrase the prompt if necessary to make it more
concise and easier to understand
"""
),
)
client = instructor.from_provider("openai/gpt-5-nano")
def generate_prompt_templates(prompt: str):
return client.create(
messages=[
{
"role": "system",
"content": (
"You are an expert prompt miner that excels at "
"generating prompt templates which are more "
"concise and easier to understand\n\nYou are "
"about to be passed a prompt to extract 3 new "
"prompt templates for"
),
},
{"role": "system", "content": prompt},
],
response_model=list[PromptTemplate],
temperature=0,
max_retries=3,
model="gpt-4o",
)
if __name__ == "__main__":
prompt = "France is the capital of Paris"
prompt_template = generate_prompt_templates(prompt)
for prompt in prompt_template:
print(prompt)
#> prompt_template='{subject} is the capital of {object}'
#> prompt_template='The capital of {object} is {subject}'
#> prompt_template="{object}'s capital is {subject}"
```
### References
<sup id="ref-1">1</sup>: [How Can We Know What Language Models Know? ](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00324/96460/How-Can-We-Know-What-Language-Models-Know)

103
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_few_shot/uncertainty_routed_cot.md Normal file
View File

@@ -0,0 +1,103 @@
---
description: "Uncertainty Routed Chain Of Thought is a technique used in the Gemini Paper to improve upon the conventional Chain Of Thought approach"
---
Uncertainty-Routed Chain Of Thought<sup><a href="https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf">1</a></sup> prompting generates multiple chain of thought reasoning chains ( This is either 8 or 32 in the original paper ).
It then takes the majority answer out of these chains as the final solution only if the proportion of chains that agreed on this answer are higher than a specific threshold.
We can implement this using `instructor` as seen below.
```python hl_lines="74-87"
from pydantic import BaseModel
import instructor
from textwrap import dedent
from typing import Literal
import asyncio
from collections import Counter
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class ChainOfThoughtResponse(BaseModel):
chain_of_thought: str
correct_answer: Literal["A", "B", "C", "D"]
async def generate_response(query: str, options: dict[str, str]):
formatted_options = "\n".join(
[f"{key}:{answer}" for key, answer in options.items()]
)
return await client.create(
model="gpt-4o",
response_model=ChainOfThoughtResponse,
messages=[
{
"role": "system",
"content": dedent(
f"""
You are a a world class AI who excels at answering
complex questions. Choose one of the options below
that best answers the question you are about to be
asked
<question>
{query}
</question>
<options>
{formatted_options}
</options>
"""
),
}
],
)
async def generate_batch_responses(
query: str, options: dict[str, str], num_chains: int
) -> list[ChainOfThoughtResponse]:
coros = [generate_response(query, options) for _ in range(num_chains)]
return await asyncio.gather(*coros)
if __name__ == "__main__":
question = """In a population of giraffes, an environmental
change occurs that favors individuals that are tallest. As a
result, more of the taller individuals are able to obtain
nutrients and survive to pass along their genetic information.
This is an example of"""
options = {
"A": "directional selection",
"B": "stabilizing selection",
"C": "sexual selection",
"D": "disruptive selection",
}
correct_answer = "A"
k = 8
threshold = 0.6
responses = asyncio.run(generate_batch_responses(question, options, k))
votes = Counter([response.correct_answer for response in responses])
print(votes)
#> Counter({'A': 8})
majority_vote_element, majority_vote_count = votes.most_common(1)[0]
print(majority_vote_element, majority_vote_count)
#> A 8
majority_threshold = majority_vote_count / k
if majority_threshold < threshold:
response = asyncio.run(generate_response(question, options))
response = response.correct_answer
else:
response = majority_vote_element
print(response)
#> A
```
### References
<sup id="ref-1">1</sup>: [Gemini: A Family of Highly Capable Multimodal Models](https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf)

124
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_zero_shot/analogical_prompting.md Normal file
View File

@@ -0,0 +1,124 @@
---
description: "Analogical Prompting aims to help improve model accuracy by getting a model to generate relevant exemplars before solving the problem"
---
Analogical Prompting<sup><a href="https://arxiv.org/pdf/2310.01714">1</a></sup> is a method that aims to get LLMs to generate examples that are relevant to the problem before starting to address the user's query.
This takes advantage of the various forms of knowledge that the LLM has acquired during training and explicitly prompts them to recall the relevant problems and solutions. We can use Analogical Prompting using the following template
![](../../../img/analogical_prompting.png)
!!! example "Analogical Prompting Prompt Template"
Problem: [User Prompt]
Relevant Problems: Recall three relevant and distinct problems. For each problem, describe it and explain the solution
Solve the problem
We can implement this using `instructor` as seen below with some slight modifications.
```python hl_lines="33-36"
from pydantic import BaseModel, Field
import instructor
from textwrap import dedent
client = instructor.from_provider("openai/gpt-5-nano")
class RelevantProblem(BaseModel):
problem_explanation: str
solution: str
class Response(BaseModel):
relevant_problems: list[RelevantProblem] = Field(
max_length=3,
min_length=3,
)
answer: RelevantProblem
def analogical_prompting(query: str):
return client.create(
messages=[
{
"role": "user",
"content": dedent(
f"""
<problem>
{query}
</problem>
Relevant Problems: Recall three relevant and
distinct problems. For each problem, describe
it and explain the solution before solving
the problem
"""
),
}
],
model="gpt-4o",
response_model=Response,
)
if __name__ == "__main__":
query = (
"What is the area of the square with the four "
"vertices at (-2, 2), (2, -2), (-2, -6), and "
"(-6, -2)?"
)
response = analogical_prompting(query)
for problem in response.relevant_problems:
print(problem.model_dump_json(indent=2))
"""
{
"problem_explanation": "Determine the distance
between two points in a coordinate plane.",
"solution": "To find the distance between two
points, use the distance formula: \\(d =
\\sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2}\\). This
formula calculates the Euclidean distance between
points (x_1, y_1) and (x_2, y_2)."
}
"""
"""
{
"problem_explanation": "Calculate the area of a
square given its side length.",
"solution": "The area of a square can be found
using the formula: \\(A = s^2\\), where \\(s\\) is
the length of one side of the square."
}
"""
"""
{
"problem_explanation": "Identify vertices and
properties of a geometry shape such as
parallelogram.",
"solution": "For any quadrilateral, verify that
all sides are equal and angles are right angles to
confirm it is a square. Use properties of
quadrilaterals and distance formula."
}
"""
print(response.answer.model_dump_json(indent=2))
"""
{
"problem_explanation": "Calculate the area of a
square given its vertices.",
"solution": "First, confirm the shape is a square by
checking the distance between consecutive vertices
and ensuring all sides are of equal length using the
distance formula. For vertices (-2,2), (2,-2),
(-2,-6), and (-6,-2), calculate distances between
consecutive points. If distances are equal, use the
side length to compute area using \\(A = s^2\\)."
}
"""
```
### References
<sup id="ref-1">1</sup>: [Large Language Models As Analogical Reasoners](https://arxiv.org/pdf/2310.01714)

149
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_zero_shot/step_back_prompting.md Normal file
View File

@@ -0,0 +1,149 @@
---
description: "Step-back prompting is a two-step prompting technique that asks the LLM a step-back question to gather context for the query"
---
How can we encourage an LLM to think through any high-level context required to answer a query? Step-back prompting encourages this in two steps:
1. **Abstraction**: Ask the LLM a generic, higher-level concept. This is generally topic-specific. This is known as the _step-back question_.
2. **Reasoning**: Ask the LLM the original question, given its answer to the abstract question. This is known as _abstracted-grounded reasoning_.
!!! example "Step-Back Prompting Example"
**Original Question**: What happens to the pressure of an ideal gas when temperature and volume are increased?
**Step-Back Question**: What are the physics concepts associated with this question?
**Reasoning Prompt**: {step-back response} {original question}
Note that the step-back question is also generated using an LLM query.
Step-back prompting has been shown to improve scores on reasoning benchmarks for PaLM-2L and GPT-4.<sup><a href="https://arxiv.org/abs/2406.06608">\*</a></sup>
```python
import openai
import instructor
from pydantic import BaseModel
from typing import Iterable, Literal
client = instructor.from_provider("openai/gpt-5-nano")
class Stepback(BaseModel):
original_question: str
abstract_question: str
class Education(BaseModel):
degree: Literal["Bachelors", "Masters", "PhD"]
school: str
topic: str
year: int
class Response(BaseModel):
school: str
def generate_stepback_question():
return client.create(
model="gpt-4o",
response_model=Stepback,
messages=[
{
"role": "user",
"content": f"""
You are an expert at world knowledge. Your task is to step back
and paraphrase a question to a more generic step-back question,
which is easier to answer.
Here are a few examples:
Original Question: Which position did Knox Cunningham hold from
May 1955 to Apr 1956?
Step-back Question: Which positions has Knox Cunningham held in
his career?
Original Question: Who was the spouse of Anna Karina from 1968
to 1974?
Step-back Question: Who were the spouses of Anna Karina?
Original Question: Which team did Thierry Audel play for from
2007 to 2008?
Step-back Question: Which teams did Thierry Audel play for in
his career?
Now, generate the step-back question for the following question:
Estella Leopold went to which school between Aug 1954 and
Nov 1954?
""",
},
],
)
def ask_stepback_question(stepback):
return client.create(
model="gpt-4o",
response_model=Iterable[Education],
messages=[
{"role": "user", "content": stepback.abstract_question},
],
)
def get_final_response(stepback, stepback_response):
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "user",
"content": f"""
Q: {stepback.abstract_question},
A: {stepback_response}
Q: {stepback.original_question}
A:
""",
},
],
)
if __name__ == "__main__":
# Generate the step-back question
stepback = generate_stepback_question()
print(stepback.original_question)
#> Estella Leopold went to which school between Aug 1954 and Nov 1954?
print(stepback.abstract_question)
#> Which schools did Estella Leopold attend in her life?
# Ask the step-back question
stepback_response = ask_stepback_question(stepback)
for item in stepback_response:
print(item)
"""
degree='Bachelors'
school='University of Wisconsin-Madison'
topic='Botany'
year=1948
"""
"""
degree='Masters'
school='University of California, Berkeley'
topic='Botany and Paleobotany'
year=1950
"""
"""
degree='PhD'
school='Yale University'
topic='Botany and Paleobotany'
year=1955
"""
# Ask the original question, appended with context from the stepback response
print(get_final_response(stepback, stepback_response))
#> school='Yale University'
```
### References
<sup id="ref-1">1</sup>: [Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models](https://arxiv.org/abs/2310.06117)
<sup id="ref-asterisk">\*</sup>: [The Prompt Report: A Systematic Survey of Prompting Techniques](https://arxiv.org/abs/2406.06608)

122
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_zero_shot/tab_cot.md Normal file
View File

@@ -0,0 +1,122 @@
---
description: "Tab-CoT encourages LLMs to output reasoning as a markdown table, improving the structure and reasoning of its output"
---
By getting language models to output their reasoning as a structured markdown table, we can improve their reasoning capabilities and the quality of their outputs. This is known as Tabular Chain Of Thought (Tab-CoT) <sup><a href="https://arxiv.org/pdf/2305.17812">1</a></sup>.
We can implement this using `instructor` as a response object as seen below to ensure we get exactly the data that we want. Each row in our table is represented here as a `ReasoningStep` object.
```python hl_lines="36-38"
import instructor
from pydantic import BaseModel, Field
from textwrap import dedent
client = instructor.from_provider("openai/gpt-5-nano")
class ReasoningStep(BaseModel):
step: int = Field(description="The step number")
subquestion: str = Field(description="Subquestion to solve")
procedure: str = Field(
description="""Any intermediate computation
that was done in the reasoning process. Leave
empty if no computation is needed""",
)
result: str
class Response(BaseModel):
reasoning: list[ReasoningStep] = Field(
description="reasoning steps to derive answer",
)
correct_answer: int
def generate_structured_reasoning_response(query: str, context: str):
response = client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "system",
"content": dedent(
f"""
<system>
<role>expert Question Answering system</role>
<instruction>Make sure to output your reasoning in structured reasoning steps before generating a response to the user's query.</instruction>
</system>
<context>
{context}
</context>
<query>
{query}
</query>
"""
),
},
],
)
return response
if __name__ == "__main__":
query = "How many loaves of bread did they have left?"
context = """
The bakers at the Beverly Hills Bakery baked
200 loaves of bread on Monday morning. They
sold 93 loaves in the morning and 39 loaves
in the afternoon. A grocery store returned 6
unsold loaves.
"""
response = generate_structured_reasoning_response(query, context)
print(response.model_dump_json(indent=2))
"""
{
"reasoning": [
{
"step": 1,
"subquestion": "How many loaves of bread were sold in the morning
and afternoon?",
"procedure": "93 (morning) + 39 (afternoon)",
"result": "132"
},
{
"step": 2,
"subquestion": "How many loaves of bread were originally baked?",
"procedure": "",
"result": "200"
},
{
"step": 3,
"subquestion": "How many loaves of bread were returned by the
grocery store?",
"procedure": "",
"result": "6"
},
{
"step": 4,
"subquestion": "How many loaves of bread were left after accounting
for sales and returns?",
"procedure": "200 (originally baked) - 132 (sold) + 6 (returned)",
"result": "74"
}
],
"correct_answer": 74
}
"""
```
This generates the following reasoning step and the correct response of 74.
| Step | Subquestion | Procedure | Result |
| ---- | -------------------------------------------------------------------------- | -------------------------------------------------- | ------ |
| 1 | How many loaves of bread were sold in the morning and afternoon? | 93 (morning) + 39 (afternoon) | 132 |
| 2 | How many loaves of bread were originally baked? | | 200 |
| 3 | How many loaves of bread were returned by the grocery store? | | 6 |
| 4 | How many loaves of bread were left after accounting for sales and returns? | 200 (originally baked) - 132 (sold) + 6 (returned) | 74 |
### References
<sup id="ref-1">1</sup>: [Tab-CoT: Zero-shot Tabular Chain of Thought](https://arxiv.org/pdf/2305.17812)

129
참고/instructor-main/docs/prompting/thought_generation/chain_of_thought_zero_shot/thread_of_thought.md Normal file
View File

@@ -0,0 +1,129 @@
---
description: "Thread of Thought helps models ignore irrelevant context in their prompt, improving overall response quality and relevance"
---
By encouraging our model to examine each source in the provided context, we can help mitigate the impact of irrelevant context. This improves reasoning performance and the final output. This is known as Thread Of Thought <sup><a href="https://arxiv.org/pdf/2311.08734">1</a></sup>.
We can implement Thread Of Thought using the following template.
!!! example "Thread Of Thought template"
**[ Input Prompt ]**
Proceed through the context systematically, zeroing in on areas that could provide the answers we’re seeking
We can implement this using `instructor` as seen below.
```python hl_lines="42-43"
import instructor
from pydantic import BaseModel, Field
from textwrap import dedent
client = instructor.from_provider("openai/gpt-5-nano")
class ThreadOfThoughtResponse(BaseModel):
analysis: list[str] = Field(
description="""An explanation for each relevant source explaining
its relevance and content""",
)
correct_answer: int
def analyze_context_and_generate_response(query: str, context: list[str]):
return client.create(
model="gpt-4o",
response_model=ThreadOfThoughtResponse,
messages=[
{
"role": "system",
"content": dedent(
f"""
You are an expert Question Answerer.
Here are all of the sources that you should refer to
for context:
{'\n'.join(context)}
"""
),
},
{
"role": "user",
"content": query,
},
{
"role": "assistant",
"content": dedent(
"""
Navigate through the context incrementally,
identifying and summarizing relevant portions.
"""
),
},
],
)
if __name__ == "__main__":
context = [
"The price of a house was $100,000 in 2024",
"""The Great Wall of China is not visible from space
with the naked eye""",
"""Honey never spoils; archaeologists have found pots
of honey in ancient Egyptian tombs that are over
3,000 years old""",
"""The world's oldest known living tree is over 5,000
years old and is located in California""",
"The price of a house was $80,000 in 2023",
]
query = "What was the increase in the price of a house from 2023 to 2024"
response = analyze_context_and_generate_response(query, context)
print(response.model_dump_json(indent=2))
"""
{
"analysis": [
"The price of a house was $80,000 in 2023",
"The price of a house was $100,000 in 2024"
],
"correct_answer": 20000
}
"""
```
## Useful Tips
Here are some alternative phrases that you can add to your prompt to generate a thread of thought before your model generates a response.
1. In a step-by-step manner, go through the context, surfacing important information that could be useful.
2. Walk me through this lengthy document segment by segment, focusing on each part's significance.
3. Guide me through the context part by part, providing insights along the way.
4. Divide the document into manageable parts and guide me through each one, providing insights as we move along.
5. Let's go through this document piece by piece, paying close attention to each section.
6. Take me through the context bit by bit, making sure we capture all important aspects.
7. Examine the document in chunks, evaluating each part critically before moving to the next.
8. Analyze the context by breaking it down into sections, summarizing each as we move forward.
9. Navigate through the context incrementally, identifying and summarizing relevant portions.
10. Proceed through the context systematically, zeroing in on areas that could provide the answers we're seeking.
11. Take me through this long document step-by-step, making sure not to miss any important details.
12. Analyze this extensive document in sections, summarizing each one and noting any key points.
13. Navigate through this long document by breaking it into smaller parts and summarizing each, so we don't miss anything.
14. Let's navigate through the context section by section, identifying key elements in each part.
15. Let's dissect the context into smaller pieces, reviewing each one for its importance and relevance.
16. Carefully analyze the context piece by piece, highlighting relevant points for each question.
17. Read the context in sections, concentrating on gathering insights that answer the question at hand.
18. Let's read through the document section by section, analyzing each part carefully as we go.
19. Let's dissect this document bit by bit, making sure to understand the nuances of each section.
20. Systematically work through this document, summarizing and analyzing each portion as we go.
21. Let's explore the context step-by-step, carefully examining each segment.
22. Systematically go through the context, focusing on each part individually.
23. Methodically examine the context, focusing on key segments that may answer the query.
24. Progressively sift through the context, ensuring we capture all pertinent details.
25. Take a modular approach to the context, summarizing each part before drawing any conclusions.
26. Examine each segment of the context meticulously, and let's discuss the findings.
27. Approach the context incrementally, taking the time to understand each portion fully.
28. Let's scrutinize the context in chunks, keeping an eye out for information that answers our queries.
29. Walk me through this context in manageable parts step by step, summarizing and analyzing as we go.
30. Let's take a segmented approach to the context, carefully evaluating each part for its relevance to the questions posed.
### References
<sup id="ref-1">1</sup>: [Thread of Thought Unraveling Chaotic Contexts](https://arxiv.org/pdf/2311.08734)

67
참고/instructor-main/docs/prompting/zero_shot/emotion_prompting.md Normal file
View File

@@ -0,0 +1,67 @@
---
title: "Emotion Prompting"
description: "Adding phrases with emotional significance to humans can help enhance the performance of a language model."
---
Do language models respond to emotional stimuli?
Adding phrases with emotional significance to humans can help enhance the performance of a language model. This includes phrases such as:
- This is very important to my career.
- Take pride in your work.
- Are you sure?
!!! info
For more examples of emotional stimuli to use in prompts, look into [EmotionPrompt](https://arxiv.org/abs/2307.11760) -- a set of prompts inspired by well-established human psychological phenomena.
## Implementation
```python hl_lines="34"
import openai
import instructor
from pydantic import BaseModel
from typing import Iterable
class Album(BaseModel):
name: str
artist: str
year: int
client = instructor.from_provider("openai/gpt-5-nano")
def emotion_prompting(query, stimuli):
return client.create(
model="gpt-4o",
response_model=Iterable[Album],
messages=[
{
"role": "user",
"content": f"""
{query}
{stimuli}
""",
}
],
)
if __name__ == "__main__":
query = "Provide me with a list of 3 musical albums from the 2000s."
stimuli = "This is very important to my career." # (1)!
albums = emotion_prompting(query, stimuli)
for album in albums:
print(album)
#> name='Kid A' artist='Radiohead' year=2000
#> name='The Marshall Mathers LP' artist='Eminem' year=2000
#> name='The College Dropout' artist='Kanye West' year=2004
```
1. The phrase `This is very important to my career` is used as emotional stimuli in the prompt.
## References
<sup id="ref-1">1</sup>: [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://arxiv.org/abs/2307.11760)

71
참고/instructor-main/docs/prompting/zero_shot/rar.md Normal file
View File

@@ -0,0 +1,71 @@
---
description: "To help the model better infer human intention from ambigious prompts, we can ask the model to rephrase and respond (RaR)."
---
How can we identify and clarify ambigious information in the prompt?
Let's say we are given the query: *Was Ed Sheeran born on an odd month?*
There are many ways a model might interpret an *odd month*:
- Februray is *odd* because of an irregular number of days.
- A month is *odd* if it has an odd number of days.
- A month is *odd* if its numberical order in the year is odd (i.e. Janurary is the 1st month).
!!! note
Ambiguities might not always be so obvious!
To help the model better infer human intention from ambigious prompts, we can ask the model to rephrase and respond (RaR).
## Implementation
```python hl_lines="19"
from pydantic import BaseModel
import instructor
client = instructor.from_provider("openai/gpt-5-nano")
class Response(BaseModel):
rephrased_question: str
answer: str
def rephrase_and_respond(query):
return client.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": f"""{query}\nRephrase and expand the question, and respond.""", # (1)!
}
],
response_model=Response,
)
if __name__ == "__main__":
query = "Take the last letters of the words in 'Edgar Bob' and concatinate them."
response = rephrase_and_respond(query)
print(response.rephrased_question)
"""
What are the last letters of each word in the name 'Edgar Bob', and what do you get when you concatenate them?
"""
print(response.answer)
"""
To find the last letters of each word in the name 'Edgar Bob', we look at 'Edgar' and 'Bob'. The last letter of 'Edgar' is 'r' and the last letter of 'Bob' is 'b'. Concatenating these letters gives us 'rb'.
"""
```
1. This prompt template comes from [this](https://arxiv.org/abs/2311.04205) paper.
This can also be implemented as two-step RaR:
1. Ask the model to rephrase the question.
2. Pass the rephrased question back to the model to generate the final response.
## References
<sup id="ref-1">1</sup>: [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://arxiv.org/abs/2311.04205)

55
참고/instructor-main/docs/prompting/zero_shot/re2.md Normal file
View File

@@ -0,0 +1,55 @@
---
description: "Re2 (Re-Reading) is a technique that asks the model to read the question again."
---
How can we enhance a model's understanding of a query?
Re2 (**Re** - **R** eading) is a technique that asks the model to read the question again.
!!! example "Re-Reading Prompting"
**Prompt Template**: Read the question again: <*query*> <*critical thinking prompt*><sup><a href="https://arxiv.org/abs/2309.06275">1</a></sup>
A common critical thinking prompt is: "Let's think step by step."
## Implementation
```python hl_lines="20"
import instructor
from pydantic import BaseModel
client = instructor.from_provider("openai/gpt-5-nano")
class Response(BaseModel):
answer: int
def re2(query, thinking_prompt):
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "system",
"content": f"Read the question again: {query} {thinking_prompt}",
},
],
)
if __name__ == "__main__":
query = """Roger has 5 tennis balls.
He buys 2 more cans of tennis balls.
Each can has 3 tennis balls.
How many tennis balls does he have now?
"""
thinking_prompt = "Let's think step by step."
response = re2(query=query, thinking_prompt=thinking_prompt)
print(response.answer)
#> 11
```
## References
<sup id="ref-1">1</sup>: [Re-Reading Improves Reasoning in Large Language Models](https://arxiv.org/abs/2309.06275)

70
참고/instructor-main/docs/prompting/zero_shot/role_prompting.md Normal file
View File

@@ -0,0 +1,70 @@
---
title: "Role Prompting"
description: "Role prompting, or persona prompting, assigns a role to the model."
---
How can we increase a model's performance on open-ended tasks?
Role prompting, or persona prompting, assigns a role to the model. Roles can be:
- **specific to the query**: *You are a talented writer. Write me a poem.*
- **general/social**: *You are a helpful AI assistant. Write me a poem.*
## Implementation
```python hl_lines="27"
import openai
import instructor
from pydantic import BaseModel
client = instructor.from_provider("openai/gpt-5-nano")
class Response(BaseModel):
poem: str
def role_prompting(query, role):
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "system",
"content": f"{role} {query}",
},
],
)
if __name__ == "__main__":
query = "Write me a short poem about coffee."
role = "You are a renowned poet."
response = role_prompting(query, role)
print(response.poem)
"""
In the morning's gentle light,
A brew of warmth, dark and bright.
Awakening dreams, so sweet,
In every sip, the day we greet.
Through the steam, stories spin,
A liquid muse, caffeine within.
Moments pause, thoughts unfold,
In coffee's embrace, we find our gold.
"""
```
!!! info "More Role Prompting"
To read about a systematic approach to choosing roles, check out [RoleLLM](https://arxiv.org/abs/2310.00746).
For more examples of social roles, check out [this](https://arxiv.org/abs/2311.10054) evaluation of social roles in system prompts..
To read about using more than one role, check out [Multi-Persona Self-Collaboration](https://arxiv.org/abs/2307.05300).
## References
<sup id="ref-1">1</sup>: [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Lanuage Models](https://arxiv.org/abs/2310.00746)
<sup id="ref-2">2</sup>: [Is "A Helpful Assistant" the Best Role for Large Language Models? A Systematic Evaluation of Social Roles in System Prompts ](https://arxiv.org/abs/2311.10054)
<sup id="ref-4">3</sup>: [Unleashing the Emergent Cognitive Synergy in Large Lanuage Models: A Task-Solving Agent through Multi-Persona Self-Collaboration ](https://arxiv.org/abs/2307.05300)

95
참고/instructor-main/docs/prompting/zero_shot/s2a.md Normal file
View File

@@ -0,0 +1,95 @@
---
title: "System 2 Attention (S2A)"
description: "The S2A (System 2 Attention) technique auto-refines a prompt by asking the model to rewrite the prompt to include only relevant information."
---
How do we remove irrelevant information from the prompt?
The S2A (System 2 Attention) technique auto-refines a prompt by asking the model to rewrite the prompt to include only *relevant* information. We implement this in two steps:
1. Ask the model to rewrite the prompt
2. Pass the rewritten prompt back to the model
## Implementation
```python hl_lines="25-28"
import openai
import instructor
from pydantic import BaseModel, Field
client = instructor.from_provider("openai/gpt-5-nano")
class Step1(BaseModel):
relevant_context: str = Field(..., description="Relevant context")
user_query: str = Field(..., description="The question from the user")
class Step2(BaseModel):
answer: int
def rewrite_prompt(query):
rewritten_prompt = client.create(
model="gpt-4o",
response_model=Step1,
messages=[
{
"role": "user",
"content": f"""
Given the following text by a user, extract the part
that is actually relevant to their question. Please
include the actual question or query that the user
is asking.
Text by user:
{query}
""", # (1)!
}
],
)
return rewritten_prompt
def generate_final_response(rewritten_prompt):
final_response = client.create(
model="gpt-4o",
response_model=Step2,
messages=[
{
"role": "user",
"content": f"""{rewritten_prompt.relevant_context}
Question: {rewritten_prompt.user_query}""",
}
],
)
return final_response
if __name__ == "__main__":
query = """Mary has 3 times as much candy as Megan.
Mary then adds 10 more pieces of candy to her collection.
Max is 5 years older than Mary.
If Megan has 5 pieces of candy, how many does Mary have in total?
"""
# Step 1: Rewrite the prompt
rewritten_prompt = rewrite_prompt(query)
print(rewritten_prompt.relevant_context)
"""
Mary has 3 times as much candy as Megan. Mary then adds 10 more pieces of candy to her collection. If Megan has 5 pieces of candy, how many does Mary have in total?
"""
print(rewritten_prompt.user_query)
#> how many does Mary have in total?
# Step 2: Generate the final response
final_response = generate_final_response(rewritten_prompt)
print(final_response.answer)
#> 25
```
1. This prompt template comes from [this](https://arxiv.org/abs/2311.11829) paper.
## References
<sup id="ref-1">1</sup>: [System 2 Attention (is something you might need too)](https://arxiv.org/abs/2311.11829)

77
참고/instructor-main/docs/prompting/zero_shot/self_ask.md Normal file
View File

@@ -0,0 +1,77 @@
---
title: "Self-Ask"
description: "Self-Ask is a technique which use a single prompt to encourage a model to use the answers to sub-problems to correctly generate the overall solution."
---
Models can sometimes correctly answer sub-problems but incorrectly answer the overall query. This is known as the *compositionality gap*<sup><a href="https://arxiv.org/abs/2210.03350">1</a></sup>.
How can we encourage a model to use the answers to sub-problems to correctly generate the overall solution?
Self-Ask is a technique which use a single prompt to:
- decide if follow-up questions are required
- generate the follow-up questions
- answer the follow-up questions
- answer the main query
## Implementation
```python hl_lines="26-29"
import instructor
from pydantic import BaseModel, Field
client = instructor.from_provider("openai/gpt-5-nano")
class FollowUp(BaseModel):
question: str = Field(description="The follow-up question")
answer: str = Field(description="The answer to the follow-up question")
class Response(BaseModel):
follow_ups_required: bool
follow_ups: list[FollowUp]
final_answer: str
def self_ask(query):
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "system",
"content": f"""Query: {query}
Are follow-up questions needed?
If so, generate follow-up questions, their answers, and then the final answer to the query.
""", # !(1)
},
],
)
if __name__ == "__main__":
query = "Who was president of the U.S. when superconductivity was discovered?"
response = self_ask(query)
print(response.follow_ups_required)
#> True
for follow_up in response.follow_ups:
print(follow_up)
"""
question='When was superconductivity discovered?' answer='Superconductivity was discovered in April 1911.'
"""
"""
question='Who was president of the U.S. in April 1911?' answer='William Howard Taft was the President of the United States in April 1911.'
"""
print(response.final_answer)
"""
William Howard Taft was president of the U.S. when superconductivity was discovered.
"""
```
1. Without `instructor`, this prompt would generally be implemented as a one-shot or few-shot prompt<sup><a href="https://arxiv.org/abs/2210.03350">1</a></sup> to encourage thinking through follow-up questions. With `instructor`, we use a zero-shot prompt!
## References
<sup id="ref-1">1</sup>: [Measuring and Narrowing the Compositionality Gap in Language Models](https://arxiv.org/abs/2210.03350)

103
참고/instructor-main/docs/prompting/zero_shot/simtom.md Normal file
View File

@@ -0,0 +1,103 @@
---
title: "SimToM (Simulated Theory of Mind)"
description: "SimToM (Simulated Theory of Mind) is a two-step prompting technique that encourages a model to consider a specific perspective."
---
How can we encourage the model to focus on relevant information?
SimToM (Simulated Theory of Mind) is a two-step prompting technique that encourages a model to consider a specific perspective.
This can be useful for complex questions with multiple entities. For example, if the prompt contains information about two individuals, we can ask the model to answer our query from the perspective of one of the individuals.
This is implemented in two steps. Given an entity:
1. Identify and isolate information relevant to the entity
2. Ask the model to answer the query from the entity's perspective
!!! example "Sample Template"
**Step 1**: Given the following context, list the facts that <*entity*> would know. Context: <*context*>
**Step 2**: You are <*entity*>. Answer the following question based only on these facts you know: <*facts*>. Question: <*query*>
## Implementation
```python hl_lines="24-25"
import openai
import instructor
from pydantic import BaseModel, Field
from typing import Iterable
client = instructor.from_provider("openai/gpt-5-nano")
class KnownFact(BaseModel):
fact: str = Field(description="A fact that the given entity would know")
class Response(BaseModel):
location: str
def generate_known_facts(entity, context, query) -> Iterable[KnownFact]:
return client.create(
model="gpt-4o",
response_model=Iterable[KnownFact],
messages=[
{
"role": "user",
"content": f"""Given the following context, list
the facts that {entity} would know:
Context:
{context}
{query}
List only the facts relevant to {entity}.
""",
}
],
)
def answer_question_based_on_facts(entity, query, known_facts) -> Response:
return client.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "system",
"content": f"""You are {entity}. Answer the following question
based only on these facts you know:
{" ".join([str(fact) for fact in known_facts])}""",
},
{
"role": "user",
"content": f"Question: {query}",
},
],
)
if __name__ == "__main__":
entity = "Alice"
context = """Alice puts the book on the table.
Alice leaves the room.
Bob moves the book to the shelf.
"""
query = f"Where does {entity} think the book is?"
known_facts = generate_known_facts(entity, context, query)
response = answer_question_based_on_facts(entity, query, known_facts)
for fact in known_facts:
print(fact)
#> fact='Alice puts the book on the table.'
#> fact='Alice leaves the room. Bob moves the book to the shelf.'
print(response.location)
#> On the table
```
## References
<sup id="ref-1">1</sup>: [Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities](https://arxiv.org/abs/2311.10227)

91
참고/instructor-main/docs/prompting/zero_shot/style_prompting.md Normal file
View File

@@ -0,0 +1,91 @@
---
title: "Style Prompting"
description: "To contrain a model's response to fit the boundaries of our task, we can specify a style."
---
How can we constrain model outputs through prompting alone?
To contrain a model's response to fit the boundaries of our task, we can specify a style.
Stylistic constraints can include:
- **writing style**: write a *flowery* poem
- **tone**: write a *dramatic* poem
- **mood**: write a *happy* poem
- **genre**: write a *mystery* poem
## Implementation
```python hl_lines="22"
import instructor
from pydantic import BaseModel
import openai
class Email(BaseModel):
subject: str
message: str
client = instructor.from_provider("openai/gpt-5-nano")
def generate_email(subject, to, sender, tone):
return client.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": f"""
Write an email about {subject} to {to} from {sender}.
The email should be {tone}.
""",
}
],
response_model=Email,
)
if __name__ == "__main__":
email = generate_email(
subject="invitation to all-hands on Monday at 6pm",
to="John Smith",
sender="Jane Doe",
tone="formal",
)
print(email.subject)
#> Invitation to All-Hands Meeting
print(email.message)
"""
Dear Mr. Smith,
I hope this message finds you well. I am writing to formally invite you to our upcoming all-hands meeting scheduled for Monday at 6:00 PM. This meeting is an important opportunity for us to come together, discuss key updates, and align on our strategic goals.
Please confirm your availability at your earliest convenience. Your presence and contributions to the discussion would be greatly valued.
Thank you and I look forward to your confirmation.
Warm regards,
Jane Doe
"""
```
## Stylistic Constraint Examples
| Constraint | Possible Phrases |
|----------------|-----------------------------------------------------------------------------------|
| Writing Style | Functional, Flowery, Candid, Prosaic, Ornate, Poetic |
| Tone | Dramatic, Humorous, Optimistic, Sad, Formal, Informal |
| Mood | Angry, Fearful, Happy, Sad |
| Genre | Historical Fiction, Literary Fiction, Science Fiction, Mystery, Dystopian, Horror |
!!! info "More Stylistic Constraints"
To see even more examples of these stylistic constraints and additional constraints (**characterization**, **pacing**, and **plot**), check out [this](https://arxiv.org/abs/2302.09185) paper.
## References
<sup id="ref-1">1</sup>: [Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints](https://arxiv.org/abs/2302.09185)