lasta/AI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

참고소스 수정본

Browse Source
This commit is contained in:
LASTA_DEV01\lasta
2026-05-12 19:40:31 +09:00
parent 0f34a451fc
commit 2e9204243d
8708 changed files with 3259488 additions and 869 deletions
Download Patch File Download Diff File Expand all files Collapse all files

270
참고/instructor-main/docs/prompting/ensembling/cosp.md Normal file
View File

@@ -0,0 +1,270 @@
---
description: "Consistency Based Self Adaptive Prompting (COSP) is a ensembling technique that aims to combine multiple Chain Of Thought reasoning calls"
---
Consistency Based Self Adaptive Prompting (COSP)<sup><a href="https://arxiv.org/pdf/2305.14106">1</a></sup> aims to improve LLM output quality by generating high quality few shot examples to be included in the final prompt. These are examples without labelled ground truth so they use self-consistency and a metric known as normalized entropy to select the best examples.
Once they've selected the examples, they then append them to the prompt and generate multiple reasoning chains before selecting the final result using [Self-Consistency](self_consistency.md).
## COSP process
![](../../img/cosp.png)
How does this look in practice? Let's dive into greater detail.
### Step 1 - Selecting Examples
In the first step, we try to generate high quality examples from questions that don't have ground truth labels. This is challenging because we want to find a way to automatically determine answer quality when sampling our model multiple times.
In this case, we have `n` questions which we want to generate `m` possible reasoning chains for each question. This gives a total of `nm` examples. We then want to filter out `k` final few shot examples from these `nm` examples to be included inside our final prompt.
1. Using chain of thought, we first generate `m` responses for each question. These responses contain a final answer and a rationale behind that answer.
2. We compute a score for each response using a weighted sum of two values - normalized entropy and repetitiveness ( How many times this rationale appears for this amswer )
3. We rank all of our `nm` responses using this score and choose the `k` examples with the lowest scores as our final few shot examples.
#### Normalized Entropy
> In the paper, the authors write that normalized entropy is a good proxy over a number of different tasks where low entropy is positively correlated with correctness. Entropy is also supposed to range from 0 to 1.
>
> Therefore in order to do so, we introduce a `-` term in our implementation so that the calculated values range from 0 to 1.
![](../../img/cosp_entropy.png)
Assuming that for a specific question $x^{(i)}$, we have generated $m$ final answers of which $u$ are unique. ( Note that this only cares about the answer itself and not the rationale )
$$
\mathcal{H}\left(x^{(i)} \mid \left\{\hat{y}_j^{(i)}\right\}_{j=1}^m\right) = \frac{\sum_{\alpha=1}^u \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right) \log \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)}{\log m},
$$
We can measure the entropy of the generated responses using the formula above where
- $x_i$ is the original question that we prompted the model with
- $y_j^{i}$ represents the $i$-th sampled response from the $m$ that we generated
- $\hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)$ is the frequency of the unique answer in all the $m$ generated answers. (Eg. if we generate 8 responses and 4 of them return the value 10, then $\hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)$ is just going to be 0.5)
#### Repetitiveness
$$
R_r(r_j^{(i)}) = \frac{2}{Q(Q-1)} \sum_{a=1}^{Q} \sum_{b=a+1}^{Q} W_{ab}
$$
In the formula above, $Q$ refers to the number of phrases in the sentence and $W_{ab}$ refers to the cosine similarity of two phrases $a$ and $b$.
Repetitiveness aims to measure how often the language model repeats itself. To do so, the paper sums up the cosine similarity between each sentence inside the generated chain of thought rationale before normalizing it.
The intuition behind this is that high repetitiveness indicates redundancy, which can lead to poorer performance. Therefore responses with a high number of similar sentences will have a larger score for repetitiveness ( since cosine similarity will be larger for each sentence ).
### Step 2 - Self Consistency
We now take our `k` responses and append them to our prompt. We then sample our model multiple times using this new prompt and take the majority vote as the answer.
## Implementation
Now that we understand what COSP is, let's see how we can implement it in instructor. Note that here we'll measure repetitiveness using cosine similarity between sentence embeddings.
```python
import instructor
from pydantic import BaseModel
from openai import AsyncOpenAI, OpenAI
from collections import defaultdict, Counter
import asyncio
from textwrap import dedent
import math
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Response(BaseModel):
chain_of_thought: list[str]
answer: int
class ResponseScore(BaseModel):
query: str
response: Response
score: float
def format_response(self):
return dedent(
f"""
Q: {self.query}
A: {''.join(self.response.chain_of_thought)}. Therefore the answer is {self.response.answer}.
"""
)
def cosine_similarity(vec1: list[float], vec2: list[float]):
dot_product = sum(a * b for a, b in zip(vec1, vec2))
magnitude1 = math.sqrt(sum(a * a for a in vec1))
magnitude2 = math.sqrt(sum(b * b for b in vec2))
if magnitude1 * magnitude2 == 0:
return 0 # Handle the case of zero vectors
return dot_product / (magnitude1 * magnitude2)
def score_repetitiveness(prediction: Response):
if len(prediction.chain_of_thought) == 1:
return 0
embedding = OpenAI().embeddings.create(
input=prediction.chain_of_thought, model="text-embedding-3-small"
)
embedding = [item.embedding for item in embedding.data]
ttl = 0
num_comparisons = 0
for idx in range(len(embedding)):
for idx2 in range(idx + 1, len(embedding)):
ttl += cosine_similarity(embedding[idx], embedding[idx2])
num_comparisons += 1
return ttl / num_comparisons if num_comparisons > 0 else 0
async def generate_cot_response(query: str) -> tuple[Response, str]:
return (
await client.create(
model="gpt-4o",
messages=[{"role": "user", "content": query}],
response_model=Response,
temperature=0.4,
),
query,
)
async def generate_batch_cot_responses(
queries: list[str], m: int
) -> list[tuple[Response, str]]:
coros = [generate_cot_response(query) for query in queries for _ in range(m)]
return await asyncio.gather(*coros)
def score_entropy(predictions: list[Response]):
counter = Counter([prediction.answer for prediction in predictions])
prob = [counter[i] / len(predictions) for i in counter]
numer = -sum([p * math.log(p) for p in prob])
denom = math.log(len(predictions))
return numer / denom
def score_responses(
predictions: list[tuple[Response, str]], trade_off_param: float
) -> list[ResponseScore]:
query_to_responses: dict[str, list[Response]] = defaultdict(list)
for prediction, query in predictions:
query_to_responses[query].append(prediction)
query_to_entropy = {
query: score_entropy(predictions)
for query, predictions in query_to_responses.items()
}
return [
ResponseScore(
query=query,
response=prediction,
score=query_to_entropy[query]
+ trade_off_param * score_repetitiveness(prediction),
)
for prediction, query in predictions
]
def get_top_k_examples(queries: list[ResponseScore], k: int):
"""
This gets the top k responses that have the minimum possible score
"""
sorted_responses = sorted(queries, key=lambda x: x.score)
return sorted_responses[:k]
async def generate_answer_with_examples(query: str, examples: list[ResponseScore]):
formatted_examples = "\n".join([example.format_response() for example in examples])
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": dedent(
f"""
You are a world class AI system that excels at answering user queries
<query>
{query}
</query>
<examples>
{formatted_examples}
</examples>
"""
),
}
],
response_model=Response,
)
async def generate_final_answers(
query: str, examples: list[ResponseScore], number_samples: int
):
coros = [
generate_answer_with_examples(query, examples) for _ in range(number_samples)
]
return await asyncio.gather(*coros)
if __name__ == "__main__":
query = (
"The schools debate team had 5 boys and 40 girls on it. "
"If they were split into groups of 9 how many groups "
"could they make?"
)
example_questions = [
(
"Debby's class is going on a field trip to the zoo. "
"If each van can hold 4 people and there are 2 students "
"and 6 adults going, how many vans will they need?"
),
(
"Nancy had 80 files on her computer. She deleted 31 of "
"them and put the rest into folders with 7 files in each "
"one. How many folders did Nancy end up with?"
),
(
"At the arcade, Tom won 32 tickets playing 'whack a mole' "
"and 25 tickets playing 'skee ball'. If he spent 7 of his "
"tickets on a hat, how many tickets does Tom have left?"
),
]
m = 2 # Number of Reasoning Chains per example ( Step 1 )
k = 3 # Number of Examples to include in final prompt (Step 2)
n = 2 # Number of Reasoning Chains For Self-Consistency ( Step 2 )
# Step 1 : Generate the examples
responses = asyncio.run(generate_batch_cot_responses(example_questions, m))
scored_responses = score_responses(responses, 0.2)
chosen_examples = get_top_k_examples(scored_responses, k)
# Step 2 : Run Self-Consistency
final_responses = asyncio.run(generate_final_answers(query, chosen_examples, n))
c = Counter([response.answer for response in final_responses])
answer = c.most_common(1)[0][0]
print(answer)
#> 5
```
### References
<sup id="ref-1">1</sup>: [Better Zero-Shot Reasoning with Self-Adaptive Prompting](https://arxiv.org/pdf/2305.14106)

103
참고/instructor-main/docs/prompting/ensembling/dense.md Normal file
View File

@@ -0,0 +1,103 @@
---
description: "Demonstration Ensembling(DENSE) creates multiple few-shot prompts, each containing a distinct subset of examples from the training set. We then use that to generate a final response"
---
We can maximise the use of our examples by prompting our model multiple times, each time using a different subset of examples. We can then take these multiple outputs and aggregate over them to generate a final response. This is known as Demonstration Ensembling ( DENSE ) <sup><a href="https://arxiv.org/pdf/2308.08780">1</a></sup>.
> For simplicity in this example, we simply iterate over the examples and partition them equally to get equally sized clusters. However, depending on your use-case you might also want to consider sampling these using some form of embedding clusering.
We can implement this using `instructor` as seen below.
```python hl_lines="26-41"
import instructor
from pydantic import BaseModel
import asyncio
from collections import Counter
from typing import Literal
from textwrap import dedent
class DemonstrationResponse(BaseModel):
correct_answer: Literal["Positive", "Negative", "Neutral"]
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
async def generate_self_consistent_response(prompt: str, examples: list[str]):
concetenated_examples = "\n".join(examples)
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": dedent(
f"""
You are an intelligent AI System that excels
at classifying user queries into three
possible labels:
- Positive
- Negative
- Neutral
You are about to be given a user query and
asked to classify it into one of the three
categories. Make sure to refer closely to
the examples provided to you, examining each
individual example before coming up with the
final answer.
Here are the examples:
{concetenated_examples}
"""
),
},
{"role": "user", "content": prompt},
],
response_model=DemonstrationResponse,
temperature=0,
)
async def generate_self_consistent_responses(
prompt: str, num_responses: int, examples: list[str]
):
assert (
len(examples) % num_responses == 0
), "The number of examples must be evenly divisible by num_responses"
# Batch the examples into num_responses batches
batch_size = len(examples) // num_responses
coros = [
generate_self_consistent_response(prompt, examples[i : i + batch_size])
for i in range(0, len(examples), batch_size)
]
responses = await asyncio.gather(*coros)
return responses
if __name__ == "__main__":
user_query = "What is the weather like today?"
examples = [
"I love this product! [Positive]",
"This is the worst service ever. [Negative]",
"The movie was okay, not great but not terrible. [Neutral]",
"I'm so happy with my new phone! [Positive]",
"The food was terrible and the service was slow. [Negative]",
"It's an average day, nothing special. [Neutral]",
"Fantastic experience, will come again! [Positive]",
"I wouldn't recommend this to anyone. [Negative]",
"The book was neither good nor bad. [Neutral]",
"Absolutely thrilled with the results! [Positive]",
]
responses = asyncio.run(generate_self_consistent_responses(user_query, 5, examples))
answer_counts = Counter([response.correct_answer for response in responses])
most_common_answer, _ = answer_counts.most_common(1)[0]
print(most_common_answer)
#> Neutral
```
### References
<sup id="ref-1">1</sup>: [Exploring Demonstration Ensembling for In Context Learning](https://arxiv.org/pdf/2308.08780)

185
참고/instructor-main/docs/prompting/ensembling/diverse.md Normal file
View File

@@ -0,0 +1,185 @@
---
description: "Diverse creates multiple prompts for a given problem before performing self-consistency for each. It then generates multiple reaosning paths before choosing the best final response"
---
Diverse Verifier On Reasoning Step (DiVeRSe)<sup><a href="https://aclanthology.org/2023.acl-long.291/">1</a></sup> is a prompting technique which provides two main improvements
1. **Diverse Prompts** : They generate multiple variations of the same prompt by varying the examples used in each prompt
2. **Verification** : They use a finetuned `Deberta-V3-Large` to determine the quality of a generated response. Instead of using majority voting, they use their model to score each generated response from 0 to 1. They then aggregate these scores for each unique answer to determine the best generated solution.
In the paper itself, they also train a step-wise verifier that is able to score each individual reasoning step. This enables much more fine-grained predictions but is challenging to obtain training data for.
We can implement this in `instructor`. However, instead of using a `deberta-v3-large` model, we'll be using gpt-4o to score its own outputs and generate a quality score.
```python
import instructor
from pydantic import BaseModel
from typing import Literal
from textwrap import dedent
import asyncio
from collections import defaultdict
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Response(BaseModel):
chain_of_thought: str
answer: int
class Grading(BaseModel):
grade: Literal["Poor", "Average", "Good", "Excellent"]
def get_score(self):
mapping = {
"Poor": 0.25,
"Average": 0.5,
"Good": 0.75,
"Excellent": 1,
}
return mapping[self.grade]
async def generate_response(query: str, examples: list[str]):
formatted_examples = "\n".join(examples)
return await client.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": dedent(
f"""
You are a world class AI that excels at answering
user queries in a succint and accurate manner.
<query>
{query}
</query>
<examples>
{formatted_examples}
</examples>
"""
),
}
],
response_model=Response,
)
async def score_response(query: str, response: Response) -> tuple[Response, Grading]:
return (
response,
await client.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": dedent(
f"""
You are a world class AI that excels at grading
responses to a user query in a succint and clear
manner.
<query>
{query}
</query>
<response>
{response}
</response>
"""
),
}
],
response_model=Grading,
),
)
async def generate_response_batch(
query: str, examples: list[str], n_examples_per_batch: int
):
batches: list[list[str]] = []
for i in range(0, len(examples), n_examples_per_batch):
batches.append(examples[i : i + n_examples_per_batch])
coros = [generate_response(query, example_batch) for example_batch in batches]
return await asyncio.gather(*coros)
async def score_responses(
query: str, responses: list[Response]
) -> list[tuple[Response, Grading]]:
coros = [score_response(query, response) for response in responses]
return await asyncio.gather(*coros)
if __name__ == "__main__":
examples = [
"""
Q: James decides to run 3 sprints 3 times a week.
He runs 60 meters each sprint. How many total
meters does he run a week?
A: James decides to run 3 sprints 3 times a week.
He runs 60 meters each sprint. So he runs 60 meters
x 3 sprints x 3 times a week. That is 60 meters x 9.
The answer is 540.
""",
"""
Q: Brandon's iPhone is four times as old as Ben's
iPhone. Ben's iPhone is two times older than Suzy's
iPhone. If Suzy's iPhone is 1 year old, how old is
Brandon's iPhone?
A: Brandon's iPhone is 4 times as old as Ben's
iPhone. Ben's iPhone is 2 times older than Suzy's
iPhone. So Brandon's iPhone is 4 x 2 = 8 times older
than Suzy's iPhone. Suzy's iPhone is 1 year old. So
Brandon's iPhone is 8 x 1 = 8 years old. The answer
is 8.
""",
"""
Q: Jean has 30 lollipops. Jean eats 2 of the
lollipops. With the remaining lollipops, Jean wants
to package 2 lollipops in one bag. How many bags can
Jean fill?
A: Jean started with 30 lollipops. She ate 2 of
them. So she has 28 lollipops left. She wants to
package 2 lollipops in one bag. So she can package
28 / 2 = 14 bags. The answer is 14.
""",
"""
Q: Weng earns $12 an hour for babysitting.
Yesterday, she just did 50 minutes of babysitting.
How much did she earn?
A: Weng earns 12/60 = $<<12/60=0.2>>0.2 per minute.
Working 50 minutes, she earned 0.2 x 50 =
$<<0.2*50=10>>10. The answer is 10
""",
]
query = """Betty is saving money for a new wallet which
costs $100. Betty has only half of the money she needs.
Her parents decided to give her $15 for that purpose,
and her grandparents twice as much as her parents. How
much more money does Betty need to buy the wallet?"""
generated_responses = asyncio.run(generate_response_batch(query, examples, 1))
scored_responses = asyncio.run(score_responses(query, generated_responses))
scores: dict[int, float] = defaultdict(int)
for response, grade in scored_responses:
scores[response.answer] += grade.get_score()
print(scores)
#> defaultdict(<class 'int'>, {5: 3.5})
answer = max(scores, key=scores.get)
print(answer)
#> 5
```
### References
<sup id="ref-1">1</sup>: [Making Language Models Better Reasoners with Step-Aware Verifier](https://aclanthology.org/2023.acl-long.291/)

259
참고/instructor-main/docs/prompting/ensembling/max_mutual_information.md Normal file
View File

@@ -0,0 +1,259 @@
---
description: "Max Mutual Information creates multiple prompt templates and then selects the optimal template as the one which maximises mutual information between the prompt and the LLM's outputs"
---
## What's Max Mutual Information?
Max Mutual Information Method is a method of prompting that aims to find the best prompt to elicit the desired response from a LLM. We do so by maximising a metric called Mutual Information - which indicates the reduction in a model's uncertainty as a result of the prompt.
### Entropy
When a language model recieves a prompt as input, it outputs a series of token probabilities sequentially until it reaches the `<EOS>` token. In the paper, they take the final probability distribution as $P(Y|X)$ where $Y$ is the final prediction of the model and $X$ the prompt.
When we have a probability distribution, we can calculate a probability known as entropy. The lower this value is, the better. This is because a lower entropy value means that the model is more confident in its prediction.
We can calculate entropy with the following formula where $P(T_i)$ represents the probability of the $i$-th token in the final output distribution.
$$
H(P(Y|X)) = \sum_{i=0}^n P(T_i) log (P(T_i))
$$
### Mutual Information
![](../../img/mutual_information.png)
We can apply this to the calculation of Mutual Information as seen above.
We'll indicate the calculate of entropy of a probability distribution as $H(X)$ where $X$ here represents a final probability distribution. We also assume you have a train dataset of $n$ examples to use.
1. First, we choose a set of tokens that are likely to be part of the final answer. This could be words that appear inside the choices we have provided.
2. Once we've chosen these tokens, we extract out the log probs for each token from our final distribution. We then normalise it so that these new log probs now sum up to 1.
3. We do this for the $n$ example inside our train set, this gives us a new distribution $P(Y_i|X_i)$ for each $i$-th example.
4. We then take the average of these $n$ distributions to get $H_{marginal}$
5. We then calculate the average of the entropy of each distribution to get $H_{conditional}$
6. We then derive the Mutual Information by taking $H_{marginal} - H_{conditional}$, the higher this metric the better.
??? info "Unsure how to calculate $H_{marginal}$ and $H\_{conditional}$"
$$
H_{marginal} = H(\frac{1}{n} \sum_{i=0}^n P(Y_i | X_i) )
$$
$$
H_{conditional} = \frac{1}{n} \sum_{i=0}^n H(P(Y_i|X_i))
$$
We can then use this new mutual information metric to compare the effectiveness of different prompts at eliciting a desired response from our train dataset.
## Implementation
Since we don't have access to the raw log probabilites of specific tokens we want in the OpenAI API, we'll instead get the language model to generate a final score from 1 - 10 of its confidence in it's prediction.
We'll then convert this to a probability distribution with two outcomes and calculate a value for the entropy off of that.
Next we'll compare the Mutual Information value for different prompts before choosing what the best prompt is. For this example, we'll be using values from the Story Cloze set.
```python
import instructor
from pydantic import BaseModel
from typing import Callable, Literal
from textwrap import dedent
import math
import asyncio
class Response(BaseModel):
chain_of_thought: str
response: Literal["A", "B"]
confidence: Literal[
"Very High Confidence",
"High Confidence",
"Moderate Confidence",
"Low Confidence",
"Very Low Confidence",
]
def generate_score(self) -> float:
confidence_scores = {
"Very High Confidence": 1,
"High Confidence": 0.8,
"Moderate Confidence": 0.6,
"Low Confidence": 0.4,
"Very Low Confidence": 0.2,
}
return confidence_scores[self.confidence]
client = instructor.from_provider("openai/gpt-4o-mini", async_client=True)
def prompt_template_1(question: str, options: list[str]):
assert len(options) == 2
a, b = options
return dedent(
f"""
You are a world class AI System which excels at understanding complex user stories and generating responses. Output your prediction and also quantify your confidence in your prediction with the following scale.
- Very High Confidence: The model is highly confident in its prediction, displaying deep understanding, flawless execution, and no noticeable errors.
- High Confidence: The model is confident in its prediction, with strong relevance and minor errors that do not detract from overall quality.
- Moderate Confidence: The model has moderate confidence in its prediction, which is generally relevant with some inaccuracies, and meets minimum requirements.
- Low Confidence: The model has low confidence in its prediction, with limited relevance and several inaccuracies.
- Very Low Confidence: The model has very low confidence in its prediction, which is largely irrelevant, inaccurate, or incomplete, needing significant improvement
Context
{question}
Options
A. {a}
B. {b}
"""
)
def prompt_template_2(question: str, options: list[str]):
assert len(options) == 2
a, b = options
return dedent(
f"""
<prompt>
<Task>
You are about to be passed a story. You are to select the correct response from the options provided.
<confidence-levels>
<level>
<name>Very High Confidence</name>
<description>The model is highly confident in its prediction, displaying deep understanding, flawless execution, and no noticeable errors.</description>
</level>
<level>
<name>High Confidence</name>
<description>The model is confident in its prediction, with strong relevance and minor errors that do not detract from overall quality.</description>
</level>
<level>
<name>Moderate Confidence</name>
<description>The model has moderate confidence in its prediction, which is generally relevant with some inaccuracies, and meets minimum requirements.</description>
</level>
<level>
<name>Low Confidence</name>
<description>The model has low confidence in its prediction, with limited relevance and several inaccuracies.</description>
</level>
<level>
<name>Very Low Confidence</name>
<description>The model has very low confidence in its prediction, which is largely irrelevant, inaccurate, or incomplete, needing significant improvement</description>
</level>
</confidence-levels>
</Task>
<Question>
{question}
</Question>
<Options>
<option>A: {a}</option>
<option>B: {b}</option>
</Options>
</prompt>
"""
)
async def generate_response(
question: str, options: list[str], prompt_template: Callable[[str, list[str]], str]
):
return await client.create(
model="gpt-3.5-turbo",
messages=[
{
"role": "system",
"content": prompt_template(question, options),
}
],
response_model=Response,
)
async def generate_responses(
questions: list[str], prompt_template: Callable[[str, list[str]], str]
):
return await asyncio.gather(
*[
generate_response(
question=question["question"],
options=question["options"],
prompt_template=prompt_template,
)
for question in questions
]
)
def calculate_entropy(probs: list[float]) -> float:
return sum([p * math.log(p) if p != 0 else 0 for p in probs])
def calculate_mutual_information(predictions: list[Response]) -> float:
probs = [
[prediction.generate_score(), 1 - prediction.generate_score()]
for prediction in predictions
]
avg_probs = [0, 0]
for p1, p2 in probs:
avg_probs[0] += p1
avg_probs[1] += p2
h_marginal = calculate_entropy([i / len(probs) for i in avg_probs])
h_conditional = sum([calculate_entropy(prob) for prob in probs]) / len(probs)
return h_marginal - h_conditional
if __name__ == "__main__":
queries = [
{
"question": "Karen was assigned a roommate her first year of college. Her roommate asked her to go to a nearby city for a concert. Karen agreed happily. The show was absolutely exhilarating.",
"options": [
"Karen became good friends with her roommate.",
"Karen hated her roommate.",
],
},
{
"question": "Jim got his first credit card in college. He didn’t have a job so he bought everything on his card. After he graduated he amounted a $10,000 debt. Jim realized that he was foolish to spend so much money. ",
"options": [
"Jim decided to devise a plan for repayment.",
"Jim decided to open another credit card.",
],
},
{
"question": "Gina misplaced her phone at her grandparents. It wasn’t anywhere in the living room. She realized she was in the car before. She grabbed her dad’s keys and ran outside.",
"options": [
"She found her phone in the car.",
"She didn’t want her phone anymore.",
],
},
]
best_mi_score = float("-inf")
best_template = None
for prompt_template in [prompt_template_1, prompt_template_2]:
responses = asyncio.run(generate_responses(queries, prompt_template))
mi_score = calculate_mutual_information(responses)
print(f"{prompt_template.__name__}: {mi_score}")
#> prompt_template_1: -0.0781292189485728
#> prompt_template_2: -0.05907285153542691
if mi_score > best_mi_score:
best_mi_score = mi_score
best_template = prompt_template.__name__
print(best_template, best_mi_score)
#> prompt_template_2 -0.05907285153542691
```

199
참고/instructor-main/docs/prompting/ensembling/meta_cot.md Normal file
View File

@@ -0,0 +1,199 @@
---
description: "Meta Chain Of Thought involves decomposing an initial query into multiple sub questions. We then aggregate the response from each of these chains as context before prompting another LLM to generate a response"
---
Meta Chain Of Thought (Meta COT) <sup><a href="https://arxiv.org/pdf/2304.13007">1</a></sup>. involves the use of multiple reasoning chains to generate a response to a given query. This helps our model evaluate multiple potential reasoning paths and from there, determine a more accurate answer.
We can implement this using `instructor` as seen below.
```python hl_lines="41-42 57-61 96-99"
import instructor
from pydantic import BaseModel, Field
import asyncio
from typing import Optional
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class ReasoningAndResponse(BaseModel):
intermediate_reasoning: str = Field(
description="""
Intermediate reasoning steps"""
)
correct_answer: str
class MaybeResponse(BaseModel):
result: Optional[ReasoningAndResponse]
error: Optional[bool]
error_message: Optional[str] = Field(
description="""Informative explanation of why
the reasoning chain was unable to generate
a result"""
)
class QueryDecomposition(BaseModel):
queries: list[str] = Field(
description="""A list of queries that need to be
answered in order to derive the final answer"""
)
async def generate_queries(query: str):
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """You are a helpful assistant that
decomposes a query into multiple sub-queries.""",
},
{"role": "user", "content": query},
],
response_model=QueryDecomposition,
)
async def generate_reasoning_chain(query: str) -> MaybeResponse:
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """
Given a question and a context,
answer the question step-by-step.
Indicate the intermediate reasoning
steps.
""",
},
{"role": "user", "content": query},
],
response_model=MaybeResponse,
)
async def batch_reasoning_chains(
queries: list[str],
) -> list[MaybeResponse]:
coros = [generate_reasoning_chain(query) for query in queries]
results = await asyncio.gather(*coros)
return results
async def generate_response(query: str, context: list[MaybeResponse]):
formatted_context = "\n".join(
[
f"""
{item.result.intermediate_reasoning}
{item.result.correct_answer}
"""
for item in context
if not item.error and item.result
]
)
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """
Given a question and a context,
answer the question step-by-step.
If you are unsure, answer Unknown.
""",
},
{
"role": "user",
"content": f"""
<question>
{query}
</question>
<context>
{formatted_context}
</context>
""",
},
],
response_model=ReasoningAndResponse,
)
if __name__ == "__main__":
query = """Would Arnold Schwarzenegger have been
able to deadlift an adult Black rhinoceros at his
peak strength?"""
decomposed_queries = asyncio.run(generate_queries(query))
for generated_query in decomposed_queries.queries:
print(generated_query)
#> How much weight could Arnold Schwarzenegger
#> deadlift at his peak strength?
#> What is the average weight of an adult Black
#> rhinoceros?
chains = asyncio.run(batch_reasoning_chains(decomposed_queries.queries))
for chain in chains:
print(chain.model_dump_json(indent=2))
"""
{
"result": {
"intermediate_reasoning": "Determining Arnold
Schwarzenegger's peak deadlift involves
researching historical records, interviews,
and Arnold’s competitive powerlifting
results.",
"correct_answer": "Arnold Schwarzenegger's
peak deadlift was reportedly 710 lbs (322
kg)."
},
"error": false,
"error_message": null
}
"""
"""
{
"result": {
"intermediate_reasoning": "To determine the
average weight of an adult Black rhinoceros,
I need to consult reliable sources such as
wildlife encyclopedias, zoological databases,
or scientific articles. Commonly, the average
weight of adult Black rhinoceros ranges
between 800 to 1,400 kg.",
"correct_answer": "The average weight of an
adult Black rhinoceros ranges between 800 to
1,400 kg."
},
"error": false,
"error_message": null
}
"""
response = asyncio.run(generate_response(query, chains))
print(response.model_dump_json(indent=2))
"""
{
"intermediate_reasoning": "Arnold Schwarzenegger's
peak deadlift was 710 lbs (322 kg). The average
weight of an adult Black rhinoceros ranges between
800 to 1,400 kg (1764 to 3086 lbs). Even at the
lower end of the rhinoceros weight range (800 kg
or 1764 lbs), it exceeds Arnold Schwarzenegger's
peak deadlift capacity of 710 lbs (322 kg).
Therefore, Arnold Schwarzenegger would not have
been able to deadlift an adult Black rhinoceros at
his peak strength.",
"correct_answer": "No"
}
"""
```
### References
<sup id="ref-1">1</sup>: [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://arxiv.org/pdf/2304.13007)

161
참고/instructor-main/docs/prompting/ensembling/more.md Normal file
View File

@@ -0,0 +1,161 @@
---
description: "MoRE creates a set of diverse reasoning experts by using different specialized prompts for different reasoning types. THe best answer from all experts is then selected using an agreement score"
---
Language Models struggle to generalize across question types that require distinct reasoning abilities. By combining a variety of different specialized language models, we can improve the quality of our responses. This is done through a technique called Mixture Of Reasoning Experts (MoRE).
In the original paper, they utilise four different experts
1. Factual Expert : This is a model that is augmented by a RAG prompting pipeline. WHen it recieves a query, it retrieves the top 10 most relevant passages from Wikipedia and appends them to the prompt right before the question.
2. Multihop Expert : This is an expert that has manually written rationales after each demo to elicit multi-step reasoning processes for the questions
3. Math Expert : This is an expert that has manually written explanations for the GSM8k Dataset to bias the model towards different reasoning steps
4. Commonsense expert: This is an expert that is provided with 10 different facts that are generated by a Codex model which are appended to the prompt right before the question
![](../../img/more.png)
Once each expert has genearted a response, they then use a random forest classifier to score it from 0 to 1. This is then used for selecting the final answer and determining if we've generated a sufficiently good answer ( Since we have the option to abstain at each point )
We can implement a simplified version of MoRE with `instructor` with a few modifications.
```python
from pydantic import BaseModel, Field
import instructor
from textwrap import dedent
client = instructor.from_provider("openai/gpt-5-nano")
class MultihopExpert(BaseModel):
chain_of_thought: str
answer: str
class FactualExpert(BaseModel):
answer: str
class ModelScore(BaseModel):
score: float = Field(ge=0, lt=1)
def query_factual_expert(query: str, evidence: list[str]):
formatted_evidence = "\n-".join(evidence)
return client.create(
model="gpt-4o",
response_model=FactualExpert,
messages=[
{
"role": "system",
"content": dedent(
f"""
<query>
{query}
</query>
<evidences>
{formatted_evidence}
</evidences>
"""
),
}
],
)
def query_multihop_expert(query: str):
return client.create(
model="gpt-4o",
response_model=MultihopExpert,
messages=[
{
"role": "system",
"content": dedent(
f"""
<query>
{query}
</query>
"""
),
}
],
)
def score_answer(query: str, answer: str):
return client.create(
model="gpt-4o",
response_model=ModelScore,
messages=[
{
"role": "system",
"content": """You are a helpful assistant that scores
answers based on well they are able to answer a
specific user query""",
},
{
"role": "user",
"content": f"""
<user query>
{query}
</user query>
<response>
{answer}
</response>
""",
},
],
)
if __name__ == "__main__":
query = """Who's the original singer of Help Me Make It
Through The Night?"""
evidences = [
"""Help Me Make It Through The Night is a country
music ballad written and composed by Kris Kristofferson
and released on his 1970 album 'Kristofferson'"""
]
threshold = 0.8
factual_expert_output = query_factual_expert(query, evidences)
print(factual_expert_output.model_dump_json(indent=2))
"""
{
"answer": "The original singer of 'Help Me Make It Through the
Night' is Kris Kristofferson, who released it on his 1970 album
'Kristofferson'."
}
"""
multihop_expert_output = query_multihop_expert(query)
print(multihop_expert_output.model_dump_json(indent=2))
"""
{
"chain_of_thought": "To identify the original singer of 'Help Me
Make It Through The Night,' I need to look for the person who
first recorded and released the song.",
"answer": "The original singer of 'Help Me Make It Through
The Night' is Kris Kristofferson."
}
"""
factual_expert_score = score_answer(query, factual_expert_output.answer)
multihop_expert_score = score_answer(query, multihop_expert_output.answer)
if max(factual_expert_score.score, multihop_expert_score.score) < threshold:
answer = "Abstaining from responding"
elif factual_expert_score.score > multihop_expert_score.score:
answer = factual_expert_output.answer
else:
answer = multihop_expert_output.answer
print(answer)
"""
The original singer of 'Help Me Make It Through the Night' is Kris
Kristofferson, who released it on his 1970 album 'Kristofferson'.
"""
```

124
참고/instructor-main/docs/prompting/ensembling/prompt_paraphrasing.md Normal file
View File

@@ -0,0 +1,124 @@
---
description: "Use Large Language Models to perform back translation in order to improve prompt performance"
---
Large Language Models are sensitive to the way that they are prompted. When prompted incorrectly, they might perform much worse despite having the information or capability to respond to the prompt. We can help find semantically similar prompts by performing back translation - where we translate our prompts to another language and back to encourage more diversity in the rephrased prompts.
Prompt paraphrasing <sup><a href="https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00324/96460/How-Can-We-Know-What-Language-Models-Know">1</a></sup>. provides some ways for us to improve on the phrasing of our prompts to do so.
We can implement this using `instructor` as seen below.
```python hl_lines="20-25"
import instructor
from pydantic import BaseModel
import random
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class TranslatedPrompt(BaseModel):
translation: str
async def translate_prompt(prompt: str, from_language: str, to_language: str):
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": f"""
You are an expert translation assistant.
You are going to be given a prompt and
asked to translate it from {from_language}
to {to_language}. Paraphrase and use
synonyms where possible, especially for
the examples.
""",
},
{"role": "user", "content": f"Prompt: {prompt}"},
],
response_model=TranslatedPrompt,
)
async def generate_permutation(prompt: str, language: str) -> str:
tranlated_prompt = await translate_prompt(prompt, "english", language)
backtranslated_prompt = await translate_prompt(
tranlated_prompt.translation, language, "english"
)
return backtranslated_prompt.translation
async def generate_prompts(
prompt: str, languages: list[str], permutations: int
) -> list[str]:
coros = [
generate_permutation(prompt, random.choice(languages))
for _ in range(permutations)
]
return await asyncio.gather(*coros)
if __name__ == "__main__":
import asyncio
prompt = """
You are an expert system that excels at Sentiment
Analysis of User Reviews.
Here are a few examples to refer to:
1. That was a fantastic experience I had! I'm
definitely recommending this to all my friends
// Positive
2. I think it was a passable evening. I don't think
there was anything remarkable or off-putting for me.
// Negative
3. I'm horrified at the state of affairs in this new
restaurant // Negative
Sentence: This was a fantastic experience!
"""
languages = ["french", "spanish", "chinese"]
permutations = 2
generated_prompts = asyncio.run(generate_prompts(prompt, languages, permutations))
for prompt in generated_prompts:
print(prompt)
"""
You are an expert system specializing in user review sentiment analysis. Here are a few examples to guide you: 1. It was an exceptional experience! I will definitely recommend it to all my friends // Positive 2. I think it was a mediocre evening. There wasn't anything outstanding or particularly bad for me // Negative 3. I am horrified by the condition of things in this new restaurant // Negative Sentence: It was an amazing experience!
"""
"""
You are an expert system that excels in User Review Sentiment Analysis.
Here are some reference examples:
1. I had an amazing experience! I will definitely recommend it to all my friends.
// Positive
2. I think it was an average evening. I don’t believe there was anything remarkable or unpleasant about it for me.
// Negative
3. I am horrified by the situation at this new restaurant.
// Negative
Sentence: This was a fantastic experience!
"""
"""
You are an expert system skilled in conducting user
review sentiment analysis.
Here are some examples for reference:
1. That was an awesome experience! I'll definitely
recommend it to all my friends // Positive
2. I think it was an okay evening. I don't find
anything particularly outstanding or unpleasant.
// Neutral
3. I am very shocked by the condition of this new
restaurant // Negative
Sentence: This was a wonderful experience!
"""
```
### References
<sup id="ref-1">1</sup>: [How Can We Know What Language Models Know? ](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00324/96460/How-Can-We-Know-What-Language-Models-Know)

73
참고/instructor-main/docs/prompting/ensembling/self_consistency.md Normal file
View File

@@ -0,0 +1,73 @@
---
description: "Self Consistency aims to help maximise llm performance by sampling multiple potential calls. We then take a majority vote on the final response to derive the answer"
---
By generating multiple candidate responses in parallel and choosing the most common answer among them, we can get a more accurate answer. This is known as Self-Consistency <sup><a href="https://arxiv.org/pdf/2203.11171">1</a></sup>
We can implement this using `instructor` as seen below.
```python hl_lines="25-29"
import instructor
from pydantic import BaseModel, Field
import asyncio
from collections import Counter
from textwrap import dedent
class SelfConsistencyResponse(BaseModel):
chain_of_thought: str = Field(
description="reasoning behind the final correct answer"
)
correct_answer: int
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
async def generate_self_consistent_response(prompt: str):
return await client.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """You are an intelligent question
answering AI system that excels at answering
user queries. Make sure to generate a
comprehensive explanation of your thought
process before providing the final answer""",
},
{"role": "user", "content": prompt},
],
response_model=SelfConsistencyResponse,
temperature=0.5,
)
async def generate_self_consistent_responses(prompt: str, num_responses: int):
coros = [generate_self_consistent_response(prompt) for _ in range(num_responses)]
responses = await asyncio.gather(*coros)
return responses
if __name__ == "__main__":
prompt = dedent(
"""
Janet's ducks lay 16 eggs per day.
She eats three for breakfast every
morning and bakes muffins for her
friends every day with four. She sells
the remainder for $2 per egg. How
much does she make every day?
"""
)
responses = asyncio.run(generate_self_consistent_responses(prompt, 5))
answer_counts = Counter([response.correct_answer for response in responses])
most_common_answer, _ = answer_counts.most_common(1)[0]
print(most_common_answer)
#> 18
```
### References
<sup id="ref-1">1</sup>: [Self-Consistency Improves Chain Of Thought
Reasoning In Language Models](https://arxiv.org/pdf/2210.03350)

180
참고/instructor-main/docs/prompting/ensembling/universal_self_consistency.md Normal file
View File

@@ -0,0 +1,180 @@
---
description: "Universal Self Consistency aims to extend Self-Consistency by using Large Language Models themselves to select the most consistent answer among multiple candidates"
---
Universal Self Consistency<sup><a href="https://arxiv.org/pdf/2311.17311">1</a></sup> aims to extend self-consistency by using a second LLM model to judge the quality of individual responses. Therefore instead of choosing the final answer based on the most frequently occuring value among each reasoning chain, we instead prompt the model to choose the most consistent answer for us relative to the prompt.
![](../../img/universal_self_consistency.png)
This enables us to support a greater variety of different response formats and answer, leading to greater diversity of outputs and hence higher accuracy.
We can implement this in `instructor` as seen below.
```python hl_lines="71-73"
from pydantic import BaseModel, Field, ValidationInfo, field_validator
import instructor
from textwrap import dedent
import asyncio
client = instructor.from_provider("openai/gpt-5-nano", async_client=True)
class Response(BaseModel):
chain_of_thought: str
answer: str
class SelectedResponse(BaseModel):
most_consistent_response_id: int = Field(
description="""The ID of the most consistent response that
was provided"""
)
@field_validator("most_consistent_response_id")
@classmethod
def validate_id(cls, v: int, info: ValidationInfo):
context = info.context
number_responses = context.get("number_responses", float("inf"))
if v > number_responses:
raise ValueError(
f"""Most consistent response ID {v} is greater than the
number of responses {number_responses}. Please return a
valid id between 0 and {number_responses-1}"""
)
return v
async def generate_response(query: str) -> Response:
return await client.create(
model="gpt-4o",
response_model=Response,
messages=[{"role": "user", "content": query}],
)
async def generate_batch_responses(query: str, no_responses: int):
coros = [generate_response(query) for _ in range(no_responses)]
return await asyncio.gather(*coros)
async def select_consistent_response(responses: list[Response], query: str):
formatted_responses = "\n".join(
[
f"Response {idx}: {response.chain_of_thought}. {response.answer}"
for idx, response in enumerate(responses)
]
)
return await client.create(
model="gpt-4o",
response_model=SelectedResponse,
messages=[
{
"role": "user",
"content": dedent(
f"""
<user query>
{query}
</user query>
{formatted_responses}
Evaluate these responses.
Select the most consistent response based on majority
consensus
"""
),
}
],
context={"number_responses": len(responses)},
)
if __name__ == "__main__":
query = """The three-digit number 'ab5' is divisible by 3. How many different
three-digit numbers can 'ab5' represent?"""
responses = asyncio.run(generate_batch_responses(query, 3))
for response in responses:
print(response.model_dump_json(indent=2))
"""
{
"chain_of_thought": "A number is divisible by 3 if
the sum of its digits is divisible by 3. Given the
number 'ab5', we need to check how many different
values of 'a' and 'b', where both are digits (0-9)
can make the sum divisible by 3.\n\nThe sum of the
digits is a + b + 5.\n\nWe need to find pairs (a, b)
such that (a + b + 5) % 3 == 0.",
"answer": "30"
}
"""
"""
{
"chain_of_thought": "A number is divisible by 3 if
the sum of its digits is divisible by 3. Let's
denote the digits a and b. The number 'ab5' has
digits a, b, and 5. Therefore, the sum of the
digits is a + b + 5. Since the number is divisible
by 3, a + b + 5 must be divisible by 3.\n\nNow,
since a and b are single digits (0-9), we need to
find pairs (a, b) such that a + b + 5 is divisible
by 3. We will evaluate all possible combinations of
values for a and b to count how many valid pairs
(a, b) exist.\n\nLet's start by considering b's
values:\n1. If b = 0, then a + 5 must be divisible
by 3.\n2. If b = 1, then a + 6 must be divisible by
3.\n3. If b = 2, then a + 7 must be divisible by
3.\n4. If b = 3, then a + 8 must be divisible by
3.\n5. If b = 4, then a + 9 must be divisible by
3.\n6. If b = 5, then a + 10 must be divisible by
3.\n7. If b = 6, then a + 11 must be divisible by
3.\n8. If b = 7, then a + 12 must be divisible by
3.\n9. If b = 8, then a + 13 must be divisible by
3.\n10. If b = 9, then a + 14 must be divisible by
3.\n\nWe will find all corresponding a values for
each b and count the valid combinations.\n",
"answer": "There are 30 different three-digit
numbers that 'ab5' can represent."
}
"""
"""
{
"chain_of_thought": "A number is divisible by 3 if
the sum of its digits is divisible by 3. The given
number is in the form 'ab5', where 'a' and 'b' are
digits from 0 to 9. To find the total number of
different three-digit numbers that 'ab5' can
represent, we need to determine all possible digit
combinations for 'a' and 'b' such that 'a + b + 5'
is divisible by 3.",
"answer": "30"
}
"""
selected_response = asyncio.run(select_consistent_response(responses, query))
print(selected_response.model_dump_json(indent=2))
"""
{
"most_consistent_response_id": 0
}
"""
print(
responses[selected_response.most_consistent_response_id].model_dump_json(
indent=2
)
)
"""
{
"chain_of_thought": "A number is divisible by 3 if the sum of its digits is divisible by 3. Given the number 'ab5', we need to
check how many different values of 'a' and 'b', where both are digits (0-9) can make the sum divisible by 3.\n\nThe sum of the
digits is a + b + 5.\n\nWe need to find pairs (a, b) such that (a + b + 5) % 3 == 0.",
"answer": "30"
}
"""
```
### References
<sup id="ref-1">1</sup>: [Universal Self-Consistency For Large Language Model Generation](https://arxiv.org/pdf/2311.17311)

236
참고/instructor-main/docs/prompting/ensembling/usp.md Normal file
View File

@@ -0,0 +1,236 @@
---
description: "Universal Self Prompting is a technique that aims to use unlabeled data to generate exemplars and a more complicated scoring function to select them."
---
Universal Self Prompting is a two stage process similar to [Consistency Based Self Adaptive Prompting (COSP)](../few_shot/cosp.md). Here is a breakdown of the two stages.
1. **Generate Examples** : LLMs are prompted to generate a collection of candidate responses using a test dataset
2. **Answer Query** : We then select a few of these model-generated responses as examples to prompt the LLM to obtain a final prediction.
Note here that the final answer is obtained using a single forward pass with greedy decoding.
## USP Process
![](../../img/universal_self_adaptive_prompting.png)
Let's see how this works in greater detail.
### Generate Few Shot Examples
We first prompt our model to generate responses for a given set of prompts. Instead of measuring the entropy and repetitiveness as in COSP, we use one of three possible methods to measure the quality of the generated responses. These methods are decided based on the three categories supported.
This category has to be specified by a user ahead of time.
Note that for Short Form and Long Form generation, we generate $m$ different samples. This is not the case for classification tasks.
- **Classification** : Classification Tasks are evaluated using the normalized probability of each label using the raw logits from the LLM.
$$
F_{CLS}(p^{(j)}|d^{(j)}) := -\sum_{c \in C} P(c|d^{(j)}) \log P(c|d^{(j)})
$$
In short, we take the raw logit for each token corresponding to the label, use a softmax to normalize each of them and then sum across the individual probabilities and their log probs. We also try to sample enough queries such that we have a balanced number of predictions across each class ( so that our model doesn't have a bias towards specific classes )
- **Short Form Generation**: This is done by using a similar formula to COSP but without the normalizing term
$$
\mathcal{H}\left(x^{(i)} \mid \left\{\hat{y}_j^{(i)}\right\}_{j=1}^m\right) = \frac{\sum_{\alpha=1}^u \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right) \log \hat{p}\left(\hat{y}_{\alpha}^{(i)}\right)}{\log m},
$$
- **Long Form Generation**: This is done by using the average pairwise ROUGE score between all pairs of the $m$ responses.
What is key here is that depending on the task specified by the user, we have a task-specific form of evaluation. This eventually allows us to better evaluate our individual generated examples. Samples of tasks for each category include
1. **Classification**: Natural Language Inference, Topic Classification and Sentiment Analysis
2. **Short Form Generation** : Question Answering and Sentence Completion
3. **Long Form Generation** : Text Summarization and Machine Translation
This helps to ultimately improve the performance of these large language models across different types of tasks.
### Generate Single Response
Once we've selected our examples, the second step is relatively simple. We just need to append a few of our chosen examples that score best on our chosen metric to append to our solution.
## Implementation
We've implemented a classification example below that tries to sample across different classes in a balanced manner before generating a response using a single inference call.
We bias this sampling towards samples that the model is more confident towards by using a confidence label.
```python
from pydantic import BaseModel
from typing import Literal
import instructor
import asyncio
from collections import defaultdict
class Classification(BaseModel):
chain_of_thought: str
label: Literal["Happy", "Angry", "Sadness"]
confidence: Literal[
"Uncertain", "Somewhat Confident", "Confident", "Highly Confident"
]
def confidence_score(self) -> int:
confidence_order = {
"Highly Confident": 4,
"Confident": 3,
"Somewhat Confident": 2,
"Uncertain": 1,
}
return confidence_order[self.confidence]
client = instructor.from_provider("openai/gpt-4o-mini", async_client=True)
async def generate_prediction(query: str):
return (
await client.create(
model="gpt-3.5-turbo",
messages=[
{
"role": "user",
"content": f"""Classify the following query {query} into
one of the following categories: Happy, Angry, Sadness""",
}
],
response_model=Classification,
),
query,
)
async def generate_predictions(queries: list[str]) -> list[tuple[Classification, str]]:
return await asyncio.gather(*[generate_prediction(query) for query in queries])
def get_balanced_sample(predictions: list[tuple[Classification, str]], k: int):
label_to_queries: dict[str, list[tuple[Classification, str]]] = defaultdict(list)
for prediction in predictions:
label_to_queries[prediction[0].label].append(prediction)
num_classes = len(label_to_queries)
num_samples_per_class = k // num_classes
res: list[str] = []
for label, label_queries in label_to_queries.items():
label_queries = sorted(
label_queries, key=lambda x: x[0].confidence_score(), reverse=True
)
label_queries = [
label_queries[1] for label_queries in label_queries[:num_samples_per_class]
]
res.extend([f"{query} ({label})" for query in label_queries])
return res
async def generate_response_with_examples(query: str, examples: list[str]):
formatted_examples = "\n".join(examples)
return await client.create(
model="gpt-4o",
response_model=Classification,
messages=[
{
"role": "system",
"content": f"""
You are a helpful assistant that classifies queries into one of the following categories: Happy, Angry, Sadness.
Here are some samples of queries and their categories:
<examples>
{formatted_examples}
</examples>
Here is a user query to classify
<query>
{query}
</query>
""",
},
],
)
if __name__ == "__main__":
examples = [
"""
i do feel that running is a divine experience and
that i can expect to have some type of spiritual
encounter
""",
"""
i get giddy over feeling elegant in a perfectly
fitted pencil skirt
""",
"""
i plan to share my everyday life stories traveling
adventures inspirations and handmade creations with
you and hope you will also feel inspired
""",
"""
i need to feel the dough to make sure its just
perfect
""",
"""
i found myself feeling a little discouraged that
morning
""",
"i didnt really feel that embarrassed",
"i feel like a miserable piece of garbage",
"""
i feel like throwing away the shitty piece of shit
paper
""",
"""
i feel irritated and rejected without anyone doing
anything or saying anything
""",
"i feel angered and firey",
"""
im feeling bitter today my mood has been strange the
entire day so i guess its that
""",
"i just feel really violent right now",
"i know there are days in which you feel distracted",
]
labels = asyncio.run(generate_predictions(examples))
balanced_sample = get_balanced_sample(labels, 3)
for sample in balanced_sample:
print(sample)
"""
i do feel that running is a divine experience and that i can
expect to have some type of spiritual encounter (Happy)
"""
#> i feel like a miserable piece of garbage (Sadness)
#> i feel like throwing away the shitty piece of shit paper (Angry)
response = asyncio.run(
generate_response_with_examples(
"""
i feel furious that right to life advocates can
and do tell me how to live and die through
lobbying and supporting those politicians
sympathic to their views
""",
balanced_sample,
)
)
print(response.model_dump_json(indent=2))
"""
{
"chain_of_thought": "The user expresses feelings of
anger and frustration specifically directed at right
to life advocates. The language used, such as
'furious,' indicates a high level of emotion
associated with anger.",
"label": "Angry",
"confidence": "Highly Confident"
}
"""
```