참고소스 수정본
This commit is contained in:
LASTA_DEV01\lasta
386
참고/instructor-main/docs/blog/posts/youtube-flashcards.md
Normal file
386
참고/instructor-main/docs/blog/posts/youtube-flashcards.md
Normal file
@@ -0,0 +1,386 @@
|
||||
---
|
||||
authors:
|
||||
- jxnl
|
||||
- zilto
|
||||
categories:
|
||||
- Data Processing
|
||||
comments: true
|
||||
date: 2024-10-18
|
||||
description: Flashcard generator application with Instructor + Burr
|
||||
draft: false
|
||||
slug: youtube-flashcards
|
||||
tags:
|
||||
- instructor
|
||||
- Burr
|
||||
- OpenAI
|
||||
- LLM
|
||||
- observability
|
||||
---
|
||||
|
||||
# Flashcard generator with Instructor + Burr
|
||||
|
||||
Flashcards help break down complex topics and learn anything from biology to a new
|
||||
language or lines for a play. This blog will show how to use LLMs to generate
|
||||
flashcards and kickstart your learning!
|
||||
|
||||
**Instructor** lets us get structured outputs from LLMs reliably, and [Burr](https://github.com/dagworks-inc/burr) helps
|
||||
create an LLM application that's easy to understand and debug. It comes with **Burr UI**,
|
||||
a free, open-source, and local-first tool for observability, annotations, and more!
|
||||
|
||||
<!-- more -->
|
||||
|
||||
??? info
|
||||
|
||||
This post expands on an earlier one: [Analyzing Youtube Transcripts with Instructor](./youtube-transcripts.md/).
|
||||
|
||||
|
||||
## Generate flashcards using LLMs with Instructor
|
||||
|
||||
```bash
|
||||
pip install openai instructor pydantic youtube_transcript_api "burr[start]"
|
||||
```
|
||||
|
||||
### 1. Define the LLM response model
|
||||
|
||||
With `instructor`, you define Pydantic models that will serve as template for the LLM to
|
||||
fill.
|
||||
|
||||
Here, we define the `QuestionAnswer` model which will store the question, the answer, and
|
||||
some metadata. Attributes without a default value will be generated by the LLM.
|
||||
|
||||
```python hl_lines="10-11 23 24-27"
|
||||
import uuid
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic.json_schema import SkipJsonSchema
|
||||
|
||||
|
||||
class QuestionAnswer(BaseModel):
|
||||
question: str = Field(description="Question about the topic")
|
||||
options: list[str] = Field(
|
||||
description="Potential answers to the question.", min_items=3, max_items=5
|
||||
)
|
||||
answer_index: int = Field(
|
||||
description="Index of the correct answer options (starting from 0).", ge=0, lt=5
|
||||
)
|
||||
difficulty: int = Field(
|
||||
description="Difficulty of this question from 1 to 5, 5 being the most difficult.",
|
||||
gt=0,
|
||||
le=5,
|
||||
)
|
||||
youtube_url: SkipJsonSchema[str | None] = None
|
||||
id: uuid.UUID = Field(description="Unique identifier", default_factory=uuid.uuid4)
|
||||
```
|
||||
|
||||
This examples shows several `instructor` features:
|
||||
|
||||
- `Field` can have a `default` or `default_factory` value to prevent the LLM from
|
||||
hallucinating the value
|
||||
- `id` generates a unique id (`uuid`)
|
||||
- The type annotation `SkipJsonSchema` also prevents the LLM from generating the value.
|
||||
- `youtube_url` is set programmatically in the application. We don't want the LLM
|
||||
to hallucinate it.
|
||||
- `Field` can set constraints on what the LLM generates.
|
||||
- `min_items=3, max_items=5` to limit the number of potential answers between 3 and 5
|
||||
- `ge=0, lt=5` to limit the difficulty between 0 and 5 with 5 being the most difficult
|
||||
|
||||
|
||||
### 2. Retrieve the YouTube transcript
|
||||
|
||||
We use `youtube-transcript-api` to get the full transcript of a video.
|
||||
|
||||
```python
|
||||
from youtube_transcript_api import YouTubeTranscriptApi
|
||||
|
||||
youtube_url = "https://www.youtube.com/watch?v=hqutVJyd3TI"
|
||||
_, _, video_id = youtube_url.partition("?v=")
|
||||
segments = YouTubeTranscriptApi.get_transcript(video_id)
|
||||
transcript = " ".join([s["text"] for s in segments])
|
||||
```
|
||||
|
||||
### 3. Generate question-answer pairs
|
||||
|
||||
Now, to produce question-answer pairs:
|
||||
|
||||
1. Create an `instructor` client by wrapping the OpenAI client
|
||||
2. Use `.create_iterable()` on the `instructor_client` to generate multiple outputs from
|
||||
the input
|
||||
3. Specify `response_model=QuestionAnswer` to ensure outputs are `QuestionAnswer` objects
|
||||
4. Use the `messages` to pass the task instructos via the `system` message, and the input
|
||||
transcript via `user` message.
|
||||
|
||||
```python hl_lines="4 10 12"
|
||||
import instructor
|
||||
|
||||
instructor_client = instructor.from_provider("openai/gpt-5-nano")
|
||||
|
||||
system_prompt = """Analyze the given YouTube transcript and generate question-answer pairs
|
||||
to help study and understand the topic better. Please rate all questions from 1 to 5
|
||||
based on their difficulty."""
|
||||
|
||||
response = instructor_client.create_iterable(
|
||||
model="gpt-4o-mini",
|
||||
response_model=QuestionAnswer,
|
||||
messages=[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": transcript},
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
This will return an generator that you can iterate over to access individual
|
||||
`QuestionAnswer` objects.
|
||||
|
||||
```python
|
||||
print("Preview:\n")
|
||||
count = 0
|
||||
for qna in response:
|
||||
if count > 2:
|
||||
break
|
||||
print(qna.question)
|
||||
print(qna.options)
|
||||
print()
|
||||
count += 1
|
||||
|
||||
"""
|
||||
Preview:
|
||||
|
||||
What is the primary purpose of the new OpenTelemetry instrumentation released with Burr?
|
||||
['To reduce code complexity', 'To provide full instrumentation without changing code', 'To couple the project with OpenAI', 'To enhance customer support']
|
||||
|
||||
What do you need to install to use the OpenTelemetry instrumentation with Burr applications?
|
||||
['Only OpenAI package', 'Specific OpenTelemetry instrumentation module', 'All available packages', 'No installation needed']
|
||||
|
||||
What advantage does OpenTelemetry provide in the context of instrumentation?
|
||||
['It is vendor agnostic', 'It requires complex integration', 'It relies on specific vendors', 'It makes applications slower']
|
||||
"""
|
||||
```
|
||||
|
||||
|
||||
## Create a flashcard application with Burr
|
||||
|
||||
Burr uses `actions` and `transitions` to define complex applications while
|
||||
preserving the simplicity of a flowchart for understanding and debugging.
|
||||
|
||||
|
||||
### 1. Define `actions`
|
||||
|
||||
Actions are what your application can do. The `@action` decorator specifies what values
|
||||
can be read from or written to `State`. The decorated function takes a `State` as
|
||||
first argument and return an updated `State` object.
|
||||
|
||||
Next, we define three actions:
|
||||
|
||||
- Process the user input to get the YouTube URL
|
||||
- Get the YouTube transcript associated with the URL
|
||||
- Generate question-answer pairs for the transcript
|
||||
|
||||
Note that this is only a light refactor from the previous code snippets.
|
||||
|
||||
```python
|
||||
from burr.core import action, State
|
||||
|
||||
|
||||
@action(reads=[], writes=["youtube_url"])
|
||||
def process_user_input(state: State, user_input: str) -> State:
|
||||
"""Process user input and update the YouTube URL."""
|
||||
youtube_url = (
|
||||
user_input # In practice, we would have more complex validation logic.
|
||||
)
|
||||
return state.update(youtube_url=youtube_url)
|
||||
|
||||
|
||||
@action(reads=["youtube_url"], writes=["transcript"])
|
||||
def get_youtube_transcript(state: State) -> State:
|
||||
"""Get the official YouTube transcript for a video given it's URL"""
|
||||
youtube_url = state["youtube_url"]
|
||||
|
||||
_, _, video_id = youtube_url.partition("?v=")
|
||||
transcript = YouTubeTranscriptApi.get_transcript(video_id)
|
||||
full_transcript = " ".join([entry["text"] for entry in transcript])
|
||||
|
||||
# store the transcript in state
|
||||
return state.update(transcript=full_transcript, youtube_url=youtube_url)
|
||||
|
||||
|
||||
@action(reads=["transcript", "youtube_url"], writes=["question_answers"])
|
||||
def generate_question_and_answers(state: State) -> State:
|
||||
"""Generate `QuestionAnswer` from a YouTube transcript using an LLM."""
|
||||
# read the transcript from state
|
||||
transcript = state["transcript"]
|
||||
youtube_url = state["youtube_url"]
|
||||
|
||||
# create the instructor client
|
||||
instructor_client = instructor.from_provider("openai/gpt-5-nano")
|
||||
system_prompt = (
|
||||
"Analyze the given YouTube transcript and generate question-answer pairs"
|
||||
" to help study and understand the topic better. Please rate all questions from 1 to 5"
|
||||
" based on their difficulty."
|
||||
)
|
||||
response = instructor_client.create_iterable(
|
||||
model="gpt-4o-mini",
|
||||
response_model=QuestionAnswer,
|
||||
messages=[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": transcript},
|
||||
],
|
||||
)
|
||||
|
||||
# iterate over QuestionAnswer, add the `youtube_url`, and append to state
|
||||
for qna in response:
|
||||
qna.youtube_url = youtube_url
|
||||
# `State` is immutable, so `.append()` returns a new object with the appended value
|
||||
state = state.append(question_answers=qna)
|
||||
|
||||
return state
|
||||
```
|
||||
|
||||
### 2. Build the `Application`
|
||||
|
||||
To create a Burr `Application`, we use the `ApplicationBuilder` object.
|
||||
|
||||
Minimally, it needs to:
|
||||
|
||||
- Use `.with_actions()` to define all possible actions. Simply pass the functions
|
||||
decorated with `@action`.
|
||||
- Use `.with_transitions()` to define possible transitions between actions. This is
|
||||
done via tuples `(from_action, to_action)`.
|
||||
- Use `.with_entrypoint()` to specify which action to run first.
|
||||
|
||||
|
||||
```python
|
||||
from burr.core import ApplicationBuilder
|
||||
|
||||
app = (
|
||||
ApplicationBuilder()
|
||||
.with_actions(
|
||||
process_user_input,
|
||||
get_youtube_transcript,
|
||||
generate_question_and_answers,
|
||||
)
|
||||
.with_transitions(
|
||||
("process_user_input", "get_youtube_transcript"),
|
||||
("get_youtube_transcript", "generate_question_and_answers"),
|
||||
("generate_question_and_answers", "process_user_input"),
|
||||
)
|
||||
.with_entrypoint("process_user_input")
|
||||
.build()
|
||||
)
|
||||
app.visualize()
|
||||
```
|
||||
|
||||
|
||||
|
||||
> You can always visualize the application graph to understand the logic's flow.
|
||||
|
||||
|
||||
### 3. Launch the application
|
||||
|
||||
Using `Application.run()` will make the application execute actions until a halt condition.
|
||||
In this case, we halt before `process_user_input` to get the YouTube URL from the user.
|
||||
|
||||
The method `.run()` returns a tuple `(action_name, result, state)`. In this case, we only
|
||||
use the state to inspect the generated question-answer pairs.
|
||||
|
||||
```python
|
||||
action_name, result, state = app.run(
|
||||
halt_before=["process_user_input"],
|
||||
inputs={"user_input": "https://www.youtube.com/watch?v=hqutVJyd3TI"},
|
||||
)
|
||||
print(state["question_answers"][0])
|
||||
```
|
||||
|
||||
You can create a simple local experience by using `.run()` in a `while` loop
|
||||
|
||||
```python
|
||||
while True:
|
||||
user_input = input("Enter a YouTube URL (q to quit): ")
|
||||
if user_input.lower() == "q":
|
||||
break
|
||||
|
||||
action_name, result, state = app.run(
|
||||
halt_before=["process_user_input"],
|
||||
inputs={"user_input": user_input},
|
||||
)
|
||||
print(f"{len(state['question_answers'])} question-answer pairs generated")
|
||||
```
|
||||
|
||||
|
||||
## Next steps
|
||||
|
||||
Now that you know how to use Instructor for reliable LLM outputs and Burr to
|
||||
structure your application, many avenues open up depending on your goals!
|
||||
|
||||
|
||||
### 1. Build complex agents
|
||||
|
||||
Instructor improves the LLM's reasoning by providing structure. Nesting models and adding
|
||||
constraints allow to [get facts with citations](../../examples/exact_citations.md)
|
||||
or [extract a knowledge graph](../../examples/knowledge_graph.md)
|
||||
in a few lines of code. Also, [retries](../../concepts/retrying.md)
|
||||
enable the LLM to self-correct.
|
||||
|
||||
Burr sets the boundaries between users, LLMs, and the rest of your system. You can add
|
||||
`Condition` on transitions to create complex workflows that remain easy to reason about.
|
||||
|
||||
### 2. Add Burr to your product
|
||||
|
||||
Your Burr `Application` is a lightweight Python object. You can run it within a notebook,
|
||||
via script, a web app (Streamlit, Gradio, etc.), or as a [web service](https://burr.dagworks.io/examples/deployment/web-server/)
|
||||
(e.g., FastAPI).
|
||||
|
||||
The `ApplicationBuilder` provides many features to productionize your app:
|
||||
|
||||
- [Persistence](https://burr.dagworks.io../../concepts/state-persistence/.md): save and restore `State`
|
||||
(e.g., store conversation history)
|
||||
- [Observability](https://burr.dagworks.io../../concepts/additional-visibility/.md): log and monitor
|
||||
application telemetry (e.g., LLM calls, number of tokens used, errors and retries)
|
||||
- [Streaming and async](https://burr.dagworks.io../../concepts/streaming-actions/.md): create snappy
|
||||
user interfaces by streaming LLM responses and running actions asynchronously.
|
||||
|
||||
For example, you can log telemetry into Burr UI in a few lines of code. First, instrument the
|
||||
OpenAI library. Then, add `.with_tracker()` the `ApplicationBuilder` with a project name and
|
||||
enabling `use_otel_tracing=True`.
|
||||
|
||||
```python hl_lines="5 19"
|
||||
from burr.core import ApplicationBuilder
|
||||
from opentelemetry.instrumentation.openai import OpenAIApiInstrumentor
|
||||
|
||||
# instrument before importing instructor or creating the OpenAI client
|
||||
OpenAIApiInstrumentor().instrument()
|
||||
|
||||
app = (
|
||||
ApplicationBuilder()
|
||||
.with_actions(
|
||||
process_user_input,
|
||||
get_youtube_transcript,
|
||||
generate_question_and_answers,
|
||||
)
|
||||
.with_transitions(
|
||||
("process_user_input", "get_youtube_transcript"),
|
||||
("get_youtube_transcript", "generate_question_and_answers"),
|
||||
("generate_question_and_answers", "process_user_input"),
|
||||
)
|
||||
.with_tracker(project="youtube-qna", use_otel_tracing=True)
|
||||
.with_entrypoint("process_user_input")
|
||||
.build()
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
> Telemetry for our OpenAI API calls with Instructor. We see the prompt, the response model, and the response content.
|
||||
|
||||
### 3. Annotate application logs
|
||||
|
||||
Burr UI has a built-in annotation tool that allows you to label, rate, or comment on
|
||||
logged data (e.g., user input, LLM response, content retrieved for RAG). This can be
|
||||
useful to create test cases and evaluation datasets.
|
||||
|
||||
|
||||
|
||||
|
||||
## Conclusion
|
||||
|
||||
We've shown how Instructor helps getting reliable outputs from LLMs and Burr provides
|
||||
the right tools to build an application. Now it's your turn to start building!
|
||||
Reference in New Issue
Block a user