207 lines
6.6 KiB
ReStructuredText
207 lines
6.6 KiB
ReStructuredText
.. _user-guide-pipeline:
|
|
|
|
User Guide: Pipeline
|
|
####################
|
|
|
|
This page provides information about how to create a pipeline.
|
|
|
|
|
|
.. note::
|
|
|
|
Pipelines run asynchronously, see examples below.
|
|
|
|
|
|
*******************
|
|
Creating Components
|
|
*******************
|
|
|
|
Components are asynchronous units of work that perform simple tasks,
|
|
such as chunking documents or saving results to Neo4j.
|
|
This package includes a few default components, but developers can create
|
|
their own by following these steps:
|
|
|
|
1. Create a subclass of the Pydantic `neo4j_graphrag.experimental.pipeline.DataModel` to represent the data being returned by the component
|
|
2. Create a subclass of `neo4j_graphrag.experimental.pipeline.Component`
|
|
3. Create a `run` method in this new class and specify the required inputs and output model using the just created `DataModel`
|
|
4. Implement the run method: it's an `async` method, allowing tasks to be parallelized and awaited within this method.
|
|
|
|
An example is given below, where a `ComponentAdd` is created to add two numbers together and return
|
|
the resulting sum:
|
|
|
|
.. code:: python
|
|
|
|
from neo4j_graphrag.experimental.pipeline import Component, DataModel
|
|
|
|
class IntResultModel(DataModel):
|
|
result: int
|
|
|
|
class ComponentAdd(Component):
|
|
async def run(self, number1: int, number2: int = 1) -> IntResultModel:
|
|
return IntResultModel(result = number1 + number2)
|
|
|
|
Read more about :ref:`components-section` in the API Documentation.
|
|
|
|
***************************************
|
|
Connecting Components within a Pipeline
|
|
***************************************
|
|
|
|
The ultimate aim of creating components is to assemble them into a complex pipeline
|
|
for a specific purpose, such as building a Knowledge Graph from text data.
|
|
|
|
Here's how to create a simple pipeline and propagate results from one component to another
|
|
(detailed explanations follow):
|
|
|
|
.. code:: python
|
|
|
|
import asyncio
|
|
from neo4j_graphrag.experimental.pipeline import Pipeline
|
|
|
|
pipe = Pipeline()
|
|
pipe.add_component(ComponentAdd(), "a")
|
|
pipe.add_component(ComponentAdd(), "b")
|
|
|
|
pipe.connect("a", "b", input_config={"number2": "a.result"})
|
|
asyncio.run(pipe.run({"a": {"number1": 10, "number2": 1}, "b": {"number1": 4}}))
|
|
# result: 10+1+4 = 15
|
|
|
|
1. First, a pipeline is created, and two components named "a" and "b" are added to it.
|
|
2. Next, the two components are connected so that "b" runs after "a", with the "number2" parameter for component "b" being the result of component "a".
|
|
3. Finally, the pipeline is run with 10 and 1 as input parameters for "a". Component "b" will receive 11 (10 + 1, the result of "a") as "number1" and 4 as "number2" (as specified in the pipeline.run parameters).
|
|
|
|
The data flow is illustrated in the diagram below:
|
|
|
|
.. code-block::
|
|
|
|
10 ---\
|
|
Component "a" -> 11
|
|
1 ----/ \
|
|
\
|
|
Component "b" -> 15
|
|
4 -------------------------/
|
|
|
|
.. warning:: Cyclic graph
|
|
|
|
Cycles are not allowed in a Pipeline.
|
|
|
|
|
|
.. warning:: Ignored user inputs
|
|
|
|
If inputs are provided both by user in the `pipeline.run` method and as
|
|
`input_config` in a connect method, the user input will be ignored. Take for
|
|
instance the following pipeline, adapted from the previous one:
|
|
|
|
.. code:: python
|
|
|
|
pipe.connect("a", "b", input_config={"number2": "a.result"})
|
|
asyncio.run(pipe.run({"a": {"number1": 10, "number2": 1}, "b": {"number1": 4, "number2": 42}}))
|
|
|
|
The result will still be **15** because the user input `"number2": 42` is ignored.
|
|
|
|
|
|
**********************
|
|
Visualising a Pipeline
|
|
**********************
|
|
|
|
Pipelines can be visualized using the `draw` method:
|
|
|
|
.. code:: python
|
|
|
|
from neo4j_graphrag.experimental.pipeline import Pipeline
|
|
|
|
pipe = Pipeline()
|
|
# ... define components and connections
|
|
|
|
pipe.draw("pipeline.html")
|
|
|
|
Here is an example pipeline rendering as an interactive HTML visualization:
|
|
|
|
.. code:: python
|
|
|
|
# To view the visualization in a browser
|
|
import webbrowser
|
|
webbrowser.open("pipeline.html")
|
|
|
|
By default, output fields which are not mapped to any component are hidden. They
|
|
can be added to the visualization by setting `hide_unused_outputs` to `False`:
|
|
|
|
.. code:: python
|
|
|
|
pipe.draw("pipeline_full.html", hide_unused_outputs=False)
|
|
|
|
# To view the full visualization in a browser
|
|
import webbrowser
|
|
webbrowser.open("pipeline_full.html")
|
|
|
|
|
|
************************
|
|
Adding an Event Callback
|
|
************************
|
|
|
|
It is possible to add a callback to receive notification about pipeline progress:
|
|
|
|
- `PIPELINE_STARTED`, when pipeline starts
|
|
- `PIPELINE_FINISHED`, when pipeline ends
|
|
- `TASK_STARTED`, when a task starts
|
|
- `TASK_PROGRESS`, sent by each component (depends on component's implementation, see below)
|
|
- `TASK_FINISHED`, when a task ends
|
|
|
|
|
|
See :ref:`pipelineevent` and :ref:`taskevent` to see what is sent in each event type.
|
|
|
|
.. code:: python
|
|
|
|
import asyncio
|
|
import logging
|
|
|
|
from neo4j_graphrag.experimental.pipeline import Pipeline
|
|
from neo4j_graphrag.experimental.pipeline.notification import Event
|
|
|
|
logger = logging.getLogger(__name__)
|
|
logging.basicConfig()
|
|
logger.setLevel(logging.WARNING)
|
|
|
|
|
|
async def event_handler(event: Event) -> None:
|
|
"""Function can do anything about the event,
|
|
here we're just logging it if it's a pipeline-level event.
|
|
"""
|
|
if event.event_type.is_pipeline_event:
|
|
logger.warning(event)
|
|
|
|
pipeline = Pipeline(
|
|
callback=event_handler,
|
|
)
|
|
# ... add components, connect them as usual
|
|
|
|
await pipeline.run(...)
|
|
|
|
|
|
Send Events from Components
|
|
===========================
|
|
|
|
Components can send progress notifications using the `notify` function from
|
|
`context_` by implementing the `run_from_context` method:
|
|
|
|
.. code:: python
|
|
|
|
from neo4j_graphrag.experimental.pipeline import Component, DataModel
|
|
from neo4j_graphrag.experimental.pipeline.types.context import RunContext
|
|
|
|
class IntResultModel(DataModel):
|
|
result: int
|
|
|
|
class ComponentAdd(Component):
|
|
async def run_with_context(self, context_: RunContext, number1: int, number2: int = 1) -> IntResultModel:
|
|
for fake_iteration in range(10):
|
|
await context_.notify(
|
|
message=f"Starting iteration {fake_iteration} out of 10",
|
|
data={"iteration": fake_iteration, "total": 10}
|
|
)
|
|
return IntResultModel(result = number1 + number2)
|
|
|
|
This will send an `TASK_PROGRESS` event to the pipeline callback.
|
|
|
|
.. note::
|
|
|
|
In a future release, the `context_` parameter will be added to the `run` method.
|