참고소스 수정본

참고/neo4j-graphrag-python-main/docs/source/user_guide_pipeline.rst

@@ -0,0 +1,206 @@
+.. _user-guide-pipeline:
+
+User Guide: Pipeline
+####################
+
+This page provides information about how to create a pipeline.
+
+
+.. note::
+
+    Pipelines run asynchronously, see examples below.
+
+
+*******************
+Creating Components
+*******************
+
+Components are asynchronous units of work that perform simple tasks,
+such as chunking documents or saving results to Neo4j.
+This package includes a few default components, but developers can create
+their own by following these steps:
+
+1. Create a subclass of the Pydantic `neo4j_graphrag.experimental.pipeline.DataModel` to represent the data being returned by the component
+2. Create a subclass of `neo4j_graphrag.experimental.pipeline.Component`
+3. Create a `run` method in this new class and specify the required inputs and output model using the just created `DataModel`
+4. Implement the run method: it's an `async` method, allowing tasks to be parallelized and awaited within this method.
+
+An example is given below, where a `ComponentAdd` is created to add two numbers together and return
+the resulting sum:
+
+.. code:: python
+
+    from neo4j_graphrag.experimental.pipeline import Component, DataModel
+
+    class IntResultModel(DataModel):
+        result: int
+
+    class ComponentAdd(Component):
+        async def run(self, number1: int, number2: int = 1) -> IntResultModel:
+            return IntResultModel(result = number1 + number2)
+
+Read more about :ref:`components-section` in the API Documentation.
+
+***************************************
+Connecting Components within a Pipeline
+***************************************
+
+The ultimate aim of creating components is to assemble them into a complex pipeline
+for a specific purpose, such as building a Knowledge Graph from text data.
+
+Here's how to create a simple pipeline and propagate results from one component to another
+(detailed explanations follow):
+
+.. code:: python
+
+    import asyncio
+    from neo4j_graphrag.experimental.pipeline import Pipeline
+
+    pipe = Pipeline()
+    pipe.add_component(ComponentAdd(), "a")
+    pipe.add_component(ComponentAdd(), "b")
+
+    pipe.connect("a", "b", input_config={"number2": "a.result"})
+    asyncio.run(pipe.run({"a": {"number1": 10, "number2": 1}, "b": {"number1": 4}}))
+    # result: 10+1+4 = 15
+
+1. First, a pipeline is created, and two components named "a" and "b" are added to it.
+2. Next, the two components are connected so that "b" runs after "a", with the "number2" parameter for component "b" being the result of component "a".
+3. Finally, the pipeline is run with 10 and 1 as input parameters for "a". Component "b" will receive 11 (10 + 1, the result of "a") as "number1" and 4 as "number2" (as specified in the pipeline.run parameters).
+
+The data flow is illustrated in the diagram below:
+
+.. code-block::
+
+    10 ---\
+            Component "a" -> 11
+    1 ----/                   \
+                               \
+                                 Component "b" -> 15
+    4 -------------------------/
+
+.. warning:: Cyclic graph
+
+    Cycles are not allowed in a Pipeline.
+
+
+.. warning:: Ignored user inputs
+
+    If inputs are provided both by user in the `pipeline.run` method and as
+    `input_config` in a connect method, the user input will be ignored. Take for
+    instance the following pipeline, adapted from the previous one:
+
+    .. code:: python
+
+            pipe.connect("a", "b", input_config={"number2": "a.result"})
+            asyncio.run(pipe.run({"a": {"number1": 10, "number2": 1}, "b": {"number1": 4, "number2": 42}}))
+
+    The result will still be **15** because the user input `"number2": 42` is ignored.
+
+
+**********************
+Visualising a Pipeline
+**********************
+
+Pipelines can be visualized using the `draw` method:
+
+.. code:: python
+
+    from neo4j_graphrag.experimental.pipeline import Pipeline
+
+    pipe = Pipeline()
+    # ... define components and connections
+
+    pipe.draw("pipeline.html")
+
+Here is an example pipeline rendering as an interactive HTML visualization:
+
+.. code:: python
+
+    # To view the visualization in a browser
+    import webbrowser
+    webbrowser.open("pipeline.html")
+
+By default, output fields which are not mapped to any component are hidden. They
+can be added to the visualization by setting `hide_unused_outputs` to `False`:
+
+.. code:: python
+
+    pipe.draw("pipeline_full.html", hide_unused_outputs=False)
+    
+    # To view the full visualization in a browser
+    import webbrowser
+    webbrowser.open("pipeline_full.html")
+
+
+************************
+Adding an Event Callback
+************************
+
+It is possible to add a callback to receive notification about pipeline progress:
+
+- `PIPELINE_STARTED`, when pipeline starts
+- `PIPELINE_FINISHED`, when pipeline ends
+- `TASK_STARTED`, when a task starts
+- `TASK_PROGRESS`, sent by each component (depends on component's implementation, see below)
+- `TASK_FINISHED`, when a task ends
+
+
+See :ref:`pipelineevent` and :ref:`taskevent` to see what is sent in each event type.
+
+.. code:: python
+
+    import asyncio
+    import logging
+
+    from neo4j_graphrag.experimental.pipeline import Pipeline
+    from neo4j_graphrag.experimental.pipeline.notification import Event
+
+    logger = logging.getLogger(__name__)
+    logging.basicConfig()
+    logger.setLevel(logging.WARNING)
+
+
+    async def event_handler(event: Event) -> None:
+        """Function can do anything about the event,
+        here we're just logging it if it's a pipeline-level event.
+        """
+        if event.event_type.is_pipeline_event:
+            logger.warning(event)
+
+    pipeline = Pipeline(
+        callback=event_handler,
+    )
+    # ... add components, connect them as usual
+
+    await pipeline.run(...)
+
+
+Send Events from Components
+===========================
+
+Components can send progress notifications using the `notify` function from
+`context_` by implementing the `run_from_context` method:
+
+.. code:: python
+
+    from neo4j_graphrag.experimental.pipeline import Component, DataModel
+    from neo4j_graphrag.experimental.pipeline.types.context import RunContext
+
+    class IntResultModel(DataModel):
+        result: int
+
+    class ComponentAdd(Component):
+        async def run_with_context(self, context_: RunContext, number1: int, number2: int = 1) -> IntResultModel:
+            for fake_iteration in range(10):
+                await context_.notify(
+                    message=f"Starting iteration {fake_iteration} out of 10",
+                    data={"iteration": fake_iteration, "total": 10}
+                )
+            return IntResultModel(result = number1 + number2)
+
+This will send an `TASK_PROGRESS` event to the pipeline callback.
+
+.. note::
+
+    In a future release, the `context_` parameter will be added to the `run` method.