참고소스 수정본
This commit is contained in:
LASTA_DEV01\lasta
279
참고/ontocast-main/docs/index.md
Normal file
279
참고/ontocast-main/docs/index.md
Normal file
@@ -0,0 +1,279 @@
|
||||
# OntoCast <img src="https://raw.githubusercontent.com/growgraph/ontocast/refs/heads/main/docs/assets/favicon.ico" alt="Agentic Ontology Triplecast logo" style="height: 32px; width:32px;"/>
|
||||
|
||||
### Agentic ontology-assisted framework for semantic triple extraction
|
||||
|
||||
|
||||
[](https://badge.fury.io/py/ontocast)
|
||||
[](https://pepy.tech/projects/ontocast)
|
||||
[](https://opensource.org/licenses/Apache-2.0)
|
||||
[](https://github.com/growgraph/ontocast/actions/workflows/pre-commit.yml)
|
||||
[](https://doi.org/10.5281/zenodo.17796467)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
OntoCast is a framework for extracting semantic triples (creating a Knowledge Graph) from documents using an agentic, ontology-driven approach. It combines ontology management, natural language processing, and knowledge graph serialization to turn unstructured text into structured, queryable data.
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Ontology-Guided Extraction**: Ensures semantic consistency and co-evolves ontologies
|
||||
- **Entity Disambiguation**: Resolves references across document chunks
|
||||
- **Multi-Format Support**: Handles text, JSON, PDF, and Markdown
|
||||
- **Semantic Chunking**: Splits text based on semantic similarity
|
||||
- **MCP Compatibility**: Implements Model Control Protocol endpoints
|
||||
- **RDF Output**: Produces standardized RDF/Turtle
|
||||
- **Triple Store Integration**: Supports Neo4j (n10s) and Apache Fuseki
|
||||
- **Automatic LLM Caching**: Built-in response caching for improved performance and cost reduction
|
||||
- **GraphUpdate Operations**: Token-efficient SPARQL-based updates instead of full graph regeneration
|
||||
- **Budget Tracking**: Comprehensive tracking of LLM usage and triple generation metrics
|
||||
- **Ontology Versioning**: Automatic semantic versioning with hash-based lineage tracking
|
||||
|
||||
---
|
||||
|
||||
## Applications
|
||||
|
||||
OntoCast can be used for:
|
||||
|
||||
- **Knowledge Graph Construction**: Build domain-specific or general-purpose knowledge graphs from documents
|
||||
- **Semantic Search**: Power search and retrieval with structured triples
|
||||
- **GraphRAG**: Enable retrieval-augmented generation over knowledge graphs (e.g., with LLMs)
|
||||
- **Ontology Management**: Automate ontology creation, validation, and refinement
|
||||
- **Data Integration**: Unify data from diverse sources into a semantic graph
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
```sh
|
||||
uv add ontocast
|
||||
# or
|
||||
pip install ontocast
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Quick Start Guide](getting_started/quickstart.md) - Get started quickly
|
||||
- [Configuration System](user_guide/configuration.md) - Detailed configuration guide
|
||||
- [LLM Caching](user_guide/llm_caching.md) - Automatic response caching
|
||||
- [Triple Store Setup](user_guide/triple_stores.md) - Triple store configuration
|
||||
- [User Guide](user_guide/concepts.md) - Core concepts and workflow
|
||||
- [API Reference](reference/onto.md) - Detailed API documentation
|
||||
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Copy the example file and edit as needed:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit with your values
|
||||
```
|
||||
|
||||
**Main options:**
|
||||
```bash
|
||||
# LLM Configuration
|
||||
# common
|
||||
LLM_PROVIDER=openai # or ollama
|
||||
LLM_MODEL_NAME=gpt-4o-mini # ollama model
|
||||
LLM_TEMPERATURE=0.0
|
||||
|
||||
# openai
|
||||
LLM_API_KEY=your_openai_api_key_here
|
||||
|
||||
# ollama
|
||||
LLM_BASE_URL=
|
||||
|
||||
# Server
|
||||
PORT=8999
|
||||
BASE_RECURSION_LIMIT=1000
|
||||
ESTIMATED_CHUNKS=30
|
||||
MAX_VISITS=3
|
||||
RENDER_MODE=ontology_and_facts
|
||||
ONTOLOGY_MAX_TRIPLES=50000
|
||||
PARALLEL_WORKERS=4
|
||||
PARALLEL_FACTS_RETRIES=3
|
||||
PARALLEL_ONTOLOGY_RETRIES=3
|
||||
ENABLE_ONTOLOGY_CONSOLIDATION=false
|
||||
|
||||
# Backend Configuration (auto-detected)
|
||||
FUSEKI_URI=http://localhost:3032/test
|
||||
FUSEKI_AUTH=admin:password
|
||||
ONTOCAST_WORKING_DIRECTORY=/path/to/working
|
||||
|
||||
# Optional: Triple Store Configuration (Fuseki preferred over Neo4j)
|
||||
FUSEKI_URI=http://localhost:3032/test
|
||||
FUSEKI_AUTH=admin/abc123-qwe
|
||||
FUSEKI_DATASET=dataset_name
|
||||
FUSEKI_ONTOLOGIES_DATASET=ontologies
|
||||
|
||||
NEO4J_URI=bolt://localhost:7689
|
||||
NEO4J_AUTH=neo4j/test!passfortesting
|
||||
|
||||
# Aggregation controls
|
||||
AGG_EMBEDDING_MODEL=paraphrase-multilingual-MiniLM-L12-v2
|
||||
AGG_SIMILARITY_THRESHOLD=0.80
|
||||
|
||||
# Optional web grounding
|
||||
WEB_SEARCH_ENABLED=false
|
||||
WEB_SEARCH_PROVIDER=duckduckgo
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Triple Store Setup
|
||||
|
||||
OntoCast supports multiple triple store backends. When both Fuseki and Neo4j are configured, **Fuseki is preferred**.
|
||||
|
||||
- See [Triple Store Setup](user_guide/triple_stores.md) for detailed Docker Compose instructions and sample `.env.example` files.
|
||||
- Quick summary: copy and edit the provided `.env.example` in `docker/fuseki` or `docker/neo4j`, then run `docker compose --env-file .env <service> up -d` in the respective directory.
|
||||
|
||||
---
|
||||
|
||||
## Running OntoCast Server
|
||||
|
||||
```bash
|
||||
# Backend automatically detected from .env configuration
|
||||
ontocast --env-path .env
|
||||
|
||||
# Process specific file
|
||||
ontocast --env-path .env --input-path ./document.pdf
|
||||
|
||||
# Process with chunk limit (for testing)
|
||||
ontocast --env-path .env --head-chunks 5
|
||||
```
|
||||
|
||||
- Backend selection is **fully automatic** based on available configuration
|
||||
- No explicit backend flags needed - just provide the required credentials/paths in .env
|
||||
- All paths and directories are configured via .env file
|
||||
|
||||
---
|
||||
|
||||
## API Usage
|
||||
|
||||
- **POST /process**: Accepts `application/json` or file uploads (`multipart/form-data`).
|
||||
- Returns: JSON with extracted facts (Turtle), ontology (Turtle), and processing metadata. Triples are also serialized to the configured triple store.
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
curl -X POST http://localhost:8999/process \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Your document text here"}'
|
||||
|
||||
# Process a PDF file
|
||||
curl -X POST http://url:port/process -F "file=@data/pdf/sample.pdf"
|
||||
|
||||
# Process a json file
|
||||
curl -X POST http://url:port/process -F "file=@test2/sample.json"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Endpoints
|
||||
|
||||
- `GET /health`: Health check
|
||||
- `GET /info`: Service info
|
||||
- `POST /process`: Document processing
|
||||
- `POST /flush`: Flush/clean triple store data (optional `dataset` query parameter for Fuseki)
|
||||
|
||||
---
|
||||
|
||||
## Filesystem Mode
|
||||
|
||||
If no triple store is configured, OntoCast stores ontologies and facts as Turtle files in the working directory.
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- JSON documents must contain a `text` field, e.g.:
|
||||
```json
|
||||
{ "text": "abc" }
|
||||
```
|
||||
- `recursion_limit` is calculated as `max_visits * estimated_chunks` (default 30, or set via `.env`)
|
||||
- Default port: 8999
|
||||
|
||||
---
|
||||
|
||||
## Docker
|
||||
|
||||
To build the OntoCast Docker image:
|
||||
```sh
|
||||
docker buildx build -t growgraph/ontocast:0.1.4 . 2>&1 | tee build.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
ontocast/
|
||||
├── agent/ # Agent workflow and orchestration
|
||||
├── cli/ # CLI utilities and server
|
||||
├── prompt/ # LLM prompt templates
|
||||
├── stategraph/ # State graph logic
|
||||
├── tool/ # Triple store, chunking, and ontology tools
|
||||
├── toolbox.py # Toolbox for agent tools
|
||||
├── onto.py # Ontology and RDF graph handling
|
||||
├── util.py # Utilities
|
||||
```
|
||||
Other directories:
|
||||
- `docker/` – Docker Compose and .env.example files for triple stores
|
||||
- `data/` – Example data, ontologies, and test files
|
||||
- `docs/` – Documentation and user guides
|
||||
- `test/` – Test suite
|
||||
|
||||
---
|
||||
|
||||
## Workflow
|
||||
|
||||
The extraction follows a multi-stage workflow:
|
||||
|
||||
<img src="https://github.com/growgraph/ontocast/blob/main/docs/assets/graph.png?raw=true" alt="Workflow diagram" width="350" style="float: right; margin-left: 20px;"/>
|
||||
|
||||
1. **Document Preparation**
|
||||
- [Optional] Convert to Markdown
|
||||
- Text chunking
|
||||
2. **Ontology Processing**
|
||||
- Ontology selection
|
||||
- Text to ontology triples
|
||||
- Ontology critique
|
||||
3. **Fact Extraction**
|
||||
- Text to facts
|
||||
- Facts critique
|
||||
- Ontology sublimation
|
||||
4. **Chunk Normalization**
|
||||
- Chunk KG aggregation
|
||||
- Entity/Property Disambiguation
|
||||
5. **Storage**
|
||||
- Triple/KG serialization
|
||||
|
||||
---
|
||||
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [x] Add Jena Fuseki triple store for triple serialization
|
||||
- [x] Add Neo4j n10s for triple serialization
|
||||
- [ ] Replace triple prompting with a tool for local graph retrieval
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please feel free to submit a Pull Request.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- Uses RDFlib for semantic triple management
|
||||
- Uses docling for pdf/pptx conversion
|
||||
- Uses OpenAI language models / open models served via Ollama for fact extraction
|
||||
- Uses langchain/langgraph
|
||||
Reference in New Issue
Block a user