# OntoCast Agentic Ontology Triplecast logo ### Agentic ontology-assisted framework for semantic triple extraction ![Python](https://img.shields.io/badge/python-3.12-blue.svg) [![PyPI version](https://badge.fury.io/py/ontocast.svg)](https://badge.fury.io/py/ontocast) [![PyPI Downloads](https://static.pepy.tech/badge/ontocast)](https://pepy.tech/projects/ontocast) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![pre-commit](https://github.com/growgraph/ontocast/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/growgraph/ontocast/actions/workflows/pre-commit.yml) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.17796467.svg)](https://doi.org/10.5281/zenodo.17796467) --- ## Overview OntoCast is a framework for extracting semantic triples (creating a Knowledge Graph) from documents using an agentic, ontology-driven approach. It combines ontology management, natural language processing, and knowledge graph serialization to turn unstructured text into structured, queryable data. --- ## Key Features - **Ontology-Guided Extraction**: Ensures semantic consistency and co-evolves ontologies - **Entity Disambiguation**: Resolves references across document chunks - **Multi-Format Support**: Handles text, JSON, PDF, and Markdown - **Semantic Chunking**: Splits text based on semantic similarity - **MCP Compatibility**: Implements Model Control Protocol endpoints - **RDF Output**: Produces standardized RDF/Turtle - **Triple Store Integration**: Supports Neo4j (n10s) and Apache Fuseki - **Hierarchical Configuration**: Type-safe configuration system with environment variable support - **CLI Parameters**: Flexible command-line interface with `--skip-ontology-critique` option - **Automatic LLM Caching**: Built-in response caching for improved performance and cost reduction - **GraphUpdate Operations**: Token-efficient SPARQL-based updates instead of full graph regeneration - **Budget Tracking**: Comprehensive tracking of LLM usage and triple generation metrics - **Ontology Versioning**: Automatic semantic versioning with hash-based lineage tracking --- ## Applications OntoCast can be used for: - **Knowledge Graph Construction**: Build domain-specific or general-purpose knowledge graphs from documents - **Semantic Search**: Power search and retrieval with structured triples - **GraphRAG**: Enable retrieval-augmented generation over knowledge graphs (e.g., with LLMs) - **Ontology Management**: Automate ontology creation, validation, and refinement - **Data Integration**: Unify data from diverse sources into a semantic graph --- ## Installation ```sh uv add ontocast[doc-processing] # or pip install ontocast ``` ### Optional features: document processing (PDFs, PPT, OCR, semantic chunking): ```sh uv add "ontocast[doc-processing]" # or pip install "ontocast[doc-processing]" ``` --- ## Quick Start ### 1. Configuration Create a `.env` file with your configuration: ```bash # LLM Configuration LLM_PROVIDER=openai LLM_API_KEY=your-api-key-here LLM_MODEL_NAME=gpt-4o-mini LLM_TEMPERATURE=0.1 # Server Configuration PORT=8999 MAX_VISITS=3 RECURSION_LIMIT=1000 ESTIMATED_CHUNKS=30 ONTOLOGY_MAX_TRIPLES=10000 # Path Configuration ONTOCAST_WORKING_DIRECTORY=/path/to/working ONTOCAST_ONTOLOGY_DIRECTORY=/path/to/ontologies ONTOCAST_CACHE_DIR=/path/to/cache # Optional: Triple Store Configuration FUSEKI_URI=http://localhost:3032/test FUSEKI_AUTH=admin:password FUSEKI_DATASET=ontocast # Optional: Skip ontology critique SKIP_ONTOLOGY_DEVELOPMENT=false # Optional: Maximum triples allowed in ontology graph (set empty for unlimited) ONTOLOGY_MAX_TRIPLES=10000 # Optional: Web search grounding (search-later mode) # Node execution starts without search; search runs only when node output requests it. WEB_SEARCH_ENABLED=false WEB_SEARCH_PROVIDER=duckduckgo WEB_SEARCH_TOP_K=3 WEB_SEARCH_TIMEOUT_SECONDS=8.0 WEB_SEARCH_MAX_SNIPPET_CHARS=400 WEB_SEARCH_MAX_TOTAL_CHARS=1800 WEB_SEARCH_ONTOLOGY_RENDER_ENABLED=true WEB_SEARCH_ONTOLOGY_CRITIC_ENABLED=true WEB_SEARCH_FACTS_RENDER_ENABLED=false WEB_SEARCH_FACTS_CRITIC_ENABLED=false WEB_SEARCH_PLANNER_ENABLED=true WEB_SEARCH_PLANNER_MAX_QUERIES=3 WEB_SEARCH_PLANNER_MIN_QUERY_CHARS=12 WEB_SEARCH_PLANNER_MIN_CONFIDENCE=0.35 WEB_SEARCH_REUSE_EVIDENCE_ACROSS_ATTEMPT=true WEB_SEARCH_MIN_SNIPPET_CHARS=40 WEB_SEARCH_ALLOWED_DOMAINS= WEB_SEARCH_BLOCKED_DOMAINS= ``` ### 2. Start Server ```bash ontocast \ --env-path .env \ --working-directory /path/to/working \ --ontology-directory /path/to/ontologies ``` ### 3. Process Documents ```bash curl -X POST http://localhost:8999/process -F "file=@document.pdf" ``` ### 4. API Endpoints The OntoCast server provides the following endpoints: - **POST /process**: Process documents and extract semantic triples ```bash curl -X POST http://localhost:8999/process -F "file=@document.pdf" ``` - **POST /flush**: Flush/clean triple store data ```bash # Clean all datasets (Fuseki) or entire database (Neo4j) curl -X POST http://localhost:8999/flush # Clean specific Fuseki dataset curl -X POST "http://localhost:8999/flush?dataset=my_dataset" ``` **Note:** For Fuseki, you can specify a `dataset` query parameter to clean a specific dataset. If omitted, all datasets are cleaned. For Neo4j, the `dataset` parameter is ignored and all data is deleted. - **GET /health**: Health check endpoint - **GET /info**: Service information endpoint --- ## LLM Caching OntoCast includes automatic LLM response caching to improve performance and reduce API costs. Caching is enabled by default and requires no configuration. ### Cache Locations - **Tests**: `.test_cache/llm/` in the current working directory - **Windows**: `%USERPROFILE%\AppData\Local\ontocast\llm\` - **Unix/Linux**: `~/.cache/ontocast/llm/` (or `$XDG_CACHE_HOME/ontocast/llm/`) ### Benefits - **Faster Execution**: Repeated queries return cached responses instantly - **Cost Reduction**: Identical requests don't hit the LLM API - **Offline Capability**: Tests can run without API access if responses are cached - **Transparent**: No configuration required - works automatically ### Custom Cache Directory If you need to specify a custom cache directory: ```python from pathlib import Path from ontocast.tool.llm import LLMTool # Cache directory is managed automatically by Cacher llm_tool = LLMTool.create( config=llm_config ) ``` ## Configuration System OntoCast uses a hierarchical configuration system built on Pydantic BaseSettings: ### Environment Variables | Variable | Description | Default | Required | |----------|-------------|---------|----------| | `LLM_API_KEY` | API key for LLM provider | - | Yes | | `LLM_PROVIDER` | LLM provider (openai, ollama) | openai | No | | `LLM_MODEL_NAME` | Model name | gpt-4o-mini | No | | `LLM_TEMPERATURE` | Temperature setting | 0.1 | No | | `ONTOCAST_WORKING_DIRECTORY` | Working directory path | - | Yes | | `ONTOCAST_ONTOLOGY_DIRECTORY` | Ontology files directory | - | No | | `PORT` | Server port | 8999 | No | | `MAX_VISITS` | Maximum visits per node | 3 | No | | `SKIP_ONTOLOGY_DEVELOPMENT` | Skip ontology critique | false | No | | `ONTOLOGY_MAX_TRIPLES` | Maximum triples allowed in ontology graph | 10000 | No | | `SKIP_FACTS_RENDERING` | Skip facts rendering and go straight to aggregation | false | No | | `ONTOCAST_CACHE_DIR` | Custom cache directory for LLM responses | Platform default | No | | `WEB_SEARCH_ENABLED` | Enable optional web grounding (search runs only on node request) | false | No | | `WEB_SEARCH_PROVIDER` | Web search provider | duckduckgo | No | | `WEB_SEARCH_TOP_K` | Number of search results used per call | 3 | No | | `WEB_SEARCH_ONTOLOGY_RENDER_ENABLED` | Allow search-eligible ontology render retries | true | No | | `WEB_SEARCH_ONTOLOGY_CRITIC_ENABLED` | Allow search-eligible ontology critic retries | true | No | | `WEB_SEARCH_FACTS_RENDER_ENABLED` | Allow search-eligible facts render retries | false | No | | `WEB_SEARCH_FACTS_CRITIC_ENABLED` | Allow search-eligible facts critic retries | false | No | | `WEB_SEARCH_PLANNER_ENABLED` | Use LLM planner for query decisioning | true | No | | `WEB_SEARCH_PLANNER_MAX_QUERIES` | Maximum planned focused queries per node | 3 | No | | `WEB_SEARCH_PLANNER_MIN_QUERY_CHARS` | Guardrail minimum query length | 12 | No | | `WEB_SEARCH_PLANNER_MIN_CONFIDENCE` | Guardrail minimum planner confidence | 0.35 | No | | `WEB_SEARCH_ALLOWED_DOMAINS` | Optional comma-separated allowlist domains | empty | No | | `WEB_SEARCH_BLOCKED_DOMAINS` | Optional comma-separated blocklist domains | empty | No | ### Triple Store Configuration ```bash # Fuseki (Preferred) FUSEKI_URI=http://localhost:3032/test FUSEKI_AUTH=admin:password FUSEKI_DATASET=dataset_name # Neo4j (Alternative) NEO4J_URI=bolt://localhost:7689 NEO4J_AUTH=neo4j:password ``` ### CLI Parameters ```bash # Skip ontology critique step ontocast --skip-ontology-critique # Process only first N chunks (for testing) ontocast --head-chunks 5 ``` --- ## Triple Store Setup OntoCast supports multiple triple store backends with automatic fallback: 1. **Apache Fuseki** (Recommended) - Native RDF with SPARQL support 2. **Neo4j with n10s** - Graph database with RDF capabilities 3. **Filesystem** (Fallback) - Local file-based storage When multiple triple stores are configured, **Fuseki is preferred over Neo4j**. ### Quick Setup with Docker **Fuseki:** ```bash cd docker/fuseki cp .env.example .env # Edit .env with your values docker compose --env-file .env fuseki up -d ``` **Neo4j:** ```bash cd docker/neo4j cp .env.example .env # Edit .env with your values docker compose --env-file .env neo4j up -d ``` See [Triple Store Setup](docs/user_guide/triple_stores.md) for detailed instructions. --- ## Documentation - [Quick Start Guide](docs/getting_started/quickstart.md) - Get started quickly - [Configuration System](docs/user_guide/configuration.md) - Detailed configuration guide - [Triple Store Setup](docs/user_guide/triple_stores.md) - Triple store configuration - [User Guide](docs/user_guide/concepts.md) - Core concepts and workflow - [API Reference](docs/reference/onto.md) - Detailed API documentation --- ## Recent Changes ### Ontology Management Improvements - **Automatic Versioning**: Semantic version increment based on change analysis (MAJOR/MINOR/PATCH) - **Hash-Based Lineage**: Git-style versioning with parent hashes for tracking ontology evolution - **Multiple Version Storage**: Versions stored as separate named graphs in Fuseki triple stores - **Timestamp Tracking**: `updated_at` field tracks when ontology was last modified - **Smart Version Analysis**: Analyzes ontology changes (classes, properties, instances) to determine appropriate version bump ### GraphUpdate System - **Token Efficiency**: LLM outputs structured SPARQL operations (insert/delete) instead of full TTL graphs - **Incremental Updates**: Only changes are generated, dramatically reducing token usage - **Structured Operations**: TripleOp operations with explicit prefix declarations for precise updates - **SPARQL Generation**: Automatic conversion of operations to executable SPARQL queries ### Budget Tracking - **LLM Statistics**: Tracks API calls, characters sent/received for cost monitoring - **Triple Metrics**: Tracks ontology and facts triples generated per operation - **Summary Reports**: Budget summaries logged at end of processing - **Integrated Tracking**: Budget tracker integrated into AgentState for clean dependency injection ### Configuration System Overhaul - **Hierarchical Configuration**: New `ToolConfig` and `ServerConfig` structure - **Environment Variables**: Support for `.env` files and environment variables - **Type Safety**: Full type safety with Python 3.12 union syntax - **API Key**: Changed from `OPENAI_API_KEY` to `LLM_API_KEY` for consistency - **Dependency Injection**: Removed global variables, implemented proper DI ### Enhanced Features - **CLI Parameters**: New `--skip-ontology-critique` and `--skip-facts-rendering` parameters - **RDFGraph Operations**: Improved `__iadd__` method with proper prefix binding - **Triple Store Management**: Better separation between filesystem and external stores - **Serialization Interface**: Unified `serialize()` method for storing Ontology and RDFGraph objects - **Error Handling**: Improved error handling and validation See [CHANGELOG.md](CHANGELOG.md) for complete details. --- ## Examples ### Basic Usage ```python from ontocast.config import Config from ontocast.toolbox import ToolBox # Load configuration config = Config() # Initialize tools tools = ToolBox(config) # Process documents # ... (use tools for processing) ``` ### Server Usage ```bash # Start server with custom configuration ontocast \ --env-path .env \ --working-directory /data/working \ --ontology-directory /data/ontologies \ --skip-ontology-critique \ --head-chunks 10 ``` --- ## Contributing We welcome contributions! Please see our [Contributing Guide](docs/contributing.md) for details. --- ## License This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details. --- ## Support - **Documentation**: [docs/](docs/) - **Issues**: [GitHub Issues](https://github.com/growgraph/ontocast/issues) - **Discussions**: [GitHub Discussions](https://github.com/growgraph/ontocast/discussions)