ontology
This commit is contained in:
LASTA_DEV01\lasta
391
ontology_platform/vendored/ontocast/README.md
Normal file
391
ontology_platform/vendored/ontocast/README.md
Normal file
@@ -0,0 +1,391 @@
|
||||
# OntoCast <img src="https://raw.githubusercontent.com/growgraph/ontocast/refs/heads/main/docs/assets/favicon.ico" alt="Agentic Ontology Triplecast logo" style="height: 32px; width:32px;"/>
|
||||
|
||||
### Agentic ontology-assisted framework for semantic triple extraction
|
||||
|
||||
|
||||
[](https://badge.fury.io/py/ontocast)
|
||||
[](https://pepy.tech/projects/ontocast)
|
||||
[](https://opensource.org/licenses/Apache-2.0)
|
||||
[](https://github.com/growgraph/ontocast/actions/workflows/pre-commit.yml)
|
||||
[](https://doi.org/10.5281/zenodo.17796467)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
OntoCast is a framework for extracting semantic triples (creating a Knowledge Graph) from documents using an agentic, ontology-driven approach. It combines ontology management, natural language processing, and knowledge graph serialization to turn unstructured text into structured, queryable data.
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Ontology-Guided Extraction**: Ensures semantic consistency and co-evolves ontologies
|
||||
- **Entity Disambiguation**: Resolves references across document chunks
|
||||
- **Multi-Format Support**: Handles text, JSON, PDF, and Markdown
|
||||
- **Semantic Chunking**: Splits text based on semantic similarity
|
||||
- **MCP Compatibility**: Implements Model Control Protocol endpoints
|
||||
- **RDF Output**: Produces standardized RDF/Turtle
|
||||
- **Triple Store Integration**: Supports Neo4j (n10s) and Apache Fuseki
|
||||
- **Hierarchical Configuration**: Type-safe configuration system with environment variable support
|
||||
- **CLI Parameters**: Flexible command-line interface with `--skip-ontology-critique` option
|
||||
- **Automatic LLM Caching**: Built-in response caching for improved performance and cost reduction
|
||||
- **GraphUpdate Operations**: Token-efficient SPARQL-based updates instead of full graph regeneration
|
||||
- **Budget Tracking**: Comprehensive tracking of LLM usage and triple generation metrics
|
||||
- **Ontology Versioning**: Automatic semantic versioning with hash-based lineage tracking
|
||||
|
||||
---
|
||||
|
||||
## Applications
|
||||
|
||||
OntoCast can be used for:
|
||||
|
||||
- **Knowledge Graph Construction**: Build domain-specific or general-purpose knowledge graphs from documents
|
||||
- **Semantic Search**: Power search and retrieval with structured triples
|
||||
- **GraphRAG**: Enable retrieval-augmented generation over knowledge graphs (e.g., with LLMs)
|
||||
- **Ontology Management**: Automate ontology creation, validation, and refinement
|
||||
- **Data Integration**: Unify data from diverse sources into a semantic graph
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
```sh
|
||||
uv add ontocast[doc-processing]
|
||||
# or
|
||||
pip install ontocast
|
||||
```
|
||||
|
||||
### Optional features: document processing (PDFs, PPT, OCR, semantic chunking):
|
||||
|
||||
```sh
|
||||
uv add "ontocast[doc-processing]"
|
||||
# or
|
||||
pip install "ontocast[doc-processing]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Configuration
|
||||
|
||||
Create a `.env` file with your configuration:
|
||||
|
||||
```bash
|
||||
# LLM Configuration
|
||||
LLM_PROVIDER=openai
|
||||
LLM_API_KEY=your-api-key-here
|
||||
LLM_MODEL_NAME=gpt-4o-mini
|
||||
LLM_TEMPERATURE=0.1
|
||||
|
||||
# Server Configuration
|
||||
PORT=8999
|
||||
MAX_VISITS=3
|
||||
RECURSION_LIMIT=1000
|
||||
ESTIMATED_CHUNKS=30
|
||||
ONTOLOGY_MAX_TRIPLES=10000
|
||||
|
||||
# Path Configuration
|
||||
ONTOCAST_WORKING_DIRECTORY=/path/to/working
|
||||
ONTOCAST_ONTOLOGY_DIRECTORY=/path/to/ontologies
|
||||
ONTOCAST_CACHE_DIR=/path/to/cache
|
||||
|
||||
# Optional: Triple Store Configuration
|
||||
FUSEKI_URI=http://localhost:3032/test
|
||||
FUSEKI_AUTH=admin:password
|
||||
FUSEKI_DATASET=ontocast
|
||||
|
||||
# Optional: Skip ontology critique
|
||||
SKIP_ONTOLOGY_DEVELOPMENT=false
|
||||
# Optional: Maximum triples allowed in ontology graph (set empty for unlimited)
|
||||
ONTOLOGY_MAX_TRIPLES=10000
|
||||
|
||||
# Optional: Web search grounding (search-later mode)
|
||||
# Node execution starts without search; search runs only when node output requests it.
|
||||
WEB_SEARCH_ENABLED=false
|
||||
WEB_SEARCH_PROVIDER=duckduckgo
|
||||
WEB_SEARCH_TOP_K=3
|
||||
WEB_SEARCH_TIMEOUT_SECONDS=8.0
|
||||
WEB_SEARCH_MAX_SNIPPET_CHARS=400
|
||||
WEB_SEARCH_MAX_TOTAL_CHARS=1800
|
||||
WEB_SEARCH_ONTOLOGY_RENDER_ENABLED=true
|
||||
WEB_SEARCH_ONTOLOGY_CRITIC_ENABLED=true
|
||||
WEB_SEARCH_FACTS_RENDER_ENABLED=false
|
||||
WEB_SEARCH_FACTS_CRITIC_ENABLED=false
|
||||
WEB_SEARCH_PLANNER_ENABLED=true
|
||||
WEB_SEARCH_PLANNER_MAX_QUERIES=3
|
||||
WEB_SEARCH_PLANNER_MIN_QUERY_CHARS=12
|
||||
WEB_SEARCH_PLANNER_MIN_CONFIDENCE=0.35
|
||||
WEB_SEARCH_REUSE_EVIDENCE_ACROSS_ATTEMPT=true
|
||||
WEB_SEARCH_MIN_SNIPPET_CHARS=40
|
||||
WEB_SEARCH_ALLOWED_DOMAINS=
|
||||
WEB_SEARCH_BLOCKED_DOMAINS=
|
||||
```
|
||||
|
||||
### 2. Start Server
|
||||
|
||||
```bash
|
||||
ontocast \
|
||||
--env-path .env \
|
||||
--working-directory /path/to/working \
|
||||
--ontology-directory /path/to/ontologies
|
||||
```
|
||||
|
||||
### 3. Process Documents
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8999/process -F "file=@document.pdf"
|
||||
```
|
||||
|
||||
### 4. API Endpoints
|
||||
|
||||
The OntoCast server provides the following endpoints:
|
||||
|
||||
- **POST /process**: Process documents and extract semantic triples
|
||||
```bash
|
||||
curl -X POST http://localhost:8999/process -F "file=@document.pdf"
|
||||
```
|
||||
|
||||
- **POST /flush**: Flush/clean triple store data
|
||||
```bash
|
||||
# Clean all datasets (Fuseki) or entire database (Neo4j)
|
||||
curl -X POST http://localhost:8999/flush
|
||||
|
||||
# Clean specific Fuseki dataset
|
||||
curl -X POST "http://localhost:8999/flush?dataset=my_dataset"
|
||||
```
|
||||
**Note:** For Fuseki, you can specify a `dataset` query parameter to clean a specific dataset. If omitted, all datasets are cleaned. For Neo4j, the `dataset` parameter is ignored and all data is deleted.
|
||||
|
||||
- **GET /health**: Health check endpoint
|
||||
- **GET /info**: Service information endpoint
|
||||
|
||||
---
|
||||
|
||||
|
||||
## LLM Caching
|
||||
|
||||
OntoCast includes automatic LLM response caching to improve performance and reduce API costs. Caching is enabled by default and requires no configuration.
|
||||
|
||||
### Cache Locations
|
||||
|
||||
- **Tests**: `.test_cache/llm/` in the current working directory
|
||||
- **Windows**: `%USERPROFILE%\AppData\Local\ontocast\llm\`
|
||||
- **Unix/Linux**: `~/.cache/ontocast/llm/` (or `$XDG_CACHE_HOME/ontocast/llm/`)
|
||||
|
||||
### Benefits
|
||||
|
||||
- **Faster Execution**: Repeated queries return cached responses instantly
|
||||
- **Cost Reduction**: Identical requests don't hit the LLM API
|
||||
- **Offline Capability**: Tests can run without API access if responses are cached
|
||||
- **Transparent**: No configuration required - works automatically
|
||||
|
||||
### Custom Cache Directory
|
||||
|
||||
If you need to specify a custom cache directory:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from ontocast.tool.llm import LLMTool
|
||||
|
||||
# Cache directory is managed automatically by Cacher
|
||||
llm_tool = LLMTool.create(
|
||||
config=llm_config
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## Configuration System
|
||||
|
||||
OntoCast uses a hierarchical configuration system built on Pydantic BaseSettings:
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default | Required |
|
||||
|----------|-------------|---------|----------|
|
||||
| `LLM_API_KEY` | API key for LLM provider | - | Yes |
|
||||
| `LLM_PROVIDER` | LLM provider (openai, ollama) | openai | No |
|
||||
| `LLM_MODEL_NAME` | Model name | gpt-4o-mini | No |
|
||||
| `LLM_TEMPERATURE` | Temperature setting | 0.1 | No |
|
||||
| `ONTOCAST_WORKING_DIRECTORY` | Working directory path | - | Yes |
|
||||
| `ONTOCAST_ONTOLOGY_DIRECTORY` | Ontology files directory | - | No |
|
||||
| `PORT` | Server port | 8999 | No |
|
||||
| `MAX_VISITS` | Maximum visits per node | 3 | No |
|
||||
| `SKIP_ONTOLOGY_DEVELOPMENT` | Skip ontology critique | false | No |
|
||||
| `ONTOLOGY_MAX_TRIPLES` | Maximum triples allowed in ontology graph | 10000 | No |
|
||||
| `SKIP_FACTS_RENDERING` | Skip facts rendering and go straight to aggregation | false | No |
|
||||
| `ONTOCAST_CACHE_DIR` | Custom cache directory for LLM responses | Platform default | No |
|
||||
| `WEB_SEARCH_ENABLED` | Enable optional web grounding (search runs only on node request) | false | No |
|
||||
| `WEB_SEARCH_PROVIDER` | Web search provider | duckduckgo | No |
|
||||
| `WEB_SEARCH_TOP_K` | Number of search results used per call | 3 | No |
|
||||
| `WEB_SEARCH_ONTOLOGY_RENDER_ENABLED` | Allow search-eligible ontology render retries | true | No |
|
||||
| `WEB_SEARCH_ONTOLOGY_CRITIC_ENABLED` | Allow search-eligible ontology critic retries | true | No |
|
||||
| `WEB_SEARCH_FACTS_RENDER_ENABLED` | Allow search-eligible facts render retries | false | No |
|
||||
| `WEB_SEARCH_FACTS_CRITIC_ENABLED` | Allow search-eligible facts critic retries | false | No |
|
||||
| `WEB_SEARCH_PLANNER_ENABLED` | Use LLM planner for query decisioning | true | No |
|
||||
| `WEB_SEARCH_PLANNER_MAX_QUERIES` | Maximum planned focused queries per node | 3 | No |
|
||||
| `WEB_SEARCH_PLANNER_MIN_QUERY_CHARS` | Guardrail minimum query length | 12 | No |
|
||||
| `WEB_SEARCH_PLANNER_MIN_CONFIDENCE` | Guardrail minimum planner confidence | 0.35 | No |
|
||||
| `WEB_SEARCH_ALLOWED_DOMAINS` | Optional comma-separated allowlist domains | empty | No |
|
||||
| `WEB_SEARCH_BLOCKED_DOMAINS` | Optional comma-separated blocklist domains | empty | No |
|
||||
|
||||
### Triple Store Configuration
|
||||
|
||||
```bash
|
||||
# Fuseki (Preferred)
|
||||
FUSEKI_URI=http://localhost:3032/test
|
||||
FUSEKI_AUTH=admin:password
|
||||
FUSEKI_DATASET=dataset_name
|
||||
|
||||
# Neo4j (Alternative)
|
||||
NEO4J_URI=bolt://localhost:7689
|
||||
NEO4J_AUTH=neo4j:password
|
||||
```
|
||||
|
||||
### CLI Parameters
|
||||
|
||||
```bash
|
||||
# Skip ontology critique step
|
||||
ontocast --skip-ontology-critique
|
||||
|
||||
# Process only first N chunks (for testing)
|
||||
ontocast --head-chunks 5
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Triple Store Setup
|
||||
|
||||
OntoCast supports multiple triple store backends with automatic fallback:
|
||||
|
||||
1. **Apache Fuseki** (Recommended) - Native RDF with SPARQL support
|
||||
2. **Neo4j with n10s** - Graph database with RDF capabilities
|
||||
3. **Filesystem** (Fallback) - Local file-based storage
|
||||
|
||||
When multiple triple stores are configured, **Fuseki is preferred over Neo4j**.
|
||||
|
||||
### Quick Setup with Docker
|
||||
|
||||
**Fuseki:**
|
||||
```bash
|
||||
cd docker/fuseki
|
||||
cp .env.example .env
|
||||
# Edit .env with your values
|
||||
docker compose --env-file .env fuseki up -d
|
||||
```
|
||||
|
||||
**Neo4j:**
|
||||
```bash
|
||||
cd docker/neo4j
|
||||
cp .env.example .env
|
||||
# Edit .env with your values
|
||||
docker compose --env-file .env neo4j up -d
|
||||
```
|
||||
|
||||
See [Triple Store Setup](docs/user_guide/triple_stores.md) for detailed instructions.
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Quick Start Guide](docs/getting_started/quickstart.md) - Get started quickly
|
||||
- [Configuration System](docs/user_guide/configuration.md) - Detailed configuration guide
|
||||
- [Triple Store Setup](docs/user_guide/triple_stores.md) - Triple store configuration
|
||||
- [User Guide](docs/user_guide/concepts.md) - Core concepts and workflow
|
||||
- [API Reference](docs/reference/onto.md) - Detailed API documentation
|
||||
|
||||
---
|
||||
|
||||
## Recent Changes
|
||||
|
||||
### Ontology Management Improvements
|
||||
|
||||
- **Automatic Versioning**: Semantic version increment based on change analysis (MAJOR/MINOR/PATCH)
|
||||
- **Hash-Based Lineage**: Git-style versioning with parent hashes for tracking ontology evolution
|
||||
- **Multiple Version Storage**: Versions stored as separate named graphs in Fuseki triple stores
|
||||
- **Timestamp Tracking**: `updated_at` field tracks when ontology was last modified
|
||||
- **Smart Version Analysis**: Analyzes ontology changes (classes, properties, instances) to determine appropriate version bump
|
||||
|
||||
### GraphUpdate System
|
||||
|
||||
- **Token Efficiency**: LLM outputs structured SPARQL operations (insert/delete) instead of full TTL graphs
|
||||
- **Incremental Updates**: Only changes are generated, dramatically reducing token usage
|
||||
- **Structured Operations**: TripleOp operations with explicit prefix declarations for precise updates
|
||||
- **SPARQL Generation**: Automatic conversion of operations to executable SPARQL queries
|
||||
|
||||
### Budget Tracking
|
||||
|
||||
- **LLM Statistics**: Tracks API calls, characters sent/received for cost monitoring
|
||||
- **Triple Metrics**: Tracks ontology and facts triples generated per operation
|
||||
- **Summary Reports**: Budget summaries logged at end of processing
|
||||
- **Integrated Tracking**: Budget tracker integrated into AgentState for clean dependency injection
|
||||
|
||||
### Configuration System Overhaul
|
||||
|
||||
- **Hierarchical Configuration**: New `ToolConfig` and `ServerConfig` structure
|
||||
- **Environment Variables**: Support for `.env` files and environment variables
|
||||
- **Type Safety**: Full type safety with Python 3.12 union syntax
|
||||
- **API Key**: Changed from `OPENAI_API_KEY` to `LLM_API_KEY` for consistency
|
||||
- **Dependency Injection**: Removed global variables, implemented proper DI
|
||||
|
||||
### Enhanced Features
|
||||
|
||||
- **CLI Parameters**: New `--skip-ontology-critique` and `--skip-facts-rendering` parameters
|
||||
- **RDFGraph Operations**: Improved `__iadd__` method with proper prefix binding
|
||||
- **Triple Store Management**: Better separation between filesystem and external stores
|
||||
- **Serialization Interface**: Unified `serialize()` method for storing Ontology and RDFGraph objects
|
||||
- **Error Handling**: Improved error handling and validation
|
||||
|
||||
See [CHANGELOG.md](CHANGELOG.md) for complete details.
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from ontocast.config import Config
|
||||
from ontocast.toolbox import ToolBox
|
||||
|
||||
# Load configuration
|
||||
config = Config()
|
||||
|
||||
# Initialize tools
|
||||
tools = ToolBox(config)
|
||||
|
||||
# Process documents
|
||||
# ... (use tools for processing)
|
||||
```
|
||||
|
||||
### Server Usage
|
||||
|
||||
```bash
|
||||
# Start server with custom configuration
|
||||
ontocast \
|
||||
--env-path .env \
|
||||
--working-directory /data/working \
|
||||
--ontology-directory /data/ontologies \
|
||||
--skip-ontology-critique \
|
||||
--head-chunks 10
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! Please see our [Contributing Guide](docs/contributing.md) for details.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
- **Documentation**: [docs/](docs/)
|
||||
- **Issues**: [GitHub Issues](https://github.com/growgraph/ontocast/issues)
|
||||
- **Discussions**: [GitHub Discussions](https://github.com/growgraph/ontocast/discussions)
|
||||
Reference in New Issue
Block a user