lasta/AI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ontology

Browse Source
This commit is contained in:
LASTA_DEV01\lasta
2026-05-13 19:57:34 +09:00
parent 2e9204243d
commit 9e88f4c7ad
4310 changed files with 48538 additions and 905279 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
ontology_platform/vendored/ontocast/docs/user_guide/concepts.md Normal file
View File

@@ -0,0 +1,61 @@
# Concepts
Here we introduce the main concepts of OntoCast, a framework for transforming data into semantic triples.
## Ontology Management
OntoCast manages ontologies with automatic versioning and timestamp tracking:
- **Semantic Versioning**: Automatic version increments (MAJOR/MINOR/PATCH) based on change analysis
- **Hash-Based Lineage**: Git-style versioning with parent hashes for tracking ontology evolution
- **Multiple Versions**: Versions stored as separate named graphs in Fuseki triple stores
- **Timestamp Tracking**: `updated_at` field tracks when ontology was last modified
- **Smart Analysis**: Analyzes ontology changes (classes, properties, instances) to determine appropriate version bump:
- **MAJOR**: Substantial breaking changes (deletions of classes/properties)
- **MINOR**: New features (new classes/properties) or any deletions
- **PATCH**: Updates to existing structures (instances, descriptions, small changes)
- **Property Syncing**: Version and timestamp are synced to the RDF graph as `owl:versionInfo` and `dcterms:modified`
- **Versioned IRIs**: Each version gets a unique IRI with hash fragment for storage organization
## GraphUpdate System
OntoCast uses a token-efficient GraphUpdate system for incremental graph modifications:
- **Structured Operations**: LLM outputs `GraphUpdate` objects containing `TripleOp` operations (insert/delete) instead of full TTL graphs
- **Token Efficiency**: Only changes are generated, dramatically reducing LLM token usage compared to full graph regeneration
- **SPARQL Generation**: Operations are automatically converted to executable SPARQL queries
- **Incremental Updates**: Graph updates are applied incrementally, allowing for precise modifications
- **Operation Types**: Supports both `insert` and `delete` operations with explicit prefix declarations
- **Custom Queries**: Also supports `GenericSparqlQuery` for complex custom SPARQL operations
### How GraphUpdate Saves Tokens
Instead of generating the entire graph in Turtle format (which can be thousands of tokens), the LLM now outputs only the changes:
- **Before**: Full TTL graph with all triples (e.g., 5000 tokens)
- **After**: Structured operations with only changes (e.g., 200 tokens)
- **Savings**: Typically 80-95% reduction in output tokens
## Budget Tracking
OntoCast provides comprehensive budget tracking for LLM usage and triple generation:
- **LLM Statistics**: Tracks API calls, characters sent/received for cost monitoring
- **Triple Metrics**: Tracks ontology and facts triples generated per operation
- **Operation Counts**: Tracks number of update operations for both ontology and facts
- **Summary Reports**: Budget summaries logged at end of processing with format:
```
LLM: X calls, Y sent, Z received | Triples: A ontology, B facts
```
- **Integrated Tracking**: Budget tracker integrated into AgentState for clean dependency injection
- **Automatic Updates**: Budget tracker automatically updated when LLM calls are made or triples are generated
## Key Components
- **Ontology**: RDF graph with properties (id, title, description, version, timestamp, hash, parent_hashes)
- **AgentState**: Central state management with budget tracking and GraphUpdate operations
- **ToolBox**: Collection of tools for processing and caching
- **Triple Stores**: Support for filesystem, Fuseki, and Neo4j storage
- **GraphUpdate**: Structured representation of graph modifications as SPARQL operations
- **BudgetTracker**: Lightweight tracker for LLM usage and triple generation statistics

149
ontology_platform/vendored/ontocast/docs/user_guide/configuration.md Normal file
View File

@@ -0,0 +1,149 @@
# Configuration System
OntoCast configuration is powered by Pydantic `BaseSettings` and is loaded from environment variables (typically via `.env`).
## Overview
- Typed config sections with defaults
- Environment variable parsing (including lists and booleans)
- Validation for provider/model compatibility
- Unified `Config` object shared across tools and server
## Configuration Shape
```python
Config
├── tool_config: ToolConfig
│ ├── llm_config: LLMConfig
│ ├── chunk_config: ChunkConfig
│ ├── path_config: PathConfig
│ ├── neo4j: Neo4jConfig
│ ├── fuseki: FusekiConfig
│ ├── domain: DomainConfig
│ ├── web_search: WebSearchConfig
│ └── aggregation: AggregationConfig
└── server: ServerConfig
```
## Environment Variables
### LLM
```bash
LLM_PROVIDER=openai # openai | ollama
LLM_MODEL_NAME=gpt-4o-mini
LLM_TEMPERATURE=0.0
LLM_API_KEY=your_openai_api_key_here # required for openai provider
LLM_BASE_URL=http://localhost:11434 # optional (mainly for ollama)
```
### Server
```bash
PORT=8999
BASE_RECURSION_LIMIT=1000
ESTIMATED_CHUNKS=30
MAX_VISITS=3 # alias for max_visits_per_node
RENDER_MODE=ontology_and_facts # ontology | facts | ontology_and_facts
ONTOLOGY_MAX_TRIPLES=50000 # empty/unset for unlimited
PARALLEL_WORKERS=4
PARALLEL_FACTS_RETRIES=3
PARALLEL_ONTOLOGY_RETRIES=3
ENABLE_ONTOLOGY_CONSOLIDATION=false
```
### Chunking
```bash
CHUNK_BREAKPOINT_THRESHOLD_TYPE=percentile # percentile | standard_deviation | interquartile | gradient
CHUNK_BREAKPOINT_THRESHOLD_AMOUNT=95.0
CHUNK_MIN_SIZE=3000
CHUNK_MAX_SIZE=12000
```
### Triple Stores
```bash
# Fuseki
FUSEKI_URI=http://localhost:3030/test
FUSEKI_AUTH=admin/admin
FUSEKI_DATASET=dataset_name
FUSEKI_ONTOLOGIES_DATASET=ontologies
# Neo4j
NEO4J_URI=bolt://localhost:7687
NEO4J_AUTH=neo4j/test
NEO4J_PORT=7476
NEO4J_BOLT_PORT=7689
```
### Paths and Domain
```bash
CURRENT_DOMAIN=https://example.com
ONTOCAST_WORKING_DIRECTORY=/path/to/working/directory
ONTOCAST_ONTOLOGY_DIRECTORY=/path/to/ontology/files
ONTOCAST_CACHE_DIR=/path/to/cache/directory
```
### Aggregation
```bash
AGG_EMBEDDING_MODEL=paraphrase-multilingual-MiniLM-L12-v2
AGG_SIMILARITY_THRESHOLD=0.80
```
### Web Search
```bash
WEB_SEARCH_ENABLED=false
WEB_SEARCH_PROVIDER=duckduckgo
WEB_SEARCH_TOP_K=3
WEB_SEARCH_TIMEOUT_SECONDS=8.0
WEB_SEARCH_MAX_SNIPPET_CHARS=400
WEB_SEARCH_MAX_TOTAL_CHARS=1800
WEB_SEARCH_ONTOLOGY_RENDER_ENABLED=true
WEB_SEARCH_ONTOLOGY_CRITIC_ENABLED=true
WEB_SEARCH_FACTS_RENDER_ENABLED=false
WEB_SEARCH_FACTS_CRITIC_ENABLED=false
WEB_SEARCH_PLANNER_ENABLED=true
WEB_SEARCH_PLANNER_MAX_QUERIES=3
WEB_SEARCH_PLANNER_MIN_QUERY_CHARS=12
WEB_SEARCH_PLANNER_MIN_CONFIDENCE=0.35
WEB_SEARCH_REUSE_EVIDENCE_ACROSS_ATTEMPT=true
WEB_SEARCH_MIN_SNIPPET_CHARS=40
WEB_SEARCH_ALLOWED_DOMAINS= # comma-separated
WEB_SEARCH_BLOCKED_DOMAINS= # comma-separated
WEB_SEARCH_REGION=wt-wt
WEB_SEARCH_SAFESEARCH=moderate
```
Search is "search-later": nodes run without search first, and only request external evidence when needed.
## Usage
```python
from ontocast.config import Config
config = Config()
tool_config = config.get_tool_config()
print(config.server.port)
print(config.server.max_visits_per_node)
print(tool_config.llm_config.provider)
print(tool_config.path_config.cache_dir)
```
## Validation Notes
- `LLM_PROVIDER=openai` requires `LLM_API_KEY`.
- `LLM_MODEL_NAME` must match the selected provider family.
- `MAX_VISITS` is supported as an alias for `max_visits_per_node`.
- `WEB_SEARCH_ALLOWED_DOMAINS` and `WEB_SEARCH_BLOCKED_DOMAINS` accept comma-separated values.
## Recommended Workflow
1. Copy `.env.example` to `.env`.
2. Fill in LLM credentials and backend settings.
3. Start with defaults for chunking/search/aggregation.
4. Tune only after inspecting extraction quality and runtime.

429
ontology_platform/vendored/ontocast/docs/user_guide/llm_caching.md Normal file
View File

@@ -0,0 +1,429 @@
# LLM Caching
OntoCast includes automatic LLM response caching to improve performance, reduce API costs, and enable offline testing capabilities.
---
## Overview
The LLM caching system automatically caches responses from language model providers, ensuring that identical queries return cached results instead of making new API calls. This provides several benefits:
- **Performance**: Cached responses return instantly
- **Cost Reduction**: Avoids duplicate API calls
- **Offline Testing**: Tests can run without API access
- **Transparency**: No configuration required - works automatically
---
## Shared Caching Architecture
OntoCast uses a **shared caching architecture** where:
- **Single Cacher Instance**: One `Cacher` object manages all caching for all tools
- **Tool-Specific Subdirectories**: Each tool gets its own subdirectory within the shared cache
- **Dependency Injection**: Tools receive the shared Cacher instance through their constructors
- **Organized Storage**: Cache files are organized by tool type (llm/, converter/, chunker/)
### Benefits
1. **Memory Efficiency**: Single cache instance instead of multiple
2. **Consistent Configuration**: All tools use the same cache directory settings
3. **Centralized Management**: Easy to clear, monitor, and manage all caches
4. **Better Organization**: Clear separation of cache files by tool type
---
## How It Works
### Shared Caching
OntoCast uses a shared caching system where all tools share a single Cacher instance:
```python
from ontocast.tool.llm import LLMTool
from ontocast.config import LLMConfig
from ontocast.tool.cache import Cacher
# Create shared cache instance
shared_cache = Cacher()
# Create LLM tool with shared cache
llm_config = LLMConfig(
provider="openai",
model_name="gpt-4o-mini",
api_key="your-api-key"
)
llm_tool = LLMTool.create(config=llm_config, cache=shared_cache)
# First call - hits API and caches response
response1 = llm_tool("What is the capital of France?")
# Second call - returns cached response instantly
response2 = llm_tool("What is the capital of France?")
```
### Cache Key Generation
Cache keys are generated based on:
- LLM provider and model
- Prompt text
- Temperature and other parameters
- API endpoint URL
This ensures that different configurations or parameters result in separate cache entries.
---
## Cache Locations
### Default Locations
The system automatically selects appropriate cache directories:
- **Tests**: `.test_cache/llm/` in the current working directory
- **Windows**: `%USERPROFILE%\AppData\Local\ontocast\llm\`
- **Unix/Linux**: `~/.cache/ontocast/llm/` (or `$XDG_CACHE_HOME/ontocast/llm/`)
### Environment Variables
Set the cache directory via environment variables:
```bash
# OntoCast cache directory (recommended)
export ONTOCAST_CACHE_DIR=/path/to/custom/cache
# Or use XDG cache home (affects all XDG-compliant applications)
export XDG_CACHE_HOME=/path/to/custom/cache
```
### CLI Parameter
Specify cache directory via command line:
```bash
ontocast --env-path .env --working-directory ./work --cache-dir /custom/cache/path
```
---
## Cache Management
### Cache Structure
The cache directory contains organized subdirectories:
```
cache_dir/
├── openai/
│ ├── gpt-4o-mini/
│ │ ├── prompt_hash_1.json
│ │ └── prompt_hash_2.json
│ └── gpt-4/
│ └── prompt_hash_3.json
└── ollama/
└── llama2/
└── prompt_hash_4.json
```
### Cache Files
Each cached response is stored as a JSON file containing:
- Original prompt and parameters
- Response content
- Metadata (timestamp, model info)
- Cache key hash
---
## Testing with Caching
### Offline Testing
Cached responses enable offline testing:
```python
# First run - with API access
pytest test_llm_functionality.py
# Subsequent runs - offline (uses cached responses)
pytest test_llm_functionality.py
```
### Test Isolation
Each test run uses a separate cache directory (`.test_cache/llm/`) to avoid interference between tests.
---
## Performance Benefits
### Speed Improvements
- **First Call**: Normal API response time
- **Cached Calls**: Near-instant response (< 1ms)
- **Batch Processing**: Significant speedup for repeated operations
### Cost Savings
- **Development**: Avoid repeated API calls during development
- **Testing**: Run tests without API costs
- **Production**: Reduce API usage for common queries
---
## Best Practices
### Development
1. **Use Default Locations**: Let the system choose appropriate cache directories
2. **Version Control**: Add cache directories to `.gitignore`
3. **Cleanup**: Periodically clean old cache files
### Production
1. **Persistent Storage**: Use persistent cache directories
2. **Monitoring**: Monitor cache hit rates
3. **Maintenance**: Implement cache cleanup strategies
### Testing
1. **Isolated Caches**: Each test run gets its own cache
2. **Deterministic**: Cached responses ensure consistent test results
3. **Offline Capability**: Tests can run without API access
---
## Troubleshooting
### Common Issues
1. **Cache Not Working**: Check directory permissions
2. **Stale Responses**: Clear cache directory
3. **Disk Space**: Monitor cache directory size
### Debug Cache
```python
from ontocast.tool.llm import LLMTool
# Check cache directory
llm_tool = LLMTool.create(config=llm_config)
print(f"Cache directory: {llm_tool.cache.tool_cache_dir}")
# List cached files
cache_files = list(llm_tool.cache.tool_cache_dir.glob("**/*.json"))
print(f"Cached responses: {len(cache_files)}")
```
### Clear Cache
```python
import shutil
from pathlib import Path
# Clear entire cache
cache_dir = Path.home() / ".cache" / "ontocast" / "llm"
if cache_dir.exists():
shutil.rmtree(cache_dir)
print("Cache cleared!")
```
---
## Advanced Usage
### Custom Cache Implementation
For advanced use cases, you can implement custom caching by extending the Cacher class:
```python
from ontocast.tool.llm import LLMTool
from ontocast.tool.cache import Cacher
from pathlib import Path
class CustomLLMTool(LLMTool):
def __init__(self, config, **kwargs):
super().__init__(config, **kwargs)
# Override with custom cache
self.cache = Cacher(subdirectory="llm", cache_dir=Path("/custom/cache"))
```
### Cache Statistics
```python
from ontocast.tool.llm import LLMTool
# Get cache statistics
llm_tool = LLMTool.create(config=llm_config)
stats = llm_tool.cache.get_cache_stats()
print(f"Cache stats: {stats}")
```
---
## Integration with Other Tools
### ToolBox Integration
Caching works seamlessly with the ToolBox through a shared Cacher instance:
```python
from ontocast.toolbox import ToolBox
from ontocast.config import Config
# ToolBox automatically creates and uses a shared Cacher
config = Config()
tools = ToolBox(config)
# All tools (LLM, Converter, Chunker) share the same cache instance
result = tools.llm("Process this document")
converted = tools.converter(document_file)
chunks = tools.chunker(text)
```
### Server Integration
The server automatically uses caching for all LLM operations:
```bash
# Start server with automatic caching
ontocast --env-path .env --working-directory /data/working
```
---
## Security Considerations
### Sensitive Data
- Cache files may contain sensitive prompt data
- Ensure proper file permissions on cache directories
- Consider encryption for sensitive deployments
### Access Control
- Restrict access to cache directories
- Use appropriate file system permissions
- Consider network security for shared cache directories
---
## Converter and Chunker Caching
In addition to LLM response caching, OntoCast also includes caching for document conversion and text chunking operations. This helps avoid redundant processing when the same documents or text are processed multiple times.
### Converter Caching
The `ConverterTool` automatically caches document conversion results based on the input file content. This means:
- **PDF files**: If the same PDF is processed multiple times, the conversion to markdown is cached
- **Other documents**: PowerPoint, Word documents, etc. are also cached after conversion
- **Plain text**: Text input is not cached as it doesn't require conversion
### Chunker Caching
The `ChunkerTool` caches chunking results based on:
- **Input text content**: The exact text being chunked
- **Chunking configuration**: All chunking parameters (max_size, min_size, model, etc.)
- **Chunking mode**: Whether semantic or naive chunking is used
This ensures that identical text with identical chunking parameters will return cached results.
### Cache Organization
Caching is organized in subdirectories:
```
~/.cache/ontocast/
├── llm/ # LLM response cache
├── converter/ # Document conversion cache
└── chunker/ # Text chunking cache
```
### Cache Benefits
1. **Faster Processing**: Repeated operations return instantly from cache
2. **Cost Reduction**: Avoids redundant LLM API calls and processing
3. **Consistency**: Identical inputs always produce identical outputs
4. **Offline Capability**: Cached operations work without API access
### Cache Management
You can access cache statistics and management through the tool instances:
```python
from ontocast.tool.converter import ConverterTool
from ontocast.tool.chunk.chunker import ChunkerTool
# Get cache statistics
converter = ConverterTool()
stats = converter.cache.get_cache_stats()
print(f"Converter cache: {stats['total_files']} files, {stats['total_size_bytes']} bytes")
# Clear cache if needed
converter.cache.clear()
# Chunker cache management
chunker = ChunkerTool()
chunker.cache.clear() # Clear chunker cache
```
### Custom Cache Directories
You can specify custom cache directories in several ways:
#### 1. Environment Variables
```bash
# OntoCast cache directory (recommended)
export ONTOCAST_CACHE_DIR=/custom/cache/path
# Or use XDG cache home (affects all XDG-compliant applications)
export XDG_CACHE_HOME=/custom/cache/path
```
#### 2. CLI Parameter
```bash
ontocast --env-path .env --working-directory ./work --cache-dir /custom/cache/path
```
#### 3. Programmatic Configuration
```python
from ontocast.toolbox import ToolBox
from ontocast.config import Config
from pathlib import Path
# Create config and set cache directory
config = Config()
config.tool_config.path_config.cache_dir = Path("/custom/cache/path")
# Create ToolBox with config (cache directory is automatically used)
tools = ToolBox(config)
# All tools will use the same custom cache directory
result = tools.llm("Process this document")
converted = tools.converter(document_file)
chunks = tools.chunker(text)
```
### Cache Key Generation
Cache keys are generated based on:
- **Content hash**: SHA256 hash of the input content
- **Configuration**: All relevant parameters that affect the output
- **Tool-specific parameters**: Model names, chunking modes, etc.
This ensures that different configurations produce different cache entries, even for the same input content.
### Best Practices
1. **Let caching work automatically**: No configuration needed for basic usage
2. **Monitor cache size**: Check cache statistics periodically
3. **Clear cache when needed**: If you change tool configurations significantly
4. **Use custom directories**: For testing or specific deployment scenarios
5. **Cache persistence**: Caches persist between runs for maximum benefit

269
ontology_platform/vendored/ontocast/docs/user_guide/triple_stores.md Normal file
View File

@@ -0,0 +1,269 @@
# Triple Store Configuration
OntoCast supports multiple triple store backends for storing and managing RDF data. This guide covers the setup and configuration of supported triple stores.
---
## Overview
OntoCast supports the following triple store backends:
1. **Apache Fuseki** (Recommended) - Native RDF triple store with SPARQL support
2. **Neo4j with n10s plugin** - Graph database with RDF capabilities
3. **Filesystem** - Local file-based storage (fallback)
When multiple triple stores are configured, OntoCast uses the following priority order:
1. Fuseki (if `FUSEKI_URI` and `FUSEKI_AUTH` are set)
2. Neo4j (if `NEO4J_URI` and `NEO4J_AUTH` are set)
3. Filesystem (default fallback)
---
## Configuration
### Environment Variables
Configure your triple store connection using environment variables in your `.env` file:
```bash
# Fuseki Configuration (Preferred)
FUSEKI_URI=http://localhost:3032/test
FUSEKI_AUTH=admin:password
FUSEKI_DATASET=dataset_name
# Neo4j Configuration (Alternative)
NEO4J_URI=bolt://localhost:7689
NEO4J_AUTH=neo4j:password
```
### Configuration Hierarchy
The new configuration system provides better organization:
```python
from ontocast.config import Config
config = Config()
# Access triple store configuration
tool_config = config.get_tool_config()
# Check which triple store is configured
if tool_config.fuseki.uri and tool_config.fuseki.auth:
print("Using Fuseki triple store")
elif tool_config.neo4j.uri and tool_config.neo4j.auth:
print("Using Neo4j triple store")
else:
print("Using filesystem storage")
```
---
## Apache Fuseki Setup
Sample configurations are provided here: [ontocast/docker](https://github.com/growgraph/ontocast/tree/main/docker).
**1. Prepare the environment file:**
```bash
cd docker/fuseki
cp .env.example .env
# Edit with your values
```
**Example `docker/fuseki/.env.example`:**
```bash
IMAGE_VERSION=secoresearch/fuseki:5.1.0
SPEC=test
CONTAINER_NAME="${SPEC}.fuseki"
STORE_FOLDER="$HOME/tmp/${CONTAINER_NAME}"
TS_PORT=3032
TS_PASSWORD="abc123-qwe"
TS_USERNAME="admin"
UID=1000
GID=1000
```
**2. Start/Stop Fuseki:**
```bash
# Start
cd docker/fuseki
docker compose --env-file .env fuseki up -d
# Stop
# (use the container name from your .env, e.g. test.fuseki)
docker compose stop test.fuseki
```
**3. Access Fuseki:**
- Web interface: http://localhost:3032
- Default dataset: `/test`
- SPARQL endpoint: http://localhost:3032/test/sparql
**4. Configure OntoCast for Fuseki:**
```bash
# In your .env file
FUSEKI_URI=http://localhost:3032/test
FUSEKI_AUTH=admin:abc123-qwe
FUSEKI_DATASET=dataset_name
```
---
## Neo4j with n10s Plugin Setup
**1. Prepare the environment file:**
```bash
cd docker/neo4j
cp .env.example .env
# Edit with your values
```
**Example `docker/neo4j/.env.example`:**
```bash
IMAGE_VERSION=neo4j:5.20
SPEC=test
CONTAINER_NAME="${SPEC}.sem.neo4j"
NEO4J_PORT=7476
NEO4J_BOLT_PORT=7689
STORE_FOLDER="$HOME/tmp/${CONTAINER_NAME}"
NEO4J_PLUGINS='["apoc", "graph-data-science", "n10s"]'
NEO4J_AUTH="neo4j/test!passfortesting"
```
**2. Start/Stop Neo4j:**
```bash
# Start
cd docker/neo4j
docker compose --env-file .env neo4j up -d
# Stop
docker compose stop neo4j
```
**3. Access Neo4j:**
- Browser: http://localhost:7476
- Username: `neo4j`
- Password: `test!passfortesting`
- Bolt: bolt://localhost:7689
**4. Configure OntoCast for Neo4j:**
```bash
# In your .env file
NEO4J_URI=bolt://localhost:7689
NEO4J_AUTH=neo4j:test!passfortesting
```
---
## Filesystem Storage (Fallback)
If neither Fuseki nor Neo4j is configured, OntoCast will store ontologies and facts as Turtle files in the working directory.
**No setup required - works out of the box.**
---
## Triple Store Comparison
| Feature | Fuseki | Neo4j + n10s | Filesystem |
|---------|--------|--------------|------------|
| **RDF Native** | ✅ Yes | ⚠️ Via plugin | ✅ Yes |
| **SPARQL** | ✅ Full 1.1 | ❌ Limited | ❌ No |
| **Setup Complexity** | ✅ Simple | ⚠️ Moderate | ✅ Very Simple |
| **Visualization** | ⚠️ Basic | ✅ Excellent | ❌ None |
| **Production Ready** | ✅ Yes | ✅ Yes | ❌ No |
| **Configuration** | ✅ Environment vars | ✅ Environment vars | ✅ Automatic |
---
## Best Practices
- Use **Filesystem** for quick setup and testing
- Use **Fuseki** for RDF-focused or production deployments
- Use **Neo4j** if you need advanced graph analytics or visualization
- Monitor triple store performance and logs
- Backup your data regularly
- Use the `/flush` API endpoint to clean triple stores when needed (see below)
---
## Troubleshooting
### Fuseki
```bash
# Check if Fuseki is running
curl http://localhost:3032/$/ping
# Restart Fuseki
docker compose restart fuseki
# Check dataset exists
curl http://localhost:3032/$/datasets
```
### Neo4j
```bash
# Check if Neo4j is running
curl http://localhost:7476
# Check n10s plugin
cypher-shell -u neo4j -p test!passfortesting "CALL n10s.graphconfig.show()"
```
### Common Problems
- **Connection Refused**: Triple store not running
- **Authentication Failed**: Incorrect credentials in environment variables
- **Dataset Not Found**: Dataset not created in Fuseki
- **Plugin Not Loaded**: n10s plugin not installed in Neo4j
- **Configuration Not Loaded**: Check `.env` file and environment variable names
---
## Flushing Triple Store Data
You can clean/flush data from the triple store using the `/flush` API endpoint. This endpoint allows you to explicitly delete data when needed.
### Using the Flush Endpoint
```bash
# Clean all datasets (Fuseki) or entire database (Neo4j)
curl -X POST http://localhost:8999/flush
# Clean specific Fuseki dataset
curl -X POST "http://localhost:8999/flush?dataset=my_dataset"
```
**For Fuseki:**
- If no `dataset` parameter is provided, both the main dataset and ontologies dataset are cleaned
- If a `dataset` parameter is provided, only that specific dataset is cleaned
**For Neo4j:**
- The `dataset` parameter is ignored (Neo4j doesn't support datasets)
- All nodes and relationships are deleted
**Warning:** This operation is irreversible and will delete all data. Use with caution in production environments!
---
## Migration from Previous Versions
If you're upgrading from a previous version of OntoCast:
1. **Update Environment Variables**: The configuration system has been refactored
2. **Check Triple Store Settings**: Ensure your triple store configuration is properly set
3. **Test Configuration**: Use the new configuration system to verify your setup
```python
# Test your configuration
from ontocast.config import Config
config = Config()
print("Configuration loaded successfully!")
print(f"LLM Provider: {config.tool_config.llm_config.provider}")
print(f"Working Directory: {config.tool_config.path_config.working_directory}")
```

314
ontology_platform/vendored/ontocast/docs/user_guide/user_instructions.md Normal file
View File

@@ -0,0 +1,314 @@
# User Instructions
User instructions allow you to provide specific guidance to OntoCast about what to focus on during ontology and facts extraction. This feature is particularly useful when you want to direct the AI's attention to specific types of entities, relationships, or concepts in your documents.
---
## Overview
User instructions work by injecting custom instructions into the AI prompts used during:
- **Ontology Extraction**: When the system extracts domain concepts and relationships
- **Facts Extraction**: When the system extracts specific facts from your documents
This allows you to customize the extraction process based on your specific needs and domain requirements.
---
## How User Instructions Work
### 1. Ontology User Instructions
Ontology user instructions guide the AI when extracting domain concepts and relationships from your documents. These instructions help focus on specific types of entities or relationships.
**Example:**
```
Focus on extracting geographical locations, organizations, and their relationships. Pay special attention to company mergers, acquisitions, and partnerships.
```
### 2. Facts User Instructions
Facts user instructions guide the AI when extracting specific facts and instances from your documents. These instructions help focus on particular types of facts or data points.
**Example:**
```
Extract financial data, dates, and numerical values. Focus on revenue, profit, and growth metrics. Include all monetary amounts with proper currency information.
```
---
## Usage Methods
### 1. JSON API Requests
When sending JSON requests to the API, include user instructions in your payload:
```json
{
"text": "Your document text here...",
"ontology_user_instruction": "Focus on extracting geographical locations and organizations",
"facts_user_instruction": "Extract financial data and numerical values with proper currency information"
}
```
### 2. Form Data (Multipart)
When using multipart form data, include user instructions as form fields:
```bash
curl -X POST http://localhost:8999/process \
-F "file=@document.pdf" \
-F "ontology_user_instruction=Focus on extracting geographical locations and organizations" \
-F "facts_user_instruction=Extract financial data and numerical values"
```
### 3. Programmatic Usage
When using OntoCast programmatically, set user instructions in the AgentState:
```python
from ontocast.onto.state import AgentState
# Create state with user instructions
state = AgentState(
input_text="Your document text...",
ontology_user_instruction="Focus on extracting geographical locations and organizations",
facts_user_instruction="Extract financial data and numerical values"
)
```
---
## Best Practices
### 1. Be Specific and Clear
**Good:**
```
Focus on extracting company names, financial metrics, and business relationships. Pay special attention to revenue, profit, and growth data.
```
**Avoid:**
```
Extract everything important.
```
### 2. Use Domain-Specific Language
**Good:**
```
Extract medical diagnoses, symptoms, treatments, and patient information. Focus on ICD-10 codes and medical terminology.
```
**Avoid:**
```
Extract medical stuff.
```
### 3. Provide Context
**Good:**
```
Extract legal entities, court cases, and legal relationships. Focus on case numbers, dates, and legal precedents mentioned in the document.
```
**Avoid:**
```
Extract legal information.
```
### 4. Specify Data Types
**Good:**
```
Extract numerical data with proper units (currency, percentages, measurements). Include dates in ISO format and geographical coordinates.
```
**Avoid:**
```
Extract numbers and dates.
```
---
## Common Use Cases
### 1. Financial Documents
**Ontology Instruction:**
```
Focus on extracting financial concepts, business entities, and economic relationships. Pay attention to revenue streams, cost structures, and financial metrics.
```
**Facts Instruction:**
```
Extract all monetary amounts with currency codes, percentages, and financial ratios. Include dates for financial periods and growth rates.
```
### 2. Medical Documents
**Ontology Instruction:**
```
Focus on extracting medical conditions, treatments, symptoms, and healthcare relationships. Pay attention to medical terminology and clinical concepts.
```
**Facts Instruction:**
```
Extract patient information, medical codes (ICD-10, CPT), dosages, and treatment timelines. Include all medical measurements and lab values.
```
### 3. Legal Documents
**Ontology Instruction:**
```
Focus on extracting legal entities, court cases, legal relationships, and regulatory frameworks. Pay attention to legal terminology and precedents.
```
**Facts Instruction:**
```
Extract case numbers, court dates, legal citations, and regulatory compliance information. Include all legal references and precedents.
```
### 4. Scientific Papers
**Ontology Instruction:**
```
Focus on extracting scientific concepts, methodologies, and research relationships. Pay attention to scientific terminology and theoretical frameworks.
```
**Facts Instruction:**
```
Extract experimental data, measurements, statistical results, and research findings. Include all numerical data with proper units and significance levels.
```
---
## Advanced Examples
### 1. Multi-Domain Extraction
```json
{
"text": "Your document text...",
"ontology_user_instruction": "Extract both business and technical concepts. Focus on companies, products, technologies, and their relationships.",
"facts_user_instruction": "Extract business metrics, technical specifications, and performance data. Include all numerical values with proper context."
}
```
### 2. Temporal Focus
```json
{
"text": "Your document text...",
"ontology_user_instruction": "Focus on extracting entities and relationships that are time-sensitive or have temporal aspects.",
"facts_user_instruction": "Extract all dates, time periods, and temporal relationships. Pay special attention to historical events and chronological data."
}
```
### 3. Geographic Focus
```json
{
"text": "Your document text...",
"ontology_user_instruction": "Focus on extracting geographical entities, locations, and spatial relationships.",
"facts_user_instruction": "Extract all geographical coordinates, addresses, and location-specific data. Include all spatial and geographical information."
}
```
---
## Integration with Workflow
User instructions are integrated into the OntoCast workflow at specific points:
1. **Document Processing**: Instructions are extracted from JSON input during document conversion
2. **Ontology Extraction**: Instructions guide the AI when extracting domain concepts
3. **Facts Extraction**: Instructions guide the AI when extracting specific facts
4. **Critique Phase**: Instructions are used during the critique and improvement phases
---
## Troubleshooting
### Common Issues
1. **Instructions Not Applied**: Ensure instructions are properly formatted in your JSON payload
2. **Vague Results**: Make instructions more specific and detailed
3. **Missing Data**: Check if instructions are too restrictive or unclear
### Debug Tips
1. **Check Logs**: Look for debug messages about user instructions in the server logs
2. **Test with Simple Instructions**: Start with basic instructions and refine
3. **Validate JSON**: Ensure your JSON payload is properly formatted
### Example Debug Output
```
DEBUG - Set ontology user instruction: Focus on extracting geographical locations and organizations
DEBUG - Set facts user instruction: Extract financial data and numerical values
```
---
## API Reference
### Request Format
```json
{
"text": "string",
"ontology_user_instruction": "string (optional)",
"facts_user_instruction": "string (optional)"
}
```
### Response Format
The response includes the extracted ontology and facts, with user instructions influencing the extraction process:
```json
{
"status": "success",
"ontology": "...",
"facts": "...",
"metadata": {
"ontology_user_instruction": "Focus on extracting geographical locations and organizations",
"facts_user_instruction": "Extract financial data and numerical values"
}
}
```
---
## Best Practices Summary
1. **Be Specific**: Provide clear, detailed instructions
2. **Use Domain Language**: Include relevant terminology
3. **Provide Context**: Explain what you're looking for
4. **Test and Refine**: Start simple and improve based on results
5. **Document Your Instructions**: Keep track of what works best for your use case
---
## Examples by Domain
### Healthcare
- **Ontology**: "Focus on medical conditions, treatments, and healthcare relationships"
- **Facts**: "Extract patient data, medical codes, and clinical measurements"
### Finance
- **Ontology**: "Focus on financial entities, business relationships, and economic concepts"
- **Facts**: "Extract monetary amounts, financial ratios, and economic indicators"
### Legal
- **Ontology**: "Focus on legal entities, court cases, and regulatory frameworks"
- **Facts**: "Extract case numbers, legal citations, and compliance information"
### Scientific
- **Ontology**: "Focus on scientific concepts, methodologies, and research relationships"
- **Facts**: "Extract experimental data, measurements, and research findings"
### Technical
- **Ontology**: "Focus on technical concepts, systems, and technological relationships"
- **Facts**: "Extract technical specifications, performance metrics, and system data"

107
ontology_platform/vendored/ontocast/docs/user_guide/workflow.md Normal file
View File

@@ -0,0 +1,107 @@
# OntoCast Workflow
This document describes the workflow of OntoCast's document processing pipeline.
## Overview
The OntoCast workflow consists of several stages that transform input documents into structured knowledge:
1. **Document Conversion**
- Input documents are converted to markdown format
- Supports various input formats (PDF, DOCX, TXT, MD)
2. **Text Chunking**
- Documents are split into manageable chunks
- Chunks are processed sequentially
- Head chunks are processed first to establish context
3. **Ontology Processing**
- **Selection**: Choose appropriate ontology for content
- **Extraction**: Extract ontological concepts from text using GraphUpdate operations
- **GraphUpdate**: LLM outputs structured SPARQL operations (insert/delete) instead of full TTL
- **Update Application**: GraphUpdate operations are applied incrementally to the ontology graph
- **Sublimation**: Refine and enhance the ontology
- **Criticism**: Validate ontology structure and relationships
- **Versioning**: Automatic semantic version increment based on changes (MAJOR/MINOR/PATCH)
- **Timestamp**: Tracks last update time with `updated_at` field
4. **Fact Processing**
- **Extraction**: Extract factual information from text using GraphUpdate operations
- **GraphUpdate**: LLM outputs structured SPARQL operations for facts updates
- **Update Application**: GraphUpdate operations are applied incrementally to the facts graph
- **Criticism**: Validate extracted facts
- **Aggregation**: Combine facts from all chunks
## Detailed Flow
### 1. Document Input
- Accepts text or file input
- Converts to markdown format
- Preserves document structure
### 2. Text Processing
- Splits text into chunks
- Processes head chunks first
- Maintains context between chunks
### 3. Ontology Management
- Selects relevant ontology
- Extracts new concepts using GraphUpdate operations (token-efficient)
- Applies incremental updates to ontology graph
- Validates relationships
- Refines structure
- Automatically increments version based on change analysis (MAJOR/MINOR/PATCH)
- Updates timestamp when ontology is modified
- Tracks version lineage with hash-based identifiers
### 4. Fact Extraction
- Identifies entities
- Extracts relationships using GraphUpdate operations (token-efficient)
- Applies incremental updates to facts graph
- Validates facts
- Combines information from all chunks
### 5. Output Generation
- Produces RDF graph
- Generates ontology with version and timestamp
- Provides extracted facts
- Reports budget usage (LLM calls, characters sent/received, triples generated)
- Logs budget summary at end of processing
## Configuration Options
The workflow can be configured through command-line parameters:
- `--head-chunks`: Number of chunks to process first
- `--max-visits`: Maximum visits per node
## Best Practices
1. **Chunk Size**
- Keep chunks manageable
- Consider context preservation
- Balance between detail and processing time
2. **Ontology Selection**
- Choose appropriate ontology
- Consider domain specificity
- Allow for ontology evolution
- Monitor version increments to track evolution
3. **Fact Validation**
- Validate extracted facts
- Check for consistency
- Handle contradictions
4. **Resource Management**
- Monitor memory usage
- Control processing time
- Handle large documents
- Review budget summaries to track LLM usage and costs
- Use budget metrics to estimate processing costs for large documents
- GraphUpdate operations significantly reduce token usage compared to full graph generation
- Monitor triple generation metrics to understand graph growth
## Next Steps
- Check [API Reference](../reference/onto.md)