144 lines
3.7 KiB
Markdown
144 lines
3.7 KiB
Markdown
|
# Crawl4AI v0.6.0 Release Notes
|
|||
|
|
|
|||
|
|
We're excited to announce the release of **Crawl4AI v0.6.0**, our biggest and most feature-rich update yet. This version introduces major architectural upgrades, brand-new capabilities for geo-aware crawling, high-efficiency scraping, and real-time streaming support for scalable deployments.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Highlights
|
|||
|
|
|
|||
|
|
### 1. **World-Aware Crawlers**
|
|||
|
|
Crawl as if you’re anywhere in the world. With v0.6.0, each crawl can simulate:
|
|||
|
|
- Specific GPS coordinates
|
|||
|
|
- Browser locale
|
|||
|
|
- Timezone
|
|||
|
|
|
|||
|
|
Example:
|
|||
|
|
```python
|
|||
|
|
CrawlerRunConfig(
|
|||
|
|
url="https://browserleaks.com/geo",
|
|||
|
|
locale="en-US",
|
|||
|
|
timezone_id="America/Los_Angeles",
|
|||
|
|
geolocation=GeolocationConfig(
|
|||
|
|
latitude=34.0522,
|
|||
|
|
longitude=-118.2437,
|
|||
|
|
accuracy=10.0
|
|||
|
|
)
|
|||
|
|
)
|
|||
|
|
```
|
|||
|
|
Great for accessing region-specific content or testing global behavior.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### 2. **Native Table Extraction**
|
|||
|
|
Extract HTML tables directly into usable formats like Pandas DataFrames or CSV with zero parsing hassle. All table data is available under `result.media["tables"]`.
|
|||
|
|
|
|||
|
|
Example:
|
|||
|
|
```python
|
|||
|
|
raw_df = pd.DataFrame(
|
|||
|
|
result.media["tables"][0]["rows"],
|
|||
|
|
columns=result.media["tables"][0]["headers"]
|
|||
|
|
)
|
|||
|
|
```
|
|||
|
|
This makes it ideal for scraping financial data, pricing pages, or anything tabular.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### 3. **Browser Pooling & Pre-Warming**
|
|||
|
|
We've overhauled browser management. Now, multiple browser instances can be pooled and pages pre-warmed for ultra-fast launches:
|
|||
|
|
- Reduces cold-start latency
|
|||
|
|
- Lowers memory spikes
|
|||
|
|
- Enhances parallel crawling stability
|
|||
|
|
|
|||
|
|
This powers the new **Docker Playground** experience and streamlines heavy-load crawling.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### 4. **Traffic & Snapshot Capture**
|
|||
|
|
Need full visibility? You can now capture:
|
|||
|
|
- Full network traffic logs
|
|||
|
|
- Console output
|
|||
|
|
- MHTML page snapshots for post-crawl audits and debugging
|
|||
|
|
|
|||
|
|
No more guesswork on what happened during your crawl.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### 5. **MCP API and Streaming Support**
|
|||
|
|
We’re exposing **MCP socket and SSE endpoints**, allowing:
|
|||
|
|
- Live streaming of crawl results
|
|||
|
|
- Real-time integration with agents or frontends
|
|||
|
|
- A new Playground UI for interactive crawling
|
|||
|
|
|
|||
|
|
This is a major step towards making Crawl4AI real-time ready.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### 6. **Stress-Test Framework**
|
|||
|
|
Want to test performance under heavy load? v0.6.0 includes a new memory stress-test suite that supports 1,000+ URL workloads. Ideal for:
|
|||
|
|
- Load testing
|
|||
|
|
- Performance benchmarking
|
|||
|
|
- Validating memory efficiency
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Core Improvements
|
|||
|
|
- Robots.txt compliance
|
|||
|
|
- Proxy rotation support
|
|||
|
|
- Improved URL normalization and session reuse
|
|||
|
|
- Shared data across crawler hooks
|
|||
|
|
- New page routing logic
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Breaking Changes & Deprecations
|
|||
|
|
- Legacy `crawl4ai/browser/*` modules are removed. Update imports accordingly.
|
|||
|
|
- `AsyncPlaywrightCrawlerStrategy.get_page` now uses a new function signature.
|
|||
|
|
- Deprecated markdown generator aliases now point to `DefaultMarkdownGenerator` with warning.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Miscellaneous Updates
|
|||
|
|
- FastAPI validators replaced custom validation logic
|
|||
|
|
- Docker build now based on a Chromium layer
|
|||
|
|
- Repo-wide cleanup: ~36,000 insertions, ~5,000 deletions
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## New Examples Included
|
|||
|
|
- Geo-location crawling
|
|||
|
|
- Network + console log capture
|
|||
|
|
- Docker MCP API usage
|
|||
|
|
- Markdown selector usage
|
|||
|
|
- Crypto project data extraction
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Watch the Release Video
|
|||
|
|
Want a visual walkthrough of all these updates? Watch the video:
|
|||
|
|
🔗 https://youtu.be/9x7nVcjOZks
|
|||
|
|
|
|||
|
|
If you're new to Crawl4AI, start here:
|
|||
|
|
🔗 https://www.youtube.com/watch?v=xo3qK6Hg9AA&t=15s
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Join the Community
|
|||
|
|
We’ve just opened up our **Discord** for the public. Join us to:
|
|||
|
|
- Ask questions
|
|||
|
|
- Share your projects
|
|||
|
|
- Get help or contribute
|
|||
|
|
|
|||
|
|
💬 https://discord.gg/wpYFACrHR4
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Install or Upgrade
|
|||
|
|
```bash
|
|||
|
|
pip install -U crawl4ai
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Live long and import crawl4ai. 🖖
|
|||
|
|
|