Search
Whole-Site Crawl
Whole-Site Crawl is a search capability available through Tavily — LLM-Optimized Search, Extract, Map, Crawl, Firecrawl — Whole-Site Crawl + Schema-Driven Extraction on Aweb. Walk a domain to a given depth and return LLM-ready content per page. Primary: Firecrawl. Fallback: Tavily Crawl.. Access it through a single unified API with automatic failover and intelligent routing.
Best for
Highest quality
Tavily — LLM-Optimized Search, Extract, Map, Crawl, Firecrawl — Whole-Site Crawl + Schema-Driven Extraction
Premium tier
Most affordable
Tavily — LLM-Optimized Search, Extract, Map, Crawl
Economy tier
Contract
Providers (2)
Public discovery and orchestration
Inspect the live capability descriptor directly, then route orchestration through a capability filter. Generic public execute examples are intentionally withheld until the canonical public execute contract is normalized.
cURL
curl "https://aweblabs.ai/api/v2/capabilities/web.crawl-site"TypeScript
import Aweb from '@aweb/sdk';
const client = new Aweb({
baseUrl: 'https://aweblabs.ai/api/v2',
});
const capability = await client.capabilities.get('web.crawl-site');
console.log(capability.data.runtime.providers);Orchestration pipeline
import Aweb from '@aweb/sdk';
const aweb = new Aweb({ apiKey: process.env.AWEB_API_KEY });
const result = await aweb.orchestrate.run({
query: 'Use Whole-Site Crawl to help with a hello-world task and summarize the output',
capabilities: ['web.crawl-site'],
policy: 'balanced',
});
console.log(result.data.status);