> For the complete documentation index, see [llms.txt](https://agpt.co/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://agpt.co/docs/integrations/block-integrations/crawl-1.md).

# Tavily Crawl

Blocks for crawling a website and extracting page content with Tavily.

## Tavily Crawl

### What it is

Crawls a website with Tavily, following links from the root URL and extracting page content

### How it works

Starting from a root URL, the block follows links up to `max_depth` and extracts the content of each page it visits, stopping at `limit` pages. Optional natural-language `instructions` steer which pages to follow (for example, "Find all the API reference pages"), and `extract_depth` and `format` control how much content is captured and whether it comes back as markdown or text.

Each crawled page is returned on `results` (and one at a time on `result`) with its extracted content. A crawl combines mapping and extraction, so its credit cost is the sum of both per Tavily's schedule; actual spend is read from the API's usage report.

### Inputs

| Input          | Description                                                                                             | Type                  | Required |
| -------------- | ------------------------------------------------------------------------------------------------------- | --------------------- | -------- |
| url            | The root URL to start crawling from                                                                     | str                   | Yes      |
| limit          | Maximum number of pages to crawl                                                                        | int                   | No       |
| max\_depth     | Maximum link depth from the root URL                                                                    | int                   | No       |
| instructions   | Natural language instructions guiding which pages to crawl (e.g. 'Find all the API reference pages')    | str                   | No       |
| extract\_depth | Depth of the extraction: basic or advanced (retrieves more data, including tables and embedded content) | "basic" \| "advanced" | No       |
| format         | The format of the extracted content                                                                     | "markdown" \| "text"  | No       |

### Outputs

| Output  | Description                              | Type                     |
| ------- | ---------------------------------------- | ------------------------ |
| error   | Error message if the crawl failed        | str                      |
| results | List of crawled pages with their content | List\[TavilyPageContent] |
| result  | Single crawled page                      | TavilyPageContent        |

### Possible use case

**Documentation Ingestion**: Crawl a docs site and extract every page to build a knowledge base or RAG pipeline.

**Targeted Site Harvesting**: Use `instructions` to gather only pricing, product, or reference pages from a large site.

**Competitive Content Snapshots**: Capture the content of a section of a site in one pass for analysis or change tracking.

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://agpt.co/docs/integrations/block-integrations/crawl-1.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.