Agent Skills › tavily-ai/skills › tavily-crawl

tavily-crawl

GitHub

使用 Tavily CLI 爬取网站多页面内容,支持路径过滤、深度控制及语义指令。可将文档批量提取为本地 Markdown 文件,适用于离线下载、批量抓取或为 LLM 提供上下文数据。

skills/tavily-crawl/SKILL.md tavily-ai/skills

触发场景

需要获取网站多个页面的内容 下载文档用于离线使用 批量提取特定目录下的所有页面 将网站保存为本地 markdown 文件

安装

npx skills add tavily-ai/skills --skill tavily-crawl -g -y
更多选项

不安装直接使用

npx skills use tavily-ai/skills@tavily-crawl

指定 Agent (Claude Code)

npx skills add tavily-ai/skills --skill tavily-crawl -a claude-code -g -y

安装 repo 全部 skill

npx skills add tavily-ai/skills --all -g -y

预览 repo 内 skill

npx skills add tavily-ai/skills --list

SKILL.md

Frontmatter
{
    "name": "tavily-crawl",
    "description": "Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says \"crawl\", \"get all the pages\", \"download the docs\", \"extract everything under \/docs\", \"bulk extract\", or needs content from many pages on the same domain. Supports depth\/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.\n",
    "allowed-tools": "Bash(tvly *)"
}

tavily crawl

Crawl a website and extract content from multiple pages. Supports saving each page as a local markdown file.

Before running

Crawl requires authentication. Run the requested command directly when tvly is already authenticated; do not add a status check to every invocation.

If tvly is missing, follow the tavily-cli setup. If an installed CLI reports an authentication error, use tvly login for authentication only, or tvly init --skip-skills when guided verification is also useful. Browser-based OAuth is preferred when an interactive user can complete it. --no-browser prints the sign-in link instead of opening it, but still waits for a localhost callback. In an unattended agent or CI environment, leave authentication to the user or use a securely provided TAVILY_API_KEY. Do not start a second login immediately after guided setup has completed.

When to use

  • You need content from many pages on a site (e.g., all /docs/)
  • You want to download documentation for offline use
  • Step 4 in the workflow: search → extract → map → crawl → research

Quick start

# Basic crawl
tvly crawl "https://docs.example.com" --json

# Save each page as a markdown file
tvly crawl "https://docs.example.com" --output-dir ./docs/

# Deeper crawl with limits
tvly crawl "https://docs.example.com" --max-depth 2 --limit 50 --json

# Filter to specific paths
tvly crawl "https://example.com" --select-paths "/api/.*,/guides/.*" --exclude-paths "/blog/.*" --json

# Semantic focus (returns relevant chunks, not full pages)
tvly crawl "https://docs.example.com" --instructions "Find authentication docs" --chunks-per-source 3 --json

Options

Option Description
--max-depth Levels deep (1-5, default: 1)
--max-breadth Links per page (default: 20)
--limit Total pages cap (default: 50)
--instructions Natural language guidance for semantic focus
--chunks-per-source Chunks per page (1-5, requires --instructions)
--extract-depth basic (default) or advanced
--format markdown (default) or text
--select-paths Comma-separated regex patterns to include
--exclude-paths Comma-separated regex patterns to exclude
--select-domains Comma-separated regex for domains to include
--exclude-domains Comma-separated regex for domains to exclude
--allow-external / --no-external Include external links (default: allow)
--include-images Include images
--timeout Max wait (10-150 seconds)
-o, --output Save JSON output to file
--output-dir Save each page as a .md file in directory
--json Structured JSON output

Crawl for context vs. data collection

For agentic use (feeding results to an LLM):

Always use --instructions + --chunks-per-source. Returns only relevant chunks instead of full pages — prevents context explosion.

tvly crawl "https://docs.example.com" --instructions "API authentication" --chunks-per-source 3 --json

For data collection (saving to files):

Use --output-dir without --chunks-per-source to get full pages as markdown files.

tvly crawl "https://docs.example.com" --max-depth 2 --output-dir ./docs/

Tips

  • Start conservative — --max-depth 1, --limit 20 — and scale up.
  • Use --select-paths to focus on the section you need.
  • Use map first to understand site structure before a full crawl.
  • Always set --limit to prevent runaway crawls.

See also

版本历史

  • 778122e 当前 2026-09-09 08:45

    优化认证指引,明确交互式OAuth操作说明,强化技能工作流与CLI设置的对齐。

  • ea5e820 2026-07-25 07:40

同 Skill 集合

skills/tasks/tavily-entity/SKILL.md
skills/tavily-best-practices/SKILL.md
skills/tavily-cli/SKILL.md
skills/tavily-dynamic-search/SKILL.md
skills/tavily-extract/SKILL.md
skills/tavily-map/SKILL.md
skills/tavily-research/SKILL.md
skills/core/tavily-search/SKILL.md
skills/tasks/tavily-list/SKILL.md
skills/tasks/tavily-monitor/SKILL.md
skills/tasks/tavily-research/SKILL.md
skills/tavily-search/SKILL.md

元信息

文件数
0
版本
778122e
Hash
7384255f
收录时间
2026-07-25 07:40

首页 - Wiki
Copyright © 2011-2026 iteam. Current version is 2.156.0. UTC+08:00, 2026-11-18 15:27
浙ICP备14020137号-1