scrape
GitHub使用Scrapling框架进行网页抓取和数据提取。支持静态HTML、反爬网站及JS渲染页面，具备TLS指纹伪装和自动绕过验证码功能，适用于商品比价、多页爬虫及状态保持场景。
Trigger Scenarios
Install
npx skills add NeverSight/learn-skills.dev --skill scrape -g -y
SKILL.md
Frontmatter
{
"name": "scrape",
"description": "Web scraping and data extraction using Scrapling. Use this skill whenever the user wants to scrape a website, crawl pages, extract data from URLs, search for products\/prices online, compare prices across sites, or pull structured data from web pages. Also use when the user mentions Coupang, Amazon, shopping search, price comparison, or any task involving fetching and parsing web content — even if they don't explicitly say \"scrape\"."
}
Scrape — Web Scraping with Scrapling
Extract data from any website using Scrapling, a Python scraping framework with TLS fingerprint spoofing and anti-bot bypass.
Environment
- Python:
~/.scrapling-venv/bin/python3 - Chromium libs:
~/.local/lib/chromium-deps/usr/lib/x86_64-linux-gnu/ - Temp scripts: write to
/tmp/scrapling_task.py, execute from there
When using StealthyFetcher or DynamicFetcher (headless Chromium), prefix the command with:
LD_LIBRARY_PATH="$HOME/.local/lib/chromium-deps/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"
Choosing a Fetcher
Pick the lightest fetcher that works. Escalate only on failure.
| Situation | Fetcher | Speed | Chromium? |
|---|---|---|---|
| Static HTML, no bot protection | Fetcher |
⚡ fast | No |
| Akamai, Cloudflare, bot walls | StealthyFetcher |
🐢 slow | Yes (LD_LIBRARY_PATH) |
| JS-rendered SPA (React, Vue) | DynamicFetcher |
🐢 slow | Yes (LD_LIBRARY_PATH) |
| Multi-page crawl | Spider |
⚡ parallel | No |
| Stateful (cookies/login) | StealthySession |
🐢 slow | Yes (LD_LIBRARY_PATH) |
Escalation order: Try Fetcher first. If 403/418/captcha → switch to StealthyFetcher. If content still missing (JS render) → DynamicFetcher.
Writing a Scraping Script
Every scraping task follows this pattern:
- Write a Python script to
/tmp/scrapling_task.py - Execute with the venv Python
- Parse results and present to user
Fetcher (fast, no browser)
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com', stealthy_headers=True, impersonate="chrome131")
print(f"Status: {page.status}")
# CSS selectors (Scrapy-style)
titles = page.css('h2::text').getall()
links = page.css('a::attr(href)').getall()
# XPath
items = page.xpath('//div[@class="item"]/text()').getall()
Key parameters: impersonate (browser TLS fingerprint), stealthy_headers (real browser headers), follow_redirects, timeout, headers, cookies.
Available impersonate values: chrome131, chrome136, firefox135, safari184, chrome_android, etc.
StealthyFetcher (beats Akamai/Cloudflare)
from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch('https://protected-site.com',
headless=True,
network_idle=True, # wait for all network requests to finish
solve_cloudflare=True, # auto-solve Turnstile
)
Session (cookie persistence)
from scrapling.fetchers import StealthySession
with StealthySession(headless=True) as session:
page1 = session.fetch('https://site.com/login')
page2 = session.fetch('https://site.com/data') # cookies carry over
Spider (multi-page)
from scrapling.spiders import Spider, Response
class MySpider(Spider):
name = "crawler"
start_urls = ["https://example.com/page/1"]
concurrent_requests = 5
async def parse(self, response: Response):
for item in response.css('.product'):
yield {
"name": item.css('.name::text').get(),
"price": item.css('.price::text').get(),
}
next_page = response.css('.next a::attr(href)').get()
if next_page:
yield response.follow(next_page)
result = MySpider().start()
result.items.to_json("/tmp/output.json")
Execution
# Fetcher (no Chromium needed)
~/.scrapling-venv/bin/python3 /tmp/scrapling_task.py
# StealthyFetcher / DynamicFetcher (Chromium needed)
LD_LIBRARY_PATH="$HOME/.local/lib/chromium-deps/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH" \
~/.scrapling-venv/bin/python3 /tmp/scrapling_task.py
Site-Specific Playbooks
Read references/sites.md for site-specific parsing patterns (Coupang, Amazon, etc.). These change over time — when a known pattern fails, investigate the HTML and update the reference.
Error Recovery
| Error | Fix |
|---|---|
| 403 / Access Denied | Escalate to StealthyFetcher |
| 418 / bot detection | StealthyFetcher + solve_cloudflare=True |
| Empty content (JS) | DynamicFetcher + network_idle=True |
libnspr4.so not found |
Add LD_LIBRARY_PATH prefix |
| Import error | ~/.scrapling-venv/bin/pip install "scrapling[all]" |
| Timeout | Increase timeout parameter |
Output
- Present extracted data in a clean table or summary
- For large datasets, save to
/tmp/as JSON and tell the user the path - Always include source URLs/links in results
Version History
- e0220ca Current 2026-07-05 21:09
