类库
› superspider
Lyx3314844-03/superspider
SuperSpider是一个企业级多语言网络爬虫框架，提供Python、Go、Rust和Java四种运行版本。核心功能包括网页抓取、浏览器自动化、AI内容提取、媒体下载、反反爬虫技术和分布式执行，支持处理动态网页并具备代理池、速率限制等完整能力。
技术栈
gospider go
查看全部依赖 (12)
依赖
github.com/PuerkitoBio/goquery
v1.8.1
github.com/antchfx/htmlquery
v1.3.4
github.com/chromedp/cdproto
v0.0.0-20231011050154-1d073bb38998
github.com/chromedp/chromedp
v0.9.3
github.com/go-chi/chi/v5
v5.2.5
github.com/go-chi/cors
v1.2.2
github.com/go-redis/redis/v8
v8.11.5
github.com/go-sql-driver/mysql
v1.8.1
github.com/lib/pq
v1.10.9
github.com/tidwall/gjson
v1.17.0
golang.org/x/net
v0.33.0
gopkg.in/yaml.v3
v3.0.1
javaspider java
查看全部依赖 (29)
依赖
ch.qos.logback:logback-classic
com.fasterxml.jackson.core:jackson-core
com.fasterxml.jackson.core:jackson-databind
com.google.code.gson:gson
com.rabbitmq:amqp-client
com.squareup.okhttp3:okhttp
io.github.bonigarcia:webdrivermanager
org.apache.commons:commons-pool2
org.apache.httpcomponents:httpclient
org.apache.httpcomponents:httpcore
org.apache.kafka:kafka-clients
org.apache.spark:spark-core_2.13
org.json:json
org.jsoup:jsoup
org.junit.jupiter:junit-jupiter-api
org.junit.jupiter:junit-jupiter-engine
org.openjdk.jmh:jmh-core
org.openjdk.jmh:jmh-generator-annprocess
org.projectlombok:lombok
org.seleniumhq.selenium:selenium-api
org.seleniumhq.selenium:selenium-chrome-driver
org.seleniumhq.selenium:selenium-chromium-driver
org.seleniumhq.selenium:selenium-firefox-driver
org.seleniumhq.selenium:selenium-remote-driver
org.seleniumhq.selenium:selenium-support
org.slf4j:slf4j-api
org.xerial:sqlite-jdbc
org.yaml:snakeyaml
redis.clients:jedis
pyspider python
框架
Flask
测试
Playwright
pytest
网络
Requests
查看全部依赖 (21)
依赖
Pydantic
aiofiles
aiohttp
anthropic
black
ffmpeg-python
flake8
flask-cors
jsonpath-ng
mkdocs
mkdocs-material
mypy
openai
psutil
pytest-asyncio
pytest-cov
pyyaml
redis
selenium
webdriver-manager
yt-dlp
