Agent Skills
› leamsigc/ShortsGenerator
› twitter-research-scrape
twitter-research-scrape
GitHub通过Chrome DevTools协议抓取X/Twitter特定话题的帖子，提取互动指标并去重排序，无需API密钥。
Trigger Scenarios
需要收集Twitter/X话题相关帖子数据
缺乏官方API访问权限需手动爬取
Install
npx skills add leamsigc/ShortsGenerator --skill twitter-research-scrape -g -y
SKILL.md
Frontmatter
{
"name": "twitter-research-scrape",
"version": "3.0.0",
"description": "Scrape state — runs the Twitter\/X topic scraper via Chrome DevTools\nProtocol. Searches X by topic, collects 10+ posts across many accounts,\nextracts engagement metrics (likes, retweets, replies, views), ranks them.\nOptionally augments with seed account profiles. No API key needed.\n",
"allowed-tools": [
"Bash",
"Read",
"Write",
"ChromeDevTools"
]
}
Twitter/X Research — Scrape State (v3)
Scrapes topic posts from X/Twitter. Requires Chrome running with --remote-debugging-port=9222 and user logged into X.
Before running
Ensure:
- Chrome is open with
--remote-debugging-port=9222flag - X/Twitter is logged in (tab stays open)
- Config exists at
.twitter-research/<timestamp>/config.json(from setup)
Run
node "$CLAUDE_SKILL_ROOT/scripts/scrape.js" <timestamp> [--data-root <path>]
# fallback: node ./skills/twitter-research/scripts/scrape.js <timestamp>
Important: Keep the X tab in the foreground (visible, not minimized) while this runs. Chrome throttles background tabs.
What the script does
- Searches X by topic (
Toptab, thenLatestif needed) usingconfig.topic+config.topicKeywords - Scrolls to load many posts from different accounts
- Extracts per post: handle, author, text, timestamp, URL, likes/retweets/replies, media
- Visits seed accounts (
config.accountswithisSeed: true) and keeps only topic-relevant tweets - Deduplicates by URL, ranks by engagement, keeps top
minPosts(min 10) - Saves
raw-posts.jsonwithposts[],distinctAccounts,accountsSummary
Manual fallback (Chrome DevTools CLI)
If the script hits an X login wall or returns too few results, do it manually:
- Navigate:
chrome-devtools navigate_page --url "https://x.com/search?q=<topic>&src=typed_query&f=top" - Scroll:
chrome-devtools evaluate_script --function "async () => { for (let i=0;i<12;i++){ window.scrollTo(0, document.body.scrollHeight); await new Promise(r=>setTimeout(r,1800)); } return 'ok'; }" - Extract tweets using
[data-testid="tweet"]selectors (tweetText,User-Name,reply/retweet/likearia-labels) - Merge results into
raw-posts.jsonwith the same schema
Output
.twitter-research/<timestamp>/
├── raw-posts.json # Ranked posts with engagement metrics
└── screenshots/ # Optional screenshots
Version History
- 7b83166 Current 2026-08-29 02:31
