Research Writing Skill

Scope

Use this skill for research writing and revision when the output is prose, LaTeX, Markdown, or manuscript text. Use office-academic-skill instead when the main deliverable is Word/PPT. Use scientific-toolkit-skill when the task is primarily MATLAB, Python, plotting, statistics, simulation, or literature search.

Writing Principles

• Default to Chinese academic expression unless the user requests English.
• Preserve English titles, formulas, variables, methods, software names, citations, and reference entries.
• Do not invent data, DOI, journal details, experiment settings, results, or author claims.
• Separate 原文/已有数据, 用户确认内容, 根据上下文推断, and 建议性扩展.
• Prefer verifiable technical statements over generic claims.
• When revising, preserve the user's intended meaning and terminology unless a change is clearly needed.

Manuscript Workflow

For a new section or paper draft:

1. Clarify target: journal/conference/thesis/course report, language, length, audience, and required format.
2. Identify source material: paper notes, experiment results, figures, tables, MATLAB/Python outputs, references, advisor comments.
3. Build an argument outline before full prose: problem, gap, method, evidence, contribution, limitation.
4. Draft in coherent paragraphs, not empty slogan bullets.
5. Add source or evidence labels for quantitative claims and literature claims.
6. Revise for logic, specificity, terminology consistency, and citation accuracy.

For revision or polishing:

• Keep claims tied to evidence.
• Replace vague words such as "显著", "先进", "有效", "鲁棒" with measured conditions, comparison baselines, or remove them.
• Check whether each paragraph advances the section's purpose.
• Keep formulas with variable definitions, units, assumptions, and applicable conditions.
• For experimental sections, state dataset/sample, hardware/software, parameters, metrics, baselines, and uncertainty when available.

Section Guides

Use these default moves unless the user's school or journal template overrides them:

• Abstract: problem, method, experiment/data, key result, contribution.
• Introduction: background, unresolved gap, why it matters, proposed approach, contributions.
• Related work: organize by technical theme, compare assumptions and limitations, avoid simple paper-by-paper summaries.
• Methods: model assumptions, variables, workflow, algorithm, implementation details needed for reproduction.
• Experiments: data/source, platform/software, parameters, metrics, baseline, repeated trials, visualization plan.
• Results and discussion: claim first, evidence second, mechanism/explanation third, limitation last.
• Conclusion: answer the research question, summarize evidence, state limitations and next steps.

Bundled References

The folder references/paper-writing/ contains external writing checklists and section patterns adapted as references. Load only the relevant file when needed:

• brainstorming_guide.md for turning an unclear idea into a paper plan.
• section_rhetorical_moves/ for section structure.
• writing_checklists/ for self-diagnosis.
• figure_templates/ for figure planning.
• author_profile/ for editorial heuristics.

These references come from an external systems/networking-oriented repository. Treat them as optional craft guidance, not binding rules, and adapt them to光电信息科学与工程, optics, optoelectronics, sensing, communication, signal processing, and MATLAB simulation work.

Final Checks

Before delivery, state:

• What was drafted or revised.
• Which source material was used.
• Which claims still need user-provided data or citation support.
• Any uncertainty about terminology, parameters, or references.

Astropy

Overview

Astropy is the core Python package for astronomy, providing essential functionality for astronomical research and data analysis. Use astropy for coordinate transformations, unit and quantity calculations, FITS file operations, cosmological calculations, precise time handling, tabular data manipulation, and astronomical image processing.

When to Use This Skill

Use astropy when tasks involve:

• Converting between celestial coordinate systems (ICRS, Galactic, FK5, AltAz, etc.)
• Working with physical units and quantities (converting Jy to mJy, parsecs to km, etc.)
• Reading, writing, or manipulating FITS files (images or tables)
• Cosmological calculations (luminosity distance, lookback time, Hubble parameter)
• Precise time handling with different time scales (UTC, TAI, TT, TDB) and formats (JD, MJD, ISO)
• Table operations (reading catalogs, cross-matching, filtering, joining)
• WCS transformations between pixel and world coordinates
• Astronomical constants and calculations

Quick Start

import astropy.units as u
from astropy.coordinates import SkyCoord
from astropy.time import Time
from astropy.io import fits
from astropy.table import Table
from astropy.cosmology import Planck18

# Units and quantities
distance = 100 * u.pc
distance_km = distance.to(u.km)

# Coordinates
coord = SkyCoord(ra=10.5*u.degree, dec=41.2*u.degree, frame='icrs')
coord_galactic = coord.galactic

# Time
t = Time('2023-01-15 12:30:00')
jd = t.jd  # Julian Date

# FITS files
data = fits.getdata('image.fits')
header = fits.getheader('image.fits')

# Tables
table = Table.read('catalog.fits')

# Cosmology
d_L = Planck18.luminosity_distance(z=1.0)

Core Capabilities

1. Units and Quantities (astropy.units)

Handle physical quantities with units, perform unit conversions, and ensure dimensional consistency in calculations.

Key operations:

• Create quantities by multiplying values with units
• Convert between units using .to() method
• Perform arithmetic with automatic unit handling
• Use equivalencies for domain-specific conversions (spectral, doppler, parallax)
• Work with logarithmic units (magnitudes, decibels)

See: references/units.md for comprehensive documentation, unit systems, equivalencies, performance optimization, and unit arithmetic.

2. Coordinate Systems (astropy.coordinates)

Represent celestial positions and transform between different coordinate frames.

Key operations:

• Create coordinates with SkyCoord in any frame (ICRS, Galactic, FK5, AltAz, etc.)
• Transform between coordinate systems
• Calculate angular separations and position angles
• Match coordinates to catalogs
• Include distance for 3D coordinate operations
• Handle proper motions and radial velocities
• Query named objects from online databases

See: references/coordinates.md for detailed coordinate frame descriptions, transformations, observer-dependent frames (AltAz), catalog matching, and performance tips.

3. Cosmological Calculations (astropy.cosmology)

Perform cosmological calculations using standard cosmological models.

Key operations:

• Use built-in cosmologies (Planck18, WMAP9, etc.)
• Create custom cosmological models
• Calculate distances (luminosity, comoving, angular diameter)
• Compute ages and lookback times
• Determine Hubble parameter at any redshift
• Calculate density parameters and volumes
• Perform inverse calculations (find z for given distance)

See: references/cosmology.md for available models, distance calculations, time calculations, density parameters, and neutrino effects.

4. FITS File Handling (astropy.io.fits)

Read, write, and manipulate FITS (Flexible Image Transport System) files.

Key operations:

• Open FITS files with context managers
• Access HDUs (Header Data Units) by index or name
• Read and modify headers (keywords, comments, history)
• Work with image data (NumPy arrays)
• Handle table data (binary and ASCII tables)
• Create new FITS files (single or multi-extension)
• Use memory mapping for large files
• Access remote FITS files (S3, HTTP)

See: references/fits.md for comprehensive file operations, header manipulation, image and table handling, multi-extension files, and performance considerations.

5. Table Operations (astropy.table)

Work with tabular data with support for units, metadata, and various file formats.

Key operations:

• Create tables from arrays, lists, or dictionaries
• Read/write tables in multiple formats (FITS, CSV, HDF5, VOTable)
• Access and modify columns and rows
• Sort, filter, and index tables
• Perform database-style operations (join, group, aggregate)
• Stack and concatenate tables
• Work with unit-aware columns (QTable)
• Handle missing data with masking

See: references/tables.md for table creation, I/O operations, data manipulation, sorting, filtering, joins, grouping, and performance tips.

6. Time Handling (astropy.time)

Precise time representation and conversion between time scales and formats.

Key operations:

• Create Time objects in various formats (ISO, JD, MJD, Unix, etc.)
• Convert between time scales (UTC, TAI, TT, TDB, etc.)
• Perform time arithmetic with TimeDelta
• Calculate sidereal time for observers
• Compute light travel time corrections (barycentric, heliocentric)
• Work with time arrays efficiently
• Handle masked (missing) times

See: references/time.md for time formats, time scales, conversions, arithmetic, observing features, and precision handling.

7. World Coordinate System (astropy.wcs)

Transform between pixel coordinates in images and world coordinates.

Key operations:

• Read WCS from FITS headers
• Convert pixel coordinates to world coordinates (and vice versa)
• Calculate image footprints
• Access WCS parameters (reference pixel, projection, scale)
• Create custom WCS objects

See: references/wcs_and_other_modules.md for WCS operations and transformations.

Additional Capabilities

The references/wcs_and_other_modules.md file also covers:

NDData and CCDData

Containers for n-dimensional datasets with metadata, uncertainty, masking, and WCS information.

Modeling

Framework for creating and fitting mathematical models to astronomical data.

Visualization

Tools for astronomical image display with appropriate stretching and scaling.

Constants

Physical and astronomical constants with proper units (speed of light, solar mass, Planck constant, etc.).

Convolution

Image processing kernels for smoothing and filtering.

Statistics

Robust statistical functions including sigma clipping and outlier rejection.

Installation

# Install astropy
uv pip install astropy

# With optional dependencies for full functionality
uv pip install astropy[all]

Common Workflows

Converting Coordinates Between Systems

from astropy.coordinates import SkyCoord
import astropy.units as u

# Create coordinate
c = SkyCoord(ra='05h23m34.5s', dec='-69d45m22s', frame='icrs')

# Transform to galactic
c_gal = c.galactic
print(f"l={c_gal.l.deg}, b={c_gal.b.deg}")

# Transform to alt-az (requires time and location)
from astropy.time import Time
from astropy.coordinates import EarthLocation, AltAz

observing_time = Time('2023-06-15 23:00:00')
observing_location = EarthLocation(lat=40*u.deg, lon=-120*u.deg)
aa_frame = AltAz(obstime=observing_time, location=observing_location)
c_altaz = c.transform_to(aa_frame)
print(f"Alt={c_altaz.alt.deg}, Az={c_altaz.az.deg}")

Reading and Analyzing FITS Files

from astropy.io import fits
import numpy as np

# Open FITS file
with fits.open('observation.fits') as hdul:
    # Display structure
    hdul.info()

    # Get image data and header
    data = hdul[1].data
    header = hdul[1].header

    # Access header values
    exptime = header['EXPTIME']
    filter_name = header['FILTER']

    # Analyze data
    mean = np.mean(data)
    median = np.median(data)
    print(f"Mean: {mean}, Median: {median}")

Cosmological Distance Calculations

from astropy.cosmology import Planck18
import astropy.units as u
import numpy as np

# Calculate distances at z=1.5
z = 1.5
d_L = Planck18.luminosity_distance(z)
d_A = Planck18.angular_diameter_distance(z)

print(f"Luminosity distance: {d_L}")
print(f"Angular diameter distance: {d_A}")

# Age of universe at that redshift
age = Planck18.age(z)
print(f"Age at z={z}: {age.to(u.Gyr)}")

# Lookback time
t_lookback = Planck18.lookback_time(z)
print(f"Lookback time: {t_lookback.to(u.Gyr)}")

Cross-Matching Catalogs

from astropy.table import Table
from astropy.coordinates import SkyCoord, match_coordinates_sky
import astropy.units as u

# Read catalogs
cat1 = Table.read('catalog1.fits')
cat2 = Table.read('catalog2.fits')

# Create coordinate objects
coords1 = SkyCoord(ra=cat1['RA']*u.degree, dec=cat1['DEC']*u.degree)
coords2 = SkyCoord(ra=cat2['RA']*u.degree, dec=cat2['DEC']*u.degree)

# Find matches
idx, sep, _ = coords1.match_to_catalog_sky(coords2)

# Filter by separation threshold
max_sep = 1 * u.arcsec
matches = sep < max_sep

# Create matched catalogs
cat1_matched = cat1[matches]
cat2_matched = cat2[idx[matches]]
print(f"Found {len(cat1_matched)} matches")

Best Practices

1. Always use units: Attach units to quantities to avoid errors and ensure dimensional consistency
2. Use context managers for FITS files: Ensures proper file closing
3. Prefer arrays over loops: Process multiple coordinates/times as arrays for better performance
4. Check coordinate frames: Verify the frame before transformations
5. Use appropriate cosmology: Choose the right cosmological model for your analysis
6. Handle missing data: Use masked columns for tables with missing values
7. Specify time scales: Be explicit about time scales (UTC, TT, TDB) for precise timing
8. Use QTable for unit-aware tables: When table columns have units
9. Check WCS validity: Verify WCS before using transformations
10. Cache frequently used values: Expensive calculations (e.g., cosmological distances) can be cached

Documentation and Resources

• Official Astropy Documentation: https://docs.astropy.org/en/stable/
• Tutorials: https://learn.astropy.org/
• GitHub: https://github.com/astropy/astropy

Reference Files

For detailed information on specific modules:

• references/units.md - Units, quantities, conversions, and equivalencies
• references/coordinates.md - Coordinate systems, transformations, and catalog matching
• references/cosmology.md - Cosmological models and calculations
• references/fits.md - FITS file operations and manipulation
• references/tables.md - Table creation, I/O, and operations
• references/time.md - Time formats, scales, and calculations
• references/wcs_and_other_modules.md - WCS, NDData, modeling, visualization, constants, and utilities

Citation Management

Overview

Manage citations systematically throughout the research and writing process. This skill provides tools and strategies for searching academic databases (Google Scholar, PubMed), extracting accurate metadata from multiple sources (CrossRef, PubMed, arXiv), validating citation information, and generating properly formatted BibTeX entries.

Critical for maintaining citation accuracy, avoiding reference errors, and ensuring reproducible research. Integrates seamlessly with the literature-review skill for comprehensive research workflows.

When to Use This Skill

Use this skill when:

• Searching for specific papers on Google Scholar or PubMed
• Converting DOIs, PMIDs, or arXiv IDs to properly formatted BibTeX
• Extracting complete metadata for citations (authors, title, journal, year, etc.)
• Validating existing citations for accuracy
• Cleaning and formatting BibTeX files
• Finding highly cited papers in a specific field
• Verifying that citation information matches the actual publication
• Building a bibliography for a manuscript or thesis
• Checking for duplicate citations
• Ensuring consistent citation formatting

Visual Enhancement with Scientific Schematics

When creating documents with this skill, always consider adding scientific diagrams and schematics to enhance visual communication.

If your document does not already contain schematics or diagrams:

• Use the scientific-schematics skill to generate AI-powered publication-quality diagrams
• Simply describe your desired diagram in natural language
• Nano Banana Pro will automatically generate, review, and refine the schematic

For new documents: Scientific schematics should be generated by default to visually represent key concepts, workflows, architectures, or relationships described in the text.

How to generate schematics:

python scripts/generate_schematic.py "your diagram description" -o figures/output.png

The AI will automatically:

• Create publication-quality images with proper formatting
• Review and refine through multiple iterations
• Ensure accessibility (colorblind-friendly, high contrast)
• Save outputs in the figures/ directory

When to add schematics:

• Citation workflow diagrams
• Literature search methodology flowcharts
• Reference management system architectures
• Citation style decision trees
• Database integration diagrams
• Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.

Core Workflow

Citation management follows a systematic process:

Phase 1: Paper Discovery and Search

Goal: Find relevant papers using academic search engines.

Google Scholar Search

Google Scholar provides the most comprehensive coverage across disciplines.

Basic Search:

# Search for papers on a topic
python scripts/search_google_scholar.py "CRISPR gene editing" \
  --limit 50 \
  --output results.json

# Search with year filter
python scripts/search_google_scholar.py "machine learning protein folding" \
  --year-start 2020 \
  --year-end 2024 \
  --limit 100 \
  --output ml_proteins.json

Advanced Search Strategies (see references/google_scholar_search.md):

• Use quotation marks for exact phrases: "deep learning"
• Search by author: author:LeCun
• Search in title: intitle:"neural networks"
• Exclude terms: machine learning -survey
• Find highly cited papers using sort options
• Filter by date ranges to get recent work

Best Practices:

• Use specific, targeted search terms
• Include key technical terms and acronyms
• Filter by recent years for fast-moving fields
• Check "Cited by" to find seminal papers
• Export top results for further analysis

PubMed Search

PubMed specializes in biomedical and life sciences literature (35+ million citations).

Basic Search:

# Search PubMed
python scripts/search_pubmed.py "Alzheimer's disease treatment" \
  --limit 100 \
  --output alzheimers.json

# Search with MeSH terms and filters
python scripts/search_pubmed.py \
  --query '"Alzheimer Disease"[MeSH] AND "Drug Therapy"[MeSH]' \
  --date-start 2020 \
  --date-end 2024 \
  --publication-types "Clinical Trial,Review" \
  --output alzheimers_trials.json

Advanced PubMed Queries (see references/pubmed_search.md):

• Use MeSH terms: "Diabetes Mellitus"[MeSH]
• Field tags: "cancer"[Title], "Smith J"[Author]
• Boolean operators: AND, OR, NOT
• Date filters: 2020:2024[Publication Date]
• Publication types: "Review"[Publication Type]
• Combine with E-utilities API for automation

Best Practices:

• Use MeSH Browser to find correct controlled vocabulary
• Construct complex queries in PubMed Advanced Search Builder first
• Include multiple synonyms with OR
• Retrieve PMIDs for easy metadata extraction
• Export to JSON or directly to BibTeX

Phase 2: Metadata Extraction

Goal: Convert paper identifiers (DOI, PMID, arXiv ID) to complete, accurate metadata.

Quick DOI to BibTeX Conversion

For single DOIs, use the quick conversion tool:

# Convert single DOI
python scripts/doi_to_bibtex.py 10.1038/s41586-021-03819-2

# Convert multiple DOIs from a file
python scripts/doi_to_bibtex.py --input dois.txt --output references.bib

# Different output formats
python scripts/doi_to_bibtex.py 10.1038/nature12345 --format json

Comprehensive Metadata Extraction

For DOIs, PMIDs, arXiv IDs, or URLs:

# Extract from DOI
python scripts/extract_metadata.py --doi 10.1038/s41586-021-03819-2

# Extract from PMID
python scripts/extract_metadata.py --pmid 34265844

# Extract from arXiv ID
python scripts/extract_metadata.py --arxiv 2103.14030

# Extract from URL
python scripts/extract_metadata.py --url "https://www.nature.com/articles/s41586-021-03819-2"

# Batch extraction from file (mixed identifiers)
python scripts/extract_metadata.py --input identifiers.txt --output citations.bib

Metadata Sources (see references/metadata_extraction.md):

1. CrossRef API: Primary source for DOIs

   • Comprehensive metadata for journal articles
   • Publisher-provided information
   • Includes authors, title, journal, volume, pages, dates
   • Free, no API key required

2. PubMed E-utilities: Biomedical literature

   • Official NCBI metadata
   • Includes MeSH terms, abstracts
   • PMID and PMCID identifiers
   • Free, API key recommended for high volume

3. arXiv API: Preprints in physics, math, CS, q-bio

   • Complete metadata for preprints
   • Version tracking
   • Author affiliations
   • Free, open access

4. DataCite API: Research datasets, software, other resources

   • Metadata for non-traditional scholarly outputs
   • DOIs for datasets and code
   • Free access

What Gets Extracted:

• Required fields: author, title, year
• Journal articles: journal, volume, number, pages, DOI
• Books: publisher, ISBN, edition
• Conference papers: booktitle, conference location, pages
• Preprints: repository (arXiv, bioRxiv), preprint ID
• Additional: abstract, keywords, URL

Phase 3: BibTeX Formatting

Goal: Generate clean, properly formatted BibTeX entries.

Understanding BibTeX Entry Types

See references/bibtex_formatting.md for complete guide.

Common Entry Types:

• @article: Journal articles (most common)
• @book: Books
• @inproceedings: Conference papers
• @incollection: Book chapters
• @phdthesis: Dissertations
• @misc: Preprints, software, datasets

Required Fields by Type:

@article{citationkey,
  author  = {Last1, First1 and Last2, First2},
  title   = {Article Title},
  journal = {Journal Name},
  year    = {2024},
  volume  = {10},
  number  = {3},
  pages   = {123--145},
  doi     = {10.1234/example}
}

@inproceedings{citationkey,
  author    = {Last, First},
  title     = {Paper Title},
  booktitle = {Conference Name},
  year      = {2024},
  pages     = {1--10}
}

@book{citationkey,
  author    = {Last, First},
  title     = {Book Title},
  publisher = {Publisher Name},
  year      = {2024}
}

Formatting and Cleaning

Use the formatter to standardize BibTeX files:

# Format and clean BibTeX file
python scripts/format_bibtex.py references.bib \
  --output formatted_references.bib

# Sort entries by citation key
python scripts/format_bibtex.py references.bib \
  --sort key \
  --output sorted_references.bib

# Sort by year (newest first)
python scripts/format_bibtex.py references.bib \
  --sort year \
  --descending \
  --output sorted_references.bib

# Remove duplicates
python scripts/format_bibtex.py references.bib \
  --deduplicate \
  --output clean_references.bib

# Validate and report issues
python scripts/format_bibtex.py references.bib \
  --validate \
  --report validation_report.txt

Formatting Operations:

• Standardize field order
• Consistent indentation and spacing
• Proper capitalization in titles (protected with {})
• Standardized author name format
• Consistent citation key format
• Remove unnecessary fields
• Fix common errors (missing commas, braces)

Phase 4: Citation Validation

Goal: Verify all citations are accurate and complete.

Comprehensive Validation

# Validate BibTeX file
python scripts/validate_citations.py references.bib

# Validate and fix common issues
python scripts/validate_citations.py references.bib \
  --auto-fix \
  --output validated_references.bib

# Generate detailed validation report
python scripts/validate_citations.py references.bib \
  --report validation_report.json \
  --verbose

Validation Checks (see references/citation_validation.md):

1. DOI Verification:

   • DOI resolves correctly via doi.org
   • Metadata matches between BibTeX and CrossRef
   • No broken or invalid DOIs

2. Required Fields:

   • All required fields present for entry type
   • No empty or missing critical information
   • Author names properly formatted

3. Data Consistency:

   • Year is valid (4 digits, reasonable range)
   • Volume/number are numeric
   • Pages formatted correctly (e.g., 123--145)
   • URLs are accessible

4. Duplicate Detection:

   • Same DOI used multiple times
   • Similar titles (possible duplicates)
   • Same author/year/title combinations

5. Format Compliance:

   • Valid BibTeX syntax
   • Proper bracing and quoting
   • Citation keys are unique
   • Special characters handled correctly

Validation Output:

{
  "total_entries": 150,
  "valid_entries": 145,
  "errors": [
    {
      "citation_key": "Smith2023",
      "error_type": "missing_field",
      "field": "journal",
      "severity": "high"
    },
    {
      "citation_key": "Jones2022",
      "error_type": "invalid_doi",
      "doi": "10.1234/broken",
      "severity": "high"
    }
  ],
  "warnings": [
    {
      "citation_key": "Brown2021",
      "warning_type": "possible_duplicate",
      "duplicate_of": "Brown2021a",
      "severity": "medium"
    }
  ]
}

Phase 5: Integration with Writing Workflow

Building References for Manuscripts

Complete workflow for creating a bibliography:

# 1. Search for papers on your topic
python scripts/search_pubmed.py \
  '"CRISPR-Cas Systems"[MeSH] AND "Gene Editing"[MeSH]' \
  --date-start 2020 \
  --limit 200 \
  --output crispr_papers.json

# 2. Extract DOIs from search results and convert to BibTeX
python scripts/extract_metadata.py \
  --input crispr_papers.json \
  --output crispr_refs.bib

# 3. Add specific papers by DOI
python scripts/doi_to_bibtex.py 10.1038/nature12345 >> crispr_refs.bib
python scripts/doi_to_bibtex.py 10.1126/science.abcd1234 >> crispr_refs.bib

# 4. Format and clean the BibTeX file
python scripts/format_bibtex.py crispr_refs.bib \
  --deduplicate \
  --sort year \
  --descending \
  --output references.bib

# 5. Validate all citations
python scripts/validate_citations.py references.bib \
  --auto-fix \
  --report validation.json \
  --output final_references.bib

# 6. Review validation report and fix any remaining issues
cat validation.json

# 7. Use in your LaTeX document
# \bibliography{final_references}

Integration with Literature Review Skill

This skill complements the literature-review skill:

Literature Review Skill → Systematic search and synthesis Citation Management Skill → Technical citation handling

Combined Workflow:

1. Use literature-review for comprehensive multi-database search
2. Use citation-management to extract and validate all citations
3. Use literature-review to synthesize findings thematically
4. Use citation-management to verify final bibliography accuracy

# After completing literature review
# Verify all citations in the review document
python scripts/validate_citations.py my_review_references.bib --report review_validation.json

# Format for specific citation style if needed
python scripts/format_bibtex.py my_review_references.bib \
  --style nature \
  --output formatted_refs.bib

Search Strategies

Google Scholar Best Practices

Finding Seminal and High-Impact Papers (CRITICAL):

Always prioritize papers based on citation count, venue quality, and author reputation:

Citation Count Thresholds:

Venue Quality Tiers:

• Tier 1 (Prefer): Nature, Science, Cell, NEJM, Lancet, JAMA, PNAS
• Tier 2 (High Priority): Impact Factor >10, top conferences (NeurIPS, ICML, ICLR)
• Tier 3 (Good): Specialized journals (IF 5-10)
• Tier 4 (Sparingly): Lower-impact peer-reviewed venues

Author Reputation Indicators:

• Senior researchers with h-index >40
• Multiple publications in Tier-1 venues
• Leadership at recognized institutions
• Awards and editorial positions

Search Strategies for High-Impact Papers:

• Sort by citation count (most cited first)
• Look for review articles from Tier-1 journals for overview
• Check "Cited by" for impact assessment and recent follow-up work
• Use citation alerts for tracking new citations to key papers
• Filter by top venues using source:Nature or source:Science
• Search for papers by known field leaders using author:LastName

Advanced Operators (full list in references/google_scholar_search.md):

"exact phrase"           # Exact phrase matching
author:lastname          # Search by author
intitle:keyword          # Search in title only
source:journal           # Search specific journal
-exclude                 # Exclude terms
OR                       # Alternative terms
2020..2024              # Year range

Example Searches:

# Find recent reviews on a topic
"CRISPR" intitle:review 2023..2024

# Find papers by specific author on topic
author:Church "synthetic biology"

# Find highly cited foundational work
"deep learning" 2012..2015 sort:citations

# Exclude surveys and focus on methods
"protein folding" -survey -review intitle:method

PubMed Best Practices

Using MeSH Terms: MeSH (Medical Subject Headings) provides controlled vocabulary for precise searching.

1. Find MeSH terms at https://meshb.nlm.nih.gov/search
2. Use in queries: "Diabetes Mellitus, Type 2"[MeSH]
3. Combine with keywords for comprehensive coverage

Field Tags:

[Title]              # Search in title only
[Title/Abstract]     # Search in title or abstract
[Author]             # Search by author name
[Journal]            # Search specific journal
[Publication Date]   # Date range
[Publication Type]   # Article type
[MeSH]              # MeSH term

Building Complex Queries:

# Clinical trials on diabetes treatment published recently
"Diabetes Mellitus, Type 2"[MeSH] AND "Drug Therapy"[MeSH] 
AND "Clinical Trial"[Publication Type] AND 2020:2024[Publication Date]

# Reviews on CRISPR in specific journal
"CRISPR-Cas Systems"[MeSH] AND "Nature"[Journal] AND "Review"[Publication Type]

# Specific author's recent work
"Smith AB"[Author] AND cancer[Title/Abstract] AND 2022:2024[Publication Date]

E-utilities for Automation: The scripts use NCBI E-utilities API for programmatic access:

• ESearch: Search and retrieve PMIDs
• EFetch: Retrieve full metadata
• ESummary: Get summary information
• ELink: Find related articles

See references/pubmed_search.md for complete API documentation.

Tools and Scripts

search_google_scholar.py

Search Google Scholar and export results.

Features:

• Automated searching with rate limiting
• Pagination support
• Year range filtering
• Export to JSON or BibTeX
• Citation count information

Usage:

# Basic search
python scripts/search_google_scholar.py "quantum computing"

# Advanced search with filters
python scripts/search_google_scholar.py "quantum computing" \
  --year-start 2020 \
  --year-end 2024 \
  --limit 100 \
  --sort-by citations \
  --output quantum_papers.json

# Export directly to BibTeX
python scripts/search_google_scholar.py "machine learning" \
  --limit 50 \
  --format bibtex \
  --output ml_papers.bib

search_pubmed.py

Search PubMed using E-utilities API.

Features:

• Complex query support (MeSH, field tags, Boolean)
• Date range filtering
• Publication type filtering
• Batch retrieval with metadata
• Export to JSON or BibTeX

Usage:

# Simple keyword search
python scripts/search_pubmed.py "CRISPR gene editing"

# Complex query with filters
python scripts/search_pubmed.py \
  --query '"CRISPR-Cas Systems"[MeSH] AND "therapeutic"[Title/Abstract]' \
  --date-start 2020-01-01 \
  --date-end 2024-12-31 \
  --publication-types "Clinical Trial,Review" \
  --limit 200 \
  --output crispr_therapeutic.json

# Export to BibTeX
python scripts/search_pubmed.py "Alzheimer's disease" \
  --limit 100 \
  --format bibtex \
  --output alzheimers.bib

extract_metadata.py

Extract complete metadata from paper identifiers.

Features:

• Supports DOI, PMID, arXiv ID, URL
• Queries CrossRef, PubMed, arXiv APIs
• Handles multiple identifier types
• Batch processing
• Multiple output formats

Usage:

# Single DOI
python scripts/extract_metadata.py --doi 10.1038/s41586-021-03819-2

# Single PMID
python scripts/extract_metadata.py --pmid 34265844

# Single arXiv ID
python scripts/extract_metadata.py --arxiv 2103.14030

# From URL
python scripts/extract_metadata.py \
  --url "https://www.nature.com/articles/s41586-021-03819-2"

# Batch processing (file with one identifier per line)
python scripts/extract_metadata.py \
  --input paper_ids.txt \
  --output references.bib

# Different output formats
python scripts/extract_metadata.py \
  --doi 10.1038/nature12345 \
  --format json  # or bibtex, yaml

validate_citations.py

Validate BibTeX entries for accuracy and completeness.

Features:

• DOI verification via doi.org and CrossRef
• Required field checking
• Duplicate detection
• Format validation
• Auto-fix common issues
• Detailed reporting

Usage:

# Basic validation
python scripts/validate_citations.py references.bib

# With auto-fix
python scripts/validate_citations.py references.bib \
  --auto-fix \
  --output fixed_references.bib

# Detailed validation report
python scripts/validate_citations.py references.bib \
  --report validation_report.json \
  --verbose

# Only check DOIs
python scripts/validate_citations.py references.bib \
  --check-dois-only

format_bibtex.py

Format and clean BibTeX files.

Features:

• Standardize formatting
• Sort entries (by key, year, author)
• Remove duplicates
• Validate syntax
• Fix common errors
• Enforce citation key conventions

Usage:

# Basic formatting
python scripts/format_bibtex.py references.bib

# Sort by year (newest first)
python scripts/format_bibtex.py references.bib \
  --sort year \
  --descending \
  --output sorted_refs.bib

# Remove duplicates
python scripts/format_bibtex.py references.bib \
  --deduplicate \
  --output clean_refs.bib

# Complete cleanup
python scripts/format_bibtex.py references.bib \
  --deduplicate \
  --sort year \
  --validate \
  --auto-fix \
  --output final_refs.bib

doi_to_bibtex.py

Quick DOI to BibTeX conversion.

Features:

• Fast single DOI conversion
• Batch processing
• Multiple output formats
• Clipboard support

Usage:

# Single DOI
python scripts/doi_to_bibtex.py 10.1038/s41586-021-03819-2

# Multiple DOIs
python scripts/doi_to_bibtex.py \
  10.1038/nature12345 \
  10.1126/science.abc1234 \
  10.1016/j.cell.2023.01.001

# From file (one DOI per line)
python scripts/doi_to_bibtex.py --input dois.txt --output references.bib

# Copy to clipboard
python scripts/doi_to_bibtex.py 10.1038/nature12345 --clipboard

Best Practices

Search Strategy

1. Start broad, then narrow:

   • Begin with general terms to understand the field
   • Refine with specific keywords and filters
   • Use synonyms and related terms

2. Use multiple sources:

   • Google Scholar for comprehensive coverage
   • PubMed for biomedical focus
   • arXiv for preprints
   • Combine results for completeness

3. Leverage citations:

   • Check "Cited by" for seminal papers
   • Review references from key papers
   • Use citation networks to discover related work

4. Document your searches:

   • Save search queries and dates
   • Record number of results
   • Note any filters or restrictions applied

Metadata Extraction

1. Always use DOIs when available:

   • Most reliable identifier
   • Permanent link to the publication
   • Best metadata source via CrossRef

2. Verify extracted metadata:

   • Check author names are correct
   • Verify journal/conference names
   • Confirm publication year
   • Validate page numbers and volume

3. Handle edge cases:

   • Preprints: Include repository and ID
   • Preprints later published: Use published version
   • Conference papers: Include conference name and location
   • Book chapters: Include book title and editors

4. Maintain consistency:

   • Use consistent author name format
   • Standardize journal abbreviations
   • Use same DOI format (URL preferred)

BibTeX Quality

1. Follow conventions:

   • Use meaningful citation keys (FirstAuthor2024keyword)
   • Protect capitalization in titles with {}
   • Use -- for page ranges (not single dash)
   • Include DOI field for all modern publications

2. Keep it clean:

   • Remove unnecessary fields
   • No redundant information
   • Consistent formatting
   • Validate syntax regularly

3. Organize systematically:

   • Sort by year or topic
   • Group related papers
   • Use separate files for different projects
   • Merge carefully to avoid duplicates

Validation

1. Validate early and often:

   • Check citations when adding them
   • Validate complete bibliography before submission
   • Re-validate after any manual edits

2. Fix issues promptly:

   • Broken DOIs: Find correct identifier
   • Missing fields: Extract from original source
   • Duplicates: Choose best version, remove others
   • Format errors: Use auto-fix when safe

3. Manual review for critical citations:

   • Verify key papers cited correctly
   • Check author names match publication
   • Confirm page numbers and volume
   • Ensure URLs are current

Common Pitfalls to Avoid

1. Single source bias: Only using Google Scholar or PubMed

   • Solution: Search multiple databases for comprehensive coverage

2. Accepting metadata blindly: Not verifying extracted information

   • Solution: Spot-check extracted metadata against original sources

3. Ignoring DOI errors: Broken or incorrect DOIs in bibliography

   • Solution: Run validation before final submission

4. Inconsistent formatting: Mixed citation key styles, formatting

   • Solution: Use format_bibtex.py to standardize

5. Duplicate entries: Same paper cited multiple times with different keys

   • Solution: Use duplicate detection in validation

6. Missing required fields: Incomplete BibTeX entries

   • Solution: Validate and ensure all required fields present

7. Outdated preprints: Citing preprint when published version exists

   • Solution: Check if preprints have been published, update to journal version

8. Special character issues: Broken LaTeX compilation due to characters

   • Solution: Use proper escaping or Unicode in BibTeX

9. No validation before submission: Submitting with citation errors

   • Solution: Always run validation as final check

10. Manual BibTeX entry: Typing entries by hand

    • Solution: Always extract from metadata sources using scripts

Example Workflows

Example 1: Building a Bibliography for a Paper

# Step 1: Find key papers on your topic
python scripts/search_google_scholar.py "transformer neural networks" \
  --year-start 2017 \
  --limit 50 \
  --output transformers_gs.json

python scripts/search_pubmed.py "deep learning medical imaging" \
  --date-start 2020 \
  --limit 50 \
  --output medical_dl_pm.json

# Step 2: Extract metadata from search results
python scripts/extract_metadata.py \
  --input transformers_gs.json \
  --output transformers.bib

python scripts/extract_metadata.py \
  --input medical_dl_pm.json \
  --output medical.bib

# Step 3: Add specific papers you already know
python scripts/doi_to_bibtex.py 10.1038/s41586-021-03819-2 >> specific.bib
python scripts/doi_to_bibtex.py 10.1126/science.aam9317 >> specific.bib

# Step 4: Combine all BibTeX files
cat transformers.bib medical.bib specific.bib > combined.bib

# Step 5: Format and deduplicate
python scripts/format_bibtex.py combined.bib \
  --deduplicate \
  --sort year \
  --descending \
  --output formatted.bib

# Step 6: Validate
python scripts/validate_citations.py formatted.bib \
  --auto-fix \
  --report validation.json \
  --output final_references.bib

# Step 7: Review any issues
cat validation.json | grep -A 3 '"errors"'

# Step 8: Use in LaTeX
# \bibliography{final_references}

Example 2: Converting a List of DOIs

# You have a text file with DOIs (one per line)
# dois.txt contains:
# 10.1038/s41586-021-03819-2
# 10.1126/science.aam9317
# 10.1016/j.cell.2023.01.001

# Convert all to BibTeX
python scripts/doi_to_bibtex.py --input dois.txt --output references.bib

# Validate the result
python scripts/validate_citations.py references.bib --verbose

Example 3: Cleaning an Existing BibTeX File

# You have a messy BibTeX file from various sources
# Clean it up systematically

# Step 1: Format and standardize
python scripts/format_bibtex.py messy_references.bib \
  --output step1_formatted.bib

# Step 2: Remove duplicates
python scripts/format_bibtex.py step1_formatted.bib \
  --deduplicate \
  --output step2_deduplicated.bib

# Step 3: Validate and auto-fix
python scripts/validate_citations.py step2_deduplicated.bib \
  --auto-fix \
  --output step3_validated.bib

# Step 4: Sort by year
python scripts/format_bibtex.py step3_validated.bib \
  --sort year \
  --descending \
  --output clean_references.bib

# Step 5: Final validation report
python scripts/validate_citations.py clean_references.bib \
  --report final_validation.json \
  --verbose

# Review report
cat final_validation.json

Example 4: Finding and Citing Seminal Papers

# Find highly cited papers on a topic
python scripts/search_google_scholar.py "AlphaFold protein structure" \
  --year-start 2020 \
  --year-end 2024 \
  --sort-by citations \
  --limit 20 \
  --output alphafold_seminal.json

# Extract the top 10 by citation count
# (script will have included citation counts in JSON)

# Convert to BibTeX
python scripts/extract_metadata.py \
  --input alphafold_seminal.json \
  --output alphafold_refs.bib

# The BibTeX file now contains the most influential papers

Integration with Other Skills

Literature Review Skill

Citation Management provides the technical infrastructure for Literature Review:

• Literature Review: Multi-database systematic search and synthesis
• Citation Management: Metadata extraction and validation

Combined workflow:

1. Use literature-review for systematic search methodology
2. Use citation-management to extract and validate citations
3. Use literature-review to synthesize findings
4. Use citation-management to ensure bibliography accuracy

Scientific Writing Skill

Citation Management ensures accurate references for Scientific Writing:

• Export validated BibTeX for use in LaTeX manuscripts
• Verify citations match publication standards
• Format references according to journal requirements

Venue Templates Skill

Citation Management works with Venue Templates for submission-ready manuscripts:

• Different venues require different citation styles
• Generate properly formatted references
• Validate citations meet venue requirements

Resources

Bundled Resources

References (in references/):

• google_scholar_search.md: Complete Google Scholar search guide
• pubmed_search.md: PubMed and E-utilities API documentation
• metadata_extraction.md: Metadata sources and field requirements
• citation_validation.md: Validation criteria and quality checks
• bibtex_formatting.md: BibTeX entry types and formatting rules

Scripts (in scripts/):

• search_google_scholar.py: Google Scholar search automation
• search_pubmed.py: PubMed E-utilities API client
• extract_metadata.py: Universal metadata extractor
• validate_citations.py: Citation validation and verification
• format_bibtex.py: BibTeX formatter and cleaner
• doi_to_bibtex.py: Quick DOI to BibTeX converter

Assets (in assets/):

• bibtex_template.bib: Example BibTeX entries for all types
• citation_checklist.md: Quality assurance checklist

External Resources

Search Engines:

• Google Scholar: https://scholar.google.com/
• PubMed: https://pubmed.ncbi.nlm.nih.gov/
• PubMed Advanced Search: https://pubmed.ncbi.nlm.nih.gov/advanced/

Metadata APIs:

• CrossRef API: https://api.crossref.org/
• PubMed E-utilities: https://www.ncbi.nlm.nih.gov/books/NBK25501/
• arXiv API: https://arxiv.org/help/api/
• DataCite API: https://api.datacite.org/

Tools and Validators:

• MeSH Browser: https://meshb.nlm.nih.gov/search
• DOI Resolver: https://doi.org/
• BibTeX Format: http://www.bibtex.org/Format/

Citation Styles:

• BibTeX documentation: http://www.bibtex.org/
• LaTeX bibliography management: https://www.overleaf.com/learn/latex/Bibliography_management

Dependencies

Required Python Packages

# Core dependencies
pip install requests  # HTTP requests for APIs
pip install bibtexparser  # BibTeX parsing and formatting
pip install biopython  # PubMed E-utilities access

# Optional (for Google Scholar)
pip install scholarly  # Google Scholar API wrapper
# or
pip install selenium  # For more robust Scholar scraping

Optional Tools

# For advanced validation
pip install crossref-commons  # Enhanced CrossRef API access
pip install pylatexenc  # LaTeX special character handling

Summary

The citation-management skill provides:

1. Comprehensive search capabilities for Google Scholar and PubMed
2. Automated metadata extraction from DOI, PMID, arXiv ID, URLs
3. Citation validation with DOI verification and completeness checking
4. BibTeX formatting with standardization and cleaning tools
5. Quality assurance through validation and reporting
6. Integration with scientific writing workflow
7. Reproducibility through documented search and extraction methods

Use this skill to maintain accurate, complete citations throughout your research and ensure publication-ready bibliographies.

Literature Review

Overview

Conduct systematic, comprehensive literature reviews following rigorous academic methodology. Search multiple literature databases, synthesize findings thematically, verify all citations for accuracy, and generate professional output documents in markdown and PDF formats.

This skill uses the parallel-web skill (parallel-cli search) as the primary web search tool for broad academic literature discovery, supplemented by specialized database access skills (gget, bioservices, datacommons-client). It provides specialized tools for citation verification, result aggregation, and document generation.

When to Use This Skill

Use this skill when:

• Conducting a systematic literature review for research or publication
• Synthesizing current knowledge on a specific topic across multiple sources
• Performing meta-analysis or scoping reviews
• Writing the literature review section of a research paper or thesis
• Investigating the state of the art in a research domain
• Identifying research gaps and future directions
• Requiring verified citations and professional formatting

Visual Enhancement with Scientific Schematics

⚠️ MANDATORY: Every literature review MUST include at least 1-2 AI-generated figures using the scientific-schematics skill.

This is not optional. Literature reviews without visual elements are incomplete. Before finalizing any document:

1. Generate at minimum ONE schematic or diagram (e.g., PRISMA flow diagram for systematic reviews)
2. Prefer 2-3 figures for comprehensive reviews (search strategy flowchart, thematic synthesis diagram, conceptual framework)

How to generate figures:

• Use the scientific-schematics skill to generate AI-powered publication-quality diagrams
• Simply describe your desired diagram in natural language
• Nano Banana Pro will automatically generate, review, and refine the schematic

How to generate schematics:

python scripts/generate_schematic.py "your diagram description" -o figures/output.png

The AI will automatically:

• Create publication-quality images with proper formatting
• Review and refine through multiple iterations
• Ensure accessibility (colorblind-friendly, high contrast)
• Save outputs in the figures/ directory

When to add schematics:

• PRISMA flow diagrams for systematic reviews
• Literature search strategy flowcharts
• Thematic synthesis diagrams
• Research gap visualization maps
• Citation network diagrams
• Conceptual framework illustrations
• Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.

Core Workflow

Literature reviews follow a structured, multi-phase workflow:

Phase 1: Planning and Scoping

1. Define Research Question: Use PICO framework (Population, Intervention, Comparison, Outcome) for clinical/biomedical reviews

   • Example: "What is the efficacy of CRISPR-Cas9 (I) for treating sickle cell disease (P) compared to standard care (C)?"

2. Establish Scope and Objectives:

   • Define clear, specific research questions
   • Determine review type (narrative, systematic, scoping, meta-analysis)
   • Set boundaries (time period, geographic scope, study types)

3. Develop Search Strategy:

   • Identify 2-4 main concepts from research question
   • List synonyms, abbreviations, and related terms for each concept
   • Plan Boolean operators (AND, OR, NOT) to combine terms
   • Select minimum 3 complementary databases
   • Use the parallel-web skill (parallel-cli search) for initial scoping to quickly gauge the landscape before formal database searches

4. Set Inclusion/Exclusion Criteria:

   • Date range (e.g., last 10 years: 2015-2024)
   • Language (typically English, or specify multilingual)
   • Publication types (peer-reviewed, preprints, reviews)
   • Study designs (RCTs, observational, in vitro, etc.)
   • Document all criteria clearly

Phase 2: Systematic Literature Search

1. Multi-Database Search:

   Select databases appropriate for the domain. Always start with parallel-web for broad academic coverage, then supplement with domain-specific databases.

   Web-Based Academic Search (parallel-web skill — START HERE):

   • Use parallel-cli search with academic domain filtering for broad scholarly coverage
   • Run two searches: academic-focused + general to catch all relevant sources

   # Academic-focused search across scholarly sources
   parallel-cli search "your research topic" -q "keyword1" -q "keyword2" \
     --json --max-results 10 --excerpt-max-chars-total 27000 \
     --include-domains "scholar.google.com,arxiv.org,pubmed.ncbi.nlm.nih.gov,semanticscholar.org,biorxiv.org,medrxiv.org,ncbi.nlm.nih.gov,nature.com,science.org,ieee.org,acm.org,springer.com,wiley.com,cell.com,pnas.org,nih.gov" \
     -o sources/litreview_<topic>-academic.json
   
   # General search for supplementary sources
   parallel-cli search "your research topic" -q "keyword1" -q "keyword2" \
     --json --max-results 10 --excerpt-max-chars-total 27000 \
     -o sources/litreview_<topic>-general.json
   

   • Use parallel-cli extract to fetch full content from specific paper URLs or PDFs found in search results

   parallel-cli extract "https://arxiv.org/abs/XXXX.XXXXX" --json
   

   Biomedical & Life Sciences:

   • Use gget skill: gget search pubmed "search terms" for PubMed/PMC
   • Use gget skill: gget search biorxiv "search terms" for preprints
   • Use bioservices skill for ChEMBL, KEGG, UniProt, etc.

   General Scientific Literature:

   • Search arXiv via direct API (preprints in physics, math, CS, q-bio)
   • Search Semantic Scholar via API (200M+ papers, cross-disciplinary)
   • Use Google Scholar for comprehensive coverage (manual or careful scraping)

   Specialized Databases:

   • Use gget alphafold for protein structures
   • Use gget cosmic for cancer genomics
   • Use datacommons-client for demographic/statistical data
   • Use specialized databases as appropriate for the domain

2. Document Search Parameters:

   ## Search Strategy
   
   ### Database: PubMed
   - **Date searched**: 2024-10-25
   - **Date range**: 2015-01-01 to 2024-10-25
   - **Search string**:
   

   ("CRISPR"[Title] OR "Cas9"[Title]) AND ("sickle cell"[MeSH] OR "SCD"[Title/Abstract]) AND 2015:2024[Publication Date]

   - **Results**: 247 articles
   

   Repeat for each database searched.

3. Export and Aggregate Results:

   • Export results in JSON format from each database
   • Combine all results into a single file
   • Use scripts/search_databases.py for post-processing:

     python search_databases.py combined_results.json \
       --deduplicate \
       --format markdown \
       --output aggregated_results.md
     

Phase 3: Screening and Selection

1. Deduplication:

   python search_databases.py results.json --deduplicate --output unique_results.json
   

   • Removes duplicates by DOI (primary) or title (fallback)
   • Document number of duplicates removed

2. Title Screening:

   • Review all titles against inclusion/exclusion criteria
   • Exclude obviously irrelevant studies
   • Document number excluded at this stage

3. Abstract Screening:

   • Read abstracts of remaining studies
   • Apply inclusion/exclusion criteria rigorously
   • Document reasons for exclusion

4. Full-Text Screening:

   • Obtain full texts of remaining studies
   • Conduct detailed review against all criteria
   • Document specific reasons for exclusion
   • Record final number of included studies

5. Create PRISMA Flow Diagram:

   Initial search: n = X
   ├─ After deduplication: n = Y
   ├─ After title screening: n = Z
   ├─ After abstract screening: n = A
   └─ Included in review: n = B
   

Phase 4: Data Extraction and Quality Assessment

1. Extract Key Data from each included study:

   • Study metadata (authors, year, journal, DOI)
   • Study design and methods
   • Sample size and population characteristics
   • Key findings and results
   • Limitations noted by authors
   • Funding sources and conflicts of interest

2. Assess Study Quality:

   • For RCTs: Use Cochrane Risk of Bias tool
   • For observational studies: Use Newcastle-Ottawa Scale
   • For systematic reviews: Use AMSTAR 2
   • Rate each study: High, Moderate, Low, or Very Low quality
   • Consider excluding very low-quality studies

3. Organize by Themes:

   • Identify 3-5 major themes across studies
   • Group studies by theme (studies may appear in multiple themes)
   • Note patterns, consensus, and controversies

Phase 5: Synthesis and Analysis

1. Create Review Document from template:

   cp assets/review_template.md my_literature_review.md
   

2. Write Thematic Synthesis (NOT study-by-study summaries):

   • Organize Results section by themes or research questions
   • Synthesize findings across multiple studies within each theme
   • Compare and contrast different approaches and results
   • Identify consensus areas and points of controversy
   • Highlight the strongest evidence

   Example structure:

   #### 3.3.1 Theme: CRISPR Delivery Methods
   
   Multiple delivery approaches have been investigated for therapeutic
   gene editing. Viral vectors (AAV) were used in 15 studies^1-15^ and
   showed high transduction efficiency (65-85%) but raised immunogenicity
   concerns^3,7,12^. In contrast, lipid nanoparticles demonstrated lower
   efficiency (40-60%) but improved safety profiles^16-23^.
   

3. Critical Analysis:

   • Evaluate methodological strengths and limitations across studies
   • Assess quality and consistency of evidence
   • Identify knowledge gaps and methodological gaps
   • Note areas requiring future research

4. Write Discussion:

   • Interpret findings in broader context
   • Discuss clinical, practical, or research implications
   • Acknowledge limitations of the review itself
   • Compare with previous reviews if applicable
   • Propose specific future research directions

Phase 6: Citation Verification

CRITICAL: All citations must be verified for accuracy before final submission.

1. Verify All DOIs:

   python scripts/verify_citations.py my_literature_review.md
   

   This script:

   • Extracts all DOIs from the document
   • Verifies each DOI resolves correctly
   • Retrieves metadata from CrossRef
   • Generates verification report
   • Outputs properly formatted citations

2. Review Verification Report:

   • Check for any failed DOIs
   • Verify author names, titles, and publication details match
   • Correct any errors in the original document
   • Re-run verification until all citations pass

3. Format Citations Consistently:

   • Choose one citation style and use throughout (see references/citation_styles.md)
   • Common styles: APA, Nature, Vancouver, Chicago, IEEE
   • Use verification script output to format citations correctly
   • Ensure in-text citations match reference list format

Phase 7: Document Generation

1. Generate PDF:

   python scripts/generate_pdf.py my_literature_review.md \
     --citation-style apa \
     --output my_review.pdf
   

   Options:

   • --citation-style: apa, nature, chicago, vancouver, ieee
   • --no-toc: Disable table of contents
   • --no-numbers: Disable section numbering
   • --check-deps: Check if pandoc/xelatex are installed

2. Review Final Output:

   • Check PDF formatting and layout
   • Verify all sections are present
   • Ensure citations render correctly
   • Check that figures/tables appear properly
   • Verify table of contents is accurate

3. Quality Checklist:

   • All DOIs verified with verify_citations.py
   • Citations formatted consistently
   • PRISMA flow diagram included (for systematic reviews)
   • Search methodology fully documented
   • Inclusion/exclusion criteria clearly stated
   • Results organized thematically (not study-by-study)
   • Quality assessment completed
   • Limitations acknowledged
   • References complete and accurate
   • PDF generates without errors

Database-Specific Search Guidance

PubMed / PubMed Central

Access via gget skill:

# Search PubMed
gget search pubmed "CRISPR gene editing" -l 100

# Search with filters
# Use PubMed Advanced Search Builder to construct complex queries
# Then execute via gget or direct Entrez API

Search tips:

• Use MeSH terms: "sickle cell disease"[MeSH]
• Field tags: [Title], [Title/Abstract], [Author]
• Date filters: 2020:2024[Publication Date]
• Boolean operators: AND, OR, NOT
• See MeSH browser: https://meshb.nlm.nih.gov/search

bioRxiv / medRxiv

Access via gget skill:

gget search biorxiv "CRISPR sickle cell" -l 50

Important considerations:

• Preprints are not peer-reviewed
• Verify findings with caution
• Check if preprint has been published (CrossRef)
• Note preprint version and date

arXiv

Access via direct API or WebFetch:

# Example search categories:
# q-bio.QM (Quantitative Methods)
# q-bio.GN (Genomics)
# q-bio.MN (Molecular Networks)
# cs.LG (Machine Learning)
# stat.ML (Machine Learning Statistics)

# Search format: category AND terms
search_query = "cat:q-bio.QM AND ti:\"single cell sequencing\""

Semantic Scholar

Access via direct API (requires API key, or use free tier):

• 200M+ papers across all fields
• Excellent for cross-disciplinary searches
• Provides citation graphs and paper recommendations
• Use for finding highly influential papers

Specialized Biomedical Databases

Use appropriate skills:

• ChEMBL: bioservices skill for chemical bioactivity
• UniProt: gget or bioservices skill for protein information
• KEGG: bioservices skill for pathways and genes
• COSMIC: gget skill for cancer mutations
• AlphaFold: gget alphafold for protein structures
• PDB: gget or direct API for experimental structures

Citation Chaining

Expand search via citation networks:

1. Forward citations (papers citing key papers):

   • Use parallel-cli search to find papers citing a specific work:

     parallel-cli search "papers citing [Author et al. Year] [paper title]" \
       -q "citing" -q "[key author]" \
       --json --max-results 10 --excerpt-max-chars-total 27000 \
       --include-domains "scholar.google.com,semanticscholar.org,arxiv.org,pubmed.ncbi.nlm.nih.gov" \
       -o sources/litreview_forward_citations.json
     

   • Use Google Scholar "Cited by"
   • Use Semantic Scholar or OpenAlex APIs
   • Identifies newer research building on seminal work

2. Backward citations (references from key papers):

   • Use parallel-cli extract to fetch full text of key papers and extract their reference lists:

     parallel-cli extract "https://doi.org/10.xxxx/yyyy" --json
     

   • Extract references from included papers
   • Identify highly cited foundational work
   • Find papers cited by multiple included studies

Citation Style Guide

Detailed formatting guidelines are in references/citation_styles.md. Quick reference:

APA (7th Edition)

• In-text: (Smith et al., 2023)
• Reference: Smith, J. D., Johnson, M. L., & Williams, K. R. (2023). Title. Journal, 22(4), 301-318. https://doi.org/10.xxx/yyy

Nature

• In-text: Superscript numbers^1,2^
• Reference: Smith, J. D., Johnson, M. L. & Williams, K. R. Title. Nat. Rev. Drug Discov. 22, 301-318 (2023).

Vancouver

• In-text: Superscript numbers^1,2^
• Reference: Smith JD, Johnson ML, Williams KR. Title. Nat Rev Drug Discov. 2023;22(4):301-18.

Always verify citations with verify_citations.py before finalizing.

Prioritizing High-Impact Papers (CRITICAL)

Always prioritize influential, highly-cited papers from reputable authors and top venues. Quality matters more than quantity in literature reviews.

Citation Count Thresholds

Use citation counts to identify the most impactful papers:

Journal and Venue Tiers

Prioritize papers from higher-tier venues:

• Tier 1 (Always Prefer): Nature, Science, Cell, NEJM, Lancet, JAMA, PNAS, Nature Medicine, Nature Biotechnology
• Tier 2 (Strong Preference): High-impact specialized journals (IF>10), top conferences (NeurIPS, ICML for ML/AI)
• Tier 3 (Include When Relevant): Respected specialized journals (IF 5-10)
• Tier 4 (Use Sparingly): Lower-impact peer-reviewed venues

Author Reputation Assessment

Prefer papers from:

• Senior researchers with high h-index (>40 in established fields)
• Leading research groups at recognized institutions (Harvard, Stanford, MIT, Oxford, etc.)
• Authors with multiple Tier-1 publications in the relevant field
• Researchers with recognized expertise (awards, editorial positions, society fellows)

Identifying Seminal Papers

For any topic, identify foundational work by:

1. High citation count (typically 500+ for papers 5+ years old)
2. Frequently cited by other included studies (appears in many reference lists)
3. Published in Tier-1 venues (Nature, Science, Cell family)
4. Written by field pioneers (often cited as establishing concepts)

Best Practices

Search Strategy

1. Start with parallel-web: Use parallel-cli search with academic domains for initial broad coverage before querying specialized databases
2. Use multiple databases (minimum 3): Ensures comprehensive coverage — parallel-web counts as one source
3. Include preprint servers: Captures latest unpublished findings
4. Document everything: Search strings, dates, result counts for reproducibility — save all parallel-cli output to sources/
5. Test and refine: Run pilot searches, review results, adjust search terms
6. Sort by citations: When available, sort search results by citation count to surface influential work first
7. Use parallel-cli extract: Fetch full content from promising URLs found during search to verify relevance before full-text screening

Screening and Selection

1. Use multiple databases (minimum 3): Ensures comprehensive coverage
2. Include preprint servers: Captures latest unpublished findings
3. Document everything: Search strings, dates, result counts for reproducibility
4. Test and refine: Run pilot searches, review results, adjust search terms

Screening and Selection

1. Use clear criteria: Document inclusion/exclusion criteria before screening
2. Screen systematically: Title → Abstract → Full text
3. Document exclusions: Record reasons for excluding studies
4. Consider dual screening: For systematic reviews, have two reviewers screen independently

Synthesis

1. Organize thematically: Group by themes, NOT by individual studies
2. Synthesize across studies: Compare, contrast, identify patterns
3. Be critical: Evaluate quality and consistency of evidence
4. Identify gaps: Note what's missing or understudied

Quality and Reproducibility

1. Assess study quality: Use appropriate quality assessment tools
2. Verify all citations: Run verify_citations.py script
3. Document methodology: Provide enough detail for others to reproduce
4. Follow guidelines: Use PRISMA for systematic reviews

Writing

1. Be objective: Present evidence fairly, acknowledge limitations
2. Be systematic: Follow structured template
3. Be specific: Include numbers, statistics, effect sizes where available
4. Be clear: Use clear headings, logical flow, thematic organization

Common Pitfalls to Avoid

1. Single database search: Misses relevant papers; always search multiple databases
2. No search documentation: Makes review irreproducible; document all searches
3. Study-by-study summary: Lacks synthesis; organize thematically instead
4. Unverified citations: Leads to errors; always run verify_citations.py
5. Too broad search: Yields thousands of irrelevant results; refine with specific terms
6. Too narrow search: Misses relevant papers; include synonyms and related terms
7. Ignoring preprints: Misses latest findings; include bioRxiv, medRxiv, arXiv
8. No quality assessment: Treats all evidence equally; assess and report quality
9. Publication bias: Only positive results published; note potential bias
10. Outdated search: Field evolves rapidly; clearly state search date

Example Workflow

Complete workflow for a biomedical literature review:

# 1. Create review document from template
cp assets/review_template.md crispr_sickle_cell_review.md

# 2. Start with parallel-web for broad academic search
parallel-cli search "CRISPR Cas9 sickle cell disease gene therapy efficacy" \
  -q "CRISPR" -q "sickle cell" -q "gene therapy" \
  --json --max-results 10 --excerpt-max-chars-total 27000 \
  --include-domains "scholar.google.com,arxiv.org,pubmed.ncbi.nlm.nih.gov,semanticscholar.org,biorxiv.org,nature.com,science.org,cell.com,pnas.org,nih.gov" \
  -o sources/litreview_crispr_scd-academic.json

parallel-cli search "CRISPR sickle cell disease clinical trials treatment" \
  -q "CRISPR" -q "sickle cell" \
  --json --max-results 10 --excerpt-max-chars-total 27000 \
  -o sources/litreview_crispr_scd-general.json

# 3. Search specialized databases using appropriate skills
# - Use gget skill for PubMed, bioRxiv
# - Use direct API access for arXiv, Semantic Scholar
# - Export results in JSON format

# 4. Aggregate and process results (combine parallel-cli + database results)
python scripts/search_databases.py combined_results.json \
  --deduplicate \
  --rank citations \
  --year-start 2015 \
  --year-end 2024 \
  --format markdown \
  --output search_results.md \
  --summary

# 5. Screen results and extract data
# - Use parallel-cli extract to fetch full content from promising URLs
# - Manually screen titles, abstracts, full texts
# - Extract key data into the review document
# - Organize by themes

# 6. Write the review following template structure
# - Introduction with clear objectives
# - Detailed methodology section
# - Results organized thematically
# - Critical discussion
# - Clear conclusions

# 7. Verify all citations
python scripts/verify_citations.py crispr_sickle_cell_review.md

# Review the citation report
cat crispr_sickle_cell_review_citation_report.json

# Fix any failed citations and re-verify
python scripts/verify_citations.py crispr_sickle_cell_review.md

# 8. Generate professional PDF
python scripts/generate_pdf.py crispr_sickle_cell_review.md \
  --citation-style nature \
  --output crispr_sickle_cell_review.pdf

# 9. Review final PDF and markdown outputs

Integration with Other Skills

This skill works seamlessly with other scientific skills:

Web Search & Extraction (parallel-web skill — PRIMARY)

• parallel-cli search: Broad academic and general web search with domain filtering — use for initial scoping, finding papers, citation chaining, and supplementary searches
• parallel-cli extract: Fetch full content from paper URLs, journal websites, and preprint servers — use for reading abstracts, extracting reference lists, and verifying paper details
• parallel-cli search --include-domains: Academic-focused search across scholarly domains (arxiv.org, pubmed, nature.com, etc.)

Database Access Skills

• gget: PubMed, bioRxiv, COSMIC, AlphaFold, Ensembl, UniProt
• bioservices: ChEMBL, KEGG, Reactome, UniProt, PubChem
• datacommons-client: Demographics, economics, health statistics

Analysis Skills

• pydeseq2: RNA-seq differential expression (for methods sections)
• scanpy: Single-cell analysis (for methods sections)
• anndata: Single-cell data (for methods sections)
• biopython: Sequence analysis (for background sections)

Visualization Skills

• matplotlib: Generate figures and plots for review
• seaborn: Statistical visualizations

Writing Skills

• brand-guidelines: Apply institutional branding to PDF
• internal-comms: Adapt review for different audiences

Resources

Bundled Resources

Scripts:

• scripts/verify_citations.py: Verify DOIs and generate formatted citations
• scripts/generate_pdf.py: Convert markdown to professional PDF
• scripts/search_databases.py: Process, deduplicate, and format search results

References:

• references/citation_styles.md: Detailed citation formatting guide (APA, Nature, Vancouver, Chicago, IEEE)
• references/database_strategies.md: Comprehensive database search strategies

Assets:

• assets/review_template.md: Complete literature review template with all sections

External Resources

Guidelines:

• PRISMA (Systematic Reviews): http://www.prisma-statement.org/
• Cochrane Handbook: https://training.cochrane.org/handbook
• AMSTAR 2 (Review Quality): https://amstar.ca/

Tools:

• MeSH Browser: https://meshb.nlm.nih.gov/search
• PubMed Advanced Search: https://pubmed.ncbi.nlm.nih.gov/advanced/
• Boolean Search Guide: https://www.ncbi.nlm.nih.gov/books/NBK3827/

Citation Styles:

• APA Style: https://apastyle.apa.org/
• Nature Portfolio: https://www.nature.com/nature-portfolio/editorial-policies/reporting-standards
• NLM/Vancouver: https://www.nlm.nih.gov/bsd/uniform_requirements.html

Dependencies

Required CLI Tools

# parallel-cli (PRIMARY — for web search and URL extraction)
curl -fsSL https://parallel.ai/install.sh | bash
# Or: uv tool install "parallel-web-tools[cli]"
# Authenticate: parallel-cli auth

Required Python Packages

pip install requests  # For citation verification

Required System Tools

# For PDF generation
brew install pandoc  # macOS
apt-get install pandoc  # Linux

# For LaTeX (PDF generation)
brew install --cask mactex  # macOS
apt-get install texlive-xetex  # Linux

Check dependencies:

python scripts/generate_pdf.py --check-deps

Summary

This literature-review skill provides:

1. Systematic methodology following academic best practices
2. Parallel-web powered search using parallel-cli search for fast, broad academic literature discovery with scholarly domain filtering
3. Multi-database integration via existing scientific skills (gget, bioservices, datacommons-client)
4. Citation verification ensuring accuracy and credibility
5. Professional output in markdown and PDF formats
6. Comprehensive guidance covering the entire review process
7. Quality assurance with verification and validation tools
8. Reproducibility through detailed documentation requirements

Conduct thorough, rigorous literature reviews that meet academic standards and provide comprehensive synthesis of current knowledge in any domain.

Matplotlib

Overview

Matplotlib is Python's foundational visualization library for creating static, animated, and interactive plots. This skill provides guidance on using matplotlib effectively, covering both the pyplot interface (MATLAB-style) and the object-oriented API (Figure/Axes), along with best practices for creating publication-quality visualizations.

When to Use This Skill

This skill should be used when:

• Creating any type of plot or chart (line, scatter, bar, histogram, heatmap, contour, etc.)
• Generating scientific or statistical visualizations
• Customizing plot appearance (colors, styles, labels, legends)
• Creating multi-panel figures with subplots
• Exporting visualizations to various formats (PNG, PDF, SVG, etc.)
• Building interactive plots or animations
• Working with 3D visualizations
• Integrating plots into Jupyter notebooks or GUI applications

Core Concepts

The Matplotlib Hierarchy

Matplotlib uses a hierarchical structure of objects:

1. Figure - The top-level container for all plot elements
2. Axes - The actual plotting area where data is displayed (one Figure can contain multiple Axes)
3. Artist - Everything visible on the figure (lines, text, ticks, etc.)
4. Axis - The number line objects (x-axis, y-axis) that handle ticks and labels

Two Interfaces

1. pyplot Interface (Implicit, MATLAB-style)

import matplotlib.pyplot as plt

plt.plot([1, 2, 3, 4])
plt.ylabel('some numbers')
plt.show()

• Convenient for quick, simple plots
• Maintains state automatically
• Good for interactive work and simple scripts

2. Object-Oriented Interface (Explicit)

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.plot([1, 2, 3, 4])
ax.set_ylabel('some numbers')
plt.show()

• Recommended for most use cases
• More explicit control over figure and axes
• Better for complex figures with multiple subplots
• Easier to maintain and debug

Common Workflows

1. Basic Plot Creation

Single plot workflow:

import matplotlib.pyplot as plt
import numpy as np

# Create figure and axes (OO interface - RECOMMENDED)
fig, ax = plt.subplots(figsize=(10, 6))

# Generate and plot data
x = np.linspace(0, 2*np.pi, 100)
ax.plot(x, np.sin(x), label='sin(x)')
ax.plot(x, np.cos(x), label='cos(x)')

# Customize
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_title('Trigonometric Functions')
ax.legend()
ax.grid(True, alpha=0.3)

# Save and/or display
plt.savefig('plot.png', dpi=300, bbox_inches='tight')
plt.show()

2. Multiple Subplots

Creating subplot layouts:

# Method 1: Regular grid
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
axes[0, 0].plot(x, y1)
axes[0, 1].scatter(x, y2)
axes[1, 0].bar(categories, values)
axes[1, 1].hist(data, bins=30)

# Method 2: Mosaic layout (more flexible)
fig, axes = plt.subplot_mosaic([['left', 'right_top'],
                                 ['left', 'right_bottom']],
                                figsize=(10, 8))
axes['left'].plot(x, y)
axes['right_top'].scatter(x, y)
axes['right_bottom'].hist(data)

# Method 3: GridSpec (maximum control)
from matplotlib.gridspec import GridSpec
fig = plt.figure(figsize=(12, 8))
gs = GridSpec(3, 3, figure=fig)
ax1 = fig.add_subplot(gs[0, :])  # Top row, all columns
ax2 = fig.add_subplot(gs[1:, 0])  # Bottom two rows, first column
ax3 = fig.add_subplot(gs[1:, 1:])  # Bottom two rows, last two columns

3. Plot Types and Use Cases

Line plots - Time series, continuous data, trends

ax.plot(x, y, linewidth=2, linestyle='--', marker='o', color='blue')

Scatter plots - Relationships between variables, correlations

ax.scatter(x, y, s=sizes, c=colors, alpha=0.6, cmap='viridis')

Bar charts - Categorical comparisons

ax.bar(categories, values, color='steelblue', edgecolor='black')
# For horizontal bars:
ax.barh(categories, values)

Histograms - Distributions

ax.hist(data, bins=30, edgecolor='black', alpha=0.7)

Heatmaps - Matrix data, correlations

im = ax.imshow(matrix, cmap='coolwarm', aspect='auto')
plt.colorbar(im, ax=ax)

Contour plots - 3D data on 2D plane

contour = ax.contour(X, Y, Z, levels=10)
ax.clabel(contour, inline=True, fontsize=8)

Box plots - Statistical distributions

ax.boxplot([data1, data2, data3], labels=['A', 'B', 'C'])

Violin plots - Distribution densities

ax.violinplot([data1, data2, data3], positions=[1, 2, 3])

For comprehensive plot type examples and variations, refer to references/plot_types.md.

4. Styling and Customization

Color specification methods:

• Named colors: 'red', 'blue', 'steelblue'
• Hex codes: '#FF5733'
• RGB tuples: (0.1, 0.2, 0.3)
• Colormaps: cmap='viridis', cmap='plasma', cmap='coolwarm'

Using style sheets:

plt.style.use('seaborn-v0_8-darkgrid')  # Apply predefined style
# Available styles: 'ggplot', 'bmh', 'fivethirtyeight', etc.
print(plt.style.available)  # List all available styles

Customizing with rcParams:

plt.rcParams['font.size'] = 12
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['axes.titlesize'] = 16
plt.rcParams['xtick.labelsize'] = 10
plt.rcParams['ytick.labelsize'] = 10
plt.rcParams['legend.fontsize'] = 12
plt.rcParams['figure.titlesize'] = 18

Text and annotations:

ax.text(x, y, 'annotation', fontsize=12, ha='center')
ax.annotate('important point', xy=(x, y), xytext=(x+1, y+1),
            arrowprops=dict(arrowstyle='->', color='red'))

For detailed styling options and colormap guidelines, see references/styling_guide.md.

5. Saving Figures

Export to various formats:

# High-resolution PNG for presentations/papers
plt.savefig('figure.png', dpi=300, bbox_inches='tight', facecolor='white')

# Vector format for publications (scalable)
plt.savefig('figure.pdf', bbox_inches='tight')
plt.savefig('figure.svg', bbox_inches='tight')

# Transparent background
plt.savefig('figure.png', dpi=300, bbox_inches='tight', transparent=True)

Important parameters:

• dpi: Resolution (300 for publications, 150 for web, 72 for screen)
• bbox_inches='tight': Removes excess whitespace
• facecolor='white': Ensures white background (useful for transparent themes)
• transparent=True: Transparent background

6. Working with 3D Plots

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')

# Surface plot
ax.plot_surface(X, Y, Z, cmap='viridis')

# 3D scatter
ax.scatter(x, y, z, c=colors, marker='o')

# 3D line plot
ax.plot(x, y, z, linewidth=2)

# Labels
ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

Best Practices

1. Interface Selection

• Use the object-oriented interface (fig, ax = plt.subplots()) for production code
• Reserve pyplot interface for quick interactive exploration only
• Always create figures explicitly rather than relying on implicit state

2. Figure Size and DPI

• Set figsize at creation: fig, ax = plt.subplots(figsize=(10, 6))
• Use appropriate DPI for output medium:
  • Screen/notebook: 72-100 dpi
  • Web: 150 dpi
  • Print/publications: 300 dpi

3. Layout Management

• Use constrained_layout=True or tight_layout() to prevent overlapping elements
• fig, ax = plt.subplots(constrained_layout=True) is recommended for automatic spacing

4. Colormap Selection

• Sequential (viridis, plasma, inferno): Ordered data with consistent progression
• Diverging (coolwarm, RdBu): Data with meaningful center point (e.g., zero)
• Qualitative (tab10, Set3): Categorical/nominal data
• Avoid rainbow colormaps (jet) - they are not perceptually uniform

5. Accessibility

• Use colorblind-friendly colormaps (viridis, cividis)
• Add patterns/hatching for bar charts in addition to colors
• Ensure sufficient contrast between elements
• Include descriptive labels and legends

6. Performance

• For large datasets, use rasterized=True in plot calls to reduce file size
• Use appropriate data reduction before plotting (e.g., downsample dense time series)
• For animations, use blitting for better performance

7. Code Organization

# Good practice: Clear structure
def create_analysis_plot(data, title):
    """Create standardized analysis plot."""
    fig, ax = plt.subplots(figsize=(10, 6), constrained_layout=True)

    # Plot data
    ax.plot(data['x'], data['y'], linewidth=2)

    # Customize
    ax.set_xlabel('X Axis Label', fontsize=12)
    ax.set_ylabel('Y Axis Label', fontsize=12)
    ax.set_title(title, fontsize=14, fontweight='bold')
    ax.grid(True, alpha=0.3)

    return fig, ax

# Use the function
fig, ax = create_analysis_plot(my_data, 'My Analysis')
plt.savefig('analysis.png', dpi=300, bbox_inches='tight')

Quick Reference Scripts

This skill includes helper scripts in the scripts/ directory:

plot_template.py

Template script demonstrating various plot types with best practices. Use this as a starting point for creating new visualizations.

Usage:

python scripts/plot_template.py

style_configurator.py

Interactive utility to configure matplotlib style preferences and generate custom style sheets.

Usage:

python scripts/style_configurator.py

Detailed References

For comprehensive information, consult the reference documents:

• references/plot_types.md - Complete catalog of plot types with code examples and use cases
• references/styling_guide.md - Detailed styling options, colormaps, and customization
• references/api_reference.md - Core classes and methods reference
• references/common_issues.md - Troubleshooting guide for common problems

Integration with Other Tools

Matplotlib integrates well with:

• NumPy/Pandas - Direct plotting from arrays and DataFrames
• Seaborn - High-level statistical visualizations built on matplotlib
• Jupyter - Interactive plotting with %matplotlib inline or %matplotlib widget
• GUI frameworks - Embedding in Tkinter, Qt, wxPython applications

Common Gotchas

1. Overlapping elements: Use constrained_layout=True or tight_layout()
2. State confusion: Use OO interface to avoid pyplot state machine issues
3. Memory issues with many figures: Close figures explicitly with plt.close(fig)
4. Font warnings: Install fonts or suppress warnings with plt.rcParams['font.sans-serif']
5. DPI confusion: Remember that figsize is in inches, not pixels: pixels = dpi * inches

Additional Resources

• Official documentation: https://matplotlib.org/
• Gallery: https://matplotlib.org/stable/gallery/index.html
• Cheatsheets: https://matplotlib.org/cheatsheets/
• Tutorials: https://matplotlib.org/stable/tutorials/index.html

NetworkX

Overview

NetworkX is a Python package for creating, manipulating, and analyzing complex networks and graphs. Use this skill when working with network or graph data structures, including social networks, biological networks, transportation systems, citation networks, knowledge graphs, or any system involving relationships between entities.

When to Use This Skill

Invoke this skill when tasks involve:

• Creating graphs: Building network structures from data, adding nodes and edges with attributes
• Graph analysis: Computing centrality measures, finding shortest paths, detecting communities, measuring clustering
• Graph algorithms: Running standard algorithms like Dijkstra's, PageRank, minimum spanning trees, maximum flow
• Network generation: Creating synthetic networks (random, scale-free, small-world models) for testing or simulation
• Graph I/O: Reading from or writing to various formats (edge lists, GraphML, JSON, CSV, adjacency matrices)
• Visualization: Drawing and customizing network visualizations with matplotlib or interactive libraries
• Network comparison: Checking isomorphism, computing graph metrics, analyzing structural properties

Core Capabilities

1. Graph Creation and Manipulation

NetworkX supports four main graph types:

• Graph: Undirected graphs with single edges
• DiGraph: Directed graphs with one-way connections
• MultiGraph: Undirected graphs allowing multiple edges between nodes
• MultiDiGraph: Directed graphs with multiple edges

Create graphs by:

import networkx as nx

# Create empty graph
G = nx.Graph()

# Add nodes (can be any hashable type)
G.add_node(1)
G.add_nodes_from([2, 3, 4])
G.add_node("protein_A", type='enzyme', weight=1.5)

# Add edges
G.add_edge(1, 2)
G.add_edges_from([(1, 3), (2, 4)])
G.add_edge(1, 4, weight=0.8, relation='interacts')

Reference: See references/graph-basics.md for comprehensive guidance on creating, modifying, examining, and managing graph structures, including working with attributes and subgraphs.

2. Graph Algorithms

NetworkX provides extensive algorithms for network analysis:

Shortest Paths:

# Find shortest path
path = nx.shortest_path(G, source=1, target=5)
length = nx.shortest_path_length(G, source=1, target=5, weight='weight')

Centrality Measures:

# Degree centrality
degree_cent = nx.degree_centrality(G)

# Betweenness centrality
betweenness = nx.betweenness_centrality(G)

# PageRank
pagerank = nx.pagerank(G)

Community Detection:

from networkx.algorithms import community

# Detect communities
communities = community.greedy_modularity_communities(G)

Connectivity:

# Check connectivity
is_connected = nx.is_connected(G)

# Find connected components
components = list(nx.connected_components(G))

Reference: See references/algorithms.md for detailed documentation on all available algorithms including shortest paths, centrality measures, clustering, community detection, flows, matching, tree algorithms, and graph traversal.

3. Graph Generators

Create synthetic networks for testing, simulation, or modeling:

Classic Graphs:

# Complete graph
G = nx.complete_graph(n=10)

# Cycle graph
G = nx.cycle_graph(n=20)

# Known graphs
G = nx.karate_club_graph()
G = nx.petersen_graph()

Random Networks:

# Erdős-Rényi random graph
G = nx.erdos_renyi_graph(n=100, p=0.1, seed=42)

# Barabási-Albert scale-free network
G = nx.barabasi_albert_graph(n=100, m=3, seed=42)

# Watts-Strogatz small-world network
G = nx.watts_strogatz_graph(n=100, k=6, p=0.1, seed=42)

Structured Networks:

# Grid graph
G = nx.grid_2d_graph(m=5, n=7)

# Random tree
G = nx.random_tree(n=100, seed=42)

Reference: See references/generators.md for comprehensive coverage of all graph generators including classic, random, lattice, bipartite, and specialized network models with detailed parameters and use cases.

4. Reading and Writing Graphs

NetworkX supports numerous file formats and data sources:

File Formats:

# Edge list
G = nx.read_edgelist('graph.edgelist')
nx.write_edgelist(G, 'graph.edgelist')

# GraphML (preserves attributes)
G = nx.read_graphml('graph.graphml')
nx.write_graphml(G, 'graph.graphml')

# GML
G = nx.read_gml('graph.gml')
nx.write_gml(G, 'graph.gml')

# JSON
data = nx.node_link_data(G)
G = nx.node_link_graph(data)

Pandas Integration:

import pandas as pd

# From DataFrame
df = pd.DataFrame({'source': [1, 2, 3], 'target': [2, 3, 4], 'weight': [0.5, 1.0, 0.75]})
G = nx.from_pandas_edgelist(df, 'source', 'target', edge_attr='weight')

# To DataFrame
df = nx.to_pandas_edgelist(G)

Matrix Formats:

import numpy as np

# Adjacency matrix
A = nx.to_numpy_array(G)
G = nx.from_numpy_array(A)

# Sparse matrix
A = nx.to_scipy_sparse_array(G)
G = nx.from_scipy_sparse_array(A)

Reference: See references/io.md for complete documentation on all I/O formats including CSV, SQL databases, Cytoscape, DOT, and guidance on format selection for different use cases.

5. Visualization

Create clear and informative network visualizations:

Basic Visualization:

import matplotlib.pyplot as plt

# Simple draw
nx.draw(G, with_labels=True)
plt.show()

# With layout
pos = nx.spring_layout(G, seed=42)
nx.draw(G, pos=pos, with_labels=True, node_color='lightblue', node_size=500)
plt.show()

Customization:

# Color by degree
node_colors = [G.degree(n) for n in G.nodes()]
nx.draw(G, node_color=node_colors, cmap=plt.cm.viridis)

# Size by centrality
centrality = nx.betweenness_centrality(G)
node_sizes = [3000 * centrality[n] for n in G.nodes()]
nx.draw(G, node_size=node_sizes)

# Edge weights
edge_widths = [3 * G[u][v].get('weight', 1) for u, v in G.edges()]
nx.draw(G, width=edge_widths)

Layout Algorithms:

# Spring layout (force-directed)
pos = nx.spring_layout(G, seed=42)

# Circular layout
pos = nx.circular_layout(G)

# Kamada-Kawai layout
pos = nx.kamada_kawai_layout(G)

# Spectral layout
pos = nx.spectral_layout(G)

Publication Quality:

plt.figure(figsize=(12, 8))
pos = nx.spring_layout(G, seed=42)
nx.draw(G, pos=pos, node_color='lightblue', node_size=500,
        edge_color='gray', with_labels=True, font_size=10)
plt.title('Network Visualization', fontsize=16)
plt.axis('off')
plt.tight_layout()
plt.savefig('network.png', dpi=300, bbox_inches='tight')
plt.savefig('network.pdf', bbox_inches='tight')  # Vector format

Reference: See references/visualization.md for extensive documentation on visualization techniques including layout algorithms, customization options, interactive visualizations with Plotly and PyVis, 3D networks, and publication-quality figure creation.

Working with NetworkX

Installation

Ensure NetworkX is installed:

# Check if installed
import networkx as nx
print(nx.__version__)

# Install if needed (via bash)
# uv pip install networkx
# uv pip install networkx[default]  # With optional dependencies

Common Workflow Pattern

Most NetworkX tasks follow this pattern:

1. Create or Load Graph:

   # From scratch
   G = nx.Graph()
   G.add_edges_from([(1, 2), (2, 3), (3, 4)])
   
   # Or load from file/data
   G = nx.read_edgelist('data.txt')
   

2. Examine Structure:

   print(f"Nodes: {G.number_of_nodes()}")
   print(f"Edges: {G.number_of_edges()}")
   print(f"Density: {nx.density(G)}")
   print(f"Connected: {nx.is_connected(G)}")
   

3. Analyze:

   # Compute metrics
   degree_cent = nx.degree_centrality(G)
   avg_clustering = nx.average_clustering(G)
   
   # Find paths
   path = nx.shortest_path(G, source=1, target=4)
   
   # Detect communities
   communities = community.greedy_modularity_communities(G)
   

4. Visualize:

   pos = nx.spring_layout(G, seed=42)
   nx.draw(G, pos=pos, with_labels=True)
   plt.show()
   

5. Export Results:

   # Save graph
   nx.write_graphml(G, 'analyzed_network.graphml')
   
   # Save metrics
   df = pd.DataFrame({
       'node': list(degree_cent.keys()),
       'centrality': list(degree_cent.values())
   })
   df.to_csv('centrality_results.csv', index=False)
   

Important Considerations

Floating Point Precision: When graphs contain floating-point numbers, all results are inherently approximate due to precision limitations. This can affect algorithm outcomes, particularly in minimum/maximum computations.

Memory and Performance: Each time a script runs, graph data must be loaded into memory. For large networks:

• Use appropriate data structures (sparse matrices for large sparse graphs)
• Consider loading only necessary subgraphs
• Use efficient file formats (pickle for Python objects, compressed formats)
• Leverage approximate algorithms for very large networks (e.g., k parameter in centrality calculations)

Node and Edge Types:

• Nodes can be any hashable Python object (numbers, strings, tuples, custom objects)
• Use meaningful identifiers for clarity
• When removing nodes, all incident edges are automatically removed

Random Seeds: Always set random seeds for reproducibility in random graph generation and force-directed layouts:

G = nx.erdos_renyi_graph(n=100, p=0.1, seed=42)
pos = nx.spring_layout(G, seed=42)

Quick Reference

Basic Operations

# Create
G = nx.Graph()
G.add_edge(1, 2)

# Query
G.number_of_nodes()
G.number_of_edges()
G.degree(1)
list(G.neighbors(1))

# Check
G.has_node(1)
G.has_edge(1, 2)
nx.is_connected(G)

# Modify
G.remove_node(1)
G.remove_edge(1, 2)
G.clear()

Essential Algorithms

# Paths
nx.shortest_path(G, source, target)
nx.all_pairs_shortest_path(G)

# Centrality
nx.degree_centrality(G)
nx.betweenness_centrality(G)
nx.closeness_centrality(G)
nx.pagerank(G)

# Clustering
nx.clustering(G)
nx.average_clustering(G)

# Components
nx.connected_components(G)
nx.strongly_connected_components(G)  # Directed

# Community
community.greedy_modularity_communities(G)

File I/O Quick Reference

# Read
nx.read_edgelist('file.txt')
nx.read_graphml('file.graphml')
nx.read_gml('file.gml')

# Write
nx.write_edgelist(G, 'file.txt')
nx.write_graphml(G, 'file.graphml')
nx.write_gml(G, 'file.gml')

# Pandas
nx.from_pandas_edgelist(df, 'source', 'target')
nx.to_pandas_edgelist(G)

Resources

This skill includes comprehensive reference documentation:

references/graph-basics.md

Detailed guide on graph types, creating and modifying graphs, adding nodes and edges, managing attributes, examining structure, and working with subgraphs.

references/algorithms.md

Complete coverage of NetworkX algorithms including shortest paths, centrality measures, connectivity, clustering, community detection, flow algorithms, tree algorithms, matching, coloring, isomorphism, and graph traversal.

references/generators.md

Comprehensive documentation on graph generators including classic graphs, random models (Erdős-Rényi, Barabási-Albert, Watts-Strogatz), lattices, trees, social network models, and specialized generators.

references/io.md

Complete guide to reading and writing graphs in various formats: edge lists, adjacency lists, GraphML, GML, JSON, CSV, Pandas DataFrames, NumPy arrays, SciPy sparse matrices, database integration, and format selection guidelines.

references/visualization.md

Extensive documentation on visualization techniques including layout algorithms, customizing node and edge appearance, labels, interactive visualizations with Plotly and PyVis, 3D networks, bipartite layouts, and creating publication-quality figures.

Additional Resources

• Official Documentation: https://networkx.org/documentation/latest/
• Tutorial: https://networkx.org/documentation/latest/tutorial.html
• Gallery: https://networkx.org/documentation/latest/auto_examples/index.html
• GitHub: https://github.com/networkx/networkx

Paper Lookup

You have access to 10 academic paper databases through their REST APIs. Your job is to figure out which database(s) best serve the user's query, call them, and return the results.

Core Workflow

1. Understand the query -- What is the user looking for? A specific paper by DOI? Papers on a topic? An author's publications? Open access PDFs? Full text? This determines which database(s) to hit.

2. Select database(s) -- Use the database selection guide below. Many queries benefit from hitting multiple databases -- for example, searching PubMed for papers and then checking Unpaywall for open access copies.

3. Read the reference file -- Each database has a reference file in references/ with endpoint details, query formats, and example calls. Read the relevant file(s) before making API calls.

4. Make the API call(s) -- See the Making API Calls section below for which HTTP fetch tool to use on your platform.

5. Return results -- Always return:

   • The raw JSON (or parsed XML for arXiv) response from each database
   • A list of databases queried with the specific endpoints used
   • If a query returned no results, say so explicitly rather than omitting it

Database Selection Guide

Match the user's intent to the right database(s).

By Use Case

Cross-Database Queries

When a query spans multiple needs (e.g., "find papers about CRISPR and get me the PDFs"), query the relevant databases in parallel.

Common Identifier Formats

Different databases use different identifier systems. If a query fails, the identifier format may be wrong.

Cross-referencing IDs: Semantic Scholar accepts DOI, PMID, PMCID, and arXiv ID via prefixes (e.g., DOI:10.1038/nature12373, PMID:34567890, ARXIV:2103.15348). OpenAlex accepts DOI and PMID via prefixes (doi:10.1038/..., pmid:34567890). Use the PMC ID Converter to translate between PMID, PMCID, and DOI.

API Keys and Access

Most of these databases are fully open. A few benefit from API keys for higher rate limits.

Databases requiring or benefiting from API keys

Fully open databases (no key needed)

Loading API keys

1. Check the environment first -- the key may already be exported (e.g., $NCBI_API_KEY).
2. Fall back to .env -- check .env in the current working directory.
3. Proceed without -- most APIs still work at lower rate limits. Tell the user which key is missing and how to get one.

Making API Calls

Use your environment's HTTP fetch tool to call REST endpoints:

If the fetch tool fails, fall back to curl via whatever shell tool is available.

Special cases

• arXiv returns Atom XML, not JSON. Parse it or use curl and extract the relevant fields. Consider piping through a simple parser if available.
• PMC eFetch returns JATS XML for full text. This is expected -- full text articles are in XML format.
• Crossref and Unpaywall benefit from including a mailto parameter or email for the polite/fast pool.

Request guidelines

• For NCBI APIs (PubMed, PMC): max 3 req/sec without key, 10 with key. Make requests sequentially.
• For arXiv: max 1 request every 3 seconds. Be patient.
• For Crossref: 5 req/sec (public), 10 req/sec (polite pool with mailto).
• For other APIs with no strict limits, you can query multiple databases in parallel.
• If you get HTTP 429 (rate limit), wait briefly and retry once.

Error recovery

1. Check the identifier format -- use the Common Identifier Formats table. A PMID won't work in arXiv, an arXiv ID won't work in PubMed directly.
2. Try alternative identifiers -- if a DOI fails in one database, try the title or PMID instead.
3. Try a different database -- if PubMed returns nothing for a CS paper, try Semantic Scholar or OpenAlex.
4. Report the failure -- tell the user which database failed, the error, and what you tried instead.

Output Format

Structure your response like this:

## Databases Queried
- **PubMed** -- esearch + esummary for "CRISPR gene therapy"
- **Unpaywall** -- DOI lookup for 10.1038/...

## Results

### PubMed
[raw JSON response or formatted results]

### Unpaywall
[raw JSON response]

If results are very large, present the most relevant portion and note that more data is available. But default to showing the full raw JSON -- the user asked for it.

Available Databases

Read the relevant reference file before making any API call.

Biomedical Literature

Preprint Servers

Multidisciplinary Indexes

Open Access & Full Text

PDF Processing Guide

Overview

This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see reference.md. If you need to fill out a PDF form, read forms.md and follow its instructions.

Quick Start

from pypdf import PdfReader, PdfWriter

# Read a PDF
reader = PdfReader("document.pdf")
print(f"Pages: {len(reader.pages)}")

# Extract text
text = ""
for page in reader.pages:
    text += page.extract_text()

Python Libraries

pypdf - Basic Operations

Merge PDFs

from pypdf import PdfWriter, PdfReader

writer = PdfWriter()
for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]:
    reader = PdfReader(pdf_file)
    for page in reader.pages:
        writer.add_page(page)

with open("merged.pdf", "wb") as output:
    writer.write(output)

Split PDF

reader = PdfReader("input.pdf")
for i, page in enumerate(reader.pages):
    writer = PdfWriter()
    writer.add_page(page)
    with open(f"page_{i+1}.pdf", "wb") as output:
        writer.write(output)

Extract Metadata

reader = PdfReader("document.pdf")
meta = reader.metadata
print(f"Title: {meta.title}")
print(f"Author: {meta.author}")
print(f"Subject: {meta.subject}")
print(f"Creator: {meta.creator}")

Rotate Pages

reader = PdfReader("input.pdf")
writer = PdfWriter()

page = reader.pages[0]
page.rotate(90)  # Rotate 90 degrees clockwise
writer.add_page(page)

with open("rotated.pdf", "wb") as output:
    writer.write(output)

pdfplumber - Text and Table Extraction

Extract Text with Layout

import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    for page in pdf.pages:
        text = page.extract_text()
        print(text)

Extract Tables

with pdfplumber.open("document.pdf") as pdf:
    for i, page in enumerate(pdf.pages):
        tables = page.extract_tables()
        for j, table in enumerate(tables):
            print(f"Table {j+1} on page {i+1}:")
            for row in table:
                print(row)

Advanced Table Extraction

import pandas as pd

with pdfplumber.open("document.pdf") as pdf:
    all_tables = []
    for page in pdf.pages:
        tables = page.extract_tables()
        for table in tables:
            if table:  # Check if table is not empty
                df = pd.DataFrame(table[1:], columns=table[0])
                all_tables.append(df)

# Combine all tables
if all_tables:
    combined_df = pd.concat(all_tables, ignore_index=True)
    combined_df.to_excel("extracted_tables.xlsx", index=False)

reportlab - Create PDFs

Basic PDF Creation

from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas

c = canvas.Canvas("hello.pdf", pagesize=letter)
width, height = letter

# Add text
c.drawString(100, height - 100, "Hello World!")
c.drawString(100, height - 120, "This is a PDF created with reportlab")

# Add a line
c.line(100, height - 140, 400, height - 140)

# Save
c.save()

Create PDF with Multiple Pages

from reportlab.lib.pagesizes import letter
from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak
from reportlab.lib.styles import getSampleStyleSheet

doc = SimpleDocTemplate("report.pdf", pagesize=letter)
styles = getSampleStyleSheet()
story = []

# Add content
title = Paragraph("Report Title", styles['Title'])
story.append(title)
story.append(Spacer(1, 12))

body = Paragraph("This is the body of the report. " * 20, styles['Normal'])
story.append(body)
story.append(PageBreak())

# Page 2
story.append(Paragraph("Page 2", styles['Heading1']))
story.append(Paragraph("Content for page 2", styles['Normal']))

# Build PDF
doc.build(story)

Subscripts and Superscripts

IMPORTANT: Never use Unicode subscript/superscript characters (₀₁₂₃₄₅₆₇₈₉, ⁰¹²³⁴⁵⁶⁷⁸⁹) in ReportLab PDFs. The built-in fonts do not include these glyphs, causing them to render as solid black boxes.

Instead, use ReportLab's XML markup tags in Paragraph objects:

from reportlab.platypus import Paragraph
from reportlab.lib.styles import getSampleStyleSheet

styles = getSampleStyleSheet()

# Subscripts: use <sub> tag
chemical = Paragraph("H<sub>2</sub>O", styles['Normal'])

# Superscripts: use <super> tag
squared = Paragraph("x<super>2</super> + y<super>2</super>", styles['Normal'])

For canvas-drawn text (not Paragraph objects), manually adjust font the size and position rather than using Unicode subscripts/superscripts.

Command-Line Tools

pdftotext (poppler-utils)

# Extract text
pdftotext input.pdf output.txt

# Extract text preserving layout
pdftotext -layout input.pdf output.txt

# Extract specific pages
pdftotext -f 1 -l 5 input.pdf output.txt  # Pages 1-5

qpdf

# Merge PDFs
qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf

# Split pages
qpdf input.pdf --pages . 1-5 -- pages1-5.pdf
qpdf input.pdf --pages . 6-10 -- pages6-10.pdf

# Rotate pages
qpdf input.pdf output.pdf --rotate=+90:1  # Rotate page 1 by 90 degrees

# Remove password
qpdf --password=mypassword --decrypt encrypted.pdf decrypted.pdf

pdftk (if available)

# Merge
pdftk file1.pdf file2.pdf cat output merged.pdf

# Split
pdftk input.pdf burst

# Rotate
pdftk input.pdf rotate 1east output rotated.pdf

Common Tasks

Extract Text from Scanned PDFs

# Requires: pip install pytesseract pdf2image
import pytesseract
from pdf2image import convert_from_path

# Convert PDF to images
images = convert_from_path('scanned.pdf')

# OCR each page
text = ""
for i, image in enumerate(images):
    text += f"Page {i+1}:\n"
    text += pytesseract.image_to_string(image)
    text += "\n\n"

print(text)

Add Watermark

from pypdf import PdfReader, PdfWriter

# Create watermark (or load existing)
watermark = PdfReader("watermark.pdf").pages[0]

# Apply to all pages
reader = PdfReader("document.pdf")
writer = PdfWriter()

for page in reader.pages:
    page.merge_page(watermark)
    writer.add_page(page)

with open("watermarked.pdf", "wb") as output:
    writer.write(output)

Extract Images

# Using pdfimages (poppler-utils)
pdfimages -j input.pdf output_prefix

# This extracts all images as output_prefix-000.jpg, output_prefix-001.jpg, etc.

Password Protection

from pypdf import PdfReader, PdfWriter

reader = PdfReader("input.pdf")
writer = PdfWriter()

for page in reader.pages:
    writer.add_page(page)

# Add password
writer.encrypt("userpassword", "ownerpassword")

with open("encrypted.pdf", "wb") as output:
    writer.write(output)

Quick Reference

Next Steps

• For advanced pypdfium2 usage, see reference.md
• For JavaScript libraries (pdf-lib), see reference.md
• If you need to fill out a PDF form, follow the instructions in forms.md
• For troubleshooting guides, see reference.md

Pymoo - Multi-Objective Optimization in Python

Overview

Pymoo is a comprehensive Python framework for optimization with emphasis on multi-objective problems. Solve single and multi-objective optimization using state-of-the-art algorithms (NSGA-II/III, MOEA/D), benchmark problems (ZDT, DTLZ), customizable genetic operators, and multi-criteria decision making methods. Excels at finding trade-off solutions (Pareto fronts) for problems with conflicting objectives.

When to Use This Skill

This skill should be used when:

• Solving optimization problems with one or multiple objectives
• Finding Pareto-optimal solutions and analyzing trade-offs
• Implementing evolutionary algorithms (GA, DE, PSO, NSGA-II/III)
• Working with constrained optimization problems
• Benchmarking algorithms on standard test problems (ZDT, DTLZ, WFG)
• Customizing genetic operators (crossover, mutation, selection)
• Visualizing high-dimensional optimization results
• Making decisions from multiple competing solutions
• Handling binary, discrete, continuous, or mixed-variable problems

Core Concepts

The Unified Interface

Pymoo uses a consistent minimize() function for all optimization tasks:

from pymoo.optimize import minimize

result = minimize(
    problem,        # What to optimize
    algorithm,      # How to optimize
    termination,    # When to stop
    seed=1,
    verbose=True
)

Result object contains:

• result.X: Decision variables of optimal solution(s)
• result.F: Objective values of optimal solution(s)
• result.G: Constraint violations (if constrained)
• result.algorithm: Algorithm object with history

Problem Types

Single-objective: One objective to minimize/maximize Multi-objective: 2-3 conflicting objectives → Pareto front Many-objective: 4+ objectives → High-dimensional Pareto front Constrained: Objectives + inequality/equality constraints Dynamic: Time-varying objectives or constraints

Quick Start Workflows

Workflow 1: Single-Objective Optimization

When: Optimizing one objective function

Steps:

1. Define or select problem
2. Choose single-objective algorithm (GA, DE, PSO, CMA-ES)
3. Configure termination criteria
4. Run optimization
5. Extract best solution

Example:

from pymoo.algorithms.soo.nonconvex.ga import GA
from pymoo.problems import get_problem
from pymoo.optimize import minimize

# Built-in problem
problem = get_problem("rastrigin", n_var=10)

# Configure Genetic Algorithm
algorithm = GA(
    pop_size=100,
    eliminate_duplicates=True
)

# Optimize
result = minimize(
    problem,
    algorithm,
    ('n_gen', 200),
    seed=1,
    verbose=True
)

print(f"Best solution: {result.X}")
print(f"Best objective: {result.F[0]}")

See: scripts/single_objective_example.py for complete example

Workflow 2: Multi-Objective Optimization (2-3 objectives)

When: Optimizing 2-3 conflicting objectives, need Pareto front

Algorithm choice: NSGA-II (standard for bi/tri-objective)

Steps:

1. Define multi-objective problem
2. Configure NSGA-II
3. Run optimization to obtain Pareto front
4. Visualize trade-offs
5. Apply decision making (optional)

Example:

from pymoo.algorithms.moo.nsga2 import NSGA2
from pymoo.problems import get_problem
from pymoo.optimize import minimize
from pymoo.visualization.scatter import Scatter

# Bi-objective benchmark problem
problem = get_problem("zdt1")

# NSGA-II algorithm
algorithm = NSGA2(pop_size=100)

# Optimize
result = minimize(problem, algorithm, ('n_gen', 200), seed=1)

# Visualize Pareto front
plot = Scatter()
plot.add(result.F, label="Obtained Front")
plot.add(problem.pareto_front(), label="True Front", alpha=0.3)
plot.show()

print(f"Found {len(result.F)} Pareto-optimal solutions")

See: scripts/multi_objective_example.py for complete example

Workflow 3: Many-Objective Optimization (4+ objectives)

When: Optimizing 4 or more objectives

Algorithm choice: NSGA-III (designed for many objectives)

Key difference: Must provide reference directions for population guidance

Steps:

1. Define many-objective problem
2. Generate reference directions
3. Configure NSGA-III with reference directions
4. Run optimization
5. Visualize using Parallel Coordinate Plot

Example:

from pymoo.algorithms.moo.nsga3 import NSGA3
from pymoo.problems import get_problem
from pymoo.optimize import minimize
from pymoo.util.ref_dirs import get_reference_directions
from pymoo.visualization.pcp import PCP

# Many-objective problem (5 objectives)
problem = get_problem("dtlz2", n_obj=5)

# Generate reference directions (required for NSGA-III)
ref_dirs = get_reference_directions("das-dennis", n_dim=5, n_partitions=12)

# Configure NSGA-III
algorithm = NSGA3(ref_dirs=ref_dirs)

# Optimize
result = minimize(problem, algorithm, ('n_gen', 300), seed=1)

# Visualize with Parallel Coordinates
plot = PCP(labels=[f"f{i+1}" for i in range(5)])
plot.add(result.F, alpha=0.3)
plot.show()

See: scripts/many_objective_example.py for complete example

Workflow 4: Custom Problem Definition

When: Solving domain-specific optimization problem

Steps:

1. Extend ElementwiseProblem class
2. Define __init__ with problem dimensions and bounds
3. Implement _evaluate method for objectives (and constraints)
4. Use with any algorithm

Unconstrained example:

from pymoo.core.problem import ElementwiseProblem
import numpy as np

class MyProblem(ElementwiseProblem):
    def __init__(self):
        super().__init__(
            n_var=2,              # Number of variables
            n_obj=2,              # Number of objectives
            xl=np.array([0, 0]),  # Lower bounds
            xu=np.array([5, 5])   # Upper bounds
        )

    def _evaluate(self, x, out, *args, **kwargs):
        # Define objectives
        f1 = x[0]**2 + x[1]**2
        f2 = (x[0]-1)**2 + (x[1]-1)**2

        out["F"] = [f1, f2]

Constrained example:

class ConstrainedProblem(ElementwiseProblem):
    def __init__(self):
        super().__init__(
            n_var=2,
            n_obj=2,
            n_ieq_constr=2,        # Inequality constraints
            n_eq_constr=1,         # Equality constraints
            xl=np.array([0, 0]),
            xu=np.array([5, 5])
        )

    def _evaluate(self, x, out, *args, **kwargs):
        # Objectives
        out["F"] = [f1, f2]

        # Inequality constraints (g <= 0)
        out["G"] = [g1, g2]

        # Equality constraints (h = 0)
        out["H"] = [h1]

Constraint formulation rules:

• Inequality: Express as g(x) <= 0 (feasible when ≤ 0)
• Equality: Express as h(x) = 0 (feasible when = 0)
• Convert g(x) >= b to -(g(x) - b) <= 0

See: scripts/custom_problem_example.py for complete examples

Workflow 5: Constraint Handling

When: Problem has feasibility constraints

Approach options:

1. Feasibility First (Default - Recommended)

from pymoo.algorithms.moo.nsga2 import NSGA2

# Works automatically with constrained problems
algorithm = NSGA2(pop_size=100)
result = minimize(problem, algorithm, termination)

# Check feasibility
feasible = result.CV[:, 0] == 0  # CV = constraint violation
print(f"Feasible solutions: {np.sum(feasible)}")

2. Penalty Method

from pymoo.constraints.as_penalty import ConstraintsAsPenalty

# Wrap problem to convert constraints to penalties
problem_penalized = ConstraintsAsPenalty(problem, penalty=1e6)

3. Constraint as Objective

from pymoo.constraints.as_obj import ConstraintsAsObjective

# Treat constraint violation as additional objective
problem_with_cv = ConstraintsAsObjective(problem)

4. Specialized Algorithms

from pymoo.algorithms.soo.nonconvex.sres import SRES

# SRES has built-in constraint handling
algorithm = SRES()

See: references/constraints_mcdm.md for comprehensive constraint handling guide

Workflow 6: Decision Making from Pareto Front

When: Have Pareto front, need to select preferred solution(s)

Steps:

1. Run multi-objective optimization
2. Normalize objectives to [0, 1]
3. Define preference weights
4. Apply MCDM method
5. Visualize selected solution

Example using Pseudo-Weights:

from pymoo.mcdm.pseudo_weights import PseudoWeights
import numpy as np

# After obtaining result from multi-objective optimization
# Normalize objectives
F_norm = (result.F - result.F.min(axis=0)) / (result.F.max(axis=0) - result.F.min(axis=0))

# Define preferences (must sum to 1)
weights = np.array([0.3, 0.7])  # 30% f1, 70% f2

# Apply decision making
dm = PseudoWeights(weights)
selected_idx = dm.do(F_norm)

# Get selected solution
best_solution = result.X[selected_idx]
best_objectives = result.F[selected_idx]

print(f"Selected solution: {best_solution}")
print(f"Objective values: {best_objectives}")

Other MCDM methods:

• Compromise Programming: Select closest to ideal point
• Knee Point: Find balanced trade-off solutions
• Hypervolume Contribution: Select most diverse subset

See:

• scripts/decision_making_example.py for complete example
• references/constraints_mcdm.md for detailed MCDM methods

Workflow 7: Visualization

Choose visualization based on number of objectives:

2 objectives: Scatter Plot

from pymoo.visualization.scatter import Scatter

plot = Scatter(title="Bi-objective Results")
plot.add(result.F, color="blue", alpha=0.7)
plot.show()

3 objectives: 3D Scatter

plot = Scatter(title="Tri-objective Results")
plot.add(result.F)  # Automatically renders in 3D
plot.show()

4+ objectives: Parallel Coordinate Plot

from pymoo.visualization.pcp import PCP

plot = PCP(
    labels=[f"f{i+1}" for i in range(n_obj)],
    normalize_each_axis=True
)
plot.add(result.F, alpha=0.3)
plot.show()

Solution comparison: Petal Diagram

from pymoo.visualization.petal import Petal

plot = Petal(
    bounds=[result.F.min(axis=0), result.F.max(axis=0)],
    labels=["Cost", "Weight", "Efficiency"]
)
plot.add(solution_A, label="Design A")
plot.add(solution_B, label="Design B")
plot.show()

See: references/visualization.md for all visualization types and usage

Algorithm Selection Guide

Single-Objective Problems

Multi-Objective Problems (2-3 objectives)

Many-Objective Problems (4+ objectives)

Constrained Problems

See: references/algorithms.md for comprehensive algorithm reference

Benchmark Problems

Quick problem access:

from pymoo.problems import get_problem

# Single-objective
problem = get_problem("rastrigin", n_var=10)
problem = get_problem("rosenbrock", n_var=10)

# Multi-objective
problem = get_problem("zdt1")        # Convex front
problem = get_problem("zdt2")        # Non-convex front
problem = get_problem("zdt3")        # Disconnected front

# Many-objective
problem = get_problem("dtlz2", n_obj=5, n_var=12)
problem = get_problem("dtlz7", n_obj=4)

See: references/problems.md for complete test problem reference

Genetic Operator Customization

Standard operator configuration:

from pymoo.algorithms.soo.nonconvex.ga import GA
from pymoo.operators.crossover.sbx import SBX
from pymoo.operators.mutation.pm import PM

algorithm = GA(
    pop_size=100,
    crossover=SBX(prob=0.9, eta=15),
    mutation=PM(eta=20),
    eliminate_duplicates=True
)

Operator selection by variable type:

Continuous variables:

• Crossover: SBX (Simulated Binary Crossover)
• Mutation: PM (Polynomial Mutation)

Binary variables:

• Crossover: TwoPointCrossover, UniformCrossover
• Mutation: BitflipMutation

Permutations (TSP, scheduling):

• Crossover: OrderCrossover (OX)
• Mutation: InversionMutation

See: references/operators.md for comprehensive operator reference

Performance and Troubleshooting

Common issues and solutions:

Problem: Algorithm not converging

• Increase population size
• Increase number of generations
• Check if problem is multimodal (try different algorithms)
• Verify constraints are correctly formulated

Problem: Poor Pareto front distribution

• For NSGA-III: Adjust reference directions
• Increase population size
• Check for duplicate elimination
• Verify problem scaling

Problem: Few feasible solutions

• Use constraint-as-objective approach
• Apply repair operators
• Try SRES/ISRES for constrained problems
• Check constraint formulation (should be g <= 0)

Problem: High computational cost

• Reduce population size
• Decrease number of generations
• Use simpler operators
• Enable parallelization (if problem supports)

Best practices:

1. Normalize objectives when scales differ significantly
2. Set random seed for reproducibility
3. Save history to analyze convergence: save_history=True
4. Visualize results to understand solution quality
5. Compare with true Pareto front when available
6. Use appropriate termination criteria (generations, evaluations, tolerance)
7. Tune operator parameters for problem characteristics

Resources

This skill includes comprehensive reference documentation and executable examples:

references/

Detailed documentation for in-depth understanding:

• algorithms.md: Complete algorithm reference with parameters, usage, and selection guidelines
• problems.md: Benchmark test problems (ZDT, DTLZ, WFG) with characteristics
• operators.md: Genetic operators (sampling, selection, crossover, mutation) with configuration
• visualization.md: All visualization types with examples and selection guide
• constraints_mcdm.md: Constraint handling techniques and multi-criteria decision making methods

Search patterns for references:

• Algorithm details: grep -r "NSGA-II\|NSGA-III\|MOEA/D" references/
• Constraint methods: grep -r "Feasibility First\|Penalty\|Repair" references/
• Visualization types: grep -r "Scatter\|PCP\|Petal" references/

scripts/

Executable examples demonstrating common workflows:

• single_objective_example.py: Basic single-objective optimization with GA
• multi_objective_example.py: Multi-objective optimization with NSGA-II, visualization
• many_objective_example.py: Many-objective optimization with NSGA-III, reference directions
• custom_problem_example.py: Defining custom problems (constrained and unconstrained)
• decision_making_example.py: Multi-criteria decision making with different preferences

Run examples:

python3 scripts/single_objective_example.py
python3 scripts/multi_objective_example.py
python3 scripts/many_objective_example.py
python3 scripts/custom_problem_example.py
python3 scripts/decision_making_example.py

Additional Notes

Installation:

uv pip install pymoo

Dependencies: NumPy, SciPy, matplotlib, autograd (optional for gradient-based)

Documentation: https://pymoo.org/

Version: This skill is based on pymoo 0.6.x

Common patterns:

• Always use ElementwiseProblem for custom problems
• Constraints formulated as g(x) <= 0 and h(x) = 0
• Reference directions required for NSGA-III
• Normalize objectives before MCDM
• Use appropriate termination: ('n_gen', N) or get_termination("f_tol", tol=0.001)

QuTiP: Quantum Toolbox in Python

Overview

QuTiP provides comprehensive tools for simulating and analyzing quantum mechanical systems. It handles both closed (unitary) and open (dissipative) quantum systems with multiple solvers optimized for different scenarios.

Installation

uv pip install qutip

Optional packages for additional functionality:

# Quantum information processing (circuits, gates)
uv pip install qutip-qip

# Quantum trajectory viewer
uv pip install qutip-qtrl

Quick Start

from qutip import *
import numpy as np
import matplotlib.pyplot as plt

# Create quantum state
psi = basis(2, 0)  # |0⟩ state

# Create operator
H = sigmaz()  # Hamiltonian

# Time evolution
tlist = np.linspace(0, 10, 100)
result = sesolve(H, psi, tlist, e_ops=[sigmaz()])

# Plot results
plt.plot(tlist, result.expect[0])
plt.xlabel('Time')
plt.ylabel('⟨σz⟩')
plt.show()

Core Capabilities

1. Quantum Objects and States

Create and manipulate quantum states and operators:

# States
psi = basis(N, n)  # Fock state |n⟩
psi = coherent(N, alpha)  # Coherent state |α⟩
rho = thermal_dm(N, n_avg)  # Thermal density matrix

# Operators
a = destroy(N)  # Annihilation operator
H = num(N)  # Number operator
sx, sy, sz = sigmax(), sigmay(), sigmaz()  # Pauli matrices

# Composite systems
psi_AB = tensor(psi_A, psi_B)  # Tensor product

See references/core_concepts.md for comprehensive coverage of quantum objects, states, operators, and tensor products.

2. Time Evolution and Dynamics

Multiple solvers for different scenarios:

# Closed systems (unitary evolution)
result = sesolve(H, psi0, tlist, e_ops=[num(N)])

# Open systems (dissipation)
c_ops = [np.sqrt(0.1) * destroy(N)]  # Collapse operators
result = mesolve(H, psi0, tlist, c_ops, e_ops=[num(N)])

# Quantum trajectories (Monte Carlo)
result = mcsolve(H, psi0, tlist, c_ops, ntraj=500, e_ops=[num(N)])

Solver selection guide:

• sesolve: Pure states, unitary evolution
• mesolve: Mixed states, dissipation, general open systems
• mcsolve: Quantum jumps, photon counting, individual trajectories
• brmesolve: Weak system-bath coupling
• fmmesolve: Time-periodic Hamiltonians (Floquet)

See references/time_evolution.md for detailed solver documentation, time-dependent Hamiltonians, and advanced options.

3. Analysis and Measurement

Compute physical quantities:

# Expectation values
n_avg = expect(num(N), psi)

# Entropy measures
S = entropy_vn(rho)  # Von Neumann entropy
C = concurrence(rho)  # Entanglement (two qubits)

# Fidelity and distance
F = fidelity(psi1, psi2)
D = tracedist(rho1, rho2)

# Correlation functions
corr = correlation_2op_1t(H, rho0, taulist, c_ops, A, B)
w, S = spectrum_correlation_fft(taulist, corr)

# Steady states
rho_ss = steadystate(H, c_ops)

See references/analysis.md for entropy, fidelity, measurements, correlation functions, and steady state calculations.

4. Visualization

Visualize quantum states and dynamics:

# Bloch sphere
b = Bloch()
b.add_states(psi)
b.show()

# Wigner function (phase space)
xvec = np.linspace(-5, 5, 200)
W = wigner(psi, xvec, xvec)
plt.contourf(xvec, xvec, W, 100, cmap='RdBu')

# Fock distribution
plot_fock_distribution(psi)

# Matrix visualization
hinton(rho)  # Hinton diagram
matrix_histogram(H.full())  # 3D bars

See references/visualization.md for Bloch sphere animations, Wigner functions, Q-functions, and matrix visualizations.

5. Advanced Methods

Specialized techniques for complex scenarios:

# Floquet theory (periodic Hamiltonians)
T = 2 * np.pi / w_drive
f_modes, f_energies = floquet_modes(H, T, args)
result = fmmesolve(H, psi0, tlist, c_ops, T=T, args=args)

# HEOM (non-Markovian, strong coupling)
from qutip.nonmarkov.heom import HEOMSolver, BosonicBath
bath = BosonicBath(Q, ck_real, vk_real)
hsolver = HEOMSolver(H_sys, [bath], max_depth=5)
result = hsolver.run(rho0, tlist)

# Permutational invariance (identical particles)
psi = dicke(N, j, m)  # Dicke states
Jz = jspin(N, 'z')  # Collective operators

See references/advanced.md for Floquet theory, HEOM, permutational invariance, stochastic solvers, superoperators, and performance optimization.

Common Workflows

Simulating a Damped Harmonic Oscillator

# System parameters
N = 20  # Hilbert space dimension
omega = 1.0  # Oscillator frequency
kappa = 0.1  # Decay rate

# Hamiltonian and collapse operators
H = omega * num(N)
c_ops = [np.sqrt(kappa) * destroy(N)]

# Initial state
psi0 = coherent(N, 3.0)

# Time evolution
tlist = np.linspace(0, 50, 200)
result = mesolve(H, psi0, tlist, c_ops, e_ops=[num(N)])

# Visualize
plt.plot(tlist, result.expect[0])
plt.xlabel('Time')
plt.ylabel('⟨n⟩')
plt.title('Photon Number Decay')
plt.show()

Two-Qubit Entanglement Dynamics

# Create Bell state
psi0 = bell_state('00')

# Local dephasing on each qubit
gamma = 0.1
c_ops = [
    np.sqrt(gamma) * tensor(sigmaz(), qeye(2)),
    np.sqrt(gamma) * tensor(qeye(2), sigmaz())
]

# Track entanglement
def compute_concurrence(t, psi):
    rho = ket2dm(psi) if psi.isket else psi
    return concurrence(rho)

tlist = np.linspace(0, 10, 100)
result = mesolve(qeye([2, 2]), psi0, tlist, c_ops)

# Compute concurrence for each state
C_t = [concurrence(state.proj()) for state in result.states]

plt.plot(tlist, C_t)
plt.xlabel('Time')
plt.ylabel('Concurrence')
plt.title('Entanglement Decay')
plt.show()

Jaynes-Cummings Model

# System parameters
N = 10  # Cavity Fock space
wc = 1.0  # Cavity frequency
wa = 1.0  # Atom frequency
g = 0.05  # Coupling strength

# Operators
a = tensor(destroy(N), qeye(2))  # Cavity
sm = tensor(qeye(N), sigmam())  # Atom

# Hamiltonian (RWA)
H = wc * a.dag() * a + wa * sm.dag() * sm + g * (a.dag() * sm + a * sm.dag())

# Initial state: cavity in coherent state, atom in ground state
psi0 = tensor(coherent(N, 2), basis(2, 0))

# Dissipation
kappa = 0.1  # Cavity decay
gamma = 0.05  # Atomic decay
c_ops = [np.sqrt(kappa) * a, np.sqrt(gamma) * sm]

# Observables
n_cav = a.dag() * a
n_atom = sm.dag() * sm

# Evolve
tlist = np.linspace(0, 50, 200)
result = mesolve(H, psi0, tlist, c_ops, e_ops=[n_cav, n_atom])

# Plot
fig, axes = plt.subplots(2, 1, figsize=(8, 6), sharex=True)
axes[0].plot(tlist, result.expect[0])
axes[0].set_ylabel('⟨n_cavity⟩')
axes[1].plot(tlist, result.expect[1])
axes[1].set_ylabel('⟨n_atom⟩')
axes[1].set_xlabel('Time')
plt.tight_layout()
plt.show()

Tips for Efficient Simulations

1. Truncate Hilbert spaces: Use smallest dimension that captures dynamics
2. Choose appropriate solver: sesolve for pure states is faster than mesolve
3. Time-dependent terms: String format (e.g., 'cos(w*t)') is fastest
4. Store only needed data: Use e_ops instead of storing all states
5. Adjust tolerances: Balance accuracy with computation time via Options
6. Parallel trajectories: mcsolve automatically uses multiple CPUs
7. Check convergence: Vary ntraj, Hilbert space size, and tolerances

Troubleshooting

Memory issues: Reduce Hilbert space dimension, use store_final_state option, or consider Krylov methods

Slow simulations: Use string-based time-dependence, increase tolerances slightly, or try method='bdf' for stiff problems

Numerical instabilities: Decrease time steps (nsteps option), increase tolerances, or check Hamiltonian/operators are properly defined

Import errors: Ensure QuTiP is installed correctly; quantum gates require qutip-qip package

References

This skill includes detailed reference documentation:

• references/core_concepts.md: Quantum objects, states, operators, tensor products, composite systems
• references/time_evolution.md: All solvers (sesolve, mesolve, mcsolve, brmesolve, etc.), time-dependent Hamiltonians, solver options
• references/visualization.md: Bloch sphere, Wigner functions, Q-functions, Fock distributions, matrix plots
• references/analysis.md: Expectation values, entropy, fidelity, entanglement measures, correlation functions, steady states
• references/advanced.md: Floquet theory, HEOM, permutational invariance, stochastic methods, superoperators, performance tips

External Resources

• Documentation: https://qutip.readthedocs.io/
• Tutorials: https://qutip.org/qutip-tutorials/
• API Reference: https://qutip.readthedocs.io/en/stable/apidoc/apidoc.html
• GitHub: https://github.com/qutip/qutip

Scientific Visualization

Overview

Scientific visualization transforms data into clear, accurate figures for publication. Create journal-ready plots with multi-panel layouts, error bars, significance markers, and colorblind-safe palettes. Export as PDF/EPS/TIFF using matplotlib, seaborn, and plotly for manuscripts.

When to Use This Skill

This skill should be used when:

• Creating plots or visualizations for scientific manuscripts
• Preparing figures for journal submission (Nature, Science, Cell, PLOS, etc.)
• Ensuring figures are colorblind-friendly and accessible
• Making multi-panel figures with consistent styling
• Exporting figures at correct resolution and format
• Following specific publication guidelines
• Improving existing figures to meet publication standards
• Creating figures that need to work in both color and grayscale

Quick Start Guide

Basic Publication-Quality Figure

import matplotlib.pyplot as plt
import numpy as np

# Apply publication style (from scripts/style_presets.py)
from style_presets import apply_publication_style
apply_publication_style('default')

# Create figure with appropriate size (single column = 3.5 inches)
fig, ax = plt.subplots(figsize=(3.5, 2.5))

# Plot data
x = np.linspace(0, 10, 100)
ax.plot(x, np.sin(x), label='sin(x)')
ax.plot(x, np.cos(x), label='cos(x)')

# Proper labeling with units
ax.set_xlabel('Time (seconds)')
ax.set_ylabel('Amplitude (mV)')
ax.legend(frameon=False)

# Remove unnecessary spines
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)

# Save in publication formats (from scripts/figure_export.py)
from figure_export import save_publication_figure
save_publication_figure(fig, 'figure1', formats=['pdf', 'png'], dpi=300)

Using Pre-configured Styles

Apply journal-specific styles using the matplotlib style files in assets/:

import matplotlib.pyplot as plt

# Option 1: Use style file directly
plt.style.use('assets/nature.mplstyle')

# Option 2: Use style_presets.py helper
from style_presets import configure_for_journal
configure_for_journal('nature', figure_width='single')

# Now create figures - they'll automatically match Nature specifications
fig, ax = plt.subplots()
# ... your plotting code ...

Quick Start with Seaborn

For statistical plots, use seaborn with publication styling:

import seaborn as sns
import matplotlib.pyplot as plt
from style_presets import apply_publication_style

# Apply publication style
apply_publication_style('default')
sns.set_theme(style='ticks', context='paper', font_scale=1.1)
sns.set_palette('colorblind')

# Create statistical comparison figure
fig, ax = plt.subplots(figsize=(3.5, 3))
sns.boxplot(data=df, x='treatment', y='response', 
            order=['Control', 'Low', 'High'], palette='Set2', ax=ax)
sns.stripplot(data=df, x='treatment', y='response',
              order=['Control', 'Low', 'High'], 
              color='black', alpha=0.3, size=3, ax=ax)
ax.set_ylabel('Response (μM)')
sns.despine()

# Save figure
from figure_export import save_publication_figure
save_publication_figure(fig, 'treatment_comparison', formats=['pdf', 'png'], dpi=300)

Core Principles and Best Practices

1. Resolution and File Format

Critical requirements (detailed in references/publication_guidelines.md):

• Raster images (photos, microscopy): 300-600 DPI
• Line art (graphs, plots): 600-1200 DPI or vector format
• Vector formats (preferred): PDF, EPS, SVG
• Raster formats: TIFF, PNG (never JPEG for scientific data)

Implementation:

# Use the figure_export.py script for correct settings
from figure_export import save_publication_figure

# Saves in multiple formats with proper DPI
save_publication_figure(fig, 'myfigure', formats=['pdf', 'png'], dpi=300)

# Or save for specific journal requirements
from figure_export import save_for_journal
save_for_journal(fig, 'figure1', journal='nature', figure_type='combination')

2. Color Selection - Colorblind Accessibility

Always use colorblind-friendly palettes (detailed in references/color_palettes.md):

Recommended: Okabe-Ito palette (distinguishable by all types of color blindness):

# Option 1: Use assets/color_palettes.py
from color_palettes import OKABE_ITO_LIST, apply_palette
apply_palette('okabe_ito')

# Option 2: Manual specification
okabe_ito = ['#E69F00', '#56B4E9', '#009E73', '#F0E442',
             '#0072B2', '#D55E00', '#CC79A7', '#000000']
plt.rcParams['axes.prop_cycle'] = plt.cycler(color=okabe_ito)

For heatmaps/continuous data:

• Use perceptually uniform colormaps: viridis, plasma, cividis
• Avoid red-green diverging maps (use PuOr, RdBu, BrBG instead)
• Never use jet or rainbow colormaps

Always test figures in grayscale to ensure interpretability.

3. Typography and Text

Font guidelines (detailed in references/publication_guidelines.md):

• Sans-serif fonts: Arial, Helvetica, Calibri
• Minimum sizes at final print size:
  • Axis labels: 7-9 pt
  • Tick labels: 6-8 pt
  • Panel labels: 8-12 pt (bold)
• Sentence case for labels: "Time (hours)" not "TIME (HOURS)"
• Always include units in parentheses

Implementation:

# Set fonts globally
import matplotlib as mpl
mpl.rcParams['font.family'] = 'sans-serif'
mpl.rcParams['font.sans-serif'] = ['Arial', 'Helvetica']
mpl.rcParams['font.size'] = 8
mpl.rcParams['axes.labelsize'] = 9
mpl.rcParams['xtick.labelsize'] = 7
mpl.rcParams['ytick.labelsize'] = 7

4. Figure Dimensions

Journal-specific widths (detailed in references/journal_requirements.md):

• Nature: Single 89 mm, Double 183 mm
• Science: Single 55 mm, Double 175 mm
• Cell: Single 85 mm, Double 178 mm

Check figure size compliance:

from figure_export import check_figure_size

fig = plt.figure(figsize=(3.5, 3))  # 89 mm for Nature
check_figure_size(fig, journal='nature')

5. Multi-Panel Figures

Best practices:

• Label panels with bold letters: A, B, C (uppercase for most journals, lowercase for Nature)
• Maintain consistent styling across all panels
• Align panels along edges where possible
• Use adequate white space between panels

Example implementation (see references/matplotlib_examples.md for complete code):

from string import ascii_uppercase

fig = plt.figure(figsize=(7, 4))
gs = fig.add_gridspec(2, 2, hspace=0.4, wspace=0.4)

ax1 = fig.add_subplot(gs[0, 0])
ax2 = fig.add_subplot(gs[0, 1])
# ... create other panels ...

# Add panel labels
for i, ax in enumerate([ax1, ax2, ...]):
    ax.text(-0.15, 1.05, ascii_uppercase[i], transform=ax.transAxes,
            fontsize=10, fontweight='bold', va='top')

Common Tasks

Task 1: Create a Publication-Ready Line Plot

See references/matplotlib_examples.md Example 1 for complete code.

Key steps:

1. Apply publication style
2. Set appropriate figure size for target journal
3. Use colorblind-friendly colors
4. Add error bars with correct representation (SEM, SD, or CI)
5. Label axes with units
6. Remove unnecessary spines
7. Save in vector format

Using seaborn for automatic confidence intervals:

import seaborn as sns
fig, ax = plt.subplots(figsize=(5, 3))
sns.lineplot(data=timeseries, x='time', y='measurement',
             hue='treatment', errorbar=('ci', 95), 
             markers=True, ax=ax)
ax.set_xlabel('Time (hours)')
ax.set_ylabel('Measurement (AU)')
sns.despine()

Task 2: Create a Multi-Panel Figure

See references/matplotlib_examples.md Example 2 for complete code.

Key steps:

1. Use GridSpec for flexible layout
2. Ensure consistent styling across panels
3. Add bold panel labels (A, B, C, etc.)
4. Align related panels
5. Verify all text is readable at final size

Task 3: Create a Heatmap with Proper Colormap

See references/matplotlib_examples.md Example 4 for complete code.

Key steps:

1. Use perceptually uniform colormap (viridis, plasma, cividis)
2. Include labeled colorbar
3. For diverging data, use colorblind-safe diverging map (RdBu_r, PuOr)
4. Set appropriate center value for diverging maps
5. Test appearance in grayscale

Using seaborn for correlation matrices:

import seaborn as sns
fig, ax = plt.subplots(figsize=(5, 4))
corr = df.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, mask=mask, annot=True, fmt='.2f',
            cmap='RdBu_r', center=0, square=True,
            linewidths=1, cbar_kws={'shrink': 0.8}, ax=ax)

Task 4: Prepare Figure for Specific Journal

Workflow:

1. Check journal requirements: references/journal_requirements.md
2. Configure matplotlib for journal:

   from style_presets import configure_for_journal
   configure_for_journal('nature', figure_width='single')
   

3. Create figure (will auto-size correctly)
4. Export with journal specifications:

   from figure_export import save_for_journal
   save_for_journal(fig, 'figure1', journal='nature', figure_type='line_art')
   

Task 5: Fix an Existing Figure to Meet Publication Standards

Checklist approach (full checklist in references/publication_guidelines.md):

1. Check resolution: Verify DPI meets journal requirements
2. Check file format: Use vector for plots, TIFF/PNG for images
3. Check colors: Ensure colorblind-friendly
4. Check fonts: Minimum 6-7 pt at final size, sans-serif
5. Check labels: All axes labeled with units
6. Check size: Matches journal column width
7. Test grayscale: Figure interpretable without color
8. Remove chart junk: No unnecessary grids, 3D effects, shadows

Task 6: Create Colorblind-Friendly Visualizations

Strategy:

1. Use approved palettes from assets/color_palettes.py
2. Add redundant encoding (line styles, markers, patterns)
3. Test with colorblind simulator
4. Ensure grayscale compatibility

Example:

from color_palettes import apply_palette
import matplotlib.pyplot as plt

apply_palette('okabe_ito')

# Add redundant encoding beyond color
line_styles = ['-', '--', '-.', ':']
markers = ['o', 's', '^', 'v']

for i, (data, label) in enumerate(datasets):
    plt.plot(x, data, linestyle=line_styles[i % 4],
             marker=markers[i % 4], label=label)

Statistical Rigor

Always include:

• Error bars (SD, SEM, or CI - specify which in caption)
• Sample size (n) in figure or caption
• Statistical significance markers (*, **, ***)
• Individual data points when possible (not just summary statistics)

Example with statistics:

# Show individual points with summary statistics
ax.scatter(x_jittered, individual_points, alpha=0.4, s=8)
ax.errorbar(x, means, yerr=sems, fmt='o', capsize=3)

# Mark significance
ax.text(1.5, max_y * 1.1, '***', ha='center', fontsize=8)

Working with Different Plotting Libraries

Matplotlib

• Most control over publication details
• Best for complex multi-panel figures
• Use provided style files for consistent formatting
• See references/matplotlib_examples.md for extensive examples

Seaborn

Seaborn provides a high-level, dataset-oriented interface for statistical graphics, built on matplotlib. It excels at creating publication-quality statistical visualizations with minimal code while maintaining full compatibility with matplotlib customization.

Key advantages for scientific visualization:

• Automatic statistical estimation and confidence intervals
• Built-in support for multi-panel figures (faceting)
• Colorblind-friendly palettes by default
• Dataset-oriented API using pandas DataFrames
• Semantic mapping of variables to visual properties

Quick Start with Publication Style

Always apply matplotlib publication styles first, then configure seaborn:

import seaborn as sns
import matplotlib.pyplot as plt
from style_presets import apply_publication_style

# Apply publication style
apply_publication_style('default')

# Configure seaborn for publication
sns.set_theme(style='ticks', context='paper', font_scale=1.1)
sns.set_palette('colorblind')  # Use colorblind-safe palette

# Create figure
fig, ax = plt.subplots(figsize=(3.5, 2.5))
sns.scatterplot(data=df, x='time', y='response', 
                hue='treatment', style='condition', ax=ax)
sns.despine()  # Remove top and right spines

Common Plot Types for Publications

Statistical comparisons:

# Box plot with individual points for transparency
fig, ax = plt.subplots(figsize=(3.5, 3))
sns.boxplot(data=df, x='treatment', y='response', 
            order=['Control', 'Low', 'High'], palette='Set2', ax=ax)
sns.stripplot(data=df, x='treatment', y='response',
              order=['Control', 'Low', 'High'], 
              color='black', alpha=0.3, size=3, ax=ax)
ax.set_ylabel('Response (μM)')
sns.despine()

Distribution analysis:

# Violin plot with split comparison
fig, ax = plt.subplots(figsize=(4, 3))
sns.violinplot(data=df, x='timepoint', y='expression',
               hue='treatment', split=True, inner='quartile', ax=ax)
ax.set_ylabel('Gene Expression (AU)')
sns.despine()

Correlation matrices:

# Heatmap with proper colormap and annotations
fig, ax = plt.subplots(figsize=(5, 4))
corr = df.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))  # Show only lower triangle
sns.heatmap(corr, mask=mask, annot=True, fmt='.2f',
            cmap='RdBu_r', center=0, square=True,
            linewidths=1, cbar_kws={'shrink': 0.8}, ax=ax)
plt.tight_layout()

Time series with confidence bands:

# Line plot with automatic CI calculation
fig, ax = plt.subplots(figsize=(5, 3))
sns.lineplot(data=timeseries, x='time', y='measurement',
             hue='treatment', style='replicate',
             errorbar=('ci', 95), markers=True, dashes=False, ax=ax)
ax.set_xlabel('Time (hours)')
ax.set_ylabel('Measurement (AU)')
sns.despine()

Multi-Panel Figures with Seaborn

Using FacetGrid for automatic faceting:

# Create faceted plot
g = sns.relplot(data=df, x='dose', y='response',
                hue='treatment', col='cell_line', row='timepoint',
                kind='line', height=2.5, aspect=1.2,
                errorbar=('ci', 95), markers=True)
g.set_axis_labels('Dose (μM)', 'Response (AU)')
g.set_titles('{row_name} | {col_name}')
sns.despine()

# Save with correct DPI
from figure_export import save_publication_figure
save_publication_figure(g.figure, 'figure_facets', 
                       formats=['pdf', 'png'], dpi=300)

Combining seaborn with matplotlib subplots:

# Create custom multi-panel layout
fig, axes = plt.subplots(2, 2, figsize=(7, 6))

# Panel A: Scatter with regression
sns.regplot(data=df, x='predictor', y='response', ax=axes[0, 0])
axes[0, 0].text(-0.15, 1.05, 'A', transform=axes[0, 0].transAxes,
                fontsize=10, fontweight='bold')

# Panel B: Distribution comparison
sns.violinplot(data=df, x='group', y='value', ax=axes[0, 1])
axes[0, 1].text(-0.15, 1.05, 'B', transform=axes[0, 1].transAxes,
                fontsize=10, fontweight='bold')

# Panel C: Heatmap
sns.heatmap(correlation_data, cmap='viridis', ax=axes[1, 0])
axes[1, 0].text(-0.15, 1.05, 'C', transform=axes[1, 0].transAxes,
                fontsize=10, fontweight='bold')

# Panel D: Time series
sns.lineplot(data=timeseries, x='time', y='signal', 
             hue='condition', ax=axes[1, 1])
axes[1, 1].text(-0.15, 1.05, 'D', transform=axes[1, 1].transAxes,
                fontsize=10, fontweight='bold')

plt.tight_layout()
sns.despine()

Color Palettes for Publications

Seaborn includes several colorblind-safe palettes:

# Use built-in colorblind palette (recommended)
sns.set_palette('colorblind')

# Or specify custom colorblind-safe colors (Okabe-Ito)
okabe_ito = ['#E69F00', '#56B4E9', '#009E73', '#F0E442',
             '#0072B2', '#D55E00', '#CC79A7', '#000000']
sns.set_palette(okabe_ito)

# For heatmaps and continuous data
sns.heatmap(data, cmap='viridis')  # Perceptually uniform
sns.heatmap(corr, cmap='RdBu_r', center=0)  # Diverging, centered

Choosing Between Axes-Level and Figure-Level Functions

Axes-level functions (e.g., scatterplot, boxplot, heatmap):

• Use when building custom multi-panel layouts
• Accept ax= parameter for precise placement
• Better integration with matplotlib subplots
• More control over figure composition

fig, ax = plt.subplots(figsize=(3.5, 2.5))
sns.scatterplot(data=df, x='x', y='y', hue='group', ax=ax)

Figure-level functions (e.g., relplot, catplot, displot):

• Use for automatic faceting by categorical variables
• Create complete figures with consistent styling
• Great for exploratory analysis
• Use height and aspect for sizing

g = sns.relplot(data=df, x='x', y='y', col='category', kind='scatter')

Statistical Rigor with Seaborn

Seaborn automatically computes and displays uncertainty:

# Line plot: shows mean ± 95% CI by default
sns.lineplot(data=df, x='time', y='value', hue='treatment',
             errorbar=('ci', 95))  # Can change to 'sd', 'se', etc.

# Bar plot: shows mean with bootstrapped CI
sns.barplot(data=df, x='treatment', y='response',
            errorbar=('ci', 95), capsize=0.1)

# Always specify error type in figure caption:
# "Error bars represent 95% confidence intervals"

Best Practices for Publication-Ready Seaborn Figures

1. Always set publication theme first:

   sns.set_theme(style='ticks', context='paper', font_scale=1.1)
   

2. Use colorblind-safe palettes:

   sns.set_palette('colorblind')
   

3. Remove unnecessary elements:

   sns.despine()  # Remove top and right spines
   

4. Control figure size appropriately:

   # Axes-level: use matplotlib figsize
   fig, ax = plt.subplots(figsize=(3.5, 2.5))
   
   # Figure-level: use height and aspect
   g = sns.relplot(..., height=3, aspect=1.2)
   

5. Show individual data points when possible:

   sns.boxplot(...)  # Summary statistics
   sns.stripplot(..., alpha=0.3)  # Individual points
   

6. Include proper labels with units:

   ax.set_xlabel('Time (hours)')
   ax.set_ylabel('Expression (AU)')
   

7. Export at correct resolution:

   from figure_export import save_publication_figure
   save_publication_figure(fig, 'figure_name', 
                          formats=['pdf', 'png'], dpi=300)
   

Advanced Seaborn Techniques

Pairwise relationships for exploratory analysis:

# Quick overview of all relationships
g = sns.pairplot(data=df, hue='condition', 
                 vars=['gene1', 'gene2', 'gene3'],
                 corner=True, diag_kind='kde', height=2)

Hierarchical clustering heatmap:

# Cluster samples and features
g = sns.clustermap(expression_data, method='ward', 
                   metric='euclidean', z_score=0,
                   cmap='RdBu_r', center=0, 
                   figsize=(10, 8), 
                   row_colors=condition_colors,
                   cbar_kws={'label': 'Z-score'})

Joint distributions with marginals:

# Bivariate distribution with context
g = sns.jointplot(data=df, x='gene1', y='gene2',
                  hue='treatment', kind='scatter',
                  height=6, ratio=4, marginal_kws={'kde': True})

Common Seaborn Issues and Solutions

Issue: Legend outside plot area

g = sns.relplot(...)
g._legend.set_bbox_to_anchor((0.9, 0.5))

Issue: Overlapping labels

plt.xticks(rotation=45, ha='right')
plt.tight_layout()

Issue: Text too small at final size

sns.set_context('paper', font_scale=1.2)  # Increase if needed

Additional Resources

For more detailed seaborn information, see:

• scientific-skills/seaborn/SKILL.md - Comprehensive seaborn documentation
• scientific-skills/seaborn/references/examples.md - Practical use cases
• scientific-skills/seaborn/references/function_reference.md - Complete API reference
• scientific-skills/seaborn/references/objects_interface.md - Modern declarative API

Plotly

• Interactive figures for exploration
• Export static images for publication
• Configure for publication quality:

fig.update_layout(
    font=dict(family='Arial, sans-serif', size=10),
    plot_bgcolor='white',
    # ... see matplotlib_examples.md Example 8
)
fig.write_image('figure.png', scale=3)  # scale=3 gives ~300 DPI

Resources

References Directory

Load these as needed for detailed information:

• publication_guidelines.md: Comprehensive best practices

  • Resolution and file format requirements
  • Typography guidelines
  • Layout and composition rules
  • Statistical rigor requirements
  • Complete publication checklist

• color_palettes.md: Color usage guide

  • Colorblind-friendly palette specifications with RGB values
  • Sequential and diverging colormap recommendations
  • Testing procedures for accessibility
  • Domain-specific palettes (genomics, microscopy)

• journal_requirements.md: Journal-specific specifications

  • Technical requirements by publisher
  • File format and DPI specifications
  • Figure dimension requirements
  • Quick reference table

• matplotlib_examples.md: Practical code examples

  • 10 complete working examples
  • Line plots, bar plots, heatmaps, multi-panel figures
  • Journal-specific figure examples
  • Tips for each library (matplotlib, seaborn, plotly)

Scripts Directory

Use these helper scripts for automation:

• figure_export.py: Export utilities

  • save_publication_figure(): Save in multiple formats with correct DPI
  • save_for_journal(): Use journal-specific requirements automatically
  • check_figure_size(): Verify dimensions meet journal specs
  • Run directly: python scripts/figure_export.py for examples

• style_presets.py: Pre-configured styles

  • apply_publication_style(): Apply preset styles (default, nature, science, cell)
  • set_color_palette(): Quick palette switching
  • configure_for_journal(): One-command journal configuration
  • Run directly: python scripts/style_presets.py to see examples

Assets Directory

Use these files in figures:

• color_palettes.py: Importable color definitions

  • All recommended palettes as Python constants
  • apply_palette() helper function
  • Can be imported directly into notebooks/scripts

• Matplotlib style files: Use with plt.style.use()

  • publication.mplstyle: General publication quality
  • nature.mplstyle: Nature journal specifications
  • presentation.mplstyle: Larger fonts for posters/slides

Workflow Summary

Recommended workflow for creating publication figures:

1. Plan: Determine target journal, figure type, and content
2. Configure: Apply appropriate style for journal

   from style_presets import configure_for_journal
   configure_for_journal('nature', 'single')
   

3. Create: Build figure with proper labels, colors, statistics
4. Verify: Check size, fonts, colors, accessibility

   from figure_export import check_figure_size
   check_figure_size(fig, journal='nature')
   

5. Export: Save in required formats

   from figure_export import save_for_journal
   save_for_journal(fig, 'figure1', 'nature', 'combination')
   

6. Review: View at final size in manuscript context

Common Pitfalls to Avoid

1. Font too small: Text unreadable when printed at final size
2. JPEG format: Never use JPEG for graphs/plots (creates artifacts)
3. Red-green colors: ~8% of males cannot distinguish
4. Low resolution: Pixelated figures in publication
5. Missing units: Always label axes with units
6. 3D effects: Distorts perception, avoid completely
7. Chart junk: Remove unnecessary gridlines, decorations
8. Truncated axes: Start bar charts at zero unless scientifically justified
9. Inconsistent styling: Different fonts/colors across figures in same manuscript
10. No error bars: Always show uncertainty

Final Checklist

Before submitting figures, verify:

• Resolution meets journal requirements (300+ DPI)
• File format is correct (vector for plots, TIFF for images)
• Figure size matches journal specifications
• All text readable at final size (≥6 pt)
• Colors are colorblind-friendly
• Figure works in grayscale
• All axes labeled with units
• Error bars present with definition in caption
• Panel labels present and consistent
• No chart junk or 3D effects
• Fonts consistent across all figures
• Statistical significance clearly marked
• Legend is clear and complete

Use this skill to ensure scientific figures meet the highest publication standards while remaining accessible to all readers.

Scikit-learn

Overview

This skill provides comprehensive guidance for machine learning tasks using scikit-learn, the industry-standard Python library for classical machine learning. Use this skill for classification, regression, clustering, dimensionality reduction, preprocessing, model evaluation, and building production-ready ML pipelines.

Installation

# Install scikit-learn using uv
uv pip install scikit-learn

# Optional: Install visualization dependencies
uv pip install matplotlib seaborn

# Commonly used with
uv pip install pandas numpy

When to Use This Skill

Use the scikit-learn skill when:

• Building classification or regression models
• Performing clustering or dimensionality reduction
• Preprocessing and transforming data for machine learning
• Evaluating model performance with cross-validation
• Tuning hyperparameters with grid or random search
• Creating ML pipelines for production workflows
• Comparing different algorithms for a task
• Working with both structured (tabular) and text data
• Need interpretable, classical machine learning approaches

Quick Start

Classification Example

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# Preprocess
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train_scaled, y_train)

# Evaluate
y_pred = model.predict(X_test_scaled)
print(classification_report(y_test, y_pred))

Complete Pipeline with Mixed Data

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier

# Define feature types
numeric_features = ['age', 'income']
categorical_features = ['gender', 'occupation']

# Create preprocessing pipelines
numeric_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

# Combine transformers
preprocessor = ColumnTransformer([
    ('num', numeric_transformer, numeric_features),
    ('cat', categorical_transformer, categorical_features)
])

# Full pipeline
model = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', GradientBoostingClassifier(random_state=42))
])

# Fit and predict
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

Core Capabilities

1. Supervised Learning

Comprehensive algorithms for classification and regression tasks.

Key algorithms:

• Linear models: Logistic Regression, Linear Regression, Ridge, Lasso, ElasticNet
• Tree-based: Decision Trees, Random Forest, Gradient Boosting
• Support Vector Machines: SVC, SVR with various kernels
• Ensemble methods: AdaBoost, Voting, Stacking
• Neural Networks: MLPClassifier, MLPRegressor
• Others: Naive Bayes, K-Nearest Neighbors

When to use:

• Classification: Predicting discrete categories (spam detection, image classification, fraud detection)
• Regression: Predicting continuous values (price prediction, demand forecasting)

See: references/supervised_learning.md for detailed algorithm documentation, parameters, and usage examples.

2. Unsupervised Learning

Discover patterns in unlabeled data through clustering and dimensionality reduction.

Clustering algorithms:

• Partition-based: K-Means, MiniBatchKMeans
• Density-based: DBSCAN, HDBSCAN, OPTICS
• Hierarchical: AgglomerativeClustering
• Probabilistic: Gaussian Mixture Models
• Others: MeanShift, SpectralClustering, BIRCH

Dimensionality reduction:

• Linear: PCA, TruncatedSVD, NMF
• Manifold learning: t-SNE, UMAP, Isomap, LLE
• Feature extraction: FastICA, LatentDirichletAllocation

When to use:

• Customer segmentation, anomaly detection, data visualization
• Reducing feature dimensions, exploratory data analysis
• Topic modeling, image compression

See: references/unsupervised_learning.md for detailed documentation.

3. Model Evaluation and Selection

Tools for robust model evaluation, cross-validation, and hyperparameter tuning.

Cross-validation strategies:

• KFold, StratifiedKFold (classification)
• TimeSeriesSplit (temporal data)
• GroupKFold (grouped samples)

Hyperparameter tuning:

• GridSearchCV (exhaustive search)
• RandomizedSearchCV (random sampling)
• HalvingGridSearchCV (successive halving)

Metrics:

• Classification: accuracy, precision, recall, F1-score, ROC AUC, confusion matrix
• Regression: MSE, RMSE, MAE, R², MAPE
• Clustering: silhouette score, Calinski-Harabasz, Davies-Bouldin

When to use:

• Comparing model performance objectively
• Finding optimal hyperparameters
• Preventing overfitting through cross-validation
• Understanding model behavior with learning curves

See: references/model_evaluation.md for comprehensive metrics and tuning strategies.

4. Data Preprocessing

Transform raw data into formats suitable for machine learning.

Scaling and normalization:

• StandardScaler (zero mean, unit variance)
• MinMaxScaler (bounded range)
• RobustScaler (robust to outliers)
• Normalizer (sample-wise normalization)

Encoding categorical variables:

• OneHotEncoder (nominal categories)
• OrdinalEncoder (ordered categories)
• LabelEncoder (target encoding)

Handling missing values:

• SimpleImputer (mean, median, most frequent)
• KNNImputer (k-nearest neighbors)
• IterativeImputer (multivariate imputation)

Feature engineering:

• PolynomialFeatures (interaction terms)
• KBinsDiscretizer (binning)
• Feature selection (RFE, SelectKBest, SelectFromModel)

When to use:

• Before training any algorithm that requires scaled features (SVM, KNN, Neural Networks)
• Converting categorical variables to numeric format
• Handling missing data systematically
• Creating non-linear features for linear models

See: references/preprocessing.md for detailed preprocessing techniques.

5. Pipelines and Composition

Build reproducible, production-ready ML workflows.

Key components:

• Pipeline: Chain transformers and estimators sequentially
• ColumnTransformer: Apply different preprocessing to different columns
• FeatureUnion: Combine multiple transformers in parallel
• TransformedTargetRegressor: Transform target variable

Benefits:

• Prevents data leakage in cross-validation
• Simplifies code and improves maintainability
• Enables joint hyperparameter tuning
• Ensures consistency between training and prediction

When to use:

• Always use Pipelines for production workflows
• When mixing numerical and categorical features (use ColumnTransformer)
• When performing cross-validation with preprocessing steps
• When hyperparameter tuning includes preprocessing parameters

See: references/pipelines_and_composition.md for comprehensive pipeline patterns.

Example Scripts

Classification Pipeline

Run a complete classification workflow with preprocessing, model comparison, hyperparameter tuning, and evaluation:

python scripts/classification_pipeline.py

This script demonstrates:

• Handling mixed data types (numeric and categorical)
• Model comparison using cross-validation
• Hyperparameter tuning with GridSearchCV
• Comprehensive evaluation with multiple metrics
• Feature importance analysis

Clustering Analysis

Perform clustering analysis with algorithm comparison and visualization:

python scripts/clustering_analysis.py

This script demonstrates:

• Finding optimal number of clusters (elbow method, silhouette analysis)
• Comparing multiple clustering algorithms (K-Means, DBSCAN, Agglomerative, Gaussian Mixture)
• Evaluating clustering quality without ground truth
• Visualizing results with PCA projection

Reference Documentation

This skill includes comprehensive reference files for deep dives into specific topics:

Quick Reference

File: references/quick_reference.md

• Common import patterns and installation instructions
• Quick workflow templates for common tasks
• Algorithm selection cheat sheets
• Common patterns and gotchas
• Performance optimization tips

Supervised Learning

File: references/supervised_learning.md

• Linear models (regression and classification)
• Support Vector Machines
• Decision Trees and ensemble methods
• K-Nearest Neighbors, Naive Bayes, Neural Networks
• Algorithm selection guide

Unsupervised Learning

File: references/unsupervised_learning.md

• All clustering algorithms with parameters and use cases
• Dimensionality reduction techniques
• Outlier and novelty detection
• Gaussian Mixture Models
• Method selection guide

Model Evaluation

File: references/model_evaluation.md

• Cross-validation strategies
• Hyperparameter tuning methods
• Classification, regression, and clustering metrics
• Learning and validation curves
• Best practices for model selection

Preprocessing

File: references/preprocessing.md

• Feature scaling and normalization
• Encoding categorical variables
• Missing value imputation
• Feature engineering techniques
• Custom transformers

Pipelines and Composition

File: references/pipelines_and_composition.md

• Pipeline construction and usage
• ColumnTransformer for mixed data types
• FeatureUnion for parallel transformations
• Complete end-to-end examples
• Best practices

Common Workflows

Building a Classification Model

1. Load and explore data

   import pandas as pd
   df = pd.read_csv('data.csv')
   X = df.drop('target', axis=1)
   y = df['target']
   

2. Split data with stratification

   from sklearn.model_selection import train_test_split
   X_train, X_test, y_train, y_test = train_test_split(
       X, y, test_size=0.2, stratify=y, random_state=42
   )
   

3. Create preprocessing pipeline

   from sklearn.pipeline import Pipeline
   from sklearn.preprocessing import StandardScaler
   from sklearn.compose import ColumnTransformer
   
   # Handle numeric and categorical features separately
   preprocessor = ColumnTransformer([
       ('num', StandardScaler(), numeric_features),
       ('cat', OneHotEncoder(), categorical_features)
   ])
   

4. Build complete pipeline

   model = Pipeline([
       ('preprocessor', preprocessor),
       ('classifier', RandomForestClassifier(random_state=42))
   ])
   

5. Tune hyperparameters

   from sklearn.model_selection import GridSearchCV
   
   param_grid = {
       'classifier__n_estimators': [100, 200],
       'classifier__max_depth': [10, 20, None]
   }
   
   grid_search = GridSearchCV(model, param_grid, cv=5)
   grid_search.fit(X_train, y_train)
   

6. Evaluate on test set

   from sklearn.metrics import classification_report
   
   best_model = grid_search.best_estimator_
   y_pred = best_model.predict(X_test)
   print(classification_report(y_test, y_pred))
   

Performing Clustering Analysis

1. Preprocess data

   from sklearn.preprocessing import StandardScaler
   
   scaler = StandardScaler()
   X_scaled = scaler.fit_transform(X)
   

2. Find optimal number of clusters

   from sklearn.cluster import KMeans
   from sklearn.metrics import silhouette_score
   
   scores = []
   for k in range(2, 11):
       kmeans = KMeans(n_clusters=k, random_state=42)
       labels = kmeans.fit_predict(X_scaled)
       scores.append(silhouette_score(X_scaled, labels))
   
   optimal_k = range(2, 11)[np.argmax(scores)]
   

3. Apply clustering

   model = KMeans(n_clusters=optimal_k, random_state=42)
   labels = model.fit_predict(X_scaled)
   

4. Visualize with dimensionality reduction

   from sklearn.decomposition import PCA
   
   pca = PCA(n_components=2)
   X_2d = pca.fit_transform(X_scaled)
   
   plt.scatter(X_2d[:, 0], X_2d[:, 1], c=labels, cmap='viridis')
   

Best Practices

Always Use Pipelines

Pipelines prevent data leakage and ensure consistency:

# Good: Preprocessing in pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', LogisticRegression())
])

# Bad: Preprocessing outside (can leak information)
X_scaled = StandardScaler().fit_transform(X)

Fit on Training Data Only

Never fit on test data:

# Good
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)  # Only transform

# Bad
scaler = StandardScaler()
X_all_scaled = scaler.fit_transform(np.vstack([X_train, X_test]))

Use Stratified Splitting for Classification

Preserve class distribution:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

Set Random State for Reproducibility

model = RandomForestClassifier(n_estimators=100, random_state=42)

Choose Appropriate Metrics

• Balanced data: Accuracy, F1-score
• Imbalanced data: Precision, Recall, ROC AUC, Balanced Accuracy
• Cost-sensitive: Define custom scorer

Scale Features When Required

Algorithms requiring feature scaling:

• SVM, KNN, Neural Networks
• PCA, Linear/Logistic Regression with regularization
• K-Means clustering

Algorithms not requiring scaling:

• Tree-based models (Decision Trees, Random Forest, Gradient Boosting)
• Naive Bayes

Troubleshooting Common Issues

ConvergenceWarning

Issue: Model didn't converge Solution: Increase max_iter or scale features

model = LogisticRegression(max_iter=1000)

Poor Performance on Test Set

Issue: Overfitting Solution: Use regularization, cross-validation, or simpler model

# Add regularization
model = Ridge(alpha=1.0)

# Use cross-validation
scores = cross_val_score(model, X, y, cv=5)

Memory Error with Large Datasets

Solution: Use algorithms designed for large data

# Use SGD for large datasets
from sklearn.linear_model import SGDClassifier
model = SGDClassifier()

# Or MiniBatchKMeans for clustering
from sklearn.cluster import MiniBatchKMeans
model = MiniBatchKMeans(n_clusters=8, batch_size=100)

Additional Resources

• Official Documentation: https://scikit-learn.org/stable/
• User Guide: https://scikit-learn.org/stable/user_guide.html
• API Reference: https://scikit-learn.org/stable/api/index.html
• Examples Gallery: https://scikit-learn.org/stable/auto_examples/index.html

Seaborn Statistical Visualization

Overview

Seaborn is a Python visualization library for creating publication-quality statistical graphics. Use this skill for dataset-oriented plotting, multivariate analysis, automatic statistical estimation, and complex multi-panel figures with minimal code.

Design Philosophy

Seaborn follows these core principles:

1. Dataset-oriented: Work directly with DataFrames and named variables rather than abstract coordinates
2. Semantic mapping: Automatically translate data values into visual properties (colors, sizes, styles)
3. Statistical awareness: Built-in aggregation, error estimation, and confidence intervals
4. Aesthetic defaults: Publication-ready themes and color palettes out of the box
5. Matplotlib integration: Full compatibility with matplotlib customization when needed

Quick Start

import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

# Load example dataset
df = sns.load_dataset('tips')

# Create a simple visualization
sns.scatterplot(data=df, x='total_bill', y='tip', hue='day')
plt.show()

Core Plotting Interfaces

Function Interface (Traditional)

The function interface provides specialized plotting functions organized by visualization type. Each category has axes-level functions (plot to single axes) and figure-level functions (manage entire figure with faceting).

When to use:

• Quick exploratory analysis
• Single-purpose visualizations
• When you need a specific plot type

Objects Interface (Modern)

The seaborn.objects interface provides a declarative, composable API similar to ggplot2. Build visualizations by chaining methods to specify data mappings, marks, transformations, and scales.

When to use:

• Complex layered visualizations
• When you need fine-grained control over transformations
• Building custom plot types
• Programmatic plot generation

from seaborn import objects as so

# Declarative syntax
(
    so.Plot(data=df, x='total_bill', y='tip')
    .add(so.Dot(), color='day')
    .add(so.Line(), so.PolyFit())
)

Plotting Functions by Category

Relational Plots (Relationships Between Variables)

Use for: Exploring how two or more variables relate to each other

• scatterplot() - Display individual observations as points
• lineplot() - Show trends and changes (automatically aggregates and computes CI)
• relplot() - Figure-level interface with automatic faceting

Key parameters:

• x, y - Primary variables
• hue - Color encoding for additional categorical/continuous variable
• size - Point/line size encoding
• style - Marker/line style encoding
• col, row - Facet into multiple subplots (figure-level only)

# Scatter with multiple semantic mappings
sns.scatterplot(data=df, x='total_bill', y='tip',
                hue='time', size='size', style='sex')

# Line plot with confidence intervals
sns.lineplot(data=timeseries, x='date', y='value', hue='category')

# Faceted relational plot
sns.relplot(data=df, x='total_bill', y='tip',
            col='time', row='sex', hue='smoker', kind='scatter')

Distribution Plots (Single and Bivariate Distributions)

Use for: Understanding data spread, shape, and probability density

• histplot() - Bar-based frequency distributions with flexible binning
• kdeplot() - Smooth density estimates using Gaussian kernels
• ecdfplot() - Empirical cumulative distribution (no parameters to tune)
• rugplot() - Individual observation tick marks
• displot() - Figure-level interface for univariate and bivariate distributions
• jointplot() - Bivariate plot with marginal distributions
• pairplot() - Matrix of pairwise relationships across dataset

Key parameters:

• x, y - Variables (y optional for univariate)
• hue - Separate distributions by category
• stat - Normalization: "count", "frequency", "probability", "density"
• bins / binwidth - Histogram binning control
• bw_adjust - KDE bandwidth multiplier (higher = smoother)
• fill - Fill area under curve
• multiple - How to handle hue: "layer", "stack", "dodge", "fill"

# Histogram with density normalization
sns.histplot(data=df, x='total_bill', hue='time',
             stat='density', multiple='stack')

# Bivariate KDE with contours
sns.kdeplot(data=df, x='total_bill', y='tip',
            fill=True, levels=5, thresh=0.1)

# Joint plot with marginals
sns.jointplot(data=df, x='total_bill', y='tip',
              kind='scatter', hue='time')

# Pairwise relationships
sns.pairplot(data=df, hue='species', corner=True)

Categorical Plots (Comparisons Across Categories)

Use for: Comparing distributions or statistics across discrete categories

Categorical scatterplots:

• stripplot() - Points with jitter to show all observations
• swarmplot() - Non-overlapping points (beeswarm algorithm)

Distribution comparisons:

• boxplot() - Quartiles and outliers
• violinplot() - KDE + quartile information
• boxenplot() - Enhanced boxplot for larger datasets

Statistical estimates:

• barplot() - Mean/aggregate with confidence intervals
• pointplot() - Point estimates with connecting lines
• countplot() - Count of observations per category

Figure-level:

• catplot() - Faceted categorical plots (set kind parameter)

Key parameters:

• x, y - Variables (one typically categorical)
• hue - Additional categorical grouping
• order, hue_order - Control category ordering
• dodge - Separate hue levels side-by-side
• orient - "v" (vertical) or "h" (horizontal)
• kind - Plot type for catplot: "strip", "swarm", "box", "violin", "bar", "point"

# Swarm plot showing all points
sns.swarmplot(data=df, x='day', y='total_bill', hue='sex')

# Violin plot with split for comparison
sns.violinplot(data=df, x='day', y='total_bill',
               hue='sex', split=True)

# Bar plot with error bars
sns.barplot(data=df, x='day', y='total_bill',
            hue='sex', estimator='mean', errorbar='ci')

# Faceted categorical plot
sns.catplot(data=df, x='day', y='total_bill',
            col='time', kind='box')

Regression Plots (Linear Relationships)

Use for: Visualizing linear regressions and residuals

• regplot() - Axes-level regression plot with scatter + fit line
• lmplot() - Figure-level with faceting support
• residplot() - Residual plot for assessing model fit

Key parameters:

• x, y - Variables to regress
• order - Polynomial regression order
• logistic - Fit logistic regression
• robust - Use robust regression (less sensitive to outliers)
• ci - Confidence interval width (default 95)
• scatter_kws, line_kws - Customize scatter and line properties

# Simple linear regression
sns.regplot(data=df, x='total_bill', y='tip')

# Polynomial regression with faceting
sns.lmplot(data=df, x='total_bill', y='tip',
           col='time', order=2, ci=95)

# Check residuals
sns.residplot(data=df, x='total_bill', y='tip')

Matrix Plots (Rectangular Data)

Use for: Visualizing matrices, correlations, and grid-structured data

• heatmap() - Color-encoded matrix with annotations
• clustermap() - Hierarchically-clustered heatmap

Key parameters:

• data - 2D rectangular dataset (DataFrame or array)
• annot - Display values in cells
• fmt - Format string for annotations (e.g., ".2f")
• cmap - Colormap name
• center - Value at colormap center (for diverging colormaps)
• vmin, vmax - Color scale limits
• square - Force square cells
• linewidths - Gap between cells

# Correlation heatmap
corr = df.corr()
sns.heatmap(corr, annot=True, fmt='.2f',
            cmap='coolwarm', center=0, square=True)

# Clustered heatmap
sns.clustermap(data, cmap='viridis',
               standard_scale=1, figsize=(10, 10))

Multi-Plot Grids

Seaborn provides grid objects for creating complex multi-panel figures:

FacetGrid

Create subplots based on categorical variables. Most useful when called through figure-level functions (relplot, displot, catplot), but can be used directly for custom plots.

g = sns.FacetGrid(df, col='time', row='sex', hue='smoker')
g.map(sns.scatterplot, 'total_bill', 'tip')
g.add_legend()

PairGrid

Show pairwise relationships between all variables in a dataset.

g = sns.PairGrid(df, hue='species')
g.map_upper(sns.scatterplot)
g.map_lower(sns.kdeplot)
g.map_diag(sns.histplot)
g.add_legend()

JointGrid

Combine bivariate plot with marginal distributions.

g = sns.JointGrid(data=df, x='total_bill', y='tip')
g.plot_joint(sns.scatterplot)
g.plot_marginals(sns.histplot)

Figure-Level vs Axes-Level Functions

Understanding this distinction is crucial for effective seaborn usage:

Axes-Level Functions

• Plot to a single matplotlib Axes object
• Integrate easily into complex matplotlib figures
• Accept ax= parameter for precise placement
• Return Axes object
• Examples: scatterplot, histplot, boxplot, regplot, heatmap

When to use:

• Building custom multi-plot layouts
• Combining different plot types
• Need matplotlib-level control
• Integrating with existing matplotlib code

fig, axes = plt.subplots(2, 2, figsize=(10, 10))
sns.scatterplot(data=df, x='x', y='y', ax=axes[0, 0])
sns.histplot(data=df, x='x', ax=axes[0, 1])
sns.boxplot(data=df, x='cat', y='y', ax=axes[1, 0])
sns.kdeplot(data=df, x='x', y='y', ax=axes[1, 1])

Figure-Level Functions

• Manage entire figure including all subplots
• Built-in faceting via col and row parameters
• Return FacetGrid, JointGrid, or PairGrid objects
• Use height and aspect for sizing (per subplot)
• Cannot be placed in existing figure
• Examples: relplot, displot, catplot, lmplot, jointplot, pairplot

When to use:

• Faceted visualizations (small multiples)
• Quick exploratory analysis
• Consistent multi-panel layouts
• Don't need to combine with other plot types

# Automatic faceting
sns.relplot(data=df, x='x', y='y', col='category', row='group',
            hue='type', height=3, aspect=1.2)

Data Structure Requirements

Long-Form Data (Preferred)

Each variable is a column, each observation is a row. This "tidy" format provides maximum flexibility:

# Long-form structure
   subject  condition  measurement
0        1    control         10.5
1        1  treatment         12.3
2        2    control          9.8
3        2  treatment         13.1

Advantages:

• Works with all seaborn functions
• Easy to remap variables to visual properties
• Supports arbitrary complexity
• Natural for DataFrame operations

Wide-Form Data

Variables are spread across columns. Useful for simple rectangular data:

# Wide-form structure
   control  treatment
0     10.5       12.3
1      9.8       13.1

Use cases:

• Simple time series
• Correlation matrices
• Heatmaps
• Quick plots of array data

Converting wide to long:

df_long = df.melt(var_name='condition', value_name='measurement')

Color Palettes

Seaborn provides carefully designed color palettes for different data types:

Qualitative Palettes (Categorical Data)

Distinguish categories through hue variation:

• "deep" - Default, vivid colors
• "muted" - Softer, less saturated
• "pastel" - Light, desaturated
• "bright" - Highly saturated
• "dark" - Dark values
• "colorblind" - Safe for color vision deficiency

sns.set_palette("colorblind")
sns.color_palette("Set2")

Sequential Palettes (Ordered Data)

Show progression from low to high values:

• "rocket", "mako" - Wide luminance range (good for heatmaps)
• "flare", "crest" - Restricted luminance (good for points/lines)
• "viridis", "magma", "plasma" - Matplotlib perceptually uniform

sns.heatmap(data, cmap='rocket')
sns.kdeplot(data=df, x='x', y='y', cmap='mako', fill=True)

Diverging Palettes (Centered Data)

Emphasize deviations from a midpoint:

• "vlag" - Blue to red
• "icefire" - Blue to orange
• "coolwarm" - Cool to warm
• "Spectral" - Rainbow diverging

sns.heatmap(correlation_matrix, cmap='vlag', center=0)

Custom Palettes

# Create custom palette
custom = sns.color_palette("husl", 8)

# Light to dark gradient
palette = sns.light_palette("seagreen", as_cmap=True)

# Diverging palette from hues
palette = sns.diverging_palette(250, 10, as_cmap=True)

Theming and Aesthetics

Set Theme

set_theme() controls overall appearance:

# Set complete theme
sns.set_theme(style='whitegrid', palette='pastel', font='sans-serif')

# Reset to defaults
sns.set_theme()

Styles

Control background and grid appearance:

• "darkgrid" - Gray background with white grid (default)
• "whitegrid" - White background with gray grid
• "dark" - Gray background, no grid
• "white" - White background, no grid
• "ticks" - White background with axis ticks

sns.set_style("whitegrid")

# Remove spines
sns.despine(left=False, bottom=False, offset=10, trim=True)

# Temporary style
with sns.axes_style("white"):
    sns.scatterplot(data=df, x='x', y='y')

Contexts

Scale elements for different use cases:

• "paper" - Smallest (default)
• "notebook" - Slightly larger
• "talk" - Presentation slides
• "poster" - Large format

sns.set_context("talk", font_scale=1.2)

# Temporary context
with sns.plotting_context("poster"):
    sns.barplot(data=df, x='category', y='value')

Best Practices

1. Data Preparation

Always use well-structured DataFrames with meaningful column names:

# Good: Named columns in DataFrame
df = pd.DataFrame({'bill': bills, 'tip': tips, 'day': days})
sns.scatterplot(data=df, x='bill', y='tip', hue='day')

# Avoid: Unnamed arrays
sns.scatterplot(x=x_array, y=y_array)  # Loses axis labels

2. Choose the Right Plot Type

Continuous x, continuous y: scatterplot, lineplot, kdeplot, regplot Continuous x, categorical y: violinplot, boxplot, stripplot, swarmplot One continuous variable: histplot, kdeplot, ecdfplot Correlations/matrices: heatmap, clustermap Pairwise relationships: pairplot, jointplot

3. Use Figure-Level Functions for Faceting

# Instead of manual subplot creation
sns.relplot(data=df, x='x', y='y', col='category', col_wrap=3)

# Not: Creating subplots manually for simple faceting

4. Leverage Semantic Mappings

Use hue, size, and style to encode additional dimensions:

sns.scatterplot(data=df, x='x', y='y',
                hue='category',      # Color by category
                size='importance',    # Size by continuous variable
                style='type')         # Marker style by type

5. Control Statistical Estimation

Many functions compute statistics automatically. Understand and customize:

# Lineplot computes mean and 95% CI by default
sns.lineplot(data=df, x='time', y='value',
             errorbar='sd')  # Use standard deviation instead

# Barplot computes mean by default
sns.barplot(data=df, x='category', y='value',
            estimator='median',  # Use median instead
            errorbar=('ci', 95))  # Bootstrapped CI

6. Combine with Matplotlib

Seaborn integrates seamlessly with matplotlib for fine-tuning:

ax = sns.scatterplot(data=df, x='x', y='y')
ax.set(xlabel='Custom X Label', ylabel='Custom Y Label',
       title='Custom Title')
ax.axhline(y=0, color='r', linestyle='--')
plt.tight_layout()

7. Save High-Quality Figures

fig = sns.relplot(data=df, x='x', y='y', col='group')
fig.savefig('figure.png', dpi=300, bbox_inches='tight')
fig.savefig('figure.pdf')  # Vector format for publications

Common Patterns

Exploratory Data Analysis

# Quick overview of all relationships
sns.pairplot(data=df, hue='target', corner=True)

# Distribution exploration
sns.displot(data=df, x='variable', hue='group',
            kind='kde', fill=True, col='category')

# Correlation analysis
corr = df.corr()
sns.heatmap(corr, annot=True, cmap='coolwarm', center=0)

Publication-Quality Figures

sns.set_theme(style='ticks', context='paper', font_scale=1.1)

g = sns.catplot(data=df, x='treatment', y='response',
                col='cell_line', kind='box', height=3, aspect=1.2)
g.set_axis_labels('Treatment Condition', 'Response (μM)')
g.set_titles('{col_name}')
sns.despine(trim=True)

g.savefig('figure.pdf', dpi=300, bbox_inches='tight')

Complex Multi-Panel Figures

# Using matplotlib subplots with seaborn
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

sns.scatterplot(data=df, x='x1', y='y', hue='group', ax=axes[0, 0])
sns.histplot(data=df, x='x1', hue='group', ax=axes[0, 1])
sns.violinplot(data=df, x='group', y='y', ax=axes[1, 0])
sns.heatmap(df.pivot_table(values='y', index='x1', columns='x2'),
            ax=axes[1, 1], cmap='viridis')

plt.tight_layout()

Time Series with Confidence Bands

# Lineplot automatically aggregates and shows CI
sns.lineplot(data=timeseries, x='date', y='measurement',
             hue='sensor', style='location', errorbar='sd')

# For more control
g = sns.relplot(data=timeseries, x='date', y='measurement',
                col='location', hue='sensor', kind='line',
                height=4, aspect=1.5, errorbar=('ci', 95))
g.set_axis_labels('Date', 'Measurement (units)')

Troubleshooting

Issue: Legend Outside Plot Area

Figure-level functions place legends outside by default. To move inside:

g = sns.relplot(data=df, x='x', y='y', hue='category')
g._legend.set_bbox_to_anchor((0.9, 0.5))  # Adjust position

Issue: Overlapping Labels

plt.xticks(rotation=45, ha='right')
plt.tight_layout()

Issue: Figure Too Small

For figure-level functions:

sns.relplot(data=df, x='x', y='y', height=6, aspect=1.5)

For axes-level functions:

fig, ax = plt.subplots(figsize=(10, 6))
sns.scatterplot(data=df, x='x', y='y', ax=ax)

Issue: Colors Not Distinct Enough

# Use a different palette
sns.set_palette("bright")

# Or specify number of colors
palette = sns.color_palette("husl", n_colors=len(df['category'].unique()))
sns.scatterplot(data=df, x='x', y='y', hue='category', palette=palette)

Issue: KDE Too Smooth or Jagged

# Adjust bandwidth
sns.kdeplot(data=df, x='x', bw_adjust=0.5)  # Less smooth
sns.kdeplot(data=df, x='x', bw_adjust=2)    # More smooth

Resources

This skill includes reference materials for deeper exploration:

references/

• function_reference.md - Comprehensive listing of all seaborn functions with parameters and examples
• objects_interface.md - Detailed guide to the modern seaborn.objects API
• examples.md - Common use cases and code patterns for different analysis scenarios

Load reference files as needed for detailed function signatures, advanced parameters, or specific examples.

SimPy - Discrete-Event Simulation

Overview

SimPy is a process-based discrete-event simulation framework based on standard Python. Use SimPy to model systems where entities (customers, vehicles, packets, etc.) interact with each other and compete for shared resources (servers, machines, bandwidth, etc.) over time.

Core capabilities:

• Process modeling using Python generator functions
• Shared resource management (servers, containers, stores)
• Event-driven scheduling and synchronization
• Real-time simulations synchronized with wall-clock time
• Comprehensive monitoring and data collection

When to Use This Skill

Use the SimPy skill when:

1. Modeling discrete-event systems - Systems where events occur at irregular intervals
2. Resource contention - Entities compete for limited resources (servers, machines, staff)
3. Queue analysis - Studying waiting lines, service times, and throughput
4. Process optimization - Analyzing manufacturing, logistics, or service processes
5. Network simulation - Packet routing, bandwidth allocation, latency analysis
6. Capacity planning - Determining optimal resource levels for desired performance
7. System validation - Testing system behavior before implementation

Not suitable for:

• Continuous simulations with fixed time steps (consider SciPy ODE solvers)
• Independent processes without resource sharing
• Pure mathematical optimization (consider SciPy optimize)

Quick Start

Basic Simulation Structure

import simpy

def process(env, name):
    """A simple process that waits and prints."""
    print(f'{name} starting at {env.now}')
    yield env.timeout(5)
    print(f'{name} finishing at {env.now}')

# Create environment
env = simpy.Environment()

# Start processes
env.process(process(env, 'Process 1'))
env.process(process(env, 'Process 2'))

# Run simulation
env.run(until=10)

Resource Usage Pattern

import simpy

def customer(env, name, resource):
    """Customer requests resource, uses it, then releases."""
    with resource.request() as req:
        yield req  # Wait for resource
        print(f'{name} got resource at {env.now}')
        yield env.timeout(3)  # Use resource
        print(f'{name} released resource at {env.now}')

env = simpy.Environment()
server = simpy.Resource(env, capacity=1)

env.process(customer(env, 'Customer 1', server))
env.process(customer(env, 'Customer 2', server))
env.run()

Core Concepts

1. Environment

The simulation environment manages time and schedules events.

import simpy

# Standard environment (runs as fast as possible)
env = simpy.Environment(initial_time=0)

# Real-time environment (synchronized with wall-clock)
import simpy.rt
env_rt = simpy.rt.RealtimeEnvironment(factor=1.0)

# Run simulation
env.run(until=100)  # Run until time 100
env.run()  # Run until no events remain

2. Processes

Processes are defined using Python generator functions (functions with yield statements).

def my_process(env, param1, param2):
    """Process that yields events to pause execution."""
    print(f'Starting at {env.now}')

    # Wait for time to pass
    yield env.timeout(5)

    print(f'Resumed at {env.now}')

    # Wait for another event
    yield env.timeout(3)

    print(f'Done at {env.now}')
    return 'result'

# Start the process
env.process(my_process(env, 'value1', 'value2'))

3. Events

Events are the fundamental mechanism for process synchronization. Processes yield events and resume when those events are triggered.

Common event types:

• env.timeout(delay) - Wait for time to pass
• resource.request() - Request a resource
• env.event() - Create a custom event
• env.process(func()) - Process as an event
• event1 & event2 - Wait for all events (AllOf)
• event1 | event2 - Wait for any event (AnyOf)

Resources

SimPy provides several resource types for different scenarios. For comprehensive details, see references/resources.md.

Resource Types Summary

Quick Reference

import simpy

env = simpy.Environment()

# Basic resource (e.g., servers)
resource = simpy.Resource(env, capacity=2)

# Priority resource
priority_resource = simpy.PriorityResource(env, capacity=1)

# Container (e.g., fuel tank)
fuel_tank = simpy.Container(env, capacity=100, init=50)

# Store (e.g., warehouse)
warehouse = simpy.Store(env, capacity=10)

Common Simulation Patterns

Pattern 1: Customer-Server Queue

import simpy
import random

def customer(env, name, server):
    arrival = env.now
    with server.request() as req:
        yield req
        wait = env.now - arrival
        print(f'{name} waited {wait:.2f}, served at {env.now}')
        yield env.timeout(random.uniform(2, 4))

def customer_generator(env, server):
    i = 0
    while True:
        yield env.timeout(random.uniform(1, 3))
        i += 1
        env.process(customer(env, f'Customer {i}', server))

env = simpy.Environment()
server = simpy.Resource(env, capacity=2)
env.process(customer_generator(env, server))
env.run(until=20)

Pattern 2: Producer-Consumer

import simpy

def producer(env, store):
    item_id = 0
    while True:
        yield env.timeout(2)
        item = f'Item {item_id}'
        yield store.put(item)
        print(f'Produced {item} at {env.now}')
        item_id += 1

def consumer(env, store):
    while True:
        item = yield store.get()
        print(f'Consumed {item} at {env.now}')
        yield env.timeout(3)

env = simpy.Environment()
store = simpy.Store(env, capacity=10)
env.process(producer(env, store))
env.process(consumer(env, store))
env.run(until=20)

Pattern 3: Parallel Task Execution

import simpy

def task(env, name, duration):
    print(f'{name} starting at {env.now}')
    yield env.timeout(duration)
    print(f'{name} done at {env.now}')
    return f'{name} result'

def coordinator(env):
    # Start tasks in parallel
    task1 = env.process(task(env, 'Task 1', 5))
    task2 = env.process(task(env, 'Task 2', 3))
    task3 = env.process(task(env, 'Task 3', 4))

    # Wait for all to complete
    results = yield task1 & task2 & task3
    print(f'All done at {env.now}')

env = simpy.Environment()
env.process(coordinator(env))
env.run()

Workflow Guide

Step 1: Define the System

Identify:

• Entities: What moves through the system? (customers, parts, packets)
• Resources: What are the constraints? (servers, machines, bandwidth)
• Processes: What are the activities? (arrival, service, departure)
• Metrics: What to measure? (wait times, utilization, throughput)

Step 2: Implement Process Functions

Create generator functions for each process type:

def entity_process(env, name, resources, parameters):
    # Arrival logic
    arrival_time = env.now

    # Request resources
    with resource.request() as req:
        yield req

        # Service logic
        service_time = calculate_service_time(parameters)
        yield env.timeout(service_time)

    # Departure logic
    collect_statistics(env.now - arrival_time)

Step 3: Set Up Monitoring

Use monitoring utilities to collect data. See references/monitoring.md for comprehensive techniques.

from scripts.resource_monitor import ResourceMonitor

# Create and monitor resource
resource = simpy.Resource(env, capacity=2)
monitor = ResourceMonitor(env, resource, "Server")

# After simulation
monitor.report()

Step 4: Run and Analyze

# Run simulation
env.run(until=simulation_time)

# Generate reports
monitor.report()
stats.report()

# Export data for further analysis
monitor.export_csv('results.csv')

Advanced Features

Process Interaction

Processes can interact through events, process yields, and interrupts. See references/process-interaction.md for detailed patterns.

Key mechanisms:

• Event signaling: Shared events for coordination
• Process yields: Wait for other processes to complete
• Interrupts: Forcefully resume processes for preemption

Real-Time Simulations

Synchronize simulation with wall-clock time for hardware-in-the-loop or interactive applications. See references/real-time.md.

import simpy.rt

env = simpy.rt.RealtimeEnvironment(factor=1.0)  # 1:1 time mapping
# factor=0.5 means 1 sim unit = 0.5 seconds (2x faster)

Comprehensive Monitoring

Monitor processes, resources, and events. See references/monitoring.md for techniques including:

• State variable tracking
• Resource monkey-patching
• Event tracing
• Statistical collection

Scripts and Templates

basic_simulation_template.py

Complete template for building queue simulations with:

• Configurable parameters
• Statistics collection
• Customer generation
• Resource usage
• Report generation

Usage:

from scripts.basic_simulation_template import SimulationConfig, run_simulation

config = SimulationConfig()
config.num_resources = 2
config.sim_time = 100
stats = run_simulation(config)
stats.report()

resource_monitor.py

Reusable monitoring utilities:

• ResourceMonitor - Track single resource
• MultiResourceMonitor - Monitor multiple resources
• ContainerMonitor - Track container levels
• Automatic statistics calculation
• CSV export functionality

Usage:

from scripts.resource_monitor import ResourceMonitor

monitor = ResourceMonitor(env, resource, "My Resource")
# ... run simulation ...
monitor.report()
monitor.export_csv('data.csv')

Reference Documentation

Detailed guides for specific topics:

• references/resources.md - All resource types with examples
• references/events.md - Event system and patterns
• references/process-interaction.md - Process synchronization
• references/monitoring.md - Data collection techniques
• references/real-time.md - Real-time simulation setup

Best Practices

1. Generator functions: Always use yield in process functions
2. Resource context managers: Use with resource.request() as req: for automatic cleanup
3. Reproducibility: Set random.seed() for consistent results
4. Monitoring: Collect data throughout simulation, not just at the end
5. Validation: Compare simple cases with analytical solutions
6. Documentation: Comment process logic and parameter choices
7. Modular design: Separate process logic, statistics, and configuration

Common Pitfalls

1. Forgetting yield: Processes must yield events to pause
2. Event reuse: Events can only be triggered once
3. Resource leaks: Use context managers or ensure release
4. Blocking operations: Avoid Python blocking calls in processes
5. Time units: Stay consistent with time unit interpretation
6. Deadlocks: Ensure at least one process can make progress

Example Use Cases

• Manufacturing: Machine scheduling, production lines, inventory management
• Healthcare: Emergency room simulation, patient flow, staff allocation
• Telecommunications: Network traffic, packet routing, bandwidth allocation
• Transportation: Traffic flow, logistics, vehicle routing
• Service operations: Call centers, retail checkout, appointment scheduling
• Computer systems: CPU scheduling, memory management, I/O operations

Statistical Analysis

Overview

Statistical analysis is a systematic process for testing hypotheses and quantifying relationships. Conduct hypothesis tests (t-test, ANOVA, chi-square), regression, correlation, and Bayesian analyses with assumption checks and APA reporting. Apply this skill for academic research.

When to Use This Skill

This skill should be used when:

• Conducting statistical hypothesis tests (t-tests, ANOVA, chi-square)
• Performing regression or correlation analyses
• Running Bayesian statistical analyses
• Checking statistical assumptions and diagnostics
• Calculating effect sizes and conducting power analyses
• Reporting statistical results in APA format
• Analyzing experimental or observational data for research

Core Capabilities

1. Test Selection and Planning

• Choose appropriate statistical tests based on research questions and data characteristics
• Conduct a priori power analyses to determine required sample sizes
• Plan analysis strategies including multiple comparison corrections

2. Assumption Checking

• Automatically verify all relevant assumptions before running tests
• Provide diagnostic visualizations (Q-Q plots, residual plots, box plots)
• Recommend remedial actions when assumptions are violated

3. Statistical Testing

• Hypothesis testing: t-tests, ANOVA, chi-square, non-parametric alternatives
• Regression: linear, multiple, logistic, with diagnostics
• Correlations: Pearson, Spearman, with confidence intervals
• Bayesian alternatives: Bayesian t-tests, ANOVA, regression with Bayes Factors

4. Effect Sizes and Interpretation

• Calculate and interpret appropriate effect sizes for all analyses
• Provide confidence intervals for effect estimates
• Distinguish statistical from practical significance

5. Professional Reporting

• Generate APA-style statistical reports
• Create publication-ready figures and tables
• Provide complete interpretation with all required statistics

Workflow Decision Tree

Use this decision tree to determine your analysis path:

START
│
├─ Need to SELECT a statistical test?
│  └─ YES → See "Test Selection Guide"
│  └─ NO → Continue
│
├─ Ready to check ASSUMPTIONS?
│  └─ YES → See "Assumption Checking"
│  └─ NO → Continue
│
├─ Ready to run ANALYSIS?
│  └─ YES → See "Running Statistical Tests"
│  └─ NO → Continue
│
└─ Need to REPORT results?
   └─ YES → See "Reporting Results"

Test Selection Guide

Quick Reference: Choosing the Right Test

Use references/test_selection_guide.md for comprehensive guidance. Quick reference:

Comparing Two Groups:

• Independent, continuous, normal → Independent t-test
• Independent, continuous, non-normal → Mann-Whitney U test
• Paired, continuous, normal → Paired t-test
• Paired, continuous, non-normal → Wilcoxon signed-rank test
• Binary outcome → Chi-square or Fisher's exact test

Comparing 3+ Groups:

• Independent, continuous, normal → One-way ANOVA
• Independent, continuous, non-normal → Kruskal-Wallis test
• Paired, continuous, normal → Repeated measures ANOVA
• Paired, continuous, non-normal → Friedman test

Relationships:

• Two continuous variables → Pearson (normal) or Spearman correlation (non-normal)
• Continuous outcome with predictor(s) → Linear regression
• Binary outcome with predictor(s) → Logistic regression

Bayesian Alternatives: All tests have Bayesian versions that provide:

• Direct probability statements about hypotheses
• Bayes Factors quantifying evidence
• Ability to support null hypothesis
• See references/bayesian_statistics.md

Assumption Checking

Systematic Assumption Verification

ALWAYS check assumptions before interpreting test results.

Use the provided scripts/assumption_checks.py module for automated checking:

from scripts.assumption_checks import comprehensive_assumption_check

# Comprehensive check with visualizations
results = comprehensive_assumption_check(
    data=df,
    value_col='score',
    group_col='group',  # Optional: for group comparisons
    alpha=0.05
)

This performs:

1. Outlier detection (IQR and z-score methods)
2. Normality testing (Shapiro-Wilk test + Q-Q plots)
3. Homogeneity of variance (Levene's test + box plots)
4. Interpretation and recommendations

Individual Assumption Checks

For targeted checks, use individual functions:

from scripts.assumption_checks import (
    check_normality,
    check_normality_per_group,
    check_homogeneity_of_variance,
    check_linearity,
    detect_outliers
)

# Example: Check normality with visualization
result = check_normality(
    data=df['score'],
    name='Test Score',
    alpha=0.05,
    plot=True
)
print(result['interpretation'])
print(result['recommendation'])

What to Do When Assumptions Are Violated

Normality violated:

• Mild violation + n > 30 per group → Proceed with parametric test (robust)
• Moderate violation → Use non-parametric alternative
• Severe violation → Transform data or use non-parametric test

Homogeneity of variance violated:

• For t-test → Use Welch's t-test
• For ANOVA → Use Welch's ANOVA or Brown-Forsythe ANOVA
• For regression → Use robust standard errors or weighted least squares

Linearity violated (regression):

• Add polynomial terms
• Transform variables
• Use non-linear models or GAM

See references/assumptions_and_diagnostics.md for comprehensive guidance.

Running Statistical Tests

Python Libraries

Primary libraries for statistical analysis:

• scipy.stats: Core statistical tests
• statsmodels: Advanced regression and diagnostics
• pingouin: User-friendly statistical testing with effect sizes
• pymc: Bayesian statistical modeling
• arviz: Bayesian visualization and diagnostics

Example Analyses

T-Test with Complete Reporting

import pingouin as pg
import numpy as np

# Run independent t-test
result = pg.ttest(group_a, group_b, correction='auto')

# Extract results
t_stat = result['T'].values[0]
df = result['dof'].values[0]
p_value = result['p-val'].values[0]
cohens_d = result['cohen-d'].values[0]
ci_lower = result['CI95%'].values[0][0]
ci_upper = result['CI95%'].values[0][1]

# Report
print(f"t({df:.0f}) = {t_stat:.2f}, p = {p_value:.3f}")
print(f"Cohen's d = {cohens_d:.2f}, 95% CI [{ci_lower:.2f}, {ci_upper:.2f}]")

ANOVA with Post-Hoc Tests

import pingouin as pg

# One-way ANOVA
aov = pg.anova(dv='score', between='group', data=df, detailed=True)
print(aov)

# If significant, conduct post-hoc tests
if aov['p-unc'].values[0] < 0.05:
    posthoc = pg.pairwise_tukey(dv='score', between='group', data=df)
    print(posthoc)

# Effect size
eta_squared = aov['np2'].values[0]  # Partial eta-squared
print(f"Partial η² = {eta_squared:.3f}")

Linear Regression with Diagnostics

import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Fit model
X = sm.add_constant(X_predictors)  # Add intercept
model = sm.OLS(y, X).fit()

# Summary
print(model.summary())

# Check multicollinearity (VIF)
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print(vif_data)

# Check assumptions
residuals = model.resid
fitted = model.fittedvalues

# Residual plots
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Residuals vs fitted
axes[0, 0].scatter(fitted, residuals, alpha=0.6)
axes[0, 0].axhline(y=0, color='r', linestyle='--')
axes[0, 0].set_xlabel('Fitted values')
axes[0, 0].set_ylabel('Residuals')
axes[0, 0].set_title('Residuals vs Fitted')

# Q-Q plot
from scipy import stats
stats.probplot(residuals, dist="norm", plot=axes[0, 1])
axes[0, 1].set_title('Normal Q-Q')

# Scale-Location
axes[1, 0].scatter(fitted, np.sqrt(np.abs(residuals / residuals.std())), alpha=0.6)
axes[1, 0].set_xlabel('Fitted values')
axes[1, 0].set_ylabel('√|Standardized residuals|')
axes[1, 0].set_title('Scale-Location')

# Residuals histogram
axes[1, 1].hist(residuals, bins=20, edgecolor='black', alpha=0.7)
axes[1, 1].set_xlabel('Residuals')
axes[1, 1].set_ylabel('Frequency')
axes[1, 1].set_title('Histogram of Residuals')

plt.tight_layout()
plt.show()

Bayesian T-Test

import pymc as pm
import arviz as az
import numpy as np

with pm.Model() as model:
    # Priors
    mu1 = pm.Normal('mu_group1', mu=0, sigma=10)
    mu2 = pm.Normal('mu_group2', mu=0, sigma=10)
    sigma = pm.HalfNormal('sigma', sigma=10)

    # Likelihood
    y1 = pm.Normal('y1', mu=mu1, sigma=sigma, observed=group_a)
    y2 = pm.Normal('y2', mu=mu2, sigma=sigma, observed=group_b)

    # Derived quantity
    diff = pm.Deterministic('difference', mu1 - mu2)

    # Sample
    trace = pm.sample(2000, tune=1000, return_inferencedata=True)

# Summarize
print(az.summary(trace, var_names=['difference']))

# Probability that group1 > group2
prob_greater = np.mean(trace.posterior['difference'].values > 0)
print(f"P(μ₁ > μ₂ | data) = {prob_greater:.3f}")

# Plot posterior
az.plot_posterior(trace, var_names=['difference'], ref_val=0)

Effect Sizes

Always Calculate Effect Sizes

Effect sizes quantify magnitude, while p-values only indicate existence of an effect.

See references/effect_sizes_and_power.md for comprehensive guidance.

Quick Reference: Common Effect Sizes

Important: Benchmarks are guidelines. Context matters!

Calculating Effect Sizes

Most effect sizes are automatically calculated by pingouin:

# T-test returns Cohen's d
result = pg.ttest(x, y)
d = result['cohen-d'].values[0]

# ANOVA returns partial eta-squared
aov = pg.anova(dv='score', between='group', data=df)
eta_p2 = aov['np2'].values[0]

# Correlation: r is already an effect size
corr = pg.corr(x, y)
r = corr['r'].values[0]

Confidence Intervals for Effect Sizes

Always report CIs to show precision:

from pingouin import compute_effsize_from_t

# For t-test
d, ci = compute_effsize_from_t(
    t_statistic,
    nx=len(group1),
    ny=len(group2),
    eftype='cohen'
)
print(f"d = {d:.2f}, 95% CI [{ci[0]:.2f}, {ci[1]:.2f}]")

Power Analysis

A Priori Power Analysis (Study Planning)

Determine required sample size before data collection:

from statsmodels.stats.power import (
    tt_ind_solve_power,
    FTestAnovaPower
)

# T-test: What n is needed to detect d = 0.5?
n_required = tt_ind_solve_power(
    effect_size=0.5,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative='two-sided'
)
print(f"Required n per group: {n_required:.0f}")

# ANOVA: What n is needed to detect f = 0.25?
anova_power = FTestAnovaPower()
n_per_group = anova_power.solve_power(
    effect_size=0.25,
    ngroups=3,
    alpha=0.05,
    power=0.80
)
print(f"Required n per group: {n_per_group:.0f}")

Sensitivity Analysis (Post-Study)

Determine what effect size you could detect:

# With n=50 per group, what effect could we detect?
detectable_d = tt_ind_solve_power(
    effect_size=None,  # Solve for this
    nobs1=50,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative='two-sided'
)
print(f"Study could detect d ≥ {detectable_d:.2f}")

Note: Post-hoc power analysis (calculating power after study) is generally not recommended. Use sensitivity analysis instead.

See references/effect_sizes_and_power.md for detailed guidance.

Reporting Results

APA Style Statistical Reporting

Follow guidelines in references/reporting_standards.md.

Essential Reporting Elements

1. Descriptive statistics: M, SD, n for all groups/variables
2. Test statistics: Test name, statistic, df, exact p-value
3. Effect sizes: With confidence intervals
4. Assumption checks: Which tests were done, results, actions taken
5. All planned analyses: Including non-significant findings

Example Report Templates

Independent T-Test

Group A (n = 48, M = 75.2, SD = 8.5) scored significantly higher than
Group B (n = 52, M = 68.3, SD = 9.2), t(98) = 3.82, p < .001, d = 0.77,
95% CI [0.36, 1.18], two-tailed. Assumptions of normality (Shapiro-Wilk:
Group A W = 0.97, p = .18; Group B W = 0.96, p = .12) and homogeneity
of variance (Levene's F(1, 98) = 1.23, p = .27) were satisfied.

One-Way ANOVA

A one-way ANOVA revealed a significant main effect of treatment condition
on test scores, F(2, 147) = 8.45, p < .001, η²_p = .10. Post hoc
comparisons using Tukey's HSD indicated that Condition A (M = 78.2,
SD = 7.3) scored significantly higher than Condition B (M = 71.5,
SD = 8.1, p = .002, d = 0.87) and Condition C (M = 70.1, SD = 7.9,
p < .001, d = 1.07). Conditions B and C did not differ significantly
(p = .52, d = 0.18).

Multiple Regression

Multiple linear regression was conducted to predict exam scores from
study hours, prior GPA, and attendance. The overall model was significant,
F(3, 146) = 45.2, p < .001, R² = .48, adjusted R² = .47. Study hours
(B = 1.80, SE = 0.31, β = .35, t = 5.78, p < .001, 95% CI [1.18, 2.42])
and prior GPA (B = 8.52, SE = 1.95, β = .28, t = 4.37, p < .001,
95% CI [4.66, 12.38]) were significant predictors, while attendance was
not (B = 0.15, SE = 0.12, β = .08, t = 1.25, p = .21, 95% CI [-0.09, 0.39]).
Multicollinearity was not a concern (all VIF < 1.5).

Bayesian Analysis

A Bayesian independent samples t-test was conducted using weakly
informative priors (Normal(0, 1) for mean difference). The posterior
distribution indicated that Group A scored higher than Group B
(M_diff = 6.8, 95% credible interval [3.2, 10.4]). The Bayes Factor
BF₁₀ = 45.3 provided very strong evidence for a difference between
groups, with a 99.8% posterior probability that Group A's mean exceeded
Group B's mean. Convergence diagnostics were satisfactory (all R̂ < 1.01,
ESS > 1000).

Bayesian Statistics

When to Use Bayesian Methods

Consider Bayesian approaches when:

• You have prior information to incorporate
• You want direct probability statements about hypotheses
• Sample size is small or planning sequential data collection
• You need to quantify evidence for the null hypothesis
• The model is complex (hierarchical, missing data)

See references/bayesian_statistics.md for comprehensive guidance on:

• Bayes' theorem and interpretation
• Prior specification (informative, weakly informative, non-informative)
• Bayesian hypothesis testing with Bayes Factors
• Credible intervals vs. confidence intervals
• Bayesian t-tests, ANOVA, regression, and hierarchical models
• Model convergence checking and posterior predictive checks

Key Advantages

1. Intuitive interpretation: "Given the data, there is a 95% probability the parameter is in this interval"
2. Evidence for null: Can quantify support for no effect
3. Flexible: No p-hacking concerns; can analyze data as it arrives
4. Uncertainty quantification: Full posterior distribution

Resources

This skill includes comprehensive reference materials:

References Directory

• test_selection_guide.md: Decision tree for choosing appropriate statistical tests
• assumptions_and_diagnostics.md: Detailed guidance on checking and handling assumption violations
• effect_sizes_and_power.md: Calculating, interpreting, and reporting effect sizes; conducting power analyses
• bayesian_statistics.md: Complete guide to Bayesian analysis methods
• reporting_standards.md: APA-style reporting guidelines with examples

Scripts Directory

• assumption_checks.py: Automated assumption checking with visualizations
  • comprehensive_assumption_check(): Complete workflow
  • check_normality(): Normality testing with Q-Q plots
  • check_homogeneity_of_variance(): Levene's test with box plots
  • check_linearity(): Regression linearity checks
  • detect_outliers(): IQR and z-score outlier detection

Best Practices

1. Pre-register analyses when possible to distinguish confirmatory from exploratory
2. Always check assumptions before interpreting results
3. Report effect sizes with confidence intervals
4. Report all planned analyses including non-significant results
5. Distinguish statistical from practical significance
6. Visualize data before and after analysis
7. Check diagnostics for regression/ANOVA (residual plots, VIF, etc.)
8. Conduct sensitivity analyses to assess robustness
9. Share data and code for reproducibility
10. Be transparent about violations, transformations, and decisions

Common Pitfalls to Avoid

1. P-hacking: Don't test multiple ways until something is significant
2. HARKing: Don't present exploratory findings as confirmatory
3. Ignoring assumptions: Check them and report violations
4. Confusing significance with importance: p < .05 ≠ meaningful effect
5. Not reporting effect sizes: Essential for interpretation
6. Cherry-picking results: Report all planned analyses
7. Misinterpreting p-values: They're NOT probability that hypothesis is true
8. Multiple comparisons: Correct for family-wise error when appropriate
9. Ignoring missing data: Understand mechanism (MCAR, MAR, MNAR)
10. Overinterpreting non-significant results: Absence of evidence ≠ evidence of absence

Getting Started Checklist

When beginning a statistical analysis:

• Define research question and hypotheses
• Determine appropriate statistical test (use test_selection_guide.md)
• Conduct power analysis to determine sample size
• Load and inspect data
• Check for missing data and outliers
• Verify assumptions using assumption_checks.py
• Run primary analysis
• Calculate effect sizes with confidence intervals
• Conduct post-hoc tests if needed (with corrections)
• Create visualizations
• Write results following reporting_standards.md
• Conduct sensitivity analyses
• Share data and code

Support and Further Reading

For questions about:

• Test selection: See references/test_selection_guide.md
• Assumptions: See references/assumptions_and_diagnostics.md
• Effect sizes: See references/effect_sizes_and_power.md
• Bayesian methods: See references/bayesian_statistics.md
• Reporting: See references/reporting_standards.md

Key textbooks:

• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences
• Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics
• Gelman, A., & Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models
• Kruschke, J. K. (2014). Doing Bayesian Data Analysis

Online resources:

• APA Style Guide: https://apastyle.apa.org/
• Statistical Consulting: Cross Validated (stats.stackexchange.com)

Statsmodels: Statistical Modeling and Econometrics

Overview

Statsmodels is Python's premier library for statistical modeling, providing tools for estimation, inference, and diagnostics across a wide range of statistical methods. Apply this skill for rigorous statistical analysis, from simple linear regression to complex time series models and econometric analyses.

When to Use This Skill

This skill should be used when:

• Fitting regression models (OLS, WLS, GLS, quantile regression)
• Performing generalized linear modeling (logistic, Poisson, Gamma, etc.)
• Analyzing discrete outcomes (binary, multinomial, count, ordinal)
• Conducting time series analysis (ARIMA, SARIMAX, VAR, forecasting)
• Running statistical tests and diagnostics
• Testing model assumptions (heteroskedasticity, autocorrelation, normality)
• Detecting outliers and influential observations
• Comparing models (AIC/BIC, likelihood ratio tests)
• Estimating causal effects
• Producing publication-ready statistical tables and inference

Quick Start Guide

Linear Regression (OLS)

import statsmodels.api as sm
import numpy as np
import pandas as pd

# Prepare data - ALWAYS add constant for intercept
X = sm.add_constant(X_data)

# Fit OLS model
model = sm.OLS(y, X)
results = model.fit()

# View comprehensive results
print(results.summary())

# Key results
print(f"R-squared: {results.rsquared:.4f}")
print(f"Coefficients:\\n{results.params}")
print(f"P-values:\\n{results.pvalues}")

# Predictions with confidence intervals
predictions = results.get_prediction(X_new)
pred_summary = predictions.summary_frame()
print(pred_summary)  # includes mean, CI, prediction intervals

# Diagnostics
from statsmodels.stats.diagnostic import het_breuschpagan
bp_test = het_breuschpagan(results.resid, X)
print(f"Breusch-Pagan p-value: {bp_test[1]:.4f}")

# Visualize residuals
import matplotlib.pyplot as plt
plt.scatter(results.fittedvalues, results.resid)
plt.axhline(y=0, color='r', linestyle='--')
plt.xlabel('Fitted values')
plt.ylabel('Residuals')
plt.show()

Logistic Regression (Binary Outcomes)

from statsmodels.discrete.discrete_model import Logit

# Add constant
X = sm.add_constant(X_data)

# Fit logit model
model = Logit(y_binary, X)
results = model.fit()

print(results.summary())

# Odds ratios
odds_ratios = np.exp(results.params)
print("Odds ratios:\\n", odds_ratios)

# Predicted probabilities
probs = results.predict(X)

# Binary predictions (0.5 threshold)
predictions = (probs > 0.5).astype(int)

# Model evaluation
from sklearn.metrics import classification_report, roc_auc_score

print(classification_report(y_binary, predictions))
print(f"AUC: {roc_auc_score(y_binary, probs):.4f}")

# Marginal effects
marginal = results.get_margeff()
print(marginal.summary())

Time Series (ARIMA)

from statsmodels.tsa.arima.model import ARIMA
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf

# Check stationarity
from statsmodels.tsa.stattools import adfuller

adf_result = adfuller(y_series)
print(f"ADF p-value: {adf_result[1]:.4f}")

if adf_result[1] > 0.05:
    # Series is non-stationary, difference it
    y_diff = y_series.diff().dropna()

# Plot ACF/PACF to identify p, q
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 8))
plot_acf(y_diff, lags=40, ax=ax1)
plot_pacf(y_diff, lags=40, ax=ax2)
plt.show()

# Fit ARIMA(p,d,q)
model = ARIMA(y_series, order=(1, 1, 1))
results = model.fit()

print(results.summary())

# Forecast
forecast = results.forecast(steps=10)
forecast_obj = results.get_forecast(steps=10)
forecast_df = forecast_obj.summary_frame()

print(forecast_df)  # includes mean and confidence intervals

# Residual diagnostics
results.plot_diagnostics(figsize=(12, 8))
plt.show()

Generalized Linear Models (GLM)

import statsmodels.api as sm

# Poisson regression for count data
X = sm.add_constant(X_data)
model = sm.GLM(y_counts, X, family=sm.families.Poisson())
results = model.fit()

print(results.summary())

# Rate ratios (for Poisson with log link)
rate_ratios = np.exp(results.params)
print("Rate ratios:\\n", rate_ratios)

# Check overdispersion
overdispersion = results.pearson_chi2 / results.df_resid
print(f"Overdispersion: {overdispersion:.2f}")

if overdispersion > 1.5:
    # Use Negative Binomial instead
    from statsmodels.discrete.count_model import NegativeBinomial
    nb_model = NegativeBinomial(y_counts, X)
    nb_results = nb_model.fit()
    print(nb_results.summary())

Core Statistical Modeling Capabilities

1. Linear Regression Models

Comprehensive suite of linear models for continuous outcomes with various error structures.

Available models:

• OLS: Standard linear regression with i.i.d. errors
• WLS: Weighted least squares for heteroskedastic errors
• GLS: Generalized least squares for arbitrary covariance structure
• GLSAR: GLS with autoregressive errors for time series
• Quantile Regression: Conditional quantiles (robust to outliers)
• Mixed Effects: Hierarchical/multilevel models with random effects
• Recursive/Rolling: Time-varying parameter estimation

Key features:

• Comprehensive diagnostic tests
• Robust standard errors (HC, HAC, cluster-robust)
• Influence statistics (Cook's distance, leverage, DFFITS)
• Hypothesis testing (F-tests, Wald tests)
• Model comparison (AIC, BIC, likelihood ratio tests)
• Prediction with confidence and prediction intervals

When to use: Continuous outcome variable, want inference on coefficients, need diagnostics

Reference: See references/linear_models.md for detailed guidance on model selection, diagnostics, and best practices.

2. Generalized Linear Models (GLM)

Flexible framework extending linear models to non-normal distributions.

Distribution families:

• Binomial: Binary outcomes or proportions (logistic regression)
• Poisson: Count data
• Negative Binomial: Overdispersed counts
• Gamma: Positive continuous, right-skewed data
• Inverse Gaussian: Positive continuous with specific variance structure
• Gaussian: Equivalent to OLS
• Tweedie: Flexible family for semi-continuous data

Link functions:

• Logit, Probit, Log, Identity, Inverse, Sqrt, CLogLog, Power
• Choose based on interpretation needs and model fit

Key features:

• Maximum likelihood estimation via IRLS
• Deviance and Pearson residuals
• Goodness-of-fit statistics
• Pseudo R-squared measures
• Robust standard errors

When to use: Non-normal outcomes, need flexible variance and link specifications

Reference: See references/glm.md for family selection, link functions, interpretation, and diagnostics.

3. Discrete Choice Models

Models for categorical and count outcomes.

Binary models:

• Logit: Logistic regression (odds ratios)
• Probit: Probit regression (normal distribution)

Multinomial models:

• MNLogit: Unordered categories (3+ levels)
• Conditional Logit: Choice models with alternative-specific variables
• Ordered Model: Ordinal outcomes (ordered categories)

Count models:

• Poisson: Standard count model
• Negative Binomial: Overdispersed counts
• Zero-Inflated: Excess zeros (ZIP, ZINB)
• Hurdle Models: Two-stage models for zero-heavy data

Key features:

• Maximum likelihood estimation
• Marginal effects at means or average marginal effects
• Model comparison via AIC/BIC
• Predicted probabilities and classification
• Goodness-of-fit tests

When to use: Binary, categorical, or count outcomes

Reference: See references/discrete_choice.md for model selection, interpretation, and evaluation.

4. Time Series Analysis

Comprehensive time series modeling and forecasting capabilities.

Univariate models:

• AutoReg (AR): Autoregressive models
• ARIMA: Autoregressive integrated moving average
• SARIMAX: Seasonal ARIMA with exogenous variables
• Exponential Smoothing: Simple, Holt, Holt-Winters
• ETS: Innovations state space models

Multivariate models:

• VAR: Vector autoregression
• VARMAX: VAR with MA and exogenous variables
• Dynamic Factor Models: Extract common factors
• VECM: Vector error correction models (cointegration)

Advanced models:

• State Space: Kalman filtering, custom specifications
• Regime Switching: Markov switching models
• ARDL: Autoregressive distributed lag

Key features:

• ACF/PACF analysis for model identification
• Stationarity tests (ADF, KPSS)
• Forecasting with prediction intervals
• Residual diagnostics (Ljung-Box, heteroskedasticity)
• Granger causality testing
• Impulse response functions (IRF)
• Forecast error variance decomposition (FEVD)

When to use: Time-ordered data, forecasting, understanding temporal dynamics

Reference: See references/time_series.md for model selection, diagnostics, and forecasting methods.

5. Statistical Tests and Diagnostics

Extensive testing and diagnostic capabilities for model validation.

Residual diagnostics:

• Autocorrelation tests (Ljung-Box, Durbin-Watson, Breusch-Godfrey)
• Heteroskedasticity tests (Breusch-Pagan, White, ARCH)
• Normality tests (Jarque-Bera, Omnibus, Anderson-Darling, Lilliefors)
• Specification tests (RESET, Harvey-Collier)

Influence and outliers:

• Leverage (hat values)
• Cook's distance
• DFFITS and DFBETAs
• Studentized residuals
• Influence plots

Hypothesis testing:

• t-tests (one-sample, two-sample, paired)
• Proportion tests
• Chi-square tests
• Non-parametric tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis)
• ANOVA (one-way, two-way, repeated measures)

Multiple comparisons:

• Tukey's HSD
• Bonferroni correction
• False Discovery Rate (FDR)

Effect sizes and power:

• Cohen's d, eta-squared
• Power analysis for t-tests, proportions
• Sample size calculations

Robust inference:

• Heteroskedasticity-consistent SEs (HC0-HC3)
• HAC standard errors (Newey-West)
• Cluster-robust standard errors

When to use: Validating assumptions, detecting problems, ensuring robust inference

Reference: See references/stats_diagnostics.md for comprehensive testing and diagnostic procedures.

Formula API (R-style)

Statsmodels supports R-style formulas for intuitive model specification:

import statsmodels.formula.api as smf

# OLS with formula
results = smf.ols('y ~ x1 + x2 + x1:x2', data=df).fit()

# Categorical variables (automatic dummy coding)
results = smf.ols('y ~ x1 + C(category)', data=df).fit()

# Interactions
results = smf.ols('y ~ x1 * x2', data=df).fit()  # x1 + x2 + x1:x2

# Polynomial terms
results = smf.ols('y ~ x + I(x**2)', data=df).fit()

# Logit
results = smf.logit('y ~ x1 + x2 + C(group)', data=df).fit()

# Poisson
results = smf.poisson('count ~ x1 + x2', data=df).fit()

# ARIMA (not available via formula, use regular API)

Model Selection and Comparison

Information Criteria

# Compare models using AIC/BIC
models = {
    'Model 1': model1_results,
    'Model 2': model2_results,
    'Model 3': model3_results
}

comparison = pd.DataFrame({
    'AIC': {name: res.aic for name, res in models.items()},
    'BIC': {name: res.bic for name, res in models.items()},
    'Log-Likelihood': {name: res.llf for name, res in models.items()}
})

print(comparison.sort_values('AIC'))
# Lower AIC/BIC indicates better model

Likelihood Ratio Test (Nested Models)

# For nested models (one is subset of the other)
from scipy import stats

lr_stat = 2 * (full_model.llf - reduced_model.llf)
df = full_model.df_model - reduced_model.df_model
p_value = 1 - stats.chi2.cdf(lr_stat, df)

print(f"LR statistic: {lr_stat:.4f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Full model significantly better")
else:
    print("Reduced model preferred (parsimony)")

Cross-Validation

from sklearn.model_selection import KFold
from sklearn.metrics import mean_squared_error

kf = KFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = []

for train_idx, val_idx in kf.split(X):
    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

    # Fit model
    model = sm.OLS(y_train, X_train).fit()

    # Predict
    y_pred = model.predict(X_val)

    # Score
    rmse = np.sqrt(mean_squared_error(y_val, y_pred))
    cv_scores.append(rmse)

print(f"CV RMSE: {np.mean(cv_scores):.4f} ± {np.std(cv_scores):.4f}")

Best Practices

Data Preparation

1. Always add constant: Use sm.add_constant() unless excluding intercept
2. Check for missing values: Handle or impute before fitting
3. Scale if needed: Improves convergence, interpretation (but not required for tree models)
4. Encode categoricals: Use formula API or manual dummy coding

Model Building

1. Start simple: Begin with basic model, add complexity as needed
2. Check assumptions: Test residuals, heteroskedasticity, autocorrelation
3. Use appropriate model: Match model to outcome type (binary→Logit, count→Poisson)
4. Consider alternatives: If assumptions violated, use robust methods or different model

Inference

1. Report effect sizes: Not just p-values
2. Use robust SEs: When heteroskedasticity or clustering present
3. Multiple comparisons: Correct when testing many hypotheses
4. Confidence intervals: Always report alongside point estimates

Model Evaluation

1. Check residuals: Plot residuals vs fitted, Q-Q plot
2. Influence diagnostics: Identify and investigate influential observations
3. Out-of-sample validation: Test on holdout set or cross-validate
4. Compare models: Use AIC/BIC for non-nested, LR test for nested

Reporting

1. Comprehensive summary: Use .summary() for detailed output
2. Document decisions: Note transformations, excluded observations
3. Interpret carefully: Account for link functions (e.g., exp(β) for log link)
4. Visualize: Plot predictions, confidence intervals, diagnostics

Common Workflows

Workflow 1: Linear Regression Analysis

1. Explore data (plots, descriptives)
2. Fit initial OLS model
3. Check residual diagnostics
4. Test for heteroskedasticity, autocorrelation
5. Check for multicollinearity (VIF)
6. Identify influential observations
7. Refit with robust SEs if needed
8. Interpret coefficients and inference
9. Validate on holdout or via CV

Workflow 2: Binary Classification

1. Fit logistic regression (Logit)
2. Check for convergence issues
3. Interpret odds ratios
4. Calculate marginal effects
5. Evaluate classification performance (AUC, confusion matrix)
6. Check for influential observations
7. Compare with alternative models (Probit)
8. Validate predictions on test set

Workflow 3: Count Data Analysis

1. Fit Poisson regression
2. Check for overdispersion
3. If overdispersed, fit Negative Binomial
4. Check for excess zeros (consider ZIP/ZINB)
5. Interpret rate ratios
6. Assess goodness of fit
7. Compare models via AIC
8. Validate predictions

Workflow 4: Time Series Forecasting

1. Plot series, check for trend/seasonality
2. Test for stationarity (ADF, KPSS)
3. Difference if non-stationary
4. Identify p, q from ACF/PACF
5. Fit ARIMA or SARIMAX
6. Check residual diagnostics (Ljung-Box)
7. Generate forecasts with confidence intervals
8. Evaluate forecast accuracy on test set

Reference Documentation

This skill includes comprehensive reference files for detailed guidance:

references/linear_models.md

Detailed coverage of linear regression models including:

• OLS, WLS, GLS, GLSAR, Quantile Regression
• Mixed effects models
• Recursive and rolling regression
• Comprehensive diagnostics (heteroskedasticity, autocorrelation, multicollinearity)
• Influence statistics and outlier detection
• Robust standard errors (HC, HAC, cluster)
• Hypothesis testing and model comparison

references/glm.md

Complete guide to generalized linear models:

• All distribution families (Binomial, Poisson, Gamma, etc.)
• Link functions and when to use each
• Model fitting and interpretation
• Pseudo R-squared and goodness of fit
• Diagnostics and residual analysis
• Applications (logistic, Poisson, Gamma regression)

references/discrete_choice.md

Comprehensive guide to discrete outcome models:

• Binary models (Logit, Probit)
• Multinomial models (MNLogit, Conditional Logit)
• Count models (Poisson, Negative Binomial, Zero-Inflated, Hurdle)
• Ordinal models
• Marginal effects and interpretation
• Model diagnostics and comparison

references/time_series.md

In-depth time series analysis guidance:

• Univariate models (AR, ARIMA, SARIMAX, Exponential Smoothing)
• Multivariate models (VAR, VARMAX, Dynamic Factor)
• State space models
• Stationarity testing and diagnostics
• Forecasting methods and evaluation
• Granger causality, IRF, FEVD

references/stats_diagnostics.md

Comprehensive statistical testing and diagnostics:

• Residual diagnostics (autocorrelation, heteroskedasticity, normality)
• Influence and outlier detection
• Hypothesis tests (parametric and non-parametric)
• ANOVA and post-hoc tests
• Multiple comparisons correction
• Robust covariance matrices
• Power analysis and effect sizes

When to reference:

• Need detailed parameter explanations
• Choosing between similar models
• Troubleshooting convergence or diagnostic issues
• Understanding specific test statistics
• Looking for code examples for advanced features

Search patterns:

# Find information about specific models
grep -r "Quantile Regression" references/

# Find diagnostic tests
grep -r "Breusch-Pagan" references/stats_diagnostics.md

# Find time series guidance
grep -r "SARIMAX" references/time_series.md

Common Pitfalls to Avoid

1. Forgetting constant term: Always use sm.add_constant() unless no intercept desired
2. Ignoring assumptions: Check residuals, heteroskedasticity, autocorrelation
3. Wrong model for outcome type: Binary→Logit/Probit, Count→Poisson/NB, not OLS
4. Not checking convergence: Look for optimization warnings
5. Misinterpreting coefficients: Remember link functions (log, logit, etc.)
6. Using Poisson with overdispersion: Check dispersion, use Negative Binomial if needed
7. Not using robust SEs: When heteroskedasticity or clustering present
8. Overfitting: Too many parameters relative to sample size
9. Data leakage: Fitting on test data or using future information
10. Not validating predictions: Always check out-of-sample performance
11. Comparing non-nested models: Use AIC/BIC, not LR test
12. Ignoring influential observations: Check Cook's distance and leverage
13. Multiple testing: Correct p-values when testing many hypotheses
14. Not differencing time series: Fit ARIMA on non-stationary data
15. Confusing prediction vs confidence intervals: Prediction intervals are wider

Getting Help

For detailed documentation and examples:

• Official docs: https://www.statsmodels.org/stable/
• User guide: https://www.statsmodels.org/stable/user-guide.html
• Examples: https://www.statsmodels.org/stable/examples/index.html
• API reference: https://www.statsmodels.org/stable/api.html

TimesFM Forecasting

Overview

TimesFM (Time Series Foundation Model) is a pretrained decoder-only foundation model developed by Google Research for time-series forecasting. It works zero-shot — feed it any univariate time series and it returns point forecasts with calibrated quantile prediction intervals, no training required.

This skill wraps TimesFM for safe, agent-friendly local inference. It includes a mandatory preflight system checker that verifies RAM, GPU memory, and disk space before the model is ever loaded so the agent never crashes a user's machine.

Key numbers: TimesFM 2.5 uses 200M parameters (~800 MB on disk, ~1.5 GB in RAM on CPU, ~1 GB VRAM on GPU). The archived v1/v2 500M-parameter model needs ~32 GB RAM. Always run the system checker first.

When to Use This Skill

Use this skill when:

• Forecasting any univariate time series (sales, demand, sensor, vitals, price, weather)
• You need zero-shot forecasting without training a custom model
• You want probabilistic forecasts with calibrated prediction intervals (quantiles)
• You have time series of any length (the model handles 1–16,384 context points)
• You need to batch-forecast hundreds or thousands of series efficiently
• You want a foundation model approach instead of hand-tuning ARIMA/ETS parameters

Do not use this skill when:

• You need classical statistical models with coefficient interpretation → use statsmodels
• You need time series classification or clustering → use aeon
• You need multivariate vector autoregression or Granger causality → use statsmodels
• Your data is tabular (not temporal) → use scikit-learn

Note on Anomaly Detection: TimesFM does not have built-in anomaly detection, but you can use the quantile forecasts as prediction intervals — values outside the 90% CI (q10–q90) are statistically unusual. See the examples/anomaly-detection/ directory for a full example.

⚠️ Mandatory Preflight: System Requirements Check

CRITICAL — ALWAYS run the system checker before loading the model for the first time.

python scripts/check_system.py

This script checks:

1. Available RAM — warns if below 4 GB, blocks if below 2 GB
2. GPU availability — detects CUDA/MPS devices and VRAM
3. Disk space — verifies room for the ~800 MB model download
4. Python version — requires 3.10+
5. Existing installation — checks if timesfm and torch are installed

Note: Model weights are NOT stored in this repository. TimesFM weights (~800 MB) download on-demand from HuggingFace on first use and cache in ~/.cache/huggingface/. The preflight checker ensures sufficient resources before any download begins.

flowchart TD
    accTitle: Preflight System Check
    accDescr: Decision flowchart showing the system requirement checks that must pass before loading TimesFM.

    start["🚀 Run check_system.py"] --> ram{"RAM ≥ 4 GB?"}
    ram -->|"Yes"| gpu{"GPU available?"}
    ram -->|"No (2-4 GB)"| warn_ram["⚠️ Warning: tight RAM<br/>CPU-only, small batches"]
    ram -->|"No (< 2 GB)"| block["🛑 BLOCKED<br/>Insufficient memory"]
    warn_ram --> disk
    gpu -->|"CUDA / MPS"| vram{"VRAM ≥ 2 GB?"}
    gpu -->|"CPU only"| cpu_ok["✅ CPU mode<br/>Slower but works"]
    vram -->|"Yes"| gpu_ok["✅ GPU mode<br/>Fast inference"]
    vram -->|"No"| cpu_ok
    gpu_ok --> disk{"Disk ≥ 2 GB free?"}
    cpu_ok --> disk
    disk -->|"Yes"| ready["✅ READY<br/>Safe to load model"]
    disk -->|"No"| block_disk["🛑 BLOCKED<br/>Need space for weights"]

    classDef ok fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
    classDef warn fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
    classDef block fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#7f1d1d
    classDef neutral fill:#f3f4f6,stroke:#6b7280,stroke-width:2px,color:#1f2937

    class ready,gpu_ok,cpu_ok ok
    class warn_ram warn
    class block,block_disk block
    class start,ram,gpu,vram,disk neutral

Hardware Requirements by Model Version

Recommendation: Always use TimesFM 2.5 unless you have a specific reason to use an older checkpoint. It is smaller, faster, and supports 8× longer context.

🔧 Installation

Step 1: Verify System (always first)

python scripts/check_system.py

Step 2: Install TimesFM

# Using uv (recommended by this repo)
uv pip install timesfm[torch]

# Or using pip
pip install timesfm[torch]

# For JAX/Flax backend (faster on TPU/GPU)
uv pip install timesfm[flax]

Step 3: Install PyTorch for Your Hardware

# CUDA 12.1 (NVIDIA GPU)
pip install torch>=2.0.0 --index-url https://download.pytorch.org/whl/cu121

# CPU only
pip install torch>=2.0.0 --index-url https://download.pytorch.org/whl/cpu

# Apple Silicon (MPS)
pip install torch>=2.0.0  # MPS support is built-in

Step 4: Verify Installation

import timesfm
import numpy as np
print(f"TimesFM version: {timesfm.__version__}")
print("Installation OK")

🎯 Quick Start

Minimal Example (5 Lines)

import torch, numpy as np, timesfm

torch.set_float32_matmul_precision("high")

model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(
    "google/timesfm-2.5-200m-pytorch"
)
model.compile(timesfm.ForecastConfig(
    max_context=1024, max_horizon=256, normalize_inputs=True,
    use_continuous_quantile_head=True, force_flip_invariance=True,
    infer_is_positive=True, fix_quantile_crossing=True,
))

point, quantiles = model.forecast(horizon=24, inputs=[
    np.sin(np.linspace(0, 20, 200)),  # any 1-D array
])
# point.shape == (1, 24)        — median forecast
# quantiles.shape == (1, 24, 10) — 10th–90th percentile bands

Forecast from CSV

import pandas as pd, numpy as np

df = pd.read_csv("monthly_sales.csv", parse_dates=["date"], index_col="date")

# Convert each column to a list of arrays
inputs = [df[col].dropna().values.astype(np.float32) for col in df.columns]

point, quantiles = model.forecast(horizon=12, inputs=inputs)

# Build a results DataFrame
for i, col in enumerate(df.columns):
    last_date = df[col].dropna().index[-1]
    future_dates = pd.date_range(last_date, periods=13, freq="MS")[1:]
    forecast_df = pd.DataFrame({
        "date": future_dates,
        "forecast": point[i],
        "lower_80": quantiles[i, :, 2],  # 20th percentile
        "upper_80": quantiles[i, :, 8],  # 80th percentile
    })
    print(f"\n--- {col} ---")
    print(forecast_df.to_string(index=False))

Forecast with Covariates (XReg)

TimesFM 2.5+ supports exogenous variables through forecast_with_covariates(). Requires timesfm[xreg].

# Requires: uv pip install timesfm[xreg]
point, quantiles = model.forecast_with_covariates(
    inputs=inputs,
    dynamic_numerical_covariates={"price": price_arrays},
    dynamic_categorical_covariates={"holiday": holiday_arrays},
    static_categorical_covariates={"region": region_labels},
    xreg_mode="xreg + timesfm",  # or "timesfm + xreg"
)

XReg Modes:

• "xreg + timesfm" (default): TimesFM forecasts first, then XReg adjusts residuals
• "timesfm + xreg": XReg fits first, then TimesFM forecasts residuals

See examples/covariates-forecasting/ for a complete example with synthetic retail data.

Anomaly Detection (via Quantile Intervals)

TimesFM does not have built-in anomaly detection, but the quantile forecasts naturally provide prediction intervals that can detect anomalies:

point, q = model.forecast(horizon=H, inputs=[values])

# 90% prediction interval
lower_90 = q[0, :, 1]  # 10th percentile
upper_90 = q[0, :, 9]  # 90th percentile

# Detect anomalies: values outside the 90% CI
actual = test_values  # your holdout data
anomalies = (actual < lower_90) | (actual > upper_90)

# Severity levels
is_warning = (actual < q[0, :, 2]) | (actual > q[0, :, 8])  # outside 80% CI
is_critical = anomalies  # outside 90% CI

See examples/anomaly-detection/ for a complete example with visualization.

# Requires: uv pip install timesfm[xreg]
point, quantiles = model.forecast_with_covariates(
    inputs=inputs,
    dynamic_numerical_covariates={"temperature": temp_arrays},
    dynamic_categorical_covariates={"day_of_week": dow_arrays},
    static_categorical_covariates={"region": region_labels},
    xreg_mode="xreg + timesfm",  # or "timesfm + xreg"
)

📊 Understanding the Output

Quantile Forecast Structure

TimesFM returns (point_forecast, quantile_forecast):

• point_forecast: shape (batch, horizon) — the median (0.5 quantile)
• quantile_forecast: shape (batch, horizon, 10) — ten slices:

Extracting Prediction Intervals

point, q = model.forecast(horizon=H, inputs=data)

# 80% prediction interval (most common)
lower_80 = q[:, :, 1]  # 10th percentile
upper_80 = q[:, :, 9]  # 90th percentile

# 60% prediction interval (tighter)
lower_60 = q[:, :, 2]  # 20th percentile
upper_60 = q[:, :, 8]  # 80th percentile

# Median (same as point forecast)
median = q[:, :, 5]

flowchart LR
    accTitle: Quantile Forecast Anatomy
    accDescr: Diagram showing how the 10-element quantile vector maps to prediction intervals.

    input["📈 Input Series<br/>1-D array"] --> model["🤖 TimesFM<br/>compile + forecast"]
    model --> point["📍 Point Forecast<br/>(batch, horizon)"]
    model --> quant["📊 Quantile Forecast<br/>(batch, horizon, 10)"]
    quant --> pi80["80% PI<br/>q[:,:,1] – q[:,:,9]"]
    quant --> pi60["60% PI<br/>q[:,:,2] – q[:,:,8]"]
    quant --> median["Median<br/>q[:,:,5]"]

    classDef data fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e3a5f
    classDef model fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#581c87
    classDef output fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d

    class input data
    class model model
    class point,quant,pi80,pi60,median output

🔧 ForecastConfig Reference

All forecasting behavior is controlled by timesfm.ForecastConfig:

timesfm.ForecastConfig(
    max_context=1024,                    # Max context window (truncates longer series)
    max_horizon=256,                     # Max forecast horizon
    normalize_inputs=True,               # Normalize inputs (RECOMMENDED for stability)
    per_core_batch_size=32,              # Batch size per device (tune for memory)
    use_continuous_quantile_head=True,   # Better quantile accuracy for long horizons
    force_flip_invariance=True,          # Ensures f(-x) = -f(x) (mathematical consistency)
    infer_is_positive=True,              # Clamp forecasts ≥ 0 when all inputs > 0
    fix_quantile_crossing=True,          # Ensure q10 ≤ q20 ≤ ... ≤ q90
    return_backcast=False,               # Return backcast (for covariate workflows)
)

📋 Common Workflows

Workflow 1: Single Series Forecast

flowchart TD
    accTitle: Single Series Forecast Workflow
    accDescr: Step-by-step workflow for forecasting a single time series with system checking.

    check["1. Run check_system.py"] --> load["2. Load model<br/>from_pretrained()"]
    load --> compile["3. Compile with ForecastConfig"]
    compile --> prep["4. Prepare data<br/>pd.read_csv → np.array"]
    prep --> forecast["5. model.forecast()<br/>horizon=N"]
    forecast --> extract["6. Extract point + PI"]
    extract --> plot["7. Plot or export results"]

    classDef step fill:#f3f4f6,stroke:#6b7280,stroke-width:2px,color:#1f2937
    class check,load,compile,prep,forecast,extract,plot step

import torch, numpy as np, pandas as pd, timesfm

# 1. System check (run once)
# python scripts/check_system.py

# 2-3. Load and compile
torch.set_float32_matmul_precision("high")
model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(
    "google/timesfm-2.5-200m-pytorch"
)
model.compile(timesfm.ForecastConfig(
    max_context=512, max_horizon=52, normalize_inputs=True,
    use_continuous_quantile_head=True, fix_quantile_crossing=True,
))

# 4. Prepare data
df = pd.read_csv("weekly_demand.csv", parse_dates=["week"])
values = df["demand"].values.astype(np.float32)

# 5. Forecast
point, quantiles = model.forecast(horizon=52, inputs=[values])

# 6. Extract prediction intervals
forecast_df = pd.DataFrame({
    "forecast": point[0],
    "lower_80": quantiles[0, :, 1],
    "upper_80": quantiles[0, :, 9],
})

# 7. Plot
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(12, 5))
ax.plot(values[-104:], label="Historical")
x_fc = range(len(values[-104:]), len(values[-104:]) + 52)
ax.plot(x_fc, forecast_df["forecast"], label="Forecast", color="tab:orange")
ax.fill_between(x_fc, forecast_df["lower_80"], forecast_df["upper_80"],
                alpha=0.2, color="tab:orange", label="80% PI")
ax.legend()
ax.set_title("52-Week Demand Forecast")
plt.tight_layout()
plt.savefig("forecast.png", dpi=150)
print("Saved forecast.png")

Workflow 2: Batch Forecasting (Many Series)

import pandas as pd, numpy as np

# Load wide-format CSV (one column per series)
df = pd.read_csv("all_stores.csv", parse_dates=["date"], index_col="date")
inputs = [df[col].dropna().values.astype(np.float32) for col in df.columns]

# Forecast all series at once (batched internally)
point, quantiles = model.forecast(horizon=30, inputs=inputs)

# Collect results
results = {}
for i, col in enumerate(df.columns):
    results[col] = {
        "forecast": point[i].tolist(),
        "lower_80": quantiles[i, :, 1].tolist(),
        "upper_80": quantiles[i, :, 9].tolist(),
    }

# Export
import json
with open("batch_forecasts.json", "w") as f:
    json.dump(results, f, indent=2)
print(f"Forecasted {len(results)} series → batch_forecasts.json")

Workflow 3: Evaluate Forecast Accuracy

import numpy as np

# Hold out the last H points for evaluation
H = 24
train = values[:-H]
actual = values[-H:]

point, quantiles = model.forecast(horizon=H, inputs=[train])
pred = point[0]

# Metrics
mae = np.mean(np.abs(actual - pred))
rmse = np.sqrt(np.mean((actual - pred) ** 2))
mape = np.mean(np.abs((actual - pred) / actual)) * 100

# Prediction interval coverage
lower = quantiles[0, :, 1]
upper = quantiles[0, :, 9]
coverage = np.mean((actual >= lower) & (actual <= upper)) * 100

print(f"MAE:  {mae:.2f}")
print(f"RMSE: {rmse:.2f}")
print(f"MAPE: {mape:.1f}%")
print(f"80% PI Coverage: {coverage:.1f}% (target: 80%)")

⚙️ Performance Tuning

GPU Acceleration

import torch

# Check GPU availability
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
    print("Apple Silicon MPS available")
else:
    print("CPU only — inference will be slower but still works")

# Always set this for Ampere+ GPUs (A100, RTX 3090, etc.)
torch.set_float32_matmul_precision("high")

Batch Size Tuning

# Start conservative, increase until OOM
# GPU with 8 GB VRAM:  per_core_batch_size=64
# GPU with 16 GB VRAM: per_core_batch_size=128
# GPU with 24 GB VRAM: per_core_batch_size=256
# CPU with 8 GB RAM:   per_core_batch_size=8
# CPU with 16 GB RAM:  per_core_batch_size=32
# CPU with 32 GB RAM:  per_core_batch_size=64

model.compile(timesfm.ForecastConfig(
    max_context=1024,
    max_horizon=256,
    per_core_batch_size=32,  # <-- tune this
    normalize_inputs=True,
    use_continuous_quantile_head=True,
    fix_quantile_crossing=True,
))

Memory-Constrained Environments

import gc, torch

# Force garbage collection before loading
gc.collect()
if torch.cuda.is_available():
    torch.cuda.empty_cache()

# Load model
model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(
    "google/timesfm-2.5-200m-pytorch"
)

# Use small batch size on low-memory machines
model.compile(timesfm.ForecastConfig(
    max_context=512,        # Reduce context if needed
    max_horizon=128,        # Reduce horizon if needed
    per_core_batch_size=4,  # Small batches
    normalize_inputs=True,
    use_continuous_quantile_head=True,
    fix_quantile_crossing=True,
))

# Process series in chunks to avoid OOM
CHUNK = 50
all_results = []
for i in range(0, len(inputs), CHUNK):
    chunk = inputs[i:i+CHUNK]
    p, q = model.forecast(horizon=H, inputs=chunk)
    all_results.append((p, q))
    gc.collect()  # Clean up between chunks

🔗 Integration with Other Skills

With statsmodels

Use statsmodels for classical models (ARIMA, SARIMAX) as a comparison baseline:

# TimesFM forecast
tfm_point, tfm_q = model.forecast(horizon=H, inputs=[values])

# statsmodels ARIMA forecast
from statsmodels.tsa.arima.model import ARIMA
arima = ARIMA(values, order=(1,1,1)).fit()
arima_forecast = arima.forecast(steps=H)

# Compare
print(f"TimesFM MAE: {np.mean(np.abs(actual - tfm_point[0])):.2f}")
print(f"ARIMA MAE:   {np.mean(np.abs(actual - arima_forecast)):.2f}")

With matplotlib / scientific-visualization

Plot forecasts with prediction intervals as publication-quality figures.

With exploratory-data-analysis

Run EDA on the time series before forecasting to understand trends, seasonality, and stationarity.

📚 Available Scripts

scripts/check_system.py

Mandatory preflight checker. Run before first model load.

python scripts/check_system.py

Output example:

=== TimesFM System Requirements Check ===

[RAM]       Total: 32.0 GB | Available: 24.3 GB  ✅ PASS
[GPU]       NVIDIA RTX 4090 | VRAM: 24.0 GB      ✅ PASS
[Disk]      Free: 142.5 GB                        ✅ PASS
[Python]    3.12.1                                 ✅ PASS
[timesfm]   Installed (2.5.0)                      ✅ PASS
[torch]     Installed (2.4.1+cu121)                ✅ PASS

VERDICT: ✅ System is ready for TimesFM 2.5 (GPU mode)
Recommended: per_core_batch_size=128

scripts/forecast_csv.py

End-to-end CSV forecasting with automatic system check.

python scripts/forecast_csv.py input.csv \
    --horizon 24 \
    --date-col date \
    --value-cols sales,revenue \
    --output forecasts.csv

📖 Reference Documentation

Detailed guides in references/:

Common Pitfalls

1. Not running system check → model load crashes on low-RAM machines. Always run check_system.py first.
2. Forgetting model.compile() → RuntimeError: Model is not compiled. Must call compile() before forecast().
3. Not setting normalize_inputs=True → unstable forecasts for series with large values.
4. Using v1/v2 on machines with < 32 GB RAM → use TimesFM 2.5 (200M params) instead.
5. Not setting fix_quantile_crossing=True → quantiles may not be monotonic (q10 > q50).
6. Huge per_core_batch_size on small GPU → CUDA OOM. Start small, increase.
7. Passing 2-D arrays → TimesFM expects a list of 1-D arrays, not a 2-D matrix.
8. Forgetting torch.set_float32_matmul_precision("high") → slower inference on Ampere+ GPUs.
9. Not handling NaN in output → edge cases with very short series. Always check np.isnan(point).any().
10. Using infer_is_positive=True for series that can be negative → clamps forecasts at zero. Set False for temperature, returns, etc.

Model Versions

timeline
    accTitle: TimesFM Version History
    accDescr: Timeline of TimesFM model releases showing parameter counts and key improvements.

    section 2024
        TimesFM 1.0 : 200M params, 2K context, JAX only
        TimesFM 2.0 : 500M params, 2K context, PyTorch + JAX
    section 2025
        TimesFM 2.5 : 200M params, 16K context, quantile head, no frequency indicator

Hugging Face checkpoints:

• google/timesfm-2.5-200m-pytorch (recommended)
• google/timesfm-2.5-200m-flax
• google/timesfm-2.0-500m-pytorch (archived)
• google/timesfm-1.0-200m-pytorch (archived)

Resources

• Paper: A Decoder-Only Foundation Model for Time-Series Forecasting (ICML 2024)
• Repository: https://github.com/google-research/timesfm
• Hugging Face: https://huggingface.co/collections/google/timesfm-release-66e4be5fdb56e960c1e482a6
• Google Blog: https://research.google/blog/a-decoder-only-foundation-model-for-time-series-forecasting/
• BigQuery Integration: https://cloud.google.com/bigquery/docs/timesfm-model

Examples

Three fully-working reference examples live in examples/. Use them as ground truth for correct API usage and expected output shape.

Running the Examples

# Global temperature (no TimesFM 2.5 needed)
cd examples/global-temperature && python run_forecast.py && python visualize_forecast.py

# Anomaly detection (uses TimesFM 1.0)
cd examples/anomaly-detection && python detect_anomalies.py

# Covariates (API demo -- requires TimesFM 2.5 + timesfm[xreg] for real inference)
cd examples/covariates-forecasting && python demo_covariates.py

Expected Outputs

Quality Checklist

Run this checklist after every TimesFM task before declaring success:

• Output shape correct -- point_fc shape is (n_series, horizon), quant_fc is (n_series, horizon, 10)
• Quantile indices -- index 0 = mean, 1 = q10, 2 = q20 ... 9 = q90. NOT 0 = q0, 1 = q10.
• Frequency flag -- TimesFM 1.0/2.0: pass freq=[0] for monthly data. TimesFM 2.5: no freq flag.
• Series length -- context must be >= 32 data points (model minimum). Warn if shorter.
• No NaN -- np.isnan(point_fc).any() should be False. Check input series for gaps first.
• Visualization axes -- if multiple panels share data, use sharex=True. All time axes must cover the same span.
• Binary outputs in Git LFS -- PNG and GIF files must be tracked via .gitattributes (repo root already configured).
• No large datasets committed -- any real dataset > 1 MB should be downloaded to tempfile.mkdtemp() and annotated in code.
• matplotlib.use('Agg') -- must appear before any pyplot import when running headless.
• infer_is_positive -- set False for temperature anomalies, financial returns, or any series that can be negative.

Common Mistakes

These bugs have appeared in this skill's examples. Learn from them:

1. Quantile index off-by-one -- The most common mistake. quant_fc[..., 0] is the mean, not q0. q10 = index 1, q90 = index 9. Always define named constants: IDX_Q10, IDX_Q20, IDX_Q80, IDX_Q90 = 1, 2, 8, 9.

2. Variable shadowing in comprehensions -- If you build per-series covariate dicts inside a loop, do NOT use the loop variable as the comprehension variable. Accumulate into separate dict[str, ndarray] outside the loop, then assign.

   # WRONG -- outer `store_id` gets shadowed:
   covariates = {store_id: arr[store_id] for store_id in stores}  # inside outer loop over store_id
   # CORRECT -- use a different name or accumulate beforehand:
   prices_by_store: dict[str, np.ndarray] = {}
   for store_id, config in stores.items():
       prices_by_store[store_id] = compute_price(config)
   

3. Wrong CSV column name -- The global-temperature CSV uses anomaly_c, not anomaly. Always print(df.columns) before accessing.

4. tight_layout() warning with sharex=True -- Harmless; suppress with plt.tight_layout(rect=[0, 0, 1, 0.97]) or ignore.

5. TimesFM 2.5 required for forecast_with_covariates() -- TimesFM 1.0 does NOT have this method. Install pip install timesfm[xreg] and use checkpoint google/timesfm-2.5-200m-pytorch.

6. Future covariates must span the full horizon -- Dynamic covariates (price, promotions, holidays) must have values for BOTH the context AND the forecast horizon. You cannot pass context-only arrays.

7. Anomaly thresholds must be defined once -- Define CRITICAL_Z = 3.0, WARNING_Z = 2.0 as module-level constants. Never hardcode 3 or 2 inline.

8. Context anomaly detection uses residuals, not raw values -- Always detrend first (np.polyfit linear, or seasonal decomposition), then Z-score the residuals. Raw-value Z-scores are misleading on trending data.

Validation & Verification

Use the example outputs as regression baselines. If you change forecasting logic, verify:

# Anomaly detection regression check:
python -c "
import json
d = json.load(open('examples/anomaly-detection/output/anomaly_detection.json'))
ctx = d['context_summary']
assert ctx['critical'] >= 1, 'Sep 2023 must be CRITICAL'
assert any(r['date'] == '2023-09' and r['severity'] == 'CRITICAL'
           for r in d['context_detections']), 'Sep 2023 not found'
print('Anomaly detection regression: PASS')"

# Covariates regression check:
python -c "
import pandas as pd
df = pd.read_csv('examples/covariates-forecasting/output/sales_with_covariates.csv')
assert len(df) == 108, f'Expected 108 rows, got {len(df)}'
prices = df.groupby('store_id')['price'].mean()
assert prices['store_A'] > prices['store_B'] > prices['store_C'], 'Store price ordering wrong'
print('Covariates regression: PASS')"

Office Academic Skill

Scope

Use this skill for:

• Word reports from PDFs, DOCX files, arXiv papers, journal articles, theses, and manuscripts.
• Chinese-first academic PPTs for literature reports, group meetings, courses, opening/midterm/defense presentations, and project presentations.
• Editable .docx and .pptx generation, inspection, repair, and style preservation.
• PPT template matching, native slide editing, speaker notes, and visual quality checks.

Do not use this skill for pure manuscript prose drafting without a Word/PPT deliverable; use research-writing-skill instead. Do not use it for MATLAB, Python analysis, statistics, or plotting unless those outputs are being inserted into Word/PPT.

Language And Evidence

• Default to Chinese for explanations, Word report prose, slide text, outlines, and speaker notes.
• Preserve English titles, formulas, variables, model names, software commands, reference entries, and direct source labels.
• Distinguish 论文原文, 图表/公式证据, 代码或仿真结果, 根据上下文推断, and 建议.
• Do not invent DOI, authors, journal details, experiment values, figure numbers, section names, page numbers, or conclusions.
• Attach source labels to claims, parameters, quantitative results, formula explanations, datasets, figures, limitations, and novelty statements.

Paper Reading To Word

Default output, unless the user asks otherwise:

1. A bilingual English-Chinese report for fast browsing.
2. A Chinese-only report for submission, teaching, or presentation preparation.
3. Optional Markdown working notes if useful.

Before writing, build a source map:

• Title, authors, venue, year, DOI/arXiv if present.
• Section headings and page spans when available.
• Figures, tables, equations, datasets, hardware/software, and evaluation settings that support key claims.
• Uncertain or missing metadata marked as 未在原文中明确给出.

Use references/report-structure.md for the default report structure and evidence-label format.

For .docx creation or editing:

• Prefer structured headings, summary tables, figure/table placeholders, and source labels.
• Use reliable Chinese fonts such as Microsoft YaHei or SimSun; use Times New Roman, Calibri, or Arial for English and numbers.
• For existing academic/legal/business Word documents, make a new version or use tracked-change style edits rather than overwriting the original.
• For advanced DOCX operations, use references/office-docx/ooxml.md, references/office-docx/docx-js.md, and the scripts under references/office-docx/.

Academic PPT Workflow

First clarify only the high-impact missing details:

• Purpose: literature report, group meeting, course report, opening/midterm/defense, project display, science communication, or other.
• Duration and slide count.
• Audience and evaluation criteria.
• Required template, school/company constraints, fonts, ratio, logo, sections, notes, or output format.
• Source files: paper, thesis, Word draft, data, MATLAB/Python/Origin figures, screenshots, old PPT, template.

If the user asks to proceed immediately, make reasonable defaults and state them briefly.

For research PPTs, use a concise structure:

1. Cover.
2. Research background and problem.
3. Related work or theoretical basis.
4. Method, model, system, or algorithm.
5. Experiment/simulation setup.
6. Results and analysis.
7. Comparison and discussion.
8. Contributions, limitations, and outlook.
9. Q&A.

For paper-reading PPTs, use:

1. Paper metadata.
2. Background.
3. Core problem.
4. Method framework.
5. Experiment setup.
6. Main results.
7. Contributions.
8. Limitations.
9. Possible improvements.
10. Relationship to the user's topic.

Slide Quality Rules

• One core point per slide.
• Prefer action titles that state the conclusion, not vague topic labels.
• Figures, diagrams, tables, and formulas should carry the technical argument; avoid long paragraphs.
• Keep axes, units, legends, formulas, assumptions, data sources, and figure captions scientifically accurate.
• Use white or restrained academic backgrounds unless a supplied template requires otherwise.
• Limit colors and decoration; use color to direct attention to evidence.
• Avoid text overflow, image stretching, Chinese garbling, missing fonts, stale template text, bad navigation labels, and overlapping elements.

The academic-pptx repository was reviewed as an external reference. Because it marks its license as proprietary, do not copy its text into outputs or this skill. Use only general academic presentation principles: argument-first structure, action titles, evidence-led slides, and the ghost-deck test.

PPTX Technical Work

For template-matched defense PPTs:

• Prefer copying native template slides and replacing content rather than rebuilding from blank slides.
• On Windows with Microsoft PowerPoint installed, PowerPoint COM can be used for cloning, export, and overflow inspection.
• Never modify the user's original PPTX directly. Work on a timestamped or versioned copy.
• Do not disable PowerPoint add-ins or change application settings unless the user explicitly approves in that task.

Useful bundled resources:

• references/thesis-defense-pptx/scripts/ for thesis context extraction, template cloning, slide export, contact sheets, text scans, and overflow inspection.
• references/office-pptx/ for OOXML-level PPTX inspection and editing.
• references/office-docx/ for OOXML-level DOCX inspection and editing.

Quality Gate

Before final delivery, verify what is feasible:

• For Word: inspect extracted text or package XML for missing text, garbled Chinese, broken images, table overflow, and source labels.
• For PPT: export or inspect slides, check page order, stale placeholders, text overflow, image aspect ratio, overlap, and readability.
• Report output file paths, source paths, extraction method, checks performed, and unresolved uncertainties.

Exploratory Data Analysis

Overview

Perform comprehensive exploratory data analysis (EDA) on scientific data files across multiple domains. This skill provides automated file type detection, format-specific analysis, data quality assessment, and generates detailed markdown reports suitable for documentation and downstream analysis planning.

Key Capabilities:

• Automatic detection and analysis of 200+ scientific file formats
• Comprehensive format-specific metadata extraction
• Data quality and integrity assessment
• Statistical summaries and distributions
• Visualization recommendations
• Downstream analysis suggestions
• Markdown report generation

When to Use This Skill

Use this skill when:

• User provides a path to a scientific data file for analysis
• User asks to "explore", "analyze", or "summarize" a data file
• User wants to understand the structure and content of scientific data
• User needs a comprehensive report of a dataset before analysis
• User wants to assess data quality or completeness
• User asks what type of analysis is appropriate for a file

Supported File Categories

The skill has comprehensive coverage of scientific file formats organized into six major categories:

1. Chemistry and Molecular Formats (60+ extensions)

Structure files, computational chemistry outputs, molecular dynamics trajectories, and chemical databases.

File types include: .pdb, .cif, .mol, .mol2, .sdf, .xyz, .smi, .gro, .log, .fchk, .cube, .dcd, .xtc, .trr, .prmtop, .psf, and more.

Reference file: references/chemistry_molecular_formats.md

2. Bioinformatics and Genomics Formats (50+ extensions)

Sequence data, alignments, annotations, variants, and expression data.

File types include: .fasta, .fastq, .sam, .bam, .vcf, .bed, .gff, .gtf, .bigwig, .h5ad, .loom, .counts, .mtx, and more.

Reference file: references/bioinformatics_genomics_formats.md

3. Microscopy and Imaging Formats (45+ extensions)

Microscopy images, medical imaging, whole slide imaging, and electron microscopy.

File types include: .tif, .nd2, .lif, .czi, .ims, .dcm, .nii, .mrc, .dm3, .vsi, .svs, .ome.tiff, and more.

Reference file: references/microscopy_imaging_formats.md

4. Spectroscopy and Analytical Chemistry Formats (35+ extensions)

NMR, mass spectrometry, IR/Raman, UV-Vis, X-ray, chromatography, and other analytical techniques.

File types include: .fid, .mzML, .mzXML, .raw, .mgf, .spc, .jdx, .xy, .cif (crystallography), .wdf, and more.

Reference file: references/spectroscopy_analytical_formats.md

5. Proteomics and Metabolomics Formats (30+ extensions)

Mass spec proteomics, metabolomics, lipidomics, and multi-omics data.

File types include: .mzML, .pepXML, .protXML, .mzid, .mzTab, .sky, .mgf, .msp, .h5ad, and more.

Reference file: references/proteomics_metabolomics_formats.md

6. General Scientific Data Formats (30+ extensions)

Arrays, tables, hierarchical data, compressed archives, and common scientific formats.

File types include: .npy, .npz, .csv, .xlsx, .json, .hdf5, .zarr, .parquet, .mat, .fits, .nc, .xml, and more.

Reference file: references/general_scientific_formats.md

Workflow

Step 1: File Type Detection

When a user provides a file path, first identify the file type:

1. Extract the file extension
2. Look up the extension in the appropriate reference file
3. Identify the file category and format description
4. Load format-specific information

Example:

User: "Analyze data.fastq"
→ Extension: .fastq
→ Category: bioinformatics_genomics
→ Format: FASTQ Format (sequence data with quality scores)
→ Reference: references/bioinformatics_genomics_formats.md

Step 2: Load Format-Specific Information

Based on the file type, read the corresponding reference file to understand:

• Typical Data: What kind of data this format contains
• Use Cases: Common applications for this format
• Python Libraries: How to read the file in Python
• EDA Approach: What analyses are appropriate for this data type

Search the reference file for the specific extension (e.g., search for "### .fastq" in bioinformatics_genomics_formats.md).

Step 3: Perform Data Analysis

Use the scripts/eda_analyzer.py script OR implement custom analysis:

Option A: Use the analyzer script

# The script automatically:
# 1. Detects file type
# 2. Loads reference information
# 3. Performs format-specific analysis
# 4. Generates markdown report

python scripts/eda_analyzer.py <filepath> [output.md]

Option B: Custom analysis in the conversation Based on the format information from the reference file, perform appropriate analysis:

For tabular data (CSV, TSV, Excel):

• Load with pandas
• Check dimensions, data types
• Analyze missing values
• Calculate summary statistics
• Identify outliers
• Check for duplicates

For sequence data (FASTA, FASTQ):

• Count sequences
• Analyze length distributions
• Calculate GC content
• Assess quality scores (FASTQ)

For images (TIFF, ND2, CZI):

• Check dimensions (X, Y, Z, C, T)
• Analyze bit depth and value range
• Extract metadata (channels, timestamps, spatial calibration)
• Calculate intensity statistics

For arrays (NPY, HDF5):

• Check shape and dimensions
• Analyze data type
• Calculate statistical summaries
• Check for missing/invalid values

Step 4: Generate Comprehensive Report

Create a markdown report with the following sections:

Required Sections:

1. Title and Metadata

   • Filename and timestamp
   • File size and location

2. Basic Information

   • File properties
   • Format identification

3. File Type Details

   • Format description from reference
   • Typical data content
   • Common use cases
   • Python libraries for reading

4. Data Analysis

   • Structure and dimensions
   • Statistical summaries
   • Quality assessment
   • Data characteristics

5. Key Findings

   • Notable patterns
   • Potential issues
   • Quality metrics

6. Recommendations

   • Preprocessing steps
   • Appropriate analyses
   • Tools and methods
   • Visualization approaches

Template Location

Use assets/report_template.md as a guide for report structure.

Step 5: Save Report

Save the markdown report with a descriptive filename:

• Pattern: {original_filename}_eda_report.md
• Example: experiment_data.fastq → experiment_data_eda_report.md

Detailed Format References

Each reference file contains comprehensive information for dozens of file types. To find information about a specific format:

1. Identify the category from the extension
2. Read the appropriate reference file
3. Search for the section heading matching the extension (e.g., "### .pdb")
4. Extract the format information

Reference File Structure

Each format entry includes:

• Description: What the format is
• Typical Data: What it contains
• Use Cases: Common applications
• Python Libraries: How to read it (with code examples)
• EDA Approach: Specific analyses to perform

Example lookup:

### .pdb - Protein Data Bank
**Description:** Standard format for 3D structures of biological macromolecules
**Typical Data:** Atomic coordinates, residue information, secondary structure
**Use Cases:** Protein structure analysis, molecular visualization, docking
**Python Libraries:**
- `Biopython`: `Bio.PDB`
- `MDAnalysis`: `MDAnalysis.Universe('file.pdb')`
**EDA Approach:**
- Structure validation (bond lengths, angles)
- B-factor distribution
- Missing residues detection
- Ramachandran plots

Best Practices

Reading Reference Files

Reference files are large (10,000+ words each). To efficiently use them:

1. Search by extension: Use grep to find the specific format

   import re
   with open('references/chemistry_molecular_formats.md', 'r') as f:
       content = f.read()
       pattern = r'### \.pdb[^#]*?(?=###|\Z)'
       match = re.search(pattern, content, re.IGNORECASE | re.DOTALL)
   

2. Extract relevant sections: Don't load entire reference files into context unnecessarily

3. Cache format info: If analyzing multiple files of the same type, reuse the format information

Data Analysis

1. Sample large files: For files with millions of records, analyze a representative sample
2. Handle errors gracefully: Many scientific formats require specific libraries; provide clear installation instructions
3. Validate metadata: Cross-check metadata consistency (e.g., stated dimensions vs actual data)
4. Consider data provenance: Note instrument, software versions, processing steps

Report Generation

1. Be comprehensive: Include all relevant information for downstream analysis
2. Be specific: Provide concrete recommendations based on the file type
3. Be actionable: Suggest specific next steps and tools
4. Include code examples: Show how to load and work with the data

Examples

Example 1: Analyzing a FASTQ file

# User provides: "Analyze reads.fastq"

# 1. Detect file type
extension = '.fastq'
category = 'bioinformatics_genomics'

# 2. Read reference info
# Search references/bioinformatics_genomics_formats.md for "### .fastq"

# 3. Perform analysis
from Bio import SeqIO
sequences = list(SeqIO.parse('reads.fastq', 'fastq'))
# Calculate: read count, length distribution, quality scores, GC content

# 4. Generate report
# Include: format description, analysis results, QC recommendations

# 5. Save as: reads_eda_report.md

Example 2: Analyzing a CSV dataset

# User provides: "Explore experiment_results.csv"

# 1. Detect: .csv → general_scientific

# 2. Load reference for CSV format

# 3. Analyze
import pandas as pd
df = pd.read_csv('experiment_results.csv')
# Dimensions, dtypes, missing values, statistics, correlations

# 4. Generate report with:
# - Data structure
# - Missing value patterns
# - Statistical summaries
# - Correlation matrix
# - Outlier detection results

# 5. Save report

Example 3: Analyzing microscopy data

# User provides: "Analyze cells.nd2"

# 1. Detect: .nd2 → microscopy_imaging (Nikon format)

# 2. Read reference for ND2 format
# Learn: multi-dimensional (XYZCT), requires nd2reader

# 3. Analyze
from nd2reader import ND2Reader
with ND2Reader('cells.nd2') as images:
    # Extract: dimensions, channels, timepoints, metadata
    # Calculate: intensity statistics, frame info

# 4. Generate report with:
# - Image dimensions (XY, Z-stacks, time, channels)
# - Channel wavelengths
# - Pixel size and calibration
# - Recommendations for image analysis

# 5. Save report

Troubleshooting

Missing Libraries

Many scientific formats require specialized libraries:

Problem: Import error when trying to read a file

Solution: Provide clear installation instructions

try:
    from Bio import SeqIO
except ImportError:
    print("Install Biopython: uv pip install biopython")

Common requirements by category:

• Bioinformatics: biopython, pysam, pyBigWig
• Chemistry: rdkit, mdanalysis, cclib
• Microscopy: tifffile, nd2reader, aicsimageio, pydicom
• Spectroscopy: nmrglue, pymzml, pyteomics
• General: pandas, numpy, h5py, scipy

Unknown File Types

If a file extension is not in the references:

1. Ask the user about the file format
2. Check if it's a vendor-specific variant
3. Attempt generic analysis based on file structure (text vs binary)
4. Provide general recommendations

Large Files

For very large files:

1. Use sampling strategies (first N records)
2. Use memory-mapped access (for HDF5, NPY)
3. Process in chunks (for CSV, FASTQ)
4. Provide estimates based on samples

Script Usage

The scripts/eda_analyzer.py can be used directly:

# Basic usage
python scripts/eda_analyzer.py data.csv

# Specify output file
python scripts/eda_analyzer.py data.csv output_report.md

# The script will:
# 1. Auto-detect file type
# 2. Load format references
# 3. Perform appropriate analysis
# 4. Generate markdown report

The script supports automatic analysis for many common formats, but custom analysis in the conversation provides more flexibility and domain-specific insights.

Advanced Usage

Multi-File Analysis

When analyzing multiple related files:

1. Perform individual EDA on each file
2. Create a summary comparison report
3. Identify relationships and dependencies
4. Suggest integration strategies

Quality Control

For data quality assessment:

1. Check format compliance
2. Validate metadata consistency
3. Assess completeness
4. Identify outliers and anomalies
5. Compare to expected ranges/distributions

Preprocessing Recommendations

Based on data characteristics, recommend:

1. Normalization strategies
2. Missing value imputation
3. Outlier handling
4. Batch correction
5. Format conversions

Resources

scripts/

• eda_analyzer.py: Comprehensive analysis script that can be run directly or imported

references/

• chemistry_molecular_formats.md: 60+ chemistry/molecular file formats
• bioinformatics_genomics_formats.md: 50+ bioinformatics formats
• microscopy_imaging_formats.md: 45+ imaging formats
• spectroscopy_analytical_formats.md: 35+ spectroscopy formats
• proteomics_metabolomics_formats.md: 30+ omics formats
• general_scientific_formats.md: 30+ general formats

assets/

• report_template.md: Comprehensive markdown template for EDA reports

MATLAB/Octave Scientific Computing

MATLAB is a numerical computing environment optimized for matrix operations and scientific computing. GNU Octave is a free, open-source alternative with high MATLAB compatibility.

Quick Start

Running MATLAB scripts:

# MATLAB (commercial)
matlab -nodisplay -nosplash -r "run('script.m'); exit;"

# GNU Octave (free, open-source)
octave script.m

Install GNU Octave:

# macOS
brew install octave

# Ubuntu/Debian
sudo apt install octave

# Windows - download from https://octave.org/download

Core Capabilities

1. Matrix Operations

MATLAB operates fundamentally on matrices and arrays:

% Create matrices
A = [1 2 3; 4 5 6; 7 8 9];  % 3x3 matrix
v = 1:10;                     % Row vector 1 to 10
v = linspace(0, 1, 100);      % 100 points from 0 to 1

% Special matrices
I = eye(3);          % Identity matrix
Z = zeros(3, 4);     % 3x4 zero matrix
O = ones(2, 3);      % 2x3 ones matrix
R = rand(3, 3);      % Random uniform
N = randn(3, 3);     % Random normal

% Matrix operations
B = A';              % Transpose
C = A * B;           % Matrix multiplication
D = A .* B;          % Element-wise multiplication
E = A \ b;           % Solve linear system Ax = b
F = inv(A);          % Matrix inverse

For complete matrix operations, see references/matrices-arrays.md.

2. Linear Algebra

% Eigenvalues and eigenvectors
[V, D] = eig(A);     % V: eigenvectors, D: diagonal eigenvalues

% Singular value decomposition
[U, S, V] = svd(A);

% Matrix decompositions
[L, U] = lu(A);      % LU decomposition
[Q, R] = qr(A);      % QR decomposition
R = chol(A);         % Cholesky (symmetric positive definite)

% Solve linear systems
x = A \ b;           % Preferred method
x = linsolve(A, b);  % With options
x = inv(A) * b;      % Less efficient

For comprehensive linear algebra, see references/mathematics.md.

3. Plotting and Visualization

% 2D Plots
x = 0:0.1:2*pi;
y = sin(x);
plot(x, y, 'b-', 'LineWidth', 2);
xlabel('x'); ylabel('sin(x)');
title('Sine Wave');
grid on;

% Multiple plots
hold on;
plot(x, cos(x), 'r--');
legend('sin', 'cos');
hold off;

% 3D Surface
[X, Y] = meshgrid(-2:0.1:2, -2:0.1:2);
Z = X.^2 + Y.^2;
surf(X, Y, Z);
colorbar;

% Save figures
saveas(gcf, 'plot.png');
print('-dpdf', 'plot.pdf');

For complete visualization guide, see references/graphics-visualization.md.

4. Data Import/Export

% Read tabular data
T = readtable('data.csv');
M = readmatrix('data.csv');

% Write data
writetable(T, 'output.csv');
writematrix(M, 'output.csv');

% MAT files (MATLAB native)
save('data.mat', 'A', 'B', 'C');  % Save variables
load('data.mat');                   % Load all
S = load('data.mat', 'A');         % Load specific

% Images
img = imread('image.png');
imwrite(img, 'output.jpg');

For complete I/O guide, see references/data-import-export.md.

5. Control Flow and Functions

% Conditionals
if x > 0
    disp('positive');
elseif x < 0
    disp('negative');
else
    disp('zero');
end

% Loops
for i = 1:10
    disp(i);
end

while x > 0
    x = x - 1;
end

% Functions (in separate .m file or same file)
function y = myfunction(x, n)
    y = x.^n;
end

% Anonymous functions
f = @(x) x.^2 + 2*x + 1;
result = f(5);  % 36

For complete programming guide, see references/programming.md.

6. Statistics and Data Analysis

% Descriptive statistics
m = mean(data);
s = std(data);
v = var(data);
med = median(data);
[minVal, minIdx] = min(data);
[maxVal, maxIdx] = max(data);

% Correlation
R = corrcoef(X, Y);
C = cov(X, Y);

% Linear regression
p = polyfit(x, y, 1);  % Linear fit
y_fit = polyval(p, x);

% Moving statistics
y_smooth = movmean(y, 5);  % 5-point moving average

For statistics reference, see references/mathematics.md.

7. Differential Equations

% ODE solving
% dy/dt = -2y, y(0) = 1
f = @(t, y) -2*y;
[t, y] = ode45(f, [0 5], 1);
plot(t, y);

% Higher-order: y'' + 2y' + y = 0
% Convert to system: y1' = y2, y2' = -2*y2 - y1
f = @(t, y) [y(2); -2*y(2) - y(1)];
[t, y] = ode45(f, [0 10], [1; 0]);

For ODE solvers guide, see references/mathematics.md.

8. Signal Processing

% FFT
Y = fft(signal);
f = (0:length(Y)-1) * fs / length(Y);
plot(f, abs(Y));

% Filtering
b = fir1(50, 0.3);           % FIR filter design
y_filtered = filter(b, 1, signal);

% Convolution
y = conv(x, h, 'same');

For signal processing, see references/mathematics.md.

Common Patterns

Pattern 1: Data Analysis Pipeline

% Load data
data = readtable('experiment.csv');

% Clean data
data = rmmissing(data);  % Remove missing values

% Analyze
grouped = groupsummary(data, 'Category', 'mean', 'Value');

% Visualize
figure;
bar(grouped.Category, grouped.mean_Value);
xlabel('Category'); ylabel('Mean Value');
title('Results by Category');

% Save
writetable(grouped, 'results.csv');
saveas(gcf, 'results.png');

Pattern 2: Numerical Simulation

% Parameters
L = 1; N = 100; T = 10; dt = 0.01;
x = linspace(0, L, N);
dx = x(2) - x(1);

% Initial condition
u = sin(pi * x);

% Time stepping (heat equation)
for t = 0:dt:T
    u_new = u;
    for i = 2:N-1
        u_new(i) = u(i) + dt/(dx^2) * (u(i+1) - 2*u(i) + u(i-1));
    end
    u = u_new;
end

plot(x, u);

Pattern 3: Batch Processing

% Process multiple files
files = dir('data/*.csv');
results = cell(length(files), 1);

for i = 1:length(files)
    data = readtable(fullfile(files(i).folder, files(i).name));
    results{i} = analyze(data);  % Custom analysis function
end

% Combine results
all_results = vertcat(results{:});

Reference Files

• matrices-arrays.md - Matrix creation, indexing, manipulation, and operations
• mathematics.md - Linear algebra, calculus, ODEs, optimization, statistics
• graphics-visualization.md - 2D/3D plotting, customization, export
• data-import-export.md - File I/O, tables, data formats
• programming.md - Functions, scripts, control flow, OOP
• python-integration.md - Calling Python from MATLAB and vice versa
• octave-compatibility.md - Differences between MATLAB and GNU Octave
• executing-scripts.md - Executing generated scripts and for testing

GNU Octave Compatibility

GNU Octave is highly compatible with MATLAB. Most scripts work without modification. Key differences:

• Use # or % for comments (MATLAB only %)
• Octave allows ++, --, += operators
• Some toolbox functions unavailable in Octave
• Use pkg load for Octave packages

For complete compatibility guide, see references/octave-compatibility.md.

Best Practices

1. Vectorize operations - Avoid loops when possible:

   % Slow
   for i = 1:1000
       y(i) = sin(x(i));
   end
   
   % Fast
   y = sin(x);
   

2. Preallocate arrays - Avoid growing arrays in loops:

   % Slow
   for i = 1:1000
       y(i) = i^2;
   end
   
   % Fast
   y = zeros(1, 1000);
   for i = 1:1000
       y(i) = i^2;
   end
   

3. Use appropriate data types - Tables for mixed data, matrices for numeric:

   % Numeric data
   M = readmatrix('numbers.csv');
   
   % Mixed data with headers
   T = readtable('mixed.csv');
   

4. Comment and document - Use function help:

   function y = myfunction(x)
   %MYFUNCTION Brief description
   %   Y = MYFUNCTION(X) detailed description
   %
   %   Example:
   %       y = myfunction(5);
       y = x.^2;
   end
   

Additional Resources