Use this when

• Editing file I/O or scans in backend/cli (the OpenScience CLI package)
• Handling directory operations or external tools

Bun file APIs (from Bun docs)

• Bun.file(path) is lazy; call text, json, stream, arrayBuffer, bytes, exists to read.
• Metadata: file.size, file.type, file.name.
• Bun.write(dest, input) writes strings, buffers, Blobs, Responses, or files.
• Bun.file(...).delete() deletes a file.
• file.writer() returns a FileSink for incremental writes.
• Bun.Glob + Array.fromAsync(glob.scan({ cwd, absolute, onlyFiles, dot })) for scans.
• Use Bun.which to find a binary, then Bun.spawn to run it.
• Bun.readableStreamToText/Bytes/JSON for stream output.

When to use node:fs

• Use node:fs/promises for directories (mkdir, readdir, recursive operations).

Repo patterns

• Prefer Bun APIs over Node fs for file access.
• Check Bun.file(...).exists() before reading.
• For binary/large files use arrayBuffer() and MIME checks via file.type.
• Use Bun.Glob + Array.fromAsync for scans.
• Decode tool stderr with Bun.readableStreamToText.
• For large writes, use Bun.write(Bun.file(path), text).

Quick checklist

• Use Bun APIs first.
• Use path.join/path.resolve for paths.
• Prefer promise .catch(...) over try/catch when possible.

AnnData

Overview

AnnData is a Python package for handling annotated data matrices, storing experimental measurements (X) alongside observation metadata (obs), variable metadata (var), and multi-dimensional annotations (obsm, varm, obsp, varp, uns). Originally designed for single-cell genomics through Scanpy, it now serves as a general-purpose framework for any annotated data requiring efficient storage, manipulation, and analysis.

When to Use This Skill

Use this skill when:

• Creating, reading, or writing AnnData objects
• Working with h5ad, zarr, or other genomics data formats
• Performing single-cell RNA-seq analysis
• Managing large datasets with sparse matrices or backed mode
• Concatenating multiple datasets or experimental batches
• Subsetting, filtering, or transforming annotated data
• Integrating with scanpy, scvi-tools, or other scverse ecosystem tools

Installation

uv pip install anndata

# With optional dependencies
uv pip install anndata[dev,test,doc]

Quick Start

Creating an AnnData object

import anndata as ad
import numpy as np
import pandas as pd

# Minimal creation
X = np.random.rand(100, 2000)  # 100 cells × 2000 genes
adata = ad.AnnData(X)

# With metadata
obs = pd.DataFrame({
    'cell_type': ['T cell', 'B cell'] * 50,
    'sample': ['A', 'B'] * 50
}, index=[f'cell_{i}' for i in range(100)])

var = pd.DataFrame({
    'gene_name': [f'Gene_{i}' for i in range(2000)]
}, index=[f'ENSG{i:05d}' for i in range(2000)])

adata = ad.AnnData(X=X, obs=obs, var=var)

Reading data

# Read h5ad file
adata = ad.read_h5ad('data.h5ad')

# Read with backed mode (for large files)
adata = ad.read_h5ad('large_data.h5ad', backed='r')

# Read other formats
adata = ad.read_csv('data.csv')
adata = ad.read_loom('data.loom')
adata = ad.read_10x_h5('filtered_feature_bc_matrix.h5')

Writing data

# Write h5ad file
adata.write_h5ad('output.h5ad')

# Write with compression
adata.write_h5ad('output.h5ad', compression='gzip')

# Write other formats
adata.write_zarr('output.zarr')
adata.write_csvs('output_dir/')

Basic operations

# Subset by conditions
t_cells = adata[adata.obs['cell_type'] == 'T cell']

# Subset by indices
subset = adata[0:50, 0:100]

# Add metadata
adata.obs['quality_score'] = np.random.rand(adata.n_obs)
adata.var['highly_variable'] = np.random.rand(adata.n_vars) > 0.8

# Access dimensions
print(f"{adata.n_obs} observations × {adata.n_vars} variables")

Core Capabilities

1. Data Structure

Understand the AnnData object structure including X, obs, var, layers, obsm, varm, obsp, varp, uns, and raw components.

See: references/data_structure.md for comprehensive information on:

• Core components (X, obs, var, layers, obsm, varm, obsp, varp, uns, raw)
• Creating AnnData objects from various sources
• Accessing and manipulating data components
• Memory-efficient practices

2. Input/Output Operations

Read and write data in various formats with support for compression, backed mode, and cloud storage.

See: references/io_operations.md for details on:

• Native formats (h5ad, zarr)
• Alternative formats (CSV, MTX, Loom, 10X, Excel)
• Backed mode for large datasets
• Remote data access
• Format conversion
• Performance optimization

Common commands:

# Read/write h5ad
adata = ad.read_h5ad('data.h5ad', backed='r')
adata.write_h5ad('output.h5ad', compression='gzip')

# Read 10X data
adata = ad.read_10x_h5('filtered_feature_bc_matrix.h5')

# Read MTX format
adata = ad.read_mtx('matrix.mtx').T

3. Concatenation

Combine multiple AnnData objects along observations or variables with flexible join strategies.

See: references/concatenation.md for comprehensive coverage of:

• Basic concatenation (axis=0 for observations, axis=1 for variables)
• Join types (inner, outer)
• Merge strategies (same, unique, first, only)
• Tracking data sources with labels
• Lazy concatenation (AnnCollection)
• On-disk concatenation for large datasets

Common commands:

# Concatenate observations (combine samples)
adata = ad.concat(
    [adata1, adata2, adata3],
    axis=0,
    join='inner',
    label='batch',
    keys=['batch1', 'batch2', 'batch3']
)

# Concatenate variables (combine modalities)
adata = ad.concat([adata_rna, adata_protein], axis=1)

# Lazy concatenation
from anndata.experimental import AnnCollection
collection = AnnCollection(
    ['data1.h5ad', 'data2.h5ad'],
    join_obs='outer',
    label='dataset'
)

4. Data Manipulation

Transform, subset, filter, and reorganize data efficiently.

See: references/manipulation.md for detailed guidance on:

• Subsetting (by indices, names, boolean masks, metadata conditions)
• Transposition
• Copying (full copies vs views)
• Renaming (observations, variables, categories)
• Type conversions (strings to categoricals, sparse/dense)
• Adding/removing data components
• Reordering
• Quality control filtering

Common commands:

# Subset by metadata
filtered = adata[adata.obs['quality_score'] > 0.8]
hv_genes = adata[:, adata.var['highly_variable']]

# Transpose
adata_T = adata.T

# Copy vs view
view = adata[0:100, :]  # View (lightweight reference)
copy = adata[0:100, :].copy()  # Independent copy

# Convert strings to categoricals
adata.strings_to_categoricals()

5. Best Practices

Follow recommended patterns for memory efficiency, performance, and reproducibility.

See: references/best_practices.md for guidelines on:

• Memory management (sparse matrices, categoricals, backed mode)
• Views vs copies
• Data storage optimization
• Performance optimization
• Working with raw data
• Metadata management
• Reproducibility
• Error handling
• Integration with other tools
• Common pitfalls and solutions

Key recommendations:

# Use sparse matrices for sparse data
from scipy.sparse import csr_matrix
adata.X = csr_matrix(adata.X)

# Convert strings to categoricals
adata.strings_to_categoricals()

# Use backed mode for large files
adata = ad.read_h5ad('large.h5ad', backed='r')

# Store raw before filtering
adata.raw = adata.copy()
adata = adata[:, adata.var['highly_variable']]

Integration with Scverse Ecosystem

AnnData serves as the foundational data structure for the scverse ecosystem:

Scanpy (Single-cell analysis)

import scanpy as sc

# Preprocessing
sc.pp.filter_cells(adata, min_genes=200)
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000)

# Dimensionality reduction
sc.pp.pca(adata, n_comps=50)
sc.pp.neighbors(adata, n_neighbors=15)
sc.tl.umap(adata)
sc.tl.leiden(adata)

# Visualization
sc.pl.umap(adata, color=['cell_type', 'leiden'])

Muon (Multimodal data)

import muon as mu

# Combine RNA and protein data
mdata = mu.MuData({'rna': adata_rna, 'protein': adata_protein})

PyTorch integration

from anndata.experimental import AnnLoader

# Create DataLoader for deep learning
dataloader = AnnLoader(adata, batch_size=128, shuffle=True)

for batch in dataloader:
    X = batch.X
    # Train model

Common Workflows

Single-cell RNA-seq analysis

import anndata as ad
import scanpy as sc

# 1. Load data
adata = ad.read_10x_h5('filtered_feature_bc_matrix.h5')

# 2. Quality control
adata.obs['n_genes'] = (adata.X > 0).sum(axis=1)
adata.obs['n_counts'] = adata.X.sum(axis=1)
adata = adata[adata.obs['n_genes'] > 200]
adata = adata[adata.obs['n_counts'] < 50000]

# 3. Store raw
adata.raw = adata.copy()

# 4. Normalize and filter
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000)
adata = adata[:, adata.var['highly_variable']]

# 5. Save processed data
adata.write_h5ad('processed.h5ad')

Batch integration

# Load multiple batches
adata1 = ad.read_h5ad('batch1.h5ad')
adata2 = ad.read_h5ad('batch2.h5ad')
adata3 = ad.read_h5ad('batch3.h5ad')

# Concatenate with batch labels
adata = ad.concat(
    [adata1, adata2, adata3],
    label='batch',
    keys=['batch1', 'batch2', 'batch3'],
    join='inner'
)

# Apply batch correction
import scanpy as sc
sc.pp.combat(adata, key='batch')

# Continue analysis
sc.pp.pca(adata)
sc.pp.neighbors(adata)
sc.tl.umap(adata)

Working with large datasets

# Open in backed mode
adata = ad.read_h5ad('100GB_dataset.h5ad', backed='r')

# Filter based on metadata (no data loading)
high_quality = adata[adata.obs['quality_score'] > 0.8]

# Load filtered subset
adata_subset = high_quality.to_memory()

# Process subset
process(adata_subset)

# Or process in chunks
chunk_size = 1000
for i in range(0, adata.n_obs, chunk_size):
    chunk = adata[i:i+chunk_size, :].to_memory()
    process(chunk)

Troubleshooting

Out of memory errors

Use backed mode or convert to sparse matrices:

# Backed mode
adata = ad.read_h5ad('file.h5ad', backed='r')

# Sparse matrices
from scipy.sparse import csr_matrix
adata.X = csr_matrix(adata.X)

Slow file reading

Use compression and appropriate formats:

# Optimize for storage
adata.strings_to_categoricals()
adata.write_h5ad('file.h5ad', compression='gzip')

# Use Zarr for cloud storage
adata.write_zarr('file.zarr', chunks=(1000, 1000))

Index alignment issues

Always align external data on index:

# Wrong
adata.obs['new_col'] = external_data['values']

# Correct
adata.obs['new_col'] = external_data.set_index('cell_id').loc[adata.obs_names, 'values']

Additional Resources

• Official documentation: https://anndata.readthedocs.io/
• Scanpy tutorials: https://scanpy.readthedocs.io/
• Scverse ecosystem: https://scverse.org/
• GitHub repository: https://github.com/scverse/anndata

Benchling Integration

Overview

Benchling is a cloud platform for life sciences R&D. Access registry entities (DNA, proteins), inventory, electronic lab notebooks, and workflows programmatically via Python SDK and REST API.

When to Use This Skill

This skill should be used when:

• Working with Benchling's Python SDK or REST API
• Managing biological sequences (DNA, RNA, proteins) and registry entities
• Automating inventory operations (samples, containers, locations, transfers)
• Creating or querying electronic lab notebook entries
• Building workflow automations or Benchling Apps
• Syncing data between Benchling and external systems
• Querying the Benchling Data Warehouse for analytics
• Setting up event-driven integrations with AWS EventBridge

Core Capabilities

1. Authentication & Setup

Python SDK Installation:

# Stable release
uv pip install benchling-sdk
# or with Poetry
poetry add benchling-sdk

Authentication Methods:

API Key Authentication (recommended for scripts):

from benchling_sdk.benchling import Benchling
from benchling_sdk.auth.api_key_auth import ApiKeyAuth

benchling = Benchling(
    url="https://your-tenant.benchling.com",
    auth_method=ApiKeyAuth("your_api_key")
)

OAuth Client Credentials (for apps):

from benchling_sdk.auth.client_credentials_oauth2 import ClientCredentialsOAuth2

auth_method = ClientCredentialsOAuth2(
    client_id="your_client_id",
    client_secret="your_client_secret"
)
benchling = Benchling(
    url="https://your-tenant.benchling.com",
    auth_method=auth_method
)

Key Points:

• API keys are obtained from Profile Settings in Benchling
• Store credentials securely (use environment variables or password managers)
• All API requests require HTTPS
• Authentication permissions mirror user permissions in the UI

For detailed authentication information including OIDC and security best practices, refer to references/authentication.md.

2. Registry & Entity Management

Registry entities include DNA sequences, RNA sequences, AA sequences, custom entities, and mixtures. The SDK provides typed classes for creating and managing these entities.

Creating DNA Sequences:

from benchling_sdk.models import DnaSequenceCreate

sequence = benchling.dna_sequences.create(
    DnaSequenceCreate(
        name="My Plasmid",
        bases="ATCGATCG",
        is_circular=True,
        folder_id="fld_abc123",
        schema_id="ts_abc123",  # optional
        fields=benchling.models.fields({"gene_name": "GFP"})
    )
)

Registry Registration:

To register an entity directly upon creation:

sequence = benchling.dna_sequences.create(
    DnaSequenceCreate(
        name="My Plasmid",
        bases="ATCGATCG",
        is_circular=True,
        folder_id="fld_abc123",
        entity_registry_id="src_abc123",  # Registry to register in
        naming_strategy="NEW_IDS"  # or "IDS_FROM_NAMES"
    )
)

Important: Use either entity_registry_id OR naming_strategy, never both.

Updating Entities:

from benchling_sdk.models import DnaSequenceUpdate

updated = benchling.dna_sequences.update(
    sequence_id="seq_abc123",
    dna_sequence=DnaSequenceUpdate(
        name="Updated Plasmid Name",
        fields=benchling.models.fields({"gene_name": "mCherry"})
    )
)

Unspecified fields remain unchanged, allowing partial updates.

Listing and Pagination:

# List all DNA sequences (returns a generator)
sequences = benchling.dna_sequences.list()
for page in sequences:
    for seq in page:
        print(f"{seq.name} ({seq.id})")

# Check total count
total = sequences.estimated_count()

Key Operations:

• Create: benchling.<entity_type>.create()
• Read: benchling.<entity_type>.get(id) or .list()
• Update: benchling.<entity_type>.update(id, update_object)
• Archive: benchling.<entity_type>.archive(id)

Entity types: dna_sequences, rna_sequences, aa_sequences, custom_entities, mixtures

For comprehensive SDK reference and advanced patterns, refer to references/sdk_reference.md.

3. Inventory Management

Manage physical samples, containers, boxes, and locations within the Benchling inventory system.

Creating Containers:

from benchling_sdk.models import ContainerCreate

container = benchling.containers.create(
    ContainerCreate(
        name="Sample Tube 001",
        schema_id="cont_schema_abc123",
        parent_storage_id="box_abc123",  # optional
        fields=benchling.models.fields({"concentration": "100 ng/μL"})
    )
)

Managing Boxes:

from benchling_sdk.models import BoxCreate

box = benchling.boxes.create(
    BoxCreate(
        name="Freezer Box A1",
        schema_id="box_schema_abc123",
        parent_storage_id="loc_abc123"
    )
)

Transferring Items:

# Transfer a container to a new location
transfer = benchling.containers.transfer(
    container_id="cont_abc123",
    destination_id="box_xyz789"
)

Key Inventory Operations:

• Create containers, boxes, locations, plates
• Update inventory item properties
• Transfer items between locations
• Check in/out items
• Batch operations for bulk transfers

4. Notebook & Documentation

Interact with electronic lab notebook (ELN) entries, protocols, and templates.

Creating Notebook Entries:

from benchling_sdk.models import EntryCreate

entry = benchling.entries.create(
    EntryCreate(
        name="Experiment 2025-10-20",
        folder_id="fld_abc123",
        schema_id="entry_schema_abc123",
        fields=benchling.models.fields({"objective": "Test gene expression"})
    )
)

Linking Entities to Entries:

# Add references to entities in an entry
entry_link = benchling.entry_links.create(
    entry_id="entry_abc123",
    entity_id="seq_xyz789"
)

Key Notebook Operations:

• Create and update lab notebook entries
• Manage entry templates
• Link entities and results to entries
• Export entries for documentation

5. Workflows & Automation

Automate laboratory processes using Benchling's workflow system.

Creating Workflow Tasks:

from benchling_sdk.models import WorkflowTaskCreate

task = benchling.workflow_tasks.create(
    WorkflowTaskCreate(
        name="PCR Amplification",
        workflow_id="wf_abc123",
        assignee_id="user_abc123",
        fields=benchling.models.fields({"template": "seq_abc123"})
    )
)

Updating Task Status:

from benchling_sdk.models import WorkflowTaskUpdate

updated_task = benchling.workflow_tasks.update(
    task_id="task_abc123",
    workflow_task=WorkflowTaskUpdate(
        status_id="status_complete_abc123"
    )
)

Asynchronous Operations:

Some operations are asynchronous and return tasks:

# Wait for task completion
from benchling_sdk.helpers.tasks import wait_for_task

result = wait_for_task(
    benchling,
    task_id="task_abc123",
    interval_wait_seconds=2,
    max_wait_seconds=300
)

Key Workflow Operations:

• Create and manage workflow tasks
• Update task statuses and assignments
• Execute bulk operations asynchronously
• Monitor task progress

6. Events & Integration

Subscribe to Benchling events for real-time integrations using AWS EventBridge.

Event Types:

• Entity creation, update, archive
• Inventory transfers
• Workflow task status changes
• Entry creation and updates
• Results registration

Integration Pattern:

1. Configure event routing to AWS EventBridge in Benchling settings
2. Create EventBridge rules to filter events
3. Route events to Lambda functions or other targets
4. Process events and update external systems

Use Cases:

• Sync Benchling data to external databases
• Trigger downstream processes on workflow completion
• Send notifications on entity changes
• Audit trail logging

Refer to Benchling's event documentation for event schemas and configuration.

7. Data Warehouse & Analytics

Query historical Benchling data using SQL through the Data Warehouse.

Access Method: The Benchling Data Warehouse provides SQL access to Benchling data for analytics and reporting. Connect using standard SQL clients with provided credentials.

Common Queries:

• Aggregate experimental results
• Analyze inventory trends
• Generate compliance reports
• Export data for external analysis

Integration with Analysis Tools:

• Jupyter notebooks for interactive analysis
• BI tools (Tableau, Looker, PowerBI)
• Custom dashboards

Best Practices

Error Handling

The SDK automatically retries failed requests:

# Automatic retry for 429, 502, 503, 504 status codes
# Up to 5 retries with exponential backoff
# Customize retry behavior if needed
from benchling_sdk.retry import RetryStrategy

benchling = Benchling(
    url="https://your-tenant.benchling.com",
    auth_method=ApiKeyAuth("your_api_key"),
    retry_strategy=RetryStrategy(max_retries=3)
)

Pagination Efficiency

Use generators for memory-efficient pagination:

# Generator-based iteration
for page in benchling.dna_sequences.list():
    for sequence in page:
        process(sequence)

# Check estimated count without loading all pages
total = benchling.dna_sequences.list().estimated_count()

Schema Fields Helper

Use the fields() helper for custom schema fields:

# Convert dict to Fields object
custom_fields = benchling.models.fields({
    "concentration": "100 ng/μL",
    "date_prepared": "2025-10-20",
    "notes": "High quality prep"
})

Forward Compatibility

The SDK handles unknown enum values and types gracefully:

• Unknown enum values are preserved
• Unrecognized polymorphic types return UnknownType
• Allows working with newer API versions

Security Considerations

• Never commit API keys to version control
• Use environment variables for credentials
• Rotate keys if compromised
• Grant minimal necessary permissions for apps
• Use OAuth for multi-user scenarios

Resources

references/

Detailed reference documentation for in-depth information:

• authentication.md - Comprehensive authentication guide including OIDC, security best practices, and credential management
• sdk_reference.md - Detailed Python SDK reference with advanced patterns, examples, and all entity types
• api_endpoints.md - REST API endpoint reference for direct HTTP calls without the SDK

Load these references as needed for specific integration requirements.

scripts/

This skill currently includes example scripts that can be removed or replaced with custom automation scripts for your specific Benchling workflows.

Common Use Cases

1. Bulk Entity Import:

# Import multiple sequences from FASTA file
from Bio import SeqIO

for record in SeqIO.parse("sequences.fasta", "fasta"):
    benchling.dna_sequences.create(
        DnaSequenceCreate(
            name=record.id,
            bases=str(record.seq),
            is_circular=False,
            folder_id="fld_abc123"
        )
    )

2. Inventory Audit:

# List all containers in a specific location
containers = benchling.containers.list(
    parent_storage_id="box_abc123"
)

for page in containers:
    for container in page:
        print(f"{container.name}: {container.barcode}")

3. Workflow Automation:

# Update all pending tasks for a workflow
tasks = benchling.workflow_tasks.list(
    workflow_id="wf_abc123",
    status="pending"
)

for page in tasks:
    for task in page:
        # Perform automated checks
        if auto_validate(task):
            benchling.workflow_tasks.update(
                task_id=task.id,
                workflow_task=WorkflowTaskUpdate(
                    status_id="status_complete"
                )
            )

4. Data Export:

# Export all sequences with specific properties
sequences = benchling.dna_sequences.list()
export_data = []

for page in sequences:
    for seq in page:
        if seq.schema_id == "target_schema_id":
            export_data.append({
                "id": seq.id,
                "name": seq.name,
                "bases": seq.bases,
                "length": len(seq.bases)
            })

# Save to CSV or database
import csv
with open("sequences.csv", "w") as f:
    writer = csv.DictWriter(f, fieldnames=export_data[0].keys())
    writer.writeheader()
    writer.writerows(export_data)

Additional Resources

• Official Documentation: https://docs.benchling.com
• Python SDK Reference: https://benchling.com/sdk-docs/
• API Reference: https://benchling.com/api/reference
• Support: [email protected]

Bioimage Analysis: Microscopy Image Analysis for Cell Biology

Overview

Bioimage Analysis provides computational tools for processing and quantifying microscopy images in cell biology research. From cell segmentation using deep learning (Cellpose) and classical methods (watershed, Otsu thresholding) to object tracking across time-lapse sequences, morphology quantification, colony counting, colocalization analysis, and cytoskeleton characterization. This skill covers the full pipeline from raw microscopy images to quantitative measurements suitable for statistical analysis.

When to Use This Skill

• Segmenting cells or nuclei from brightfield or fluorescence microscopy images
• Tracking cell migration or particle movement in time-lapse sequences
• Quantifying cell morphology (area, eccentricity, circularity, aspect ratio)
• Counting bacterial or mammalian cell colonies on plates
• Analyzing colocalization of proteins from multi-channel fluorescence images
• Characterizing cytoskeleton fiber orientation and alignment
• Quantifying mitochondrial morphology from JC-1 or TMRM staining
• Batch processing multiple fields of view or multi-well plate images
• Preprocessing microscopy images (denoising, background subtraction, contrast enhancement)

Related Skills: For whole-slide pathology image analysis use pathml or histolab. For flow cytometry data use flow-cytometry-analysis. For clinical imaging (MRI, CT) use clinical-imaging.

Installation

uv pip install cellpose scikit-image opencv-python trackpy tifffile aicsimageio numpy pandas matplotlib

Quick Start

from cellpose import models
import skimage.io
import skimage.measure
import numpy as np

# Load image
image = skimage.io.imread('cells.tif')

# Segment with Cellpose
model = models.Cellpose(model_type='cyto2')
masks, flows, styles, diams = model.eval(image, diameter=None, channels=[0, 0])

# Measure properties
props = skimage.measure.regionprops_table(masks, image,
    properties=['label', 'area', 'eccentricity', 'mean_intensity'])

import pandas as pd
df = pd.DataFrame(props)
print(f"Detected {len(df)} cells")
print(df.describe())

Core Capabilities

1. Image Loading & Preprocessing

Load microscopy images from common formats and prepare them for analysis.

import tifffile
import skimage.io
import skimage.filters
import skimage.exposure
import numpy as np

# Load multi-channel TIFF
image = tifffile.imread('experiment.tif')
print(f"Shape: {image.shape}, dtype: {image.dtype}")

# For multi-dimensional images (TCZYX)
from aicsimageio import AICSImage
img = AICSImage('multi_dim.czi')
data = img.get_image_data("ZYX", C=0, T=0)

# Contrast enhancement (CLAHE)
enhanced = skimage.exposure.equalize_adapthist(image, clip_limit=0.03)

# Gaussian denoising
denoised = skimage.filters.gaussian(image, sigma=1.0)

# Background subtraction (rolling ball approximation)
from skimage.morphology import white_tophat, disk
background_removed = white_tophat(image, disk(50))

# Median filter for salt-and-pepper noise
from skimage.filters import median
from skimage.morphology import square
cleaned = median(image, square(3))

2. Cell Segmentation

Segment individual cells using deep learning or classical approaches.

Cellpose (deep learning):

from cellpose import models

# Cytoplasm segmentation
model = models.Cellpose(model_type='cyto2')
masks, flows, styles, diams = model.eval(
    image,
    diameter=None,       # Auto-detect cell diameter
    channels=[0, 0],     # Grayscale; [2,3] for green cytoplasm + blue nuclei
    flow_threshold=0.4,
    cellprob_threshold=0.0
)

# Nuclei-only segmentation
nuc_model = models.Cellpose(model_type='nuclei')
nuc_masks, _, _, _ = nuc_model.eval(image, diameter=None, channels=[0, 0])

print(f"Segmented {masks.max()} cells")

Classical watershed:

import skimage.segmentation
import skimage.feature
import skimage.filters
import skimage.morphology
from scipy import ndimage

# Threshold
thresh = skimage.filters.threshold_otsu(image)
binary = image > thresh

# Distance transform for watershed seeds
distance = ndimage.distance_transform_edt(binary)
local_max = skimage.feature.peak_local_max(distance, min_distance=20, labels=binary)
markers = np.zeros_like(image, dtype=int)
for i, (r, c) in enumerate(local_max, start=1):
    markers[r, c] = i
markers = ndimage.label(markers)[0]

# Watershed
labels = skimage.segmentation.watershed(-distance, markers, mask=binary)
print(f"Segmented {labels.max()} objects")

Adaptive thresholding:

# For uneven illumination
block_size = 51
adaptive_thresh = skimage.filters.threshold_local(image, block_size, offset=10)
binary = image > adaptive_thresh

# Connected component labeling
from skimage.measure import label
labeled = label(binary)
print(f"Found {labeled.max()} connected components")

3. Object Tracking

Track cells or particles across time-lapse frames.

import trackpy as tp
import pandas as pd

# Load time-lapse as 3D array (T, Y, X)
frames = tifffile.imread('timelapse.tif')

# Locate features in each frame
all_features = []
for t, frame in enumerate(frames):
    features = tp.locate(frame, diameter=11, minmass=1000)
    features['frame'] = t
    all_features.append(features)

features_df = pd.concat(all_features, ignore_index=True)

# Link particles across frames
trajectories = tp.link(features_df, search_range=15, memory=3)

# Filter spurious tracks (require minimum length)
trajectories = tp.filter_stubs(trajectories, threshold=10)
print(f"Tracked {trajectories['particle'].nunique()} objects over {len(frames)} frames")

# Calculate mean squared displacement
msd = tp.emsd(trajectories, mpp=0.65, fps=1)  # microns per pixel, frames per second
print(msd.head())

4. Morphology Quantification

Extract shape and intensity measurements from segmented objects.

import skimage.measure
import pandas as pd

# Measure region properties
props = skimage.measure.regionprops_table(masks, intensity_image=image,
    properties=[
        'label', 'area', 'perimeter', 'eccentricity', 'solidity',
        'major_axis_length', 'minor_axis_length', 'mean_intensity',
        'max_intensity', 'min_intensity', 'centroid'
    ])

df = pd.DataFrame(props)

# Calculate derived metrics
df['circularity'] = 4 * np.pi * df['area'] / (df['perimeter'] ** 2)
df['aspect_ratio'] = df['major_axis_length'] / df['minor_axis_length']

print(f"Measured {len(df)} objects")
print(df[['area', 'eccentricity', 'circularity', 'mean_intensity']].describe())

5. Colony Counting

Count colonies from plate images.

import skimage.io
import skimage.filters
import skimage.segmentation
import skimage.measure
from scipy import ndimage

# Load plate image
plate = skimage.io.imread('agar_plate.jpg', as_gray=True)

# Preprocessing
smoothed = skimage.filters.gaussian(plate, sigma=2)

# Threshold
thresh = skimage.filters.threshold_otsu(smoothed)
binary = smoothed < thresh  # Colonies are darker

# Remove small noise
from skimage.morphology import remove_small_objects
cleaned = remove_small_objects(binary, min_size=50)

# Separate touching colonies with watershed
distance = ndimage.distance_transform_edt(cleaned)
local_max = skimage.feature.peak_local_max(distance, min_distance=10, labels=cleaned)
markers = np.zeros_like(plate, dtype=int)
for i, (r, c) in enumerate(local_max, start=1):
    markers[r, c] = i
markers = ndimage.label(markers)[0]
separated = skimage.segmentation.watershed(-distance, markers, mask=cleaned)

# Count and measure
regions = skimage.measure.regionprops(separated)
colony_count = len(regions)
areas = [r.area for r in regions]
print(f"Colony count: {colony_count}")
print(f"Mean colony area: {np.mean(areas):.1f} px^2")

6. Colocalization Analysis

Quantify spatial overlap between fluorescence channels.

import numpy as np
from scipy.stats import pearsonr

# Load dual-channel image
ch1 = skimage.io.imread('green_channel.tif').astype(float)
ch2 = skimage.io.imread('red_channel.tif').astype(float)

# Mask background (Costes automatic thresholding approximation)
mask = (ch1 > skimage.filters.threshold_otsu(ch1)) | \
       (ch2 > skimage.filters.threshold_otsu(ch2))

ch1_masked = ch1[mask]
ch2_masked = ch2[mask]

# Pearson correlation coefficient
pcc, pval = pearsonr(ch1_masked, ch2_masked)
print(f"Pearson correlation: {pcc:.4f} (p={pval:.2e})")

# Manders overlap coefficients
ch1_coloc = ch1_masked[ch2_masked > 0]
ch2_coloc = ch2_masked[ch1_masked > 0]
M1 = ch1_coloc.sum() / ch1_masked.sum()  # Fraction of ch1 overlapping with ch2
M2 = ch2_coloc.sum() / ch2_masked.sum()  # Fraction of ch2 overlapping with ch1
print(f"Manders M1: {M1:.4f}, M2: {M2:.4f}")

7. Cytoskeleton & Fiber Analysis

Characterize fiber orientation and alignment.

import skimage.feature
import skimage.transform
import numpy as np

# Edge detection
edges = skimage.feature.canny(image, sigma=2)

# Hough line transform for fiber orientation
tested_angles = np.linspace(-np.pi / 2, np.pi / 2, 180, endpoint=False)
h, theta, d = skimage.transform.hough_line(edges, theta=tested_angles)

# Extract peaks (dominant orientations)
peaks = skimage.transform.hough_line_peaks(h, theta, d, num_peaks=50)

# Orientation distribution
angles_deg = np.degrees(peaks[1])
mean_angle = np.mean(angles_deg)
std_angle = np.std(angles_deg)

# Order parameter S = 2 * <cos^2(theta)> - 1 (1=aligned, 0=random)
cos2 = np.cos(2 * np.radians(angles_deg))
order_parameter = np.mean(cos2)
print(f"Mean orientation: {mean_angle:.1f} +/- {std_angle:.1f} degrees")
print(f"Order parameter S: {order_parameter:.3f}")

8. Mitochondrial Morphology

Analyze mitochondrial shape and membrane potential.

import skimage.measure
import numpy as np

# JC-1 staining: red=healthy (aggregate), green=depolarized (monomer)
red_channel = skimage.io.imread('jc1_red.tif').astype(float)
green_channel = skimage.io.imread('jc1_green.tif').astype(float)

# Membrane potential ratio
ratio = red_channel / (green_channel + 1e-6)  # Avoid division by zero
mean_ratio = np.mean(ratio[ratio > 0])
print(f"Mean red/green ratio: {mean_ratio:.3f}")

# Segment mitochondria from green channel
thresh = skimage.filters.threshold_otsu(green_channel)
mito_mask = green_channel > thresh
mito_labels = skimage.measure.label(mito_mask)

# Morphology metrics
props = skimage.measure.regionprops_table(mito_labels,
    properties=['label', 'area', 'major_axis_length', 'minor_axis_length',
                'eccentricity', 'euler_number'])
df = pd.DataFrame(props)
df['aspect_ratio'] = df['major_axis_length'] / (df['minor_axis_length'] + 1e-6)

# Classify: round (AR<2), intermediate (2-4), elongated (>4)
df['morphology'] = pd.cut(df['aspect_ratio'], bins=[0, 2, 4, np.inf],
                          labels=['round', 'intermediate', 'elongated'])
print(df['morphology'].value_counts())

9. Batch Processing

Process multiple fields of view or wells.

from pathlib import Path
import pandas as pd

image_dir = Path('plate_images/')
all_results = []

for img_path in sorted(image_dir.glob('*.tif')):
    image = skimage.io.imread(str(img_path))

    # Segment
    model = models.Cellpose(model_type='cyto2')
    masks, _, _, _ = model.eval(image, diameter=None, channels=[0, 0])

    # Measure
    props = skimage.measure.regionprops_table(masks, image,
        properties=['label', 'area', 'eccentricity', 'mean_intensity'])
    df = pd.DataFrame(props)
    df['image'] = img_path.stem

    all_results.append(df)

results = pd.concat(all_results, ignore_index=True)
summary = results.groupby('image').agg(
    cell_count=('label', 'count'),
    mean_area=('area', 'mean'),
    mean_intensity=('mean_intensity', 'mean')
).reset_index()
print(summary)

Typical Workflows

Workflow 1: Segment and Count Cells from Brightfield Microscopy

from cellpose import models
import skimage.io, skimage.measure
import pandas as pd

# Load
image = skimage.io.imread('brightfield_cells.tif')

# Segment
model = models.Cellpose(model_type='cyto2')
masks, _, _, diams = model.eval(image, diameter=None, channels=[0, 0])

# Quantify
props = skimage.measure.regionprops_table(masks, image,
    properties=['label', 'area', 'eccentricity', 'mean_intensity', 'centroid'])
df = pd.DataFrame(props)
print(f"Cell count: {masks.max()}")
print(f"Mean area: {df['area'].mean():.1f} px^2")
print(f"Mean eccentricity: {df['eccentricity'].mean():.3f}")

Workflow 2: Track Cell Migration and Compute Velocity

import trackpy as tp
import tifffile
import numpy as np

# Load time-lapse
frames = tifffile.imread('migration_timelapse.tif')
pixel_size = 0.65  # um/px
dt = 5  # minutes between frames

# Detect and track
features = tp.batch(frames, diameter=15, minmass=500)
tracks = tp.link(features, search_range=20, memory=3)
tracks = tp.filter_stubs(tracks, threshold=10)

# Calculate velocity per track
velocities = []
for pid, group in tracks.groupby('particle'):
    group = group.sort_values('frame')
    dx = np.diff(group['x'].values) * pixel_size
    dy = np.diff(group['y'].values) * pixel_size
    speed = np.sqrt(dx**2 + dy**2) / dt  # um/min
    velocities.append({'particle': pid, 'mean_speed': np.mean(speed),
                       'max_speed': np.max(speed), 'n_frames': len(group)})

vel_df = pd.DataFrame(velocities)
print(f"Mean migration speed: {vel_df['mean_speed'].mean():.3f} um/min")

Workflow 3: Quantify Protein Colocalization from Dual-Channel Fluorescence

import skimage.io
import numpy as np
from scipy.stats import pearsonr

# Load channels
green = skimage.io.imread('protein_A_green.tif').astype(float)
red = skimage.io.imread('protein_B_red.tif').astype(float)

# Background subtraction
green -= np.percentile(green, 5)
red -= np.percentile(red, 5)
green = np.clip(green, 0, None)
red = np.clip(red, 0, None)

# Threshold mask
mask = (green > green.mean()) | (red > red.mean())
g_masked, r_masked = green[mask], red[mask]

# Pearson
pcc, pval = pearsonr(g_masked, r_masked)

# Manders
M1 = g_masked[r_masked > 0].sum() / g_masked.sum()
M2 = r_masked[g_masked > 0].sum() / r_masked.sum()

print(f"Pearson r = {pcc:.4f} (p = {pval:.2e})")
print(f"Manders M1 = {M1:.4f}, M2 = {M2:.4f}")

Workflow 4: Count Bacterial Colonies on Agar Plate Image

import skimage.io, skimage.filters, skimage.measure
from skimage.morphology import remove_small_objects
from scipy import ndimage
import numpy as np

plate = skimage.io.imread('agar_plate.jpg', as_gray=True)
smoothed = skimage.filters.gaussian(plate, sigma=2)
binary = smoothed < skimage.filters.threshold_otsu(smoothed)
binary = remove_small_objects(binary, min_size=30)

# Watershed to split touching colonies
distance = ndimage.distance_transform_edt(binary)
from skimage.feature import peak_local_max
local_max = peak_local_max(distance, min_distance=8, labels=binary)
markers = np.zeros_like(binary, dtype=int)
for i, (r, c) in enumerate(local_max, start=1):
    markers[r, c] = i
markers = ndimage.label(markers)[0]
labels = skimage.segmentation.watershed(-distance, markers, mask=binary)

regions = skimage.measure.regionprops(labels)
print(f"Colony count: {len(regions)}")
print(f"Mean area: {np.mean([r.area for r in regions]):.0f} px^2")

Best Practices

1. Always inspect images first — check bit depth, dimensions, and channel order before processing
2. Use appropriate preprocessing — CLAHE for low contrast, Gaussian blur for noisy images, background subtraction for uneven illumination
3. Validate segmentation visually — overlay masks on original images to verify quality before batch processing
4. Cellpose model selection — use cyto2 for whole-cell, nuclei for nuclear segmentation; adjust diameter parameter if auto-detection fails
5. Trackpy parameter tuning — diameter must be odd integer; search_range should be < typical inter-particle distance; use memory for blinking objects
6. Colocalization controls — always include single-stained controls; report both Pearson and Manders coefficients
7. Batch consistency — use identical preprocessing and segmentation parameters across all images in an experiment
8. Scale bars — always record pixel size (um/px) from microscope metadata for absolute measurements

Troubleshooting

Problem: Cellpose segments too many or too few cells Solution: Adjust diameter parameter manually instead of auto-detection. Increase cellprob_threshold (e.g., 0.5) to reduce false positives, decrease for more detections.

Problem: Touching colonies not separated by watershed Solution: Decrease min_distance in peak_local_max. Try morphological erosion before distance transform.

Problem: Trackpy links wrong particles across frames Solution: Decrease search_range to limit maximum displacement. Increase minmass to exclude dim objects that cause mislinks.

Problem: TIFF file won't load or has wrong dimensions Solution: Use tifffile.imread for raw loading. Check image.shape and image.dtype. For multi-series files use aicsimageio.

Problem: Colocalization values seem artificially high Solution: Ensure proper background subtraction. Use Costes automatic thresholding. Check for bleed-through between channels.

Resources

• Cellpose Documentation
• scikit-image Documentation
• trackpy Documentation
• AICS ImageIO
• Bio-image Analysis Notebooks

BioServices

Overview

BioServices is a Python package providing programmatic access to approximately 40 bioinformatics web services and databases. Retrieve biological data, perform cross-database queries, map identifiers, analyze sequences, and integrate multiple biological resources in Python workflows. The package handles both REST and SOAP/WSDL protocols transparently.

When to Use This Skill

This skill should be used when:

• Retrieving protein sequences, annotations, or structures from UniProt, PDB, Pfam
• Analyzing metabolic pathways and gene functions via KEGG or Reactome
• Searching compound databases (ChEBI, ChEMBL, PubChem) for chemical information
• Converting identifiers between different biological databases (KEGG↔UniProt, compound IDs)
• Running sequence similarity searches (BLAST, MUSCLE alignment)
• Querying gene ontology terms (QuickGO, GO annotations)
• Accessing protein-protein interaction data (PSICQUIC, IntactComplex)
• Mining genomic data (BioMart, ArrayExpress, ENA)
• Integrating data from multiple bioinformatics resources in a single workflow

Core Capabilities

1. Protein Analysis

Retrieve protein information, sequences, and functional annotations:

from bioservices import UniProt

u = UniProt(verbose=False)

# Search for protein by name
results = u.search("ZAP70_HUMAN", frmt="tab", columns="id,genes,organism")

# Retrieve FASTA sequence
sequence = u.retrieve("P43403", "fasta")

# Map identifiers between databases
kegg_ids = u.mapping(fr="UniProtKB_AC-ID", to="KEGG", query="P43403")

Key methods:

• search(): Query UniProt with flexible search terms
• retrieve(): Get protein entries in various formats (FASTA, XML, tab)
• mapping(): Convert identifiers between databases

Reference: references/services_reference.md for complete UniProt API details.

2. Pathway Discovery and Analysis

Access KEGG pathway information for genes and organisms:

from bioservices import KEGG

k = KEGG()
k.organism = "hsa"  # Set to human

# Search for organisms
k.lookfor_organism("droso")  # Find Drosophila species

# Find pathways by name
k.lookfor_pathway("B cell")  # Returns matching pathway IDs

# Get pathways containing specific genes
pathways = k.get_pathway_by_gene("7535", "hsa")  # ZAP70 gene

# Retrieve and parse pathway data
data = k.get("hsa04660")
parsed = k.parse(data)

# Extract pathway interactions
interactions = k.parse_kgml_pathway("hsa04660")
relations = interactions['relations']  # Protein-protein interactions

# Convert to Simple Interaction Format
sif_data = k.pathway2sif("hsa04660")

Key methods:

• lookfor_organism(), lookfor_pathway(): Search by name
• get_pathway_by_gene(): Find pathways containing genes
• parse_kgml_pathway(): Extract structured pathway data
• pathway2sif(): Get protein interaction networks

Reference: references/workflow_patterns.md for complete pathway analysis workflows.

3. Compound Database Searches

Search and cross-reference compounds across multiple databases:

from bioservices import KEGG, UniChem

k = KEGG()

# Search compounds by name
results = k.find("compound", "Geldanamycin")  # Returns cpd:C11222

# Get compound information with database links
compound_info = k.get("cpd:C11222")  # Includes ChEBI links

# Cross-reference KEGG → ChEMBL using UniChem
u = UniChem()
chembl_id = u.get_compound_id_from_kegg("C11222")  # Returns CHEMBL278315

Common workflow:

1. Search compound by name in KEGG
2. Extract KEGG compound ID
3. Use UniChem for KEGG → ChEMBL mapping
4. ChEBI IDs are often provided in KEGG entries

Reference: references/identifier_mapping.md for complete cross-database mapping guide.

4. Sequence Analysis

Run BLAST searches and sequence alignments:

from bioservices import NCBIblast

s = NCBIblast(verbose=False)

# Run BLASTP against UniProtKB
jobid = s.run(
    program="blastp",
    sequence=protein_sequence,
    stype="protein",
    database="uniprotkb",
    email="your.email@example.com"  # Required by NCBI
)

# Check job status and retrieve results
s.getStatus(jobid)
results = s.getResult(jobid, "out")

Note: BLAST jobs are asynchronous. Check status before retrieving results.

5. Identifier Mapping

Convert identifiers between different biological databases:

from bioservices import UniProt, KEGG

# UniProt mapping (many database pairs supported)
u = UniProt()
results = u.mapping(
    fr="UniProtKB_AC-ID",  # Source database
    to="KEGG",              # Target database
    query="P43403"          # Identifier(s) to convert
)

# KEGG gene ID → UniProt
kegg_to_uniprot = u.mapping(fr="KEGG", to="UniProtKB_AC-ID", query="hsa:7535")

# For compounds, use UniChem
from bioservices import UniChem
u = UniChem()
chembl_from_kegg = u.get_compound_id_from_kegg("C11222")

Supported mappings (UniProt):

• UniProtKB ↔ KEGG
• UniProtKB ↔ Ensembl
• UniProtKB ↔ PDB
• UniProtKB ↔ RefSeq
• And many more (see references/identifier_mapping.md)

6. Gene Ontology Queries

Access GO terms and annotations:

from bioservices import QuickGO

g = QuickGO(verbose=False)

# Retrieve GO term information
term_info = g.Term("GO:0003824", frmt="obo")

# Search annotations
annotations = g.Annotation(protein="P43403", format="tsv")

7. Protein-Protein Interactions

Query interaction databases via PSICQUIC:

from bioservices import PSICQUIC

s = PSICQUIC(verbose=False)

# Query specific database (e.g., MINT)
interactions = s.query("mint", "ZAP70 AND species:9606")

# List available interaction databases
databases = s.activeDBs

Available databases: MINT, IntAct, BioGRID, DIP, and 30+ others.

Multi-Service Integration Workflows

BioServices excels at combining multiple services for comprehensive analysis. Common integration patterns:

Complete Protein Analysis Pipeline

Execute a full protein characterization workflow:

python scripts/protein_analysis_workflow.py ZAP70_HUMAN your.email@example.com

This script demonstrates:

1. UniProt search for protein entry
2. FASTA sequence retrieval
3. BLAST similarity search
4. KEGG pathway discovery
5. PSICQUIC interaction mapping

Pathway Network Analysis

Analyze all pathways for an organism:

python scripts/pathway_analysis.py hsa output_directory/

Extracts and analyzes:

• All pathway IDs for organism
• Protein-protein interactions per pathway
• Interaction type distributions
• Exports to CSV/SIF formats

Cross-Database Compound Search

Map compound identifiers across databases:

python scripts/compound_cross_reference.py Geldanamycin

Retrieves:

• KEGG compound ID
• ChEBI identifier
• ChEMBL identifier
• Basic compound properties

Batch Identifier Conversion

Convert multiple identifiers at once:

python scripts/batch_id_converter.py input_ids.txt --from UniProtKB_AC-ID --to KEGG

Best Practices

Output Format Handling

Different services return data in various formats:

• XML: Parse using BeautifulSoup (most SOAP services)
• Tab-separated (TSV): Pandas DataFrames for tabular data
• Dictionary/JSON: Direct Python manipulation
• FASTA: BioPython integration for sequence analysis

Rate Limiting and Verbosity

Control API request behavior:

from bioservices import KEGG

k = KEGG(verbose=False)  # Suppress HTTP request details
k.TIMEOUT = 30  # Adjust timeout for slow connections

Error Handling

Wrap service calls in try-except blocks:

try:
    results = u.search("ambiguous_query")
    if results:
        # Process results
        pass
except Exception as e:
    print(f"Search failed: {e}")

Organism Codes

Use standard organism abbreviations:

• hsa: Homo sapiens (human)
• mmu: Mus musculus (mouse)
• dme: Drosophila melanogaster
• sce: Saccharomyces cerevisiae (yeast)

List all organisms: k.list("organism") or k.organismIds

Integration with Other Tools

BioServices works well with:

• BioPython: Sequence analysis on retrieved FASTA data
• Pandas: Tabular data manipulation
• PyMOL: 3D structure visualization (retrieve PDB IDs)
• NetworkX: Network analysis of pathway interactions
• Galaxy: Custom tool wrappers for workflow platforms

Resources

scripts/

Executable Python scripts demonstrating complete workflows:

• protein_analysis_workflow.py: End-to-end protein characterization
• pathway_analysis.py: KEGG pathway discovery and network extraction
• compound_cross_reference.py: Multi-database compound searching
• batch_id_converter.py: Bulk identifier mapping utility

Scripts can be executed directly or adapted for specific use cases.

references/

Detailed documentation loaded as needed:

• services_reference.md: Comprehensive list of all 40+ services with methods
• workflow_patterns.md: Detailed multi-step analysis workflows
• identifier_mapping.md: Complete guide to cross-database ID conversion

Load references when working with specific services or complex integration tasks.

Installation

uv pip install bioservices

Dependencies are automatically managed. Package is tested on Python 3.9-3.12.

Additional Information

For detailed API documentation and advanced features, refer to:

• Official documentation: https://bioservices.readthedocs.io/
• Source code: https://github.com/cokelaer/bioservices
• Service-specific references in references/services_reference.md

Cancer Genomics Analysis: Computational Workflows

Overview

Cancer Genomics Analysis provides computational pipelines for processing and interpreting cancer genomics data. This skill covers somatic mutation detection and annotation (GATK Mutect2 integration), structural variation characterization, copy number analysis (CNVkit workflows), tumor purity and ploidy estimation, NMF-based metagene extraction from expression data, DNA damage response network analysis, and tumor mutational burden calculation. All workflows produce quantitative outputs suitable for clinical interpretation and publication.

When to Use This Skill

• Processing somatic variant calls from tumor-normal paired sequencing
• Annotating VCF files with gene names, functional impact, and clinical significance
• Detecting and classifying structural variants (deletions, duplications, inversions, translocations)
• Running copy number analysis pipelines (coverage, segmentation, calling)
• Estimating tumor purity and ploidy from sequencing data
• Extracting gene expression signatures via NMF (metagene programs)
• Analyzing DNA damage response pathway disruption in tumors
• Calculating tumor mutational burden for immunotherapy biomarker assessment

Related Skills: For cancer mutation databases use cosmic-database. For variant clinical significance use clinvar-database. For gene annotations use ensembl-database. For pathway enrichment use kegg-database or reactome-database.

Installation

uv pip install pyvcf3 cyvcf2 pysam scikit-learn networkx gseapy pandas numpy matplotlib

For command-line tools (optional):

# GATK, SnpEff, CNVkit are installed separately
# conda install -c bioconda gatk4 snpeff cnvkit

Quick Start

import cyvcf2
import pandas as pd

# Parse somatic VCF
vcf = cyvcf2.VCF('somatic_mutations.vcf.gz')
variants = []
for v in vcf:
    if v.FILTER is None or v.FILTER == 'PASS':
        variants.append({
            'chrom': v.CHROM, 'pos': v.POS,
            'ref': v.REF, 'alt': ','.join(v.ALT),
            'qual': v.QUAL,
            'depth': v.INFO.get('DP'),
            'af': v.INFO.get('AF')
        })

df = pd.DataFrame(variants)
print(f"PASS variants: {len(df)}")
print(df.head())

Core Capabilities

1. VCF Parsing & Variant Processing

Read, filter, and annotate variant calls.

import cyvcf2
import pandas as pd

def parse_vcf(vcf_path, min_qual=30, min_dp=10, min_af=0.05):
    """Parse VCF with quality filters."""
    vcf = cyvcf2.VCF(vcf_path)
    variants = []

    for v in vcf:
        # Apply filters
        if v.FILTER is not None and v.FILTER != 'PASS':
            continue

        dp = v.INFO.get('DP', 0)
        af_values = v.INFO.get('AF')
        af = af_values if isinstance(af_values, float) else (af_values[0] if af_values else 0)

        if v.QUAL and v.QUAL < min_qual:
            continue
        if dp < min_dp:
            continue
        if af < min_af:
            continue

        variants.append({
            'chrom': v.CHROM, 'pos': v.POS,
            'ref': v.REF, 'alt': ','.join(v.ALT),
            'qual': v.QUAL, 'dp': dp, 'af': af,
            'gene': v.INFO.get('ANN', '').split('|')[3] if v.INFO.get('ANN') else ''
        })

    return pd.DataFrame(variants)

df = parse_vcf('tumor_somatic.vcf.gz')
print(f"Filtered variants: {len(df)}")
print(f"Genes affected: {df['gene'].nunique()}")

2. Somatic Mutation Detection

GATK Mutect2 workflow patterns.

import subprocess

def run_mutect2(tumor_bam, normal_bam, reference, output_vcf,
                gnomad_resource=None, pon=None):
    """Run GATK Mutect2 for somatic variant calling."""
    cmd = [
        'gatk', 'Mutect2',
        '-R', reference,
        '-I', tumor_bam,
        '-I', normal_bam,
        '-tumor', 'TUMOR',
        '-normal', 'NORMAL',
        '-O', output_vcf
    ]
    if gnomad_resource:
        cmd.extend(['--germline-resource', gnomad_resource])
    if pon:
        cmd.extend(['-pon', pon])

    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"Mutect2 failed: {result.stderr}")
    return output_vcf

def filter_mutect_calls(raw_vcf, filtered_vcf, reference):
    """Apply Mutect2 filters."""
    cmd = [
        'gatk', 'FilterMutectCalls',
        '-R', reference,
        '-V', raw_vcf,
        '-O', filtered_vcf
    ]
    subprocess.run(cmd, capture_output=True, text=True, check=True)
    return filtered_vcf

def annotate_with_snpeff(vcf_path, output_vcf, genome='GRCh38.105'):
    """Annotate variants with SnpEff."""
    cmd = f"snpEff ann {genome} {vcf_path} > {output_vcf}"
    subprocess.run(cmd, shell=True, capture_output=True, text=True, check=True)
    return output_vcf

3. Structural Variation Detection

Classify and annotate structural variants.

import cyvcf2
import pandas as pd

def parse_sv_vcf(vcf_path):
    """Parse structural variant VCF (LUMPY/Manta/Delly format)."""
    vcf = cyvcf2.VCF(vcf_path)
    svs = []

    for v in vcf:
        svtype = v.INFO.get('SVTYPE', 'UNKNOWN')
        svlen = abs(v.INFO.get('SVLEN', 0)) if v.INFO.get('SVLEN') else 0
        end = v.INFO.get('END', v.POS)
        pe = v.INFO.get('PE', 0)  # Paired-end support
        sr = v.INFO.get('SR', 0)  # Split-read support

        svs.append({
            'chrom': v.CHROM, 'pos': v.POS, 'end': end,
            'svtype': svtype, 'svlen': svlen,
            'pe_support': pe, 'sr_support': sr,
            'qual': v.QUAL, 'filter': v.FILTER or 'PASS'
        })

    df = pd.DataFrame(svs)
    return df

sv_df = parse_sv_vcf('structural_variants.vcf')
print("SV type distribution:")
print(sv_df['svtype'].value_counts())
print(f"\nMedian SV length: {sv_df[sv_df['svlen'] > 0]['svlen'].median():.0f} bp")

4. Copy Number Analysis

CNVkit-based workflow for copy number profiling.

import subprocess
import pandas as pd
import numpy as np

def cnvkit_pipeline(tumor_bam, normal_bam, reference, target_bed, output_dir):
    """Run CNVkit copy number analysis pipeline."""
    # Step 1: Coverage
    subprocess.run([
        'cnvkit.py', 'coverage', tumor_bam, target_bed,
        '-o', f'{output_dir}/tumor.targetcoverage.cnn'
    ], check=True)

    # Step 2: Reference from normal
    subprocess.run([
        'cnvkit.py', 'reference', f'{output_dir}/normal.targetcoverage.cnn',
        '-f', reference, '-o', f'{output_dir}/reference.cnn'
    ], check=True)

    # Step 3: Fix and segment
    subprocess.run([
        'cnvkit.py', 'fix', f'{output_dir}/tumor.targetcoverage.cnn',
        f'{output_dir}/tumor.antitargetcoverage.cnn',
        f'{output_dir}/reference.cnn',
        '-o', f'{output_dir}/tumor.cnr'
    ], check=True)

    subprocess.run([
        'cnvkit.py', 'segment', f'{output_dir}/tumor.cnr',
        '-o', f'{output_dir}/tumor.cns'
    ], check=True)

    return f'{output_dir}/tumor.cns'

def parse_cnvkit_segments(cns_path):
    """Parse CNVkit segmentation output."""
    df = pd.read_csv(cns_path, sep='\t')
    # Classify events
    df['call'] = 'neutral'
    df.loc[df['log2'] > 0.3, 'call'] = 'gain'
    df.loc[df['log2'] > 0.8, 'call'] = 'amplification'
    df.loc[df['log2'] < -0.3, 'call'] = 'loss'
    df.loc[df['log2'] < -1.0, 'call'] = 'deep_deletion'

    print("Copy number events:")
    print(df['call'].value_counts())
    return df

def estimate_purity_ploidy(segments_df):
    """Estimate tumor purity and ploidy from segments."""
    # Simplified approach using segment log2 ratios
    log2_values = segments_df['log2'].values
    weights = segments_df['end'] - segments_df['start']

    # Weighted median for ploidy shift
    weighted_median = np.average(log2_values, weights=weights)
    estimated_ploidy = 2 * (2 ** weighted_median)

    # Purity from deviation of peaks from integer CN
    # (simplified — full methods use allele frequencies)
    deviation = np.std(log2_values)
    estimated_purity = min(1.0, deviation * 2)  # Rough heuristic

    return {'purity': estimated_purity, 'ploidy': estimated_ploidy}

5. NMF Metagene Extraction

Extract gene expression programs using Non-negative Matrix Factorization.

from sklearn.decomposition import NMF
import numpy as np
import pandas as pd

def extract_metagenes(expression_matrix, n_components=5, top_genes=50):
    """Extract metagene programs from expression matrix via NMF.

    Args:
        expression_matrix: genes x samples DataFrame (non-negative values)
        n_components: number of metagene programs to extract
        top_genes: number of top genes to report per metagene
    """
    # Ensure non-negative
    X = expression_matrix.values
    X = np.clip(X, 0, None)

    # Fit NMF
    model = NMF(n_components=n_components, init='nndsvda', random_state=42,
                max_iter=500, l1_ratio=0.5)
    W = model.fit_transform(X)  # genes x components (gene weights)
    H = model.components_        # components x samples (sample coefficients)

    # Extract top genes per metagene
    metagenes = {}
    for k in range(n_components):
        gene_weights = pd.Series(W[:, k], index=expression_matrix.index)
        top = gene_weights.nlargest(top_genes)
        metagenes[f'Metagene_{k+1}'] = top

    # Reconstruction error
    recon_error = model.reconstruction_err_
    print(f"Reconstruction error: {recon_error:.4f}")

    return metagenes, W, H, model

def optimal_rank_selection(expression_matrix, k_range=range(2, 11)):
    """Select optimal NMF rank using cophenetic correlation."""
    from scipy.cluster.hierarchy import cophenet, linkage
    from scipy.spatial.distance import pdist

    X = np.clip(expression_matrix.values, 0, None)
    scores = {}

    for k in k_range:
        # Run NMF multiple times
        consensus = np.zeros((X.shape[1], X.shape[1]))
        n_runs = 20
        for i in range(n_runs):
            model = NMF(n_components=k, init='random', random_state=i, max_iter=300)
            H = model.fit_transform(X.T).T  # Transpose for sample clustering
            assignments = np.argmax(H, axis=0)
            for a in range(X.shape[1]):
                for b in range(X.shape[1]):
                    if assignments[a] == assignments[b]:
                        consensus[a, b] += 1
        consensus /= n_runs

        # Cophenetic correlation
        Z = linkage(pdist(1 - consensus), method='average')
        coph_corr, _ = cophenet(Z, pdist(1 - consensus))
        scores[k] = coph_corr
        print(f"k={k}: cophenetic correlation = {coph_corr:.4f}")

    optimal_k = max(scores, key=scores.get)
    print(f"\nOptimal rank: {optimal_k}")
    return optimal_k, scores

6. DNA Damage Response Network

Analyze DDR pathway disruption in tumors.

import networkx as nx
import numpy as np
import pandas as pd

# Core DDR genes
DDR_GENES = [
    'TP53', 'BRCA1', 'BRCA2', 'ATM', 'ATR', 'CHEK1', 'CHEK2',
    'RAD51', 'PALB2', 'XRCC1', 'PARP1', 'MLH1', 'MSH2', 'MSH6',
    'ERCC1', 'XPA', 'XPC', 'POLH', 'REV3L', 'FANCA', 'FANCD2'
]

def build_ddr_network(expression_df, ddr_genes=DDR_GENES, threshold=0.5):
    """Build DDR correlation network from expression data."""
    # Filter to DDR genes present in data
    available = [g for g in ddr_genes if g in expression_df.index]
    ddr_expr = expression_df.loc[available]

    # Compute correlation matrix
    corr = ddr_expr.T.corr(method='spearman')

    # Build network
    G = nx.Graph()
    for i, g1 in enumerate(available):
        for j, g2 in enumerate(available):
            if i < j and abs(corr.loc[g1, g2]) > threshold:
                G.add_edge(g1, g2, weight=corr.loc[g1, g2])

    return G, corr

def compare_ddr_networks(tumor_expr, normal_expr, ddr_genes=DDR_GENES):
    """Identify disrupted DDR edges in tumor vs normal."""
    G_tumor, corr_tumor = build_ddr_network(tumor_expr, ddr_genes)
    G_normal, corr_normal = build_ddr_network(normal_expr, ddr_genes)

    # Find disrupted edges
    disrupted = []
    for u, v, d in G_normal.edges(data=True):
        normal_corr = d['weight']
        tumor_corr = corr_tumor.loc[u, v] if u in corr_tumor.index and v in corr_tumor.columns else 0
        delta = abs(normal_corr - tumor_corr)
        if delta > 0.3:
            disrupted.append({
                'gene1': u, 'gene2': v,
                'normal_corr': normal_corr, 'tumor_corr': tumor_corr,
                'delta': delta
            })

    return pd.DataFrame(disrupted).sort_values('delta', ascending=False)

7. Tumor Mutational Burden

Calculate TMB for immunotherapy biomarker assessment.

import cyvcf2

def calculate_tmb(vcf_path, exome_size_mb=35.0, min_af=0.05, min_dp=10):
    """Calculate tumor mutational burden (mutations per Mb)."""
    vcf = cyvcf2.VCF(vcf_path)
    nonsynonymous_count = 0
    total_pass = 0

    for v in vcf:
        if v.FILTER and v.FILTER != 'PASS':
            continue

        dp = v.INFO.get('DP', 0)
        af = v.INFO.get('AF', 0)
        if isinstance(af, tuple):
            af = af[0]

        if dp < min_dp or af < min_af:
            continue

        total_pass += 1

        # Check for nonsynonymous (requires SnpEff/VEP annotation)
        ann = v.INFO.get('ANN', '')
        if ann and ('missense' in ann.lower() or 'nonsense' in ann.lower() or
                    'frameshift' in ann.lower() or 'stop_gained' in ann.lower()):
            nonsynonymous_count += 1

    tmb = total_pass / exome_size_mb
    tmb_nonsynonymous = nonsynonymous_count / exome_size_mb

    print(f"Total PASS variants: {total_pass}")
    print(f"Nonsynonymous variants: {nonsynonymous_count}")
    print(f"TMB (all): {tmb:.1f} mut/Mb")
    print(f"TMB (nonsynonymous): {tmb_nonsynonymous:.1f} mut/Mb")

    # Classification
    if tmb >= 10:
        classification = 'TMB-High'
    elif tmb >= 5:
        classification = 'TMB-Intermediate'
    else:
        classification = 'TMB-Low'
    print(f"Classification: {classification}")

    return {'tmb': tmb, 'tmb_nonsyn': tmb_nonsynonymous, 'class': classification}

Typical Workflows

Workflow 1: Complete Somatic Mutation Calling and Annotation

import subprocess

# 1. Call variants
run_mutect2('tumor.bam', 'normal.bam', 'ref.fa', 'raw.vcf')

# 2. Filter
filter_mutect_calls('raw.vcf', 'filtered.vcf', 'ref.fa')

# 3. Annotate
annotate_with_snpeff('filtered.vcf', 'annotated.vcf')

# 4. Parse and analyze
df = parse_vcf('annotated.vcf')
print(f"Somatic mutations: {len(df)}")
print(f"Most mutated genes:")
print(df['gene'].value_counts().head(10))

Workflow 2: Copy Number Analysis with Purity Estimation

# 1. Run CNVkit pipeline
cns_file = cnvkit_pipeline('tumor.bam', 'normal.bam', 'ref.fa', 'targets.bed', 'cnv_output/')

# 2. Parse segments
segments = parse_cnvkit_segments(cns_file)

# 3. Estimate purity/ploidy
estimates = estimate_purity_ploidy(segments)
print(f"Estimated purity: {estimates['purity']:.2f}")
print(f"Estimated ploidy: {estimates['ploidy']:.2f}")

# 4. Identify focal events
focal = segments[(segments['call'].isin(['amplification', 'deep_deletion'])) &
                  (segments['end'] - segments['start'] < 5e6)]
print(f"\nFocal events: {len(focal)}")
print(focal[['chromosome', 'start', 'end', 'gene', 'log2', 'call']])

Workflow 3: NMF Extraction of Gene Expression Signatures

import pandas as pd

# 1. Load expression matrix (genes x samples, non-negative)
expr = pd.read_csv('tpm_matrix.csv', index_col=0)
expr = expr.clip(lower=0)

# 2. Select optimal rank
optimal_k, scores = optimal_rank_selection(expr, k_range=range(2, 8))

# 3. Extract metagenes
metagenes, W, H, model = extract_metagenes(expr, n_components=optimal_k)

# 4. Interpret metagenes with enrichment
import gseapy as gp
for name, genes in metagenes.items():
    enr = gp.enrichr(gene_list=list(genes.index), gene_sets='KEGG_2021_Human', outdir=None)
    top_pathway = enr.results.iloc[0]['Term'] if len(enr.results) > 0 else 'None'
    print(f"{name}: top pathway = {top_pathway}")
    print(f"  Top genes: {', '.join(genes.index[:5])}")

Best Practices

1. Always use paired tumor-normal for somatic calling — tumor-only mode has high false positive rates
2. Filter aggressively — apply PASS filter, minimum depth (>10x), minimum allele frequency (>5% for WES)
3. Annotate with standard tools — SnpEff or VEP for functional annotation; validate key variants in ClinVar
4. Check purity before CNV analysis — low purity tumors underestimate copy number changes
5. NMF requires non-negative input — use TPM or RPKM, not log-transformed values
6. TMB calculation — use consistent exome size (typically 30-40 Mb); nonsynonymous variants only for clinical interpretation
7. Validate key findings in COSMIC and ClinVar databases

Troubleshooting

Problem: VCF parsing fails with cyvcf2 Solution: Ensure VCF is bgzip-compressed and tabix-indexed. Use bcftools view -O z -o out.vcf.gz in.vcf && tabix -p vcf out.vcf.gz.

Problem: CNVkit segmentation produces too many small segments Solution: Increase segmentation threshold with --threshold parameter. Merge adjacent segments with similar log2 ratios.

Problem: NMF produces unstable results across runs Solution: Use init='nndsvda' for deterministic initialization. Run cophenetic correlation analysis to verify rank stability.

Problem: TMB calculation gives unexpectedly high values Solution: Verify exome capture size. Check for germline contamination (apply germline resource filter). Ensure proper Mutect2 filtering.

Resources

• GATK Best Practices
• CNVkit Documentation
• SnpEff Documentation
• COSMIC Database
• ClinVar

Clinical Imaging: Clinical & Physiological Imaging Analysis

Overview

Clinical Imaging provides computational tools for analyzing clinical and physiological imaging data. This skill covers diffusion MRI apparent diffusion coefficient (ADC) map computation, micro-CT bone morphometry (BV/TV, trabecular thickness), hemodynamic parameter analysis from blood pressure waveforms, circadian rhythm cosinor analysis, ciliary beat frequency measurement via FFT, tissue deformation analysis using optical flow, and amyloid plaque quantification from fluorescence microscopy.

When to Use This Skill

• Computing ADC maps from multi-b-value diffusion MRI data
• Analyzing bone microarchitecture from micro-CT volumes
• Processing blood pressure or hemodynamic waveform data
• Fitting circadian rhythm data with cosinor models
• Measuring ciliary beat frequency from high-speed video
• Quantifying tissue deformation from image sequences
• Counting and measuring amyloid plaques in fluorescence images

Related Skills: For DICOM file handling use pydicom. For biosignal processing (ECG, EMG, EDA) use neurokit2. For general image analysis use bioimage-analysis.

Installation

uv pip install nibabel SimpleITK scipy opencv-python scikit-image numpy pandas matplotlib

Quick Start

import numpy as np
from scipy.optimize import curve_fit

# Circadian rhythm cosinor analysis
def cosinor(t, mesor, amplitude, acrophase, period=24):
    return mesor + amplitude * np.cos(2 * np.pi * t / period + acrophase)

# Example: body temperature over 48 hours
time_hours = np.arange(0, 48, 1)
temperature = 36.8 + 0.3 * np.cos(2 * np.pi * time_hours / 24 - 1.0) + \
              np.random.normal(0, 0.1, len(time_hours))

popt, pcov = curve_fit(cosinor, time_hours, temperature,
                       p0=[36.8, 0.3, -1.0], maxfev=10000)
print(f"MESOR: {popt[0]:.2f} C")
print(f"Amplitude: {popt[1]:.3f} C")
print(f"Acrophase: {np.degrees(popt[2]):.1f} degrees ({popt[2]/(2*np.pi)*24:.1f} h)")

Core Capabilities

1. Diffusion MRI ADC Maps

Compute apparent diffusion coefficient maps from multi-b-value images.

import nibabel as nib
import numpy as np

def compute_adc_map(dwi_paths, b_values):
    """Compute ADC map from multi-b-value DWI images.

    ADC = -ln(S/S0) / b, where S0 is the b=0 image.

    Args:
        dwi_paths: list of NIfTI file paths (one per b-value)
        b_values: list of b-values in s/mm²
    """
    # Load images
    images = [nib.load(p).get_fdata() for p in dwi_paths]
    b_values = np.array(b_values)

    # Identify b=0 image
    b0_idx = np.argmin(b_values)
    S0 = images[b0_idx].astype(float)
    S0[S0 == 0] = 1e-10  # Avoid division by zero

    # Compute ADC using linear regression in log space
    # ln(S) = ln(S0) - b * ADC
    shape = S0.shape
    adc_map = np.zeros(shape)

    # Use non-zero b-values
    nonzero_mask = b_values > 0
    b_nonzero = b_values[nonzero_mask]

    for idx, (img, b) in enumerate(zip(images, b_values)):
        if b > 0:
            ratio = img.astype(float) / S0
            ratio = np.clip(ratio, 1e-10, None)
            adc_contribution = -np.log(ratio) / b
            adc_map += adc_contribution

    adc_map /= nonzero_mask.sum()

    # Mask out background (where S0 is very low)
    tissue_mask = S0 > np.percentile(S0[S0 > 0], 10)
    adc_map[~tissue_mask] = 0

    # Clip to physiological range (0 - 3.5 x 10^-3 mm²/s)
    adc_map = np.clip(adc_map, 0, 3.5e-3)

    print(f"ADC map shape: {adc_map.shape}")
    print(f"Mean ADC (tissue): {adc_map[tissue_mask].mean()*1e3:.3f} x10⁻³ mm²/s")
    print(f"Median ADC (tissue): {np.median(adc_map[tissue_mask])*1e3:.3f} x10⁻³ mm²/s")

    return adc_map, tissue_mask

def regional_adc_stats(adc_map, roi_mask):
    """Calculate ADC statistics within a region of interest."""
    roi_values = adc_map[roi_mask & (adc_map > 0)]
    return {
        'mean': roi_values.mean() * 1e3,
        'median': np.median(roi_values) * 1e3,
        'std': roi_values.std() * 1e3,
        'min': roi_values.min() * 1e3,
        'max': roi_values.max() * 1e3,
        'n_voxels': len(roi_values),
        'unit': '10⁻³ mm²/s'
    }

2. Micro-CT Bone Morphometry

Quantify bone microarchitecture from 3D micro-CT volumes.

import numpy as np
from scipy import ndimage
import skimage.filters

def bone_morphometry(volume, voxel_size_um, roi_mask=None):
    """Compute trabecular bone morphometric parameters.

    Args:
        volume: 3D numpy array (micro-CT volume)
        voxel_size_um: voxel size in micrometers
        roi_mask: optional ROI mask (binary 3D array)
    """
    if roi_mask is not None:
        vol = volume * roi_mask
    else:
        vol = volume.copy()
        roi_mask = np.ones_like(volume, dtype=bool)

    # Segment bone (Otsu thresholding)
    threshold = skimage.filters.threshold_otsu(vol[roi_mask])
    bone_mask = vol > threshold

    # BV/TV: Bone Volume / Total Volume
    total_voxels = roi_mask.sum()
    bone_voxels = (bone_mask & roi_mask).sum()
    bv_tv = bone_voxels / total_voxels

    # Trabecular Thickness (Tb.Th) via distance transform
    # Mean thickness = 2 * mean distance transform value within bone
    bone_distance = ndimage.distance_transform_edt(bone_mask & roi_mask)
    tb_th = 2 * bone_distance[bone_mask & roi_mask].mean() * voxel_size_um / 1000  # mm

    # Trabecular Spacing (Tb.Sp) via distance transform of marrow
    marrow_mask = ~bone_mask & roi_mask
    marrow_distance = ndimage.distance_transform_edt(marrow_mask)
    tb_sp = 2 * marrow_distance[marrow_mask].mean() * voxel_size_um / 1000  # mm

    # Trabecular Number (Tb.N)
    tb_n = bv_tv / tb_th if tb_th > 0 else 0  # 1/mm

    # Cortical thickness (if applicable)
    # Use morphological operations to identify cortex
    from skimage.morphology import binary_erosion, ball
    eroded = binary_erosion(bone_mask & roi_mask, ball(3))
    cortex = (bone_mask & roi_mask) & ~eroded
    cortex_dist = ndimage.distance_transform_edt(cortex)
    ct_th = cortex_dist[cortex].mean() * voxel_size_um / 1000 if cortex.sum() > 0 else 0

    results = {
        'BV/TV': bv_tv,
        'Tb.Th (mm)': tb_th,
        'Tb.Sp (mm)': tb_sp,
        'Tb.N (1/mm)': tb_n,
        'Ct.Th (mm)': ct_th,
        'bone_voxels': bone_voxels,
        'total_voxels': total_voxels
    }

    for key, val in results.items():
        if isinstance(val, float):
            print(f"{key}: {val:.4f}")

    return results

3. Hemodynamic Analysis

Process blood pressure waveforms.

import numpy as np
from scipy.signal import find_peaks

def analyze_blood_pressure(pressure_signal, sampling_rate_hz):
    """Analyze blood pressure waveform.

    Args:
        pressure_signal: 1D array of pressure values (mmHg)
        sampling_rate_hz: sampling frequency
    """
    # Detect systolic peaks
    min_distance = int(0.5 * sampling_rate_hz)  # Min 0.5s between beats
    peaks, props = find_peaks(pressure_signal, distance=min_distance,
                               prominence=20, height=60)

    # Detect diastolic troughs
    troughs, _ = find_peaks(-pressure_signal, distance=min_distance)

    # Systolic and diastolic pressures
    systolic = pressure_signal[peaks]
    # Match each peak to next trough
    diastolic = []
    for p in peaks:
        next_troughs = troughs[troughs > p]
        if len(next_troughs) > 0:
            diastolic.append(pressure_signal[next_troughs[0]])

    diastolic = np.array(diastolic[:len(systolic)])

    # Pulse pressure
    pulse_pressure = systolic[:len(diastolic)] - diastolic

    # Mean arterial pressure (MAP)
    map_pressure = diastolic + pulse_pressure / 3

    # Heart rate
    rr_intervals = np.diff(peaks) / sampling_rate_hz  # seconds
    heart_rate = 60 / rr_intervals  # BPM

    results = {
        'systolic_mean': np.mean(systolic),
        'systolic_std': np.std(systolic),
        'diastolic_mean': np.mean(diastolic),
        'diastolic_std': np.std(diastolic),
        'pulse_pressure_mean': np.mean(pulse_pressure),
        'MAP_mean': np.mean(map_pressure),
        'heart_rate_mean': np.mean(heart_rate),
        'heart_rate_std': np.std(heart_rate),
        'n_beats': len(peaks)
    }

    print(f"BP: {results['systolic_mean']:.0f}/{results['diastolic_mean']:.0f} mmHg")
    print(f"MAP: {results['MAP_mean']:.0f} mmHg")
    print(f"HR: {results['heart_rate_mean']:.0f} ± {results['heart_rate_std']:.0f} BPM")
    print(f"Pulse pressure: {results['pulse_pressure_mean']:.0f} mmHg")

    return results

4. Circadian Rhythm Cosinor

Fit circadian data with cosinor model.

import numpy as np
from scipy.optimize import curve_fit
from scipy import stats

def cosinor_analysis(time_hours, measurements, period=24):
    """Cosinor analysis: Y = MESOR + Amplitude * cos(2*pi*t/T + acrophase).

    Args:
        time_hours: time points in hours
        measurements: observed values
        period: assumed period in hours (default: 24)
    """
    def cosinor(t, mesor, amplitude, acrophase):
        return mesor + amplitude * np.cos(2 * np.pi * t / period + acrophase)

    # Initial estimates
    mesor_init = np.mean(measurements)
    amp_init = (np.max(measurements) - np.min(measurements)) / 2
    acro_init = 0

    popt, pcov = curve_fit(cosinor, time_hours, measurements,
                           p0=[mesor_init, amp_init, acro_init],
                           maxfev=10000)
    perr = np.sqrt(np.diag(pcov))

    mesor, amplitude, acrophase = popt

    # Ensure amplitude is positive
    if amplitude < 0:
        amplitude = -amplitude
        acrophase += np.pi

    # Normalize acrophase to [0, 2*pi]
    acrophase = acrophase % (2 * np.pi)

    # R-squared
    predicted = cosinor(time_hours, *popt)
    ss_res = np.sum((measurements - predicted) ** 2)
    ss_tot = np.sum((measurements - np.mean(measurements)) ** 2)
    r_squared = 1 - ss_res / ss_tot

    # F-test for rhythm significance
    n = len(measurements)
    f_stat = (ss_tot - ss_res) / 2 / (ss_res / (n - 3))
    p_value = 1 - stats.f.cdf(f_stat, 2, n - 3)

    # Convert acrophase to clock time
    acrophase_hours = (-acrophase / (2 * np.pi) * period) % period
    acro_h = int(acrophase_hours)
    acro_m = int((acrophase_hours - acro_h) * 60)

    results = {
        'MESOR': mesor,
        'amplitude': amplitude,
        'acrophase_rad': acrophase,
        'acrophase_hours': acrophase_hours,
        'acrophase_clock': f"{acro_h:02d}:{acro_m:02d}",
        'period': period,
        'r_squared': r_squared,
        'f_statistic': f_stat,
        'p_value': p_value,
        'significant': p_value < 0.05
    }

    print(f"MESOR: {mesor:.3f} ± {perr[0]:.3f}")
    print(f"Amplitude: {amplitude:.3f} ± {perr[1]:.3f}")
    print(f"Acrophase: {results['acrophase_clock']} ({acrophase_hours:.1f} h)")
    print(f"R²: {r_squared:.4f}")
    print(f"Rhythm p-value: {p_value:.2e} ({'significant' if p_value < 0.05 else 'not significant'})")

    return results

5. Ciliary Beat Frequency

Measure CBF from high-speed video using FFT.

import numpy as np
import cv2

def measure_ciliary_beat_frequency(video_path, fps, roi=None):
    """Measure ciliary beat frequency from high-speed video via FFT.

    Args:
        video_path: path to video file
        fps: frames per second of recording
        roi: (x, y, w, h) region of interest tuple, or None for full frame
    """
    cap = cv2.VideoCapture(video_path)
    frames = []

    while True:
        ret, frame = cap.read()
        if not ret:
            break
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        if roi:
            x, y, w, h = roi
            gray = gray[y:y+h, x:x+w]
        frames.append(gray.astype(float))

    cap.release()
    frames = np.array(frames)
    n_frames = len(frames)
    print(f"Loaded {n_frames} frames at {fps} fps ({n_frames/fps:.1f} seconds)")

    # Mean intensity time series per pixel
    mean_intensity = frames.mean(axis=(1, 2))

    # FFT of mean intensity
    fft_vals = np.fft.rfft(mean_intensity - mean_intensity.mean())
    freqs = np.fft.rfftfreq(n_frames, d=1/fps)
    power = np.abs(fft_vals) ** 2

    # Find dominant frequency (excluding DC and very low frequencies)
    min_freq_idx = max(1, int(2 * n_frames / fps))  # Ignore < 2 Hz
    max_freq_idx = len(freqs) - 1

    dominant_idx = min_freq_idx + np.argmax(power[min_freq_idx:max_freq_idx])
    cbf = freqs[dominant_idx]

    print(f"Dominant ciliary beat frequency: {cbf:.1f} Hz")
    print(f"Period: {1/cbf*1000:.1f} ms")

    # Spatiotemporal frequency map
    cbf_map = np.zeros(frames[0].shape)
    for y in range(0, frames.shape[1], 4):
        for x in range(0, frames.shape[2], 4):
            pixel_ts = frames[:, y, x]
            pixel_fft = np.fft.rfft(pixel_ts - pixel_ts.mean())
            pixel_power = np.abs(pixel_fft) ** 2
            peak_idx = min_freq_idx + np.argmax(pixel_power[min_freq_idx:max_freq_idx])
            cbf_map[y, x] = freqs[peak_idx]

    return cbf, cbf_map, freqs, power

6. Tissue Deformation Flow

Quantify tissue deformation using optical flow.

import cv2
import numpy as np

def analyze_tissue_deformation(frame1, frame2, method='lucas_kanade'):
    """Compute optical flow between two tissue images.

    Args:
        frame1, frame2: consecutive grayscale frames
        method: 'lucas_kanade' or 'farneback'
    """
    if method == 'farneback':
        flow = cv2.calcOpticalFlowFarneback(
            frame1, frame2, None,
            pyr_scale=0.5, levels=3, winsize=15,
            iterations=3, poly_n=5, poly_sigma=1.2, flags=0
        )
    else:  # Lucas-Kanade (sparse)
        pts = cv2.goodFeaturesToTrack(frame1, maxCorners=500,
                                      qualityLevel=0.01, minDistance=10)
        if pts is None:
            return None

        pts_new, status, err = cv2.calcOpticalFlowPyrLK(frame1, frame2, pts, None)
        good_old = pts[status.ravel() == 1]
        good_new = pts_new[status.ravel() == 1]

        # Compute displacement field
        dx = good_new[:, 0] - good_old[:, 0]
        dy = good_new[:, 1] - good_old[:, 1]
        magnitude = np.sqrt(dx**2 + dy**2)

        print(f"Tracked {len(good_old)} points")
        print(f"Mean displacement: {magnitude.mean():.2f} px")
        print(f"Max displacement: {magnitude.max():.2f} px")

        return {'dx': dx, 'dy': dy, 'magnitude': magnitude,
                'old_pts': good_old, 'new_pts': good_new}

    # For dense flow (Farneback)
    u = flow[:, :, 0]
    v = flow[:, :, 1]
    magnitude = np.sqrt(u**2 + v**2)

    # Divergence (expansion/contraction)
    du_dx = np.gradient(u, axis=1)
    dv_dy = np.gradient(v, axis=0)
    divergence = du_dx + dv_dy

    # Curl (rotation)
    du_dy = np.gradient(u, axis=0)
    dv_dx = np.gradient(v, axis=1)
    curl = dv_dx - du_dy

    # Strain tensor components
    exx = du_dx
    eyy = dv_dy
    exy = 0.5 * (du_dy + dv_dx)

    print(f"Mean displacement: {magnitude.mean():.3f} px")
    print(f"Mean divergence: {divergence.mean():.6f}")
    print(f"Mean curl: {curl.mean():.6f}")

    return {
        'flow': flow, 'magnitude': magnitude,
        'divergence': divergence, 'curl': curl,
        'strain_xx': exx, 'strain_yy': eyy, 'strain_xy': exy
    }

7. Amyloid Plaque Quantification

Segment and measure amyloid plaques from fluorescence images.

import numpy as np
import skimage.io
import skimage.filters
import skimage.measure
from skimage.morphology import remove_small_objects

def quantify_amyloid_plaques(image_path, min_plaque_area=50):
    """Quantify amyloid plaques from fluorescence microscopy.

    Args:
        image_path: path to fluorescence image (ThT/Congo Red staining)
        min_plaque_area: minimum plaque area in pixels
    """
    image = skimage.io.imread(image_path)
    if image.ndim == 3:
        image = image[:, :, 0]  # Take first channel

    # Preprocessing
    smoothed = skimage.filters.gaussian(image, sigma=2)

    # Threshold
    thresh = skimage.filters.threshold_otsu(smoothed)
    binary = smoothed > thresh

    # Remove small objects (noise)
    cleaned = remove_small_objects(binary, min_size=min_plaque_area)

    # Label and measure
    labels = skimage.measure.label(cleaned)
    regions = skimage.measure.regionprops(labels, intensity_image=image)

    # Extract measurements
    plaques = []
    for r in regions:
        plaques.append({
            'label': r.label,
            'area_px': r.area,
            'perimeter': r.perimeter,
            'eccentricity': r.eccentricity,
            'mean_intensity': r.mean_intensity,
            'max_intensity': r.max_intensity,
            'centroid_y': r.centroid[0],
            'centroid_x': r.centroid[1]
        })

    import pandas as pd
    df = pd.DataFrame(plaques)

    # Summary
    total_area = image.shape[0] * image.shape[1]
    plaque_area = df['area_px'].sum()
    plaque_density = len(df) / (total_area / 1e6)  # per million pixels

    print(f"Plaque count: {len(df)}")
    print(f"Total plaque area: {plaque_area} px ({100*plaque_area/total_area:.2f}%)")
    print(f"Mean plaque area: {df['area_px'].mean():.0f} px")
    print(f"Plaque density: {plaque_density:.1f} per Mpx")

    return df

Typical Workflows

Workflow 1: Compute ADC Map from Multi-b-Value Diffusion MRI

adc_map, mask = compute_adc_map(
    dwi_paths=['b0.nii.gz', 'b500.nii.gz', 'b1000.nii.gz'],
    b_values=[0, 500, 1000]
)
stats = regional_adc_stats(adc_map, mask)
print(f"Mean ADC: {stats['mean']:.3f} x10⁻³ mm²/s")

Workflow 2: Analyze Bone Microarchitecture from Micro-CT

import tifffile
volume = tifffile.imread('microct_stack.tif')
results = bone_morphometry(volume, voxel_size_um=10)
print(f"BV/TV: {results['BV/TV']:.4f}")
print(f"Tb.Th: {results['Tb.Th (mm)']:.4f} mm")

Workflow 3: Fit Circadian Rhythm Data with Cosinor Analysis

import numpy as np
time = np.arange(0, 48, 2)  # Every 2 hours for 48 hours
activity = 100 + 40 * np.cos(2 * np.pi * time / 24 - 0.5) + np.random.normal(0, 10, len(time))

results = cosinor_analysis(time, activity)
print(f"Peak activity at: {results['acrophase_clock']}")

Best Practices

1. ADC maps — use at least 2 non-zero b-values for reliable fitting; b=0 and b=1000 s/mm² is standard for brain
2. Bone morphometry — validate Otsu threshold against manual segmentation; report voxel size and scan parameters
3. Cosinor analysis — collect data spanning at least 2 full cycles; report MESOR, amplitude, acrophase, and p-value
4. Ciliary beat — recording must be at least 2x the expected CBF (Nyquist); typical CBF is 8-15 Hz requiring >30 fps
5. Optical flow — use Farneback for dense deformation fields, Lucas-Kanade for sparse tracking of features
6. Always report units — ADC in 10^-3 mm²/s, bone metrics in mm, blood pressure in mmHg, CBF in Hz

Troubleshooting

Problem: ADC map has noisy regions Solution: Apply Gaussian smoothing to DWI images before ADC calculation. Increase b-value range for better SNR. Mask out background using S0 threshold.

Problem: Bone segmentation includes soft tissue Solution: Apply manual ROI selection before thresholding. Use adaptive thresholding for inhomogeneous CT values. Verify Hounsfield unit calibration.

Problem: Cosinor analysis not significant Solution: Insufficient data points or too much noise. Collect more timepoints. Try different period values if 24h is not assumed. Check for ultradian rhythms.

Problem: CBF measurement gives wrong frequency Solution: Verify recording fps is correct. Check ROI contains actively beating cilia. Increase recording duration for better frequency resolution.

Resources

• nibabel Documentation
• SimpleITK Documentation
• Cosinor Analysis Method
• Bone Morphometry Standards (ASBMR)
• OpenCV Optical Flow Tutorial

Clinical Report Writing

Overview

Clinical report writing is the process of documenting medical information with precision, accuracy, and compliance with regulatory standards. This skill covers four major categories of clinical reports: case reports for journal publication, diagnostic reports for clinical practice, clinical trial reports for regulatory submission, and patient documentation for medical records. Apply this skill for healthcare documentation, research dissemination, and regulatory compliance.

Critical Principle: Clinical reports must be accurate, complete, objective, and compliant with applicable regulations (HIPAA, FDA, ICH-GCP). Patient privacy and data integrity are paramount. All clinical documentation must support evidence-based decision-making and meet professional standards.

When to Use This Skill

This skill should be used when:

• Writing clinical case reports for journal submission (CARE guidelines)
• Creating diagnostic reports (radiology, pathology, laboratory)
• Documenting clinical trial data and adverse events
• Preparing clinical study reports (CSR) for regulatory submission
• Writing patient progress notes, SOAP notes, and clinical summaries
• Drafting discharge summaries, H&P documents, or consultation notes
• Ensuring HIPAA compliance and proper de-identification
• Validating clinical documentation for completeness and accuracy
• Preparing serious adverse event (SAE) reports
• Creating data safety monitoring board (DSMB) reports

Visual Enhancement with Scientific Schematics

⚠️ MANDATORY: Every clinical report MUST include at least 1 AI-generated figure using the scientific-schematics skill.

This is not optional. Clinical reports benefit greatly from visual elements. Before finalizing any document:

1. Generate at minimum ONE schematic or diagram (e.g., patient timeline, diagnostic algorithm, or treatment workflow)
2. For case reports: include clinical progression timeline
3. For trial reports: include CONSORT flow diagram

How to generate figures:

• Use the scientific-schematics skill to generate AI-powered publication-quality diagrams
• Simply describe your desired diagram in natural language
• Nano Banana Pro will automatically generate, review, and refine the schematic

How to generate schematics:

python scripts/generate_schematic.py "your diagram description" -o figures/output.png

The AI will automatically:

• Create publication-quality images with proper formatting
• Review and refine through multiple iterations
• Ensure accessibility (colorblind-friendly, high contrast)
• Save outputs in the figures/ directory

When to add schematics:

• Patient case timelines and clinical progression diagrams
• Diagnostic algorithm flowcharts
• Treatment protocol workflows
• Anatomical diagrams for case reports
• Clinical trial participant flow diagrams (CONSORT)
• Adverse event classification trees
• Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.

Core Capabilities

1. Clinical Case Reports for Journal Publication

Clinical case reports describe unusual clinical presentations, novel diagnoses, or rare complications. They contribute to medical knowledge and are published in peer-reviewed journals.

CARE Guidelines Compliance

The CARE (CAse REport) guidelines provide a standardized framework for case report writing. All case reports should follow this checklist:

Title

• Include the words "case report" or "case study"
• Indicate the area of focus
• Example: "Unusual Presentation of Acute Myocardial Infarction in a Young Patient: A Case Report"

Keywords

• 2-5 keywords for indexing and searchability
• Use MeSH (Medical Subject Headings) terms when possible

Abstract (structured or unstructured, 150-250 words)

• Introduction: What is unique or novel about the case?
• Patient concerns: Primary symptoms and key medical history
• Diagnoses: Primary and secondary diagnoses
• Interventions: Key treatments and procedures
• Outcomes: Clinical outcome and follow-up
• Conclusions: Main takeaway or clinical lesson

Introduction

• Brief background on the medical condition
• Why this case is novel or important
• Literature review of similar cases (brief)
• What makes this case worth reporting

Patient Information

• Demographics (age, sex, race/ethnicity if relevant)
• Medical history, family history, social history
• Relevant comorbidities
• De-identification: Remove or alter 18 HIPAA identifiers
• Patient consent: Document informed consent for publication

Clinical Findings

• Chief complaint and presenting symptoms
• Physical examination findings
• Timeline of symptoms (consider timeline figure or table)
• Relevant clinical observations

Timeline

• Chronological summary of key events
• Dates of symptoms, diagnosis, interventions, outcomes
• Can be presented as a table or figure
• Example format:
  • Day 0: Initial presentation with symptoms X, Y, Z
  • Day 2: Diagnostic test A performed, revealed finding B
  • Day 5: Treatment initiated with drug C
  • Day 14: Clinical improvement noted
  • Month 3: Follow-up examination shows complete resolution

Diagnostic Assessment

• Diagnostic tests performed (labs, imaging, procedures)
• Results and interpretation
• Differential diagnosis considered
• Rationale for final diagnosis
• Challenges in diagnosis

Therapeutic Interventions

• Medications (names, dosages, routes, duration)
• Procedures or surgeries performed
• Non-pharmacological interventions
• Reasoning for treatment choices
• Alternative treatments considered

Follow-up and Outcomes

• Clinical outcome (resolution, improvement, unchanged, worsened)
• Follow-up duration and frequency
• Long-term outcomes if available
• Patient-reported outcomes
• Adherence to treatment

Discussion

• Strengths and novelty of the case
• How this case compares to existing literature
• Limitations of the case report
• Potential mechanisms or explanations
• Clinical implications and lessons learned
• Unanswered questions or areas for future research

Patient Perspective (optional but encouraged)

• Patient's experience and viewpoint
• Impact on quality of life
• Patient-reported outcomes
• Quote from patient if appropriate

Informed Consent

• Statement documenting patient consent for publication
• If patient deceased or unable to consent, describe proxy consent
• For pediatric cases, parental/guardian consent
• Example: "Written informed consent was obtained from the patient for publication of this case report and accompanying images. A copy of the written consent is available for review by the Editor-in-Chief of this journal."

For detailed CARE guidelines, refer to references/case_report_guidelines.md.

Journal-Specific Requirements

Different journals have specific formatting requirements:

• Word count limits (typically 1500-3000 words)
• Number of figures/tables allowed
• Reference style (AMA, Vancouver, APA)
• Structured vs. unstructured abstract
• Supplementary materials policies

Check journal instructions for authors before submission.

De-identification and Privacy

18 HIPAA Identifiers to Remove or Alter:

1. Names
2. Geographic subdivisions smaller than state
3. Dates (except year)
4. Telephone numbers
5. Fax numbers
6. Email addresses
7. Social Security numbers
8. Medical record numbers
9. Health plan beneficiary numbers
10. Account numbers
11. Certificate/license numbers
12. Vehicle identifiers and serial numbers
13. Device identifiers and serial numbers
14. Web URLs
15. IP addresses
16. Biometric identifiers
17. Full-face photographs
18. Any other unique identifying characteristic

Best Practices:

• Use "the patient" instead of names
• Report age ranges (e.g., "a woman in her 60s") or exact age if relevant
• Use approximate dates or time intervals (e.g., "3 months prior")
• Remove institution names unless necessary
• Blur or crop identifying features in images
• Obtain explicit consent for any potentially identifying information

2. Clinical Diagnostic Reports

Diagnostic reports communicate findings from imaging studies, pathological examinations, and laboratory tests. They must be clear, accurate, and actionable.

Radiology Reports

Radiology reports follow a standardized structure to ensure clarity and completeness.

Standard Structure:

1. Patient Demographics

• Patient name (or ID in research contexts)
• Date of birth or age
• Medical record number
• Examination date and time

2. Clinical Indication

• Reason for examination
• Relevant clinical history
• Specific clinical question to be answered
• Example: "Rule out pulmonary embolism in patient with acute dyspnea"

3. Technique

• Imaging modality (X-ray, CT, MRI, ultrasound, PET, etc.)
• Anatomical region examined
• Contrast administration (type, route, volume)
• Protocol or sequence used
• Technical quality and limitations
• Example: "Contrast-enhanced CT of the chest, abdomen, and pelvis was performed using 100 mL of intravenous iodinated contrast. Oral contrast was not administered."

4. Comparison

• Prior imaging studies available for comparison
• Dates of prior studies
• Stability or change from prior imaging
• Example: "Comparison: CT chest from [date]"

5. Findings

• Systematic description of imaging findings
• Organ-by-organ or region-by-region approach
• Positive findings first, then pertinent negatives
• Measurements of lesions or abnormalities
• Use of standardized terminology (ACR lexicon, RadLex)
• Example:
  • Lungs: Bilateral ground-glass opacities, predominant in the lower lobes. No consolidation or pleural effusion.
  • Mediastinum: No lymphadenopathy. Heart size normal.
  • Abdomen: Liver, spleen, pancreas unremarkable. No free fluid.

6. Impression/Conclusion

• Concise summary of key findings
• Answers to the clinical question
• Differential diagnosis if applicable
• Recommendations for follow-up or additional studies
• Level of suspicion or diagnostic certainty
• Example:
  • "1. Bilateral ground-glass opacities consistent with viral pneumonia or atypical infection. COVID-19 cannot be excluded. Clinical correlation recommended.
  • 2. No evidence of pulmonary embolism.
  • 3. Recommend follow-up imaging in 4-6 weeks to assess resolution."

Structured Reporting:

Many radiology departments use structured reporting templates for common examinations:

• Lung nodule reporting (Lung-RADS)
• Breast imaging (BI-RADS)
• Liver imaging (LI-RADS)
• Prostate imaging (PI-RADS)
• CT colonography (C-RADS)

Structured reports improve consistency, reduce ambiguity, and facilitate data extraction.

For radiology reporting standards, see references/diagnostic_reports_standards.md.

Pathology Reports

Pathology reports document microscopic findings from tissue specimens and provide diagnostic conclusions.

Surgical Pathology Report Structure:

1. Patient Information

• Patient name and identifiers
• Date of birth, age, sex
• Ordering physician
• Medical record number
• Specimen received date

2. Specimen Information

• Specimen type (biopsy, excision, resection)
• Anatomical site
• Laterality if applicable
• Number of specimens/blocks/slides
• Example: "Skin, left forearm, excisional biopsy"

3. Clinical History

• Relevant clinical information
• Indication for biopsy
• Prior diagnoses
• Example: "History of melanoma. New pigmented lesion, rule out recurrence."

4. Gross Description

• Macroscopic appearance of specimen
• Size, weight, color, consistency
• Orientation markers if present
• Sectioning and sampling approach
• Example: "The specimen consists of an ellipse of skin measuring 2.5 x 1.0 x 0.5 cm. A pigmented lesion measuring 0.6 cm in diameter is present on the surface. The specimen is serially sectioned and entirely submitted in cassettes A1-A3."

5. Microscopic Description

• Histological findings
• Cellular characteristics
• Architectural patterns
• Presence of malignancy
• Margins if applicable
• Special stains or immunohistochemistry results

6. Diagnosis

• Primary diagnosis
• Grade and stage if applicable (cancer)
• Margin status
• Lymph node status if applicable
• Synoptic reporting for cancers (CAP protocols)
• Example:
  • "MALIGNANT MELANOMA, SUPERFICIAL SPREADING TYPE
  • Breslow thickness: 1.2 mm
  • Clark level: IV
  • Mitotic rate: 3/mm²
  • Ulceration: Absent
  • Margins: Negative (closest margin 0.4 cm)
  • Lymphovascular invasion: Not identified"

7. Comment (if needed)

• Additional context or interpretation
• Differential diagnosis
• Recommendations for additional studies
• Clinical correlation suggestions

Synoptic Reporting:

The College of American Pathologists (CAP) provides synoptic reporting templates for cancer specimens. These checklists ensure all relevant diagnostic elements are documented.

Key elements for cancer reporting:

• Tumor site
• Tumor size
• Histologic type
• Histologic grade
• Extent of invasion
• Lymph-vascular invasion
• Perineural invasion
• Margins
• Lymph nodes (number examined, number positive)
• Pathologic stage (TNM classification)
• Ancillary studies (molecular markers, biomarkers)

Laboratory Reports

Laboratory reports communicate test results for clinical specimens (blood, urine, tissue, etc.).

Standard Components:

1. Patient and Specimen Information

• Patient identifiers
• Specimen type (blood, serum, urine, CSF, etc.)
• Collection date and time
• Received date and time
• Ordering provider

2. Test Name and Method

• Full test name
• Methodology (immunoassay, spectrophotometry, PCR, etc.)
• Laboratory accession number

3. Results

• Quantitative or qualitative result
• Units of measurement
• Reference range (normal values)
• Flags for abnormal values (H = high, L = low)
• Critical values highlighted
• Example:
  • Hemoglobin: 8.5 g/dL (L) [Reference: 12.0-16.0 g/dL]
  • White Blood Cell Count: 15.2 x10³/μL (H) [Reference: 4.5-11.0 x10³/μL]

4. Interpretation (when applicable)

• Clinical significance of results
• Suggested follow-up or additional testing
• Correlation with diagnosis
• Drug levels and therapeutic ranges

5. Quality Control Information

• Specimen adequacy
• Specimen quality issues (hemolyzed, lipemic, clotted)
• Delays in processing
• Technical limitations

Critical Value Reporting:

• Life-threatening results require immediate notification
• Examples: glucose <40 or >500 mg/dL, potassium <2.5 or >6.5 mEq/L
• Document notification time and recipient

For laboratory standards and terminology, see references/diagnostic_reports_standards.md.

3. Clinical Trial Reports

Clinical trial reports document the conduct, results, and safety of clinical research studies. These reports are essential for regulatory submissions and scientific publication.

Serious Adverse Event (SAE) Reports

SAE reports document unexpected serious adverse reactions during clinical trials. Regulatory requirements mandate timely reporting to IRBs, sponsors, and regulatory agencies.

Definition of Serious Adverse Event: An adverse event is serious if it:

• Results in death
• Is life-threatening
• Requires inpatient hospitalization or prolongation of existing hospitalization
• Results in persistent or significant disability/incapacity
• Is a congenital anomaly/birth defect
• Requires intervention to prevent permanent impairment or damage

SAE Report Components:

1. Study Information

• Protocol number and title
• Study phase
• Sponsor name
• Principal investigator
• IND/IDE number (if applicable)
• Clinical trial registry number (NCT number)

2. Patient Information (De-identified)

• Subject ID or randomization number
• Age, sex, race/ethnicity
• Study arm or treatment group
• Date of informed consent
• Date of first study intervention

3. Event Information

• Event description (narrative)
• Date of onset
• Date of resolution (or ongoing)
• Severity (mild, moderate, severe)
• Seriousness criteria met
• Outcome (recovered, recovering, not recovered, fatal, unknown)

4. Causality Assessment

• Relationship to study intervention (unrelated, unlikely, possible, probable, definite)
• Relationship to study procedures
• Relationship to underlying disease
• Rationale for causality determination

5. Action Taken

• Modification of study intervention (dose reduction, temporary hold, permanent discontinuation)
• Concomitant medications or treatments administered
• Hospitalization details
• Outcome and follow-up plan

6. Expectedness

• Expected per protocol or investigator's brochure
• Unexpected event requiring expedited reporting
• Comparison to known safety profile

7. Narrative

• Detailed description of the event
• Timeline of events
• Clinical course and management
• Laboratory and diagnostic test results
• Final diagnosis or conclusion

8. Reporter Information

• Name and contact of reporter
• Report date
• Signature

Regulatory Timelines:

• Fatal or life-threatening unexpected SAEs: 7 days for preliminary report, 15 days for complete report
• Other serious unexpected events: 15 days
• IRB notification: per institutional policy, typically within 5-10 days

For detailed SAE reporting guidance, see references/clinical_trial_reporting.md.

Clinical Study Reports (CSR)

Clinical study reports are comprehensive documents summarizing the design, conduct, and results of clinical trials. They are submitted to regulatory agencies as part of drug approval applications.

ICH-E3 Structure:

The ICH E3 guideline defines the structure and content of clinical study reports.

Main Sections:

1. Title Page

• Study title and protocol number
• Sponsor and investigator information
• Report date and version

2. Synopsis (5-15 pages)

• Brief summary of entire study
• Objectives, methods, results, conclusions
• Key efficacy and safety findings
• Can stand alone

3. Table of Contents

4. List of Abbreviations and Definitions

5. Ethics (Section 2)

• IRB/IEC approvals
• Informed consent process
• GCP compliance statement

6. Investigators and Study Administrative Structure (Section 3)

• List of investigators and sites
• Study organization
• Monitoring and quality assurance

7. Introduction (Section 4)

• Background and rationale
• Study objectives and purpose

8. Study Objectives and Plan (Section 5)

• Overall design and plan
• Objectives (primary and secondary)
• Endpoints (efficacy and safety)
• Sample size determination

9. Study Patients (Section 6)

• Inclusion and exclusion criteria
• Patient disposition
• Protocol deviations
• Demographic and baseline characteristics

10. Efficacy Evaluation (Section 7)

• Data sets analyzed (ITT, PP, safety)
• Demographic and other baseline characteristics
• Efficacy results for primary and secondary endpoints
• Subgroup analyses
• Dropouts and missing data

11. Safety Evaluation (Section 8)

• Extent of exposure
• Adverse events (summary tables)
• Serious adverse events (narratives)
• Laboratory values
• Vital signs and physical findings
• Deaths and other serious events

12. Discussion and Overall Conclusions (Section 9)

• Interpretation of results
• Benefit-risk assessment
• Clinical implications

13. Tables, Figures, and Graphs (Section 10)

14. Reference List (Section 11)

15. Appendices (Section 12)

• Study protocol and amendments
• Sample case report forms
• List of investigators and ethics committees
• Patient information and consent forms
• Investigator's brochure references
• Publications based on the study

Key Principles:

• Objectivity and transparency
• Comprehensive data presentation
• Adherence to statistical analysis plan
• Clear presentation of safety data
• Integration of appendices

For ICH-E3 templates and detailed guidance, see references/clinical_trial_reporting.md and assets/clinical_trial_csr_template.md.

Protocol Deviations

Protocol deviations are departures from the approved study protocol. They must be documented, assessed, and reported.

Categories:

• Minor deviation: Does not significantly impact patient safety or data integrity
• Major deviation: May impact patient safety, data integrity, or study conduct
• Violation: Serious deviation requiring immediate action and reporting

Documentation Requirements:

• Description of deviation
• Date of occurrence
• Subject ID affected
• Impact on safety and data
• Corrective and preventive actions (CAPA)
• Root cause analysis
• Preventive measures implemented

4. Patient Clinical Documentation

Patient documentation records clinical encounters, progress, and care plans. Accurate documentation supports continuity of care, billing, and legal protection.

SOAP Notes

SOAP notes are the most common format for progress notes in clinical practice.

Structure:

S - Subjective

• Patient's reported symptoms and concerns
• History of present illness (HPI)
• Review of systems (ROS) relevant to visit
• Patient's own words (use quotes when helpful)
• Example: "Patient reports worsening shortness of breath over the past 3 days, particularly with exertion. Denies chest pain, fever, or cough."

O - Objective

• Measurable clinical findings
• Vital signs (temperature, blood pressure, heart rate, respiratory rate, oxygen saturation)
• Physical examination findings (organized by system)
• Laboratory and imaging results
• Example:
  • Vitals: T 98.6°F, BP 142/88, HR 92, RR 22, SpO2 91% on room air
  • General: Mild respiratory distress
  • Cardiovascular: Regular rhythm, no murmurs
  • Pulmonary: Bilateral crackles at bases
  • Extremities: 2+ pitting edema bilaterally

A - Assessment

• Clinical impression or diagnosis
• Differential diagnosis
• Severity and stability
• Progress toward treatment goals
• Example:
  • "1. Acute decompensated heart failure, NYHA Class III
  • 2. Hypertension, poorly controlled
  • 3. Chronic kidney disease, stage 3"

P - Plan

• Diagnostic plan (further testing)
• Therapeutic plan (medications, procedures)
• Patient education and counseling
• Follow-up arrangements
• Example:
  • "Diagnostics: BNP, chest X-ray, echocardiogram
  • Therapeutics: Increase furosemide to 40 mg PO BID, continue lisinopril 10 mg daily, strict fluid restriction to 1.5 L/day
  • Education: Signs of worsening heart failure, daily weights
  • Follow-up: Cardiology appointment in 1 week, call if weight gain >2 lbs in 1 day"

Documentation Tips:

• Be concise but complete
• Use standard medical abbreviations
• Document time of encounter
• Sign and date all notes
• Avoid speculation or judgment
• Document medical necessity for billing
• Include patient's response to treatment

For SOAP note templates and examples, see assets/soap_note_template.md.

History and Physical (H&P)

The H&P is a comprehensive assessment performed at admission or initial encounter.

Components:

1. Chief Complaint (CC)

• Brief statement of why patient is seeking care
• Use patient's own words
• Example: "Chest pain for 2 hours"

2. History of Present Illness (HPI)

• Detailed chronological narrative of current problem
• Use OPQRST mnemonic for pain:
  • Onset: When did it start?
  • Provocation/Palliation: What makes it better or worse?
  • Quality: What does it feel like?
  • Region/Radiation: Where is it? Does it spread?
  • Severity: How bad is it (0-10 scale)?
  • Timing: Constant or intermittent? Duration?
• Associated symptoms
• Prior evaluations or treatments

3. Past Medical History (PMH)

• Chronic medical conditions
• Previous hospitalizations
• Surgeries and procedures
• Example: "Hypertension (diagnosed 2015), type 2 diabetes mellitus (diagnosed 2018), prior appendectomy (2010)"

4. Medications

• Current medications with doses and frequencies
• Over-the-counter medications
• Herbal supplements
• Allergies and reactions

5. Allergies

• Drug allergies with type of reaction
• Food allergies
• Environmental allergies
• Example: "Penicillin (rash), shellfish (anaphylaxis)"

6. Family History (FH)

• Medical conditions in first-degree relatives
• Age and cause of death of parents
• Hereditary conditions
• Example: "Father with coronary artery disease (MI at age 55), mother with breast cancer (diagnosed age 62)"

7. Social History (SH)

• Tobacco use (pack-years)
• Alcohol use (drinks per week)
• Illicit drug use
• Occupation
• Living situation
• Sexual history if relevant
• Example: "Former smoker, quit 5 years ago (20 pack-year history). Occasional alcohol (2-3 drinks/week). Works as accountant. Lives with spouse."

8. Review of Systems (ROS)

• Systematic review of symptoms by organ system
• Typically 10-14 systems
• Pertinent positives and negatives
• Systems: Constitutional, Eyes, ENT, Cardiovascular, Respiratory, GI, GU, Musculoskeletal, Skin, Neurological, Psychiatric, Endocrine, Hematologic/Lymphatic, Allergic/Immunologic

9. Physical Examination

• Vital signs
• General appearance
• Systematic examination by organ system
• HEENT, Neck, Cardiovascular, Pulmonary, Abdomen, Extremities, Neurological, Skin
• Use standard terminology and abbreviations

10. Assessment and Plan

• Problem list with assessment and plan for each
• Numbered list format
• Diagnostic and therapeutic plans
• Disposition (admit, discharge, transfer)

For H&P templates, see assets/history_physical_template.md.

Discharge Summaries

Discharge summaries document the hospital stay and communicate care plan to outpatient providers.

Required Elements:

1. Patient Identification

• Name, date of birth, medical record number
• Admission and discharge dates
• Attending physician
• Admitting and discharge diagnoses

2. Reason for Hospitalization

• Brief description of presenting problem
• Chief complaint

3. Hospital Course

• Chronological narrative of key events
• Significant findings and procedures
• Response to treatment
• Complications
• Consultations obtained
• Organized by problem or chronologically

4. Discharge Diagnoses

• Primary diagnosis
• Secondary diagnoses
• Complications
• Comorbidities

5. Procedures Performed

• Surgeries
• Invasive procedures
• Diagnostic procedures

6. Discharge Medications

• Complete medication list with instructions
• Changes from admission medications
• New medications with indications

7. Discharge Condition

• Stable, improved, unchanged, expired
• Functional status
• Mental status

8. Discharge Disposition

• Home, skilled nursing facility, rehabilitation, hospice
• With or without services

9. Follow-up Plans

• Appointments scheduled
• Recommended follow-up timing
• Pending tests or studies
• Referrals

10. Patient Instructions

• Activity restrictions
• Dietary restrictions
• Wound care
• Warning signs to seek care
• Medication instructions

Best Practices:

• Complete within 24-48 hours of discharge
• Use clear language for outpatient providers
• Highlight important pending results
• Document code status discussions
• Include patient education provided

For discharge summary templates, see assets/discharge_summary_template.md.

Regulatory Compliance and Privacy

HIPAA Compliance

The Health Insurance Portability and Accountability Act (HIPAA) mandates protection of patient health information.

Key Requirements:

• Minimum necessary disclosure
• Patient authorization for use beyond treatment/payment/operations
• Secure storage and transmission
• Audit trails for electronic records
• Breach notification procedures

De-identification Methods:

1. Safe Harbor Method: Remove 18 identifiers
2. Expert Determination: Statistical method confirming low re-identification risk

Business Associate Agreements: Required when PHI is shared with third parties for services

For detailed HIPAA guidance, see references/regulatory_compliance.md.

FDA Regulations

Clinical trial documentation must comply with FDA regulations:

• 21 CFR Part 11 (Electronic Records and Signatures)
• 21 CFR Part 50 (Informed Consent)
• 21 CFR Part 56 (IRB Standards)
• 21 CFR Part 312 (IND Regulations)

ICH-GCP Guidelines

Good Clinical Practice (GCP) guidelines ensure quality and ethical standards in clinical trials:

• Protocol adherence
• Informed consent documentation
• Source document requirements
• Audit trails and data integrity
• Investigator responsibilities

For ICH-GCP compliance, see references/regulatory_compliance.md.

Medical Terminology and Standards

Standardized Nomenclature

SNOMED CT (Systematized Nomenclature of Medicine - Clinical Terms)

• Comprehensive clinical terminology
• Used for electronic health records
• Enables semantic interoperability

LOINC (Logical Observation Identifiers Names and Codes)

• Standard for laboratory and clinical observations
• Facilitates data exchange and reporting

ICD-10-CM (International Classification of Diseases, 10th Revision, Clinical Modification)

• Diagnosis coding for billing and epidemiology
• Required for reimbursement

CPT (Current Procedural Terminology)

• Procedure coding for billing
• Maintained by AMA

Abbreviation Standards

Acceptable Abbreviations: Use standard abbreviations to improve efficiency while maintaining clarity.

Do Not Use List (Joint Commission):

• U (unit) - write "unit"
• IU (international unit) - write "international unit"
• QD, QOD (daily, every other day) - write "daily" or "every other day"
• Trailing zero (X.0 mg) - never use after decimal
• Lack of leading zero (.X mg) - always use before decimal (0.X mg)
• MS, MSO4, MgSO4 - write "morphine sulfate" or "magnesium sulfate"

For comprehensive terminology standards, see references/medical_terminology.md.

Quality Assurance and Validation

Documentation Quality Principles

Completeness:

• All required elements present
• No missing data fields
• Comprehensive patient information

Accuracy:

• Factually correct information
• Verified data sources
• Appropriate clinical reasoning

Timeliness:

• Documented contemporaneously or shortly after encounter
• Time-sensitive reports prioritized
• Regulatory deadlines met

Clarity:

• Clear and unambiguous language
• Organized logical structure
• Appropriate use of medical terminology

Compliance:

• Regulatory requirements met
• Privacy protections in place
• Institutional policies followed

Validation Checklists

For each report type, use validation checklists to ensure quality:

• Case report CARE checklist
• Diagnostic report completeness
• SAE report regulatory compliance
• Clinical documentation billing requirements

Validation scripts are available in the scripts/ directory.

Data Presentation in Clinical Reports

Tables and Figures

Tables for Clinical Data:

• Demographic and baseline characteristics
• Adverse events summary
• Laboratory values over time
• Efficacy outcomes

Table Design Principles:

• Clear column headers with units
• Footnotes for abbreviations and statistical notes
• Consistent formatting
• Appropriate precision (significant figures)

Figures for Clinical Data:

• Kaplan-Meier survival curves
• Forest plots for subgroup analyses
• Patient flow diagrams (CONSORT)
• Timeline figures for case reports
• Before-and-after images

Image Guidelines:

• High resolution (300 dpi minimum)
• Appropriate scale bars
• Annotations for key features
• De-identified (no patient identifiers visible)
• Informed consent for recognizable images

For data presentation standards, see references/data_presentation.md.

Integration with Other Skills

This clinical reports skill integrates with:

• Scientific Writing: For clear, professional medical writing
• Peer Review: For quality assessment of case reports
• Citation Management: For literature references in case reports
• Research Grants: For clinical trial protocol development
• Literature Review: For background sections in case reports

Workflow for Clinical Report Writing

Case Report Workflow

Phase 1: Case Identification and Consent (Week 1)

• Identify novel or educational case
• Obtain patient informed consent
• De-identify patient information
• Collect clinical data and images

Phase 2: Literature Review (Week 1-2)

• Search for similar cases
• Review relevant pathophysiology
• Identify knowledge gaps
• Determine novelty and significance

Phase 3: Drafting (Week 2-3)

• Write structured outline following CARE guidelines
• Draft all sections (abstract through discussion)
• Create timeline and figures
• Format references

Phase 4: Internal Review (Week 3-4)

• Co-author review
• Attending physician review
• Institutional review if required
• Patient review of de-identified draft

Phase 5: Journal Selection and Submission (Week 4-5)

• Select appropriate journal
• Format per journal guidelines
• Prepare cover letter
• Submit manuscript

Phase 6: Revision (Variable)

• Respond to peer reviewer comments
• Revise manuscript
• Resubmit

Diagnostic Report Workflow

Real-time Workflow:

• Review clinical indication and prior studies
• Interpret imaging, pathology, or laboratory findings
• Dictate or type report using structured format
• Peer review for complex cases
• Final sign-out and distribution
• Critical value notification if applicable

Turnaround Time Benchmarks:

• STAT reports: <1 hour
• Routine reports: 24-48 hours
• Complex cases: 2-5 days
• Pending additional studies: documented delay

Clinical Trial Report Workflow

SAE Report: 24 hours to 15 days

• Event identified by site
• Initial assessment and documentation
• Causality and expectedness determination
• Report completion and review
• Submission to sponsor, IRB, FDA (as required)
• Follow-up reporting until resolution

CSR: 6-12 months post-study completion

• Database lock and data cleaning
• Statistical analysis per SAP
• Drafting by medical writer
• Review by biostatistician and clinical team
• Quality control review
• Final approval and regulatory submission

Resources

This skill includes comprehensive reference files and templates:

Reference Files

• references/case_report_guidelines.md - CARE guidelines, journal requirements, writing tips
• references/diagnostic_reports_standards.md - ACR, CAP, laboratory reporting standards
• references/clinical_trial_reporting.md - ICH-E3, CONSORT, SAE reporting, CSR structure
• references/patient_documentation.md - SOAP notes, H&P, discharge summaries, coding
• references/regulatory_compliance.md - HIPAA, 21 CFR Part 11, ICH-GCP, FDA requirements
• references/medical_terminology.md - SNOMED, LOINC, ICD-10, abbreviations, nomenclature
• references/data_presentation.md - Tables, figures, safety data, CONSORT diagrams
• references/peer_review_standards.md - Review criteria for clinical manuscripts

Template Assets

• assets/case_report_template.md - Structured case report following CARE guidelines
• assets/radiology_report_template.md - Standard radiology report format
• assets/pathology_report_template.md - Surgical pathology report with synoptic elements
• assets/lab_report_template.md - Clinical laboratory report format
• assets/clinical_trial_sae_template.md - Serious adverse event report form
• assets/clinical_trial_csr_template.md - Clinical study report outline per ICH-E3
• assets/soap_note_template.md - SOAP progress note format
• assets/history_physical_template.md - Comprehensive H&P template
• assets/discharge_summary_template.md - Hospital discharge summary
• assets/consult_note_template.md - Consultation note format
• assets/quality_checklist.md - Quality assurance checklist for all report types
• assets/hipaa_compliance_checklist.md - Privacy and de-identification checklist

Automation Scripts

• scripts/validate_case_report.py - Check CARE guideline compliance and completeness
• scripts/validate_trial_report.py - Verify ICH-E3 structure and required elements
• scripts/check_deidentification.py - Scan for 18 HIPAA identifiers in text
• scripts/format_adverse_events.py - Generate AE summary tables from data
• scripts/generate_report_template.py - Interactive template selection and generation
• scripts/extract_clinical_data.py - Parse structured data from clinical reports
• scripts/compliance_checker.py - Verify regulatory compliance requirements
• scripts/terminology_validator.py - Validate medical terminology and coding

Load these resources as needed when working on specific clinical reports.

Common Pitfalls to Avoid

Case Reports

• Privacy violations: Inadequate de-identification or missing consent
• Lack of novelty: Reporting common or well-documented cases
• Insufficient detail: Missing key clinical information
• Poor literature review: Failure to contextualize within existing knowledge
• Overgeneralization: Drawing broad conclusions from single case

Diagnostic Reports

• Vague language: Using ambiguous terms like "unremarkable" without specifics
• Incomplete comparison: Not reviewing prior imaging
• Missing clinical correlation: Failing to answer clinical question
• Technical jargon: Overuse of terminology without explanation
• Delayed critical value notification: Not communicating urgent findings

Clinical Trial Reports

• Late reporting: Missing regulatory deadlines for SAE reporting
• Incomplete causality: Inadequate causality assessment
• Data inconsistencies: Discrepancies between data sources
• Protocol deviations: Unreported or inadequately documented deviations
• Selective reporting: Omitting negative or unfavorable results

Patient Documentation

• Illegibility: Poor handwriting in paper records
• Copy-forward errors: Propagating outdated information
• Insufficient detail: Vague or incomplete documentation affecting billing
• Lack of medical necessity: Not documenting indication for services
• Missing signatures: Unsigned or undated notes

Final Checklist

Before finalizing any clinical report, verify:

• All required sections complete
• Patient privacy protected (HIPAA compliance)
• Informed consent obtained (if applicable)
• Accurate and verified clinical data
• Appropriate medical terminology and coding
• Clear, professional language
• Proper formatting per guidelines
• References cited appropriately
• Figures and tables labeled correctly
• Spell-checked and proofread
• Regulatory requirements met
• Institutional policies followed
• Signatures and dates present
• Quality assurance review completed

Final Note: Clinical report writing requires attention to detail, medical accuracy, regulatory compliance, and clear communication. Whether documenting patient care, reporting research findings, or communicating diagnostic results, the quality of clinical reports directly impacts patient safety, healthcare delivery, and medical knowledge advancement. Always prioritize accuracy, privacy, and professionalism in all clinical documentation.

COBRApy - Constraint-Based Reconstruction and Analysis

Overview

COBRApy is a Python library for constraint-based reconstruction and analysis (COBRA) of metabolic models, essential for systems biology research. Work with genome-scale metabolic models, perform computational simulations of cellular metabolism, conduct metabolic engineering analyses, and predict phenotypic behaviors.

Core Capabilities

COBRApy provides comprehensive tools organized into several key areas:

1. Model Management

Load existing models from repositories or files:

from cobra.io import load_model

# Load bundled test models
model = load_model("textbook")  # E. coli core model
model = load_model("ecoli")     # Full E. coli model
model = load_model("salmonella")

# Load from files
from cobra.io import read_sbml_model, load_json_model, load_yaml_model
model = read_sbml_model("path/to/model.xml")
model = load_json_model("path/to/model.json")
model = load_yaml_model("path/to/model.yml")

Save models in various formats:

from cobra.io import write_sbml_model, save_json_model, save_yaml_model
write_sbml_model(model, "output.xml")  # Preferred format
save_json_model(model, "output.json")  # For Escher compatibility
save_yaml_model(model, "output.yml")   # Human-readable

2. Model Structure and Components

Access and inspect model components:

# Access components
model.reactions      # DictList of all reactions
model.metabolites    # DictList of all metabolites
model.genes          # DictList of all genes

# Get specific items by ID or index
reaction = model.reactions.get_by_id("PFK")
metabolite = model.metabolites[0]

# Inspect properties
print(reaction.reaction)        # Stoichiometric equation
print(reaction.bounds)          # Flux constraints
print(reaction.gene_reaction_rule)  # GPR logic
print(metabolite.formula)       # Chemical formula
print(metabolite.compartment)   # Cellular location

3. Flux Balance Analysis (FBA)

Perform standard FBA simulation:

# Basic optimization
solution = model.optimize()
print(f"Objective value: {solution.objective_value}")
print(f"Status: {solution.status}")

# Access fluxes
print(solution.fluxes["PFK"])
print(solution.fluxes.head())

# Fast optimization (objective value only)
objective_value = model.slim_optimize()

# Change objective
model.objective = "ATPM"
solution = model.optimize()

Parsimonious FBA (minimize total flux):

from cobra.flux_analysis import pfba
solution = pfba(model)

Geometric FBA (find central solution):

from cobra.flux_analysis import geometric_fba
solution = geometric_fba(model)

4. Flux Variability Analysis (FVA)

Determine flux ranges for all reactions:

from cobra.flux_analysis import flux_variability_analysis

# Standard FVA
fva_result = flux_variability_analysis(model)

# FVA at 90% optimality
fva_result = flux_variability_analysis(model, fraction_of_optimum=0.9)

# Loopless FVA (eliminates thermodynamically infeasible loops)
fva_result = flux_variability_analysis(model, loopless=True)

# FVA for specific reactions
fva_result = flux_variability_analysis(
    model,
    reaction_list=["PFK", "FBA", "PGI"]
)

5. Gene and Reaction Deletion Studies

Perform knockout analyses:

from cobra.flux_analysis import (
    single_gene_deletion,
    single_reaction_deletion,
    double_gene_deletion,
    double_reaction_deletion
)

# Single deletions
gene_results = single_gene_deletion(model)
reaction_results = single_reaction_deletion(model)

# Double deletions (uses multiprocessing)
double_gene_results = double_gene_deletion(
    model,
    processes=4  # Number of CPU cores
)

# Manual knockout using context manager
with model:
    model.genes.get_by_id("b0008").knock_out()
    solution = model.optimize()
    print(f"Growth after knockout: {solution.objective_value}")
# Model automatically reverts after context exit

6. Growth Media and Minimal Media

Manage growth medium:

# View current medium
print(model.medium)

# Modify medium (must reassign entire dict)
medium = model.medium
medium["EX_glc__D_e"] = 10.0  # Set glucose uptake
medium["EX_o2_e"] = 0.0       # Anaerobic conditions
model.medium = medium

# Calculate minimal media
from cobra.medium import minimal_medium

# Minimize total import flux
min_medium = minimal_medium(model, minimize_components=False)

# Minimize number of components (uses MILP, slower)
min_medium = minimal_medium(
    model,
    minimize_components=True,
    open_exchanges=True
)

7. Flux Sampling

Sample the feasible flux space:

from cobra.sampling import sample

# Sample using OptGP (default, supports parallel processing)
samples = sample(model, n=1000, method="optgp", processes=4)

# Sample using ACHR
samples = sample(model, n=1000, method="achr")

# Validate samples
from cobra.sampling import OptGPSampler
sampler = OptGPSampler(model, processes=4)
sampler.sample(1000)
validation = sampler.validate(sampler.samples)
print(validation.value_counts())  # Should be all 'v' for valid

8. Production Envelopes

Calculate phenotype phase planes:

from cobra.flux_analysis import production_envelope

# Standard production envelope
envelope = production_envelope(
    model,
    reactions=["EX_glc__D_e", "EX_o2_e"],
    objective="EX_ac_e"  # Acetate production
)

# With carbon yield
envelope = production_envelope(
    model,
    reactions=["EX_glc__D_e", "EX_o2_e"],
    carbon_sources="EX_glc__D_e"
)

# Visualize (use matplotlib or pandas plotting)
import matplotlib.pyplot as plt
envelope.plot(x="EX_glc__D_e", y="EX_o2_e", kind="scatter")
plt.show()

9. Gapfilling

Add reactions to make models feasible:

from cobra.flux_analysis import gapfill

# Prepare universal model with candidate reactions
universal = load_model("universal")

# Perform gapfilling
with model:
    # Remove reactions to create gaps for demonstration
    model.remove_reactions([model.reactions.PGI])

    # Find reactions needed
    solution = gapfill(model, universal)
    print(f"Reactions to add: {solution}")

10. Model Building

Build models from scratch:

from cobra import Model, Reaction, Metabolite

# Create model
model = Model("my_model")

# Create metabolites
atp_c = Metabolite("atp_c", formula="C10H12N5O13P3",
                   name="ATP", compartment="c")
adp_c = Metabolite("adp_c", formula="C10H12N5O10P2",
                   name="ADP", compartment="c")
pi_c = Metabolite("pi_c", formula="HO4P",
                  name="Phosphate", compartment="c")

# Create reaction
reaction = Reaction("ATPASE")
reaction.name = "ATP hydrolysis"
reaction.subsystem = "Energy"
reaction.lower_bound = 0.0
reaction.upper_bound = 1000.0

# Add metabolites with stoichiometry
reaction.add_metabolites({
    atp_c: -1.0,
    adp_c: 1.0,
    pi_c: 1.0
})

# Add gene-reaction rule
reaction.gene_reaction_rule = "(gene1 and gene2) or gene3"

# Add to model
model.add_reactions([reaction])

# Add boundary reactions
model.add_boundary(atp_c, type="exchange")
model.add_boundary(adp_c, type="demand")

# Set objective
model.objective = "ATPASE"

Common Workflows

Workflow 1: Load Model and Predict Growth

from cobra.io import load_model

# Load model
model = load_model("ecoli")

# Run FBA
solution = model.optimize()
print(f"Growth rate: {solution.objective_value:.3f} /h")

# Show active pathways
print(solution.fluxes[solution.fluxes.abs() > 1e-6])

Workflow 2: Gene Knockout Screen

from cobra.io import load_model
from cobra.flux_analysis import single_gene_deletion

# Load model
model = load_model("ecoli")

# Perform single gene deletions
results = single_gene_deletion(model)

# Find essential genes (growth < threshold)
essential_genes = results[results["growth"] < 0.01]
print(f"Found {len(essential_genes)} essential genes")

# Find genes with minimal impact
neutral_genes = results[results["growth"] > 0.9 * solution.objective_value]

Workflow 3: Media Optimization

from cobra.io import load_model
from cobra.medium import minimal_medium

# Load model
model = load_model("ecoli")

# Calculate minimal medium for 50% of max growth
target_growth = model.slim_optimize() * 0.5
min_medium = minimal_medium(
    model,
    target_growth,
    minimize_components=True
)

print(f"Minimal medium components: {len(min_medium)}")
print(min_medium)

Workflow 4: Flux Uncertainty Analysis

from cobra.io import load_model
from cobra.flux_analysis import flux_variability_analysis
from cobra.sampling import sample

# Load model
model = load_model("ecoli")

# First check flux ranges at optimality
fva = flux_variability_analysis(model, fraction_of_optimum=1.0)

# For reactions with large ranges, sample to understand distribution
samples = sample(model, n=1000)

# Analyze specific reaction
reaction_id = "PFK"
import matplotlib.pyplot as plt
samples[reaction_id].hist(bins=50)
plt.xlabel(f"Flux through {reaction_id}")
plt.ylabel("Frequency")
plt.show()

Workflow 5: Context Manager for Temporary Changes

Use context managers to make temporary modifications:

# Model remains unchanged outside context
with model:
    # Temporarily change objective
    model.objective = "ATPM"

    # Temporarily modify bounds
    model.reactions.EX_glc__D_e.lower_bound = -5.0

    # Temporarily knock out genes
    model.genes.b0008.knock_out()

    # Optimize with changes
    solution = model.optimize()
    print(f"Modified growth: {solution.objective_value}")

# All changes automatically reverted
solution = model.optimize()
print(f"Original growth: {solution.objective_value}")

Key Concepts

DictList Objects

Models use DictList objects for reactions, metabolites, and genes - behaving like both lists and dictionaries:

# Access by index
first_reaction = model.reactions[0]

# Access by ID
pfk = model.reactions.get_by_id("PFK")

# Query methods
atp_reactions = model.reactions.query("atp")

Flux Constraints

Reaction bounds define feasible flux ranges:

• Irreversible: lower_bound = 0, upper_bound > 0
• Reversible: lower_bound < 0, upper_bound > 0
• Set both bounds simultaneously with .bounds to avoid inconsistencies

Gene-Reaction Rules (GPR)

Boolean logic linking genes to reactions:

# AND logic (both required)
reaction.gene_reaction_rule = "gene1 and gene2"

# OR logic (either sufficient)
reaction.gene_reaction_rule = "gene1 or gene2"

# Complex logic
reaction.gene_reaction_rule = "(gene1 and gene2) or (gene3 and gene4)"

Exchange Reactions

Special reactions representing metabolite import/export:

• Named with prefix EX_ by convention
• Positive flux = secretion, negative flux = uptake
• Managed through model.medium dictionary

Best Practices

1. Use context managers for temporary modifications to avoid state management issues
2. Validate models before analysis using model.slim_optimize() to ensure feasibility
3. Check solution status after optimization - optimal indicates successful solve
4. Use loopless FVA when thermodynamic feasibility matters
5. Set fraction_of_optimum appropriately in FVA to explore suboptimal space
6. Parallelize computationally expensive operations (sampling, double deletions)
7. Prefer SBML format for model exchange and long-term storage
8. Use slim_optimize() when only objective value needed for performance
9. Validate flux samples to ensure numerical stability

Troubleshooting

Infeasible solutions: Check medium constraints, reaction bounds, and model consistency Slow optimization: Try different solvers (GLPK, CPLEX, Gurobi) via model.solver Unbounded solutions: Verify exchange reactions have appropriate upper bounds Import errors: Ensure correct file format and valid SBML identifiers

References

For detailed workflows and API patterns, refer to:

• references/workflows.md - Comprehensive step-by-step workflow examples
• references/api_quick_reference.md - Common function signatures and patterns

Official documentation: https://cobrapy.readthedocs.io/en/latest/

Curated Bio-Datasets: Biological Datasets Guide

Overview

Curated Bio-Datasets provides a comprehensive guide to accessing and working with major curated biological datasets. This skill covers COSMIC cancer genomics data, GTEx tissue expression data, GWAS Catalog SNP-trait associations, GeneBass exome-wide association results, BioGRID protein-protein interaction data, MSigDB gene set collections, DisGeNET disease-gene associations, and Gene Ontology resources. Each section includes download patterns, file formats, parsing code, and integration examples.

When to Use This Skill

• Downloading and parsing COSMIC cancer gene census data
• Accessing GTEx tissue-level expression data (TPM matrices, eQTLs)
• Querying the GWAS Catalog for SNP-trait associations
• Working with GeneBass exome-wide burden test results
• Building protein-protein interaction networks from BioGRID
• Loading MSigDB gene sets for pathway enrichment analysis
• Querying DisGeNET for disease-gene associations
• Working with Gene Ontology terms and hierarchies

Related Skills: For specific database API access use dedicated skills: cosmic-database, gwas-database, ensembl-database, kegg-database, reactome-database.

Installation

uv pip install pandas requests networkx gseapy numpy

Quick Start

import pandas as pd

# Load MSigDB gene sets (GMT format) for enrichment analysis
def parse_gmt(gmt_path):
    gene_sets = {}
    with open(gmt_path) as f:
        for line in f:
            parts = line.strip().split('\t')
            name = parts[0]
            genes = parts[2:]  # Skip description
            gene_sets[name] = genes
    return gene_sets

# Example: run enrichment with gseapy
import gseapy as gp
enr = gp.enrichr(gene_list=['TP53', 'BRCA1', 'ATM', 'CHEK2', 'PTEN'],
                 gene_sets='MSigDB_Hallmark_2020', outdir=None)
print(enr.results[['Term', 'Adjusted P-value', 'Overlap']].head())

Core Capabilities

1. COSMIC Cancer Datasets

Access the Catalogue of Somatic Mutations in Cancer.

import pandas as pd

# COSMIC Cancer Gene Census
# Download from: https://cancer.sanger.ac.uk/census (requires free registration)
# File: cancer_gene_census.csv

def load_cosmic_census(census_path='cancer_gene_census.csv'):
    """Load and parse COSMIC Cancer Gene Census."""
    df = pd.read_csv(census_path)

    print(f"Total cancer genes: {len(df)}")
    print(f"\nTier distribution:")
    print(df['Tier'].value_counts())

    print(f"\nRole in Cancer:")
    roles = df['Role in Cancer'].str.split(', ').explode()
    print(roles.value_counts())

    print(f"\nTop mutation types:")
    mut_types = df['Mutation Types'].str.split(', ').explode()
    print(mut_types.value_counts().head(5))

    return df

def filter_census_by_cancer(census_df, cancer_type):
    """Filter census for a specific cancer type."""
    mask = census_df['Tumour Types(Somatic)'].str.contains(cancer_type, case=False, na=False)
    filtered = census_df[mask]
    print(f"Genes associated with '{cancer_type}': {len(filtered)}")
    return filtered

# COSMIC somatic mutations
# Download: CosmicMutantExportCensus.tsv.gz
def load_cosmic_mutations(mutations_path):
    """Load COSMIC somatic mutation data."""
    df = pd.read_csv(mutations_path, sep='\t', low_memory=False)
    print(f"Total mutations: {len(df)}")
    print(f"Unique genes: {df['Gene name'].nunique()}")
    print(f"Mutation types:\n{df['Mutation Description'].value_counts().head()}")
    return df

# COSMIC resistance mutations
# Useful for pharmacogenomics studies
def get_resistance_genes(census_df):
    """Extract genes with known drug resistance mutations."""
    mask = census_df['Mutation Types'].str.contains('resistance', case=False, na=False) | \
           census_df['Role in Cancer'].str.contains('resistance', case=False, na=False)
    return census_df[mask][['Gene Symbol', 'Mutation Types', 'Role in Cancer']]

2. GTEx Tissue Expression

Access the Genotype-Tissue Expression project data.

import pandas as pd
import requests

# GTEx Portal API
GTEX_API = 'https://gtexportal.org/api/v2'

def get_gtex_gene_expression(gene_symbol, dataset_id='gtex_v8'):
    """Get median gene expression across tissues from GTEx API."""
    url = f"{GTEX_API}/expression/medianGeneExpression"
    params = {
        'geneSymbol': gene_symbol,
        'datasetId': dataset_id,
    }
    response = requests.get(url, params=params)
    if response.status_code != 200:
        print(f"Error: {response.status_code}")
        return None

    data = response.json()
    if 'data' not in data:
        print("No data returned")
        return None

    df = pd.DataFrame(data['data'])
    df = df.sort_values('median', ascending=False)

    print(f"Expression of {gene_symbol} across {len(df)} tissues:")
    print(df[['tissueSiteDetailId', 'median']].head(10).to_string(index=False))
    return df

# GTEx bulk download patterns
# TPM matrix: GTEx_Analysis_2017-06-05_v8_RNASeQCv1.1.9_gene_tpm.gct.gz
# Sample annotations: GTEx_Analysis_v8_Annotations_SampleAttributesDS.txt
# eQTL: GTEx_Analysis_v8_eQTL.tar

def load_gtex_tpm(tpm_path):
    """Load GTEx TPM matrix (GCT format)."""
    df = pd.read_csv(tpm_path, sep='\t', skiprows=2)
    genes = df['Name']
    descriptions = df['Description']
    expression = df.iloc[:, 2:]

    print(f"Genes: {len(genes)}")
    print(f"Samples: {expression.shape[1]}")
    return df

3. GWAS Catalog

Access SNP-trait association data.

import pandas as pd

# Download: https://www.ebi.ac.uk/gwas/api/search/downloads/full
# File: gwas_catalog_v1.0.2-associations_e*.tsv

def load_gwas_catalog(catalog_path):
    """Load GWAS Catalog associations."""
    df = pd.read_csv(catalog_path, sep='\t', low_memory=False)

    print(f"Total associations: {len(df)}")
    print(f"Unique traits: {df['DISEASE/TRAIT'].nunique()}")
    print(f"Unique SNPs: {df['SNPS'].nunique()}")

    return df

def search_gwas_by_trait(catalog_df, trait_keyword, pvalue_threshold=5e-8):
    """Search GWAS catalog for a trait."""
    mask = catalog_df['DISEASE/TRAIT'].str.contains(trait_keyword, case=False, na=False)

    # Filter by p-value
    filtered = catalog_df[mask].copy()
    filtered['P-VALUE'] = pd.to_numeric(filtered['P-VALUE'], errors='coerce')
    filtered = filtered[filtered['P-VALUE'] < pvalue_threshold]

    filtered = filtered.sort_values('P-VALUE')

    print(f"Associations for '{trait_keyword}': {len(filtered)}")
    if len(filtered) > 0:
        print(filtered[['SNPS', 'MAPPED_GENE', 'P-VALUE', 'OR or BETA']].head(10))
    return filtered

def search_gwas_by_gene(catalog_df, gene_symbol):
    """Search GWAS catalog for associations near a gene."""
    mask = catalog_df['MAPPED_GENE'].str.contains(gene_symbol, case=False, na=False) | \
           catalog_df['REPORTED GENE(S)'].str.contains(gene_symbol, case=False, na=False)
    results = catalog_df[mask].sort_values('P-VALUE')
    print(f"GWAS hits near {gene_symbol}: {len(results)}")
    return results

4. GeneBass Exome Data

Access exome-wide association results.

import pandas as pd

# GeneBass: https://genebass.org/
# Bulk download: results per phenotype in TSV format

def load_genebass_results(results_path, gene=None, pvalue_threshold=2.5e-6):
    """Load GeneBass exome-wide association results.

    Args:
        results_path: path to GeneBass results TSV
        gene: optional gene filter
        pvalue_threshold: exome-wide significance threshold
    """
    df = pd.read_csv(results_path, sep='\t')

    if gene:
        df = df[df['gene_symbol'] == gene]

    # Filter significant results
    sig = df[df['pvalue'] < pvalue_threshold]

    print(f"Total results: {len(df)}")
    print(f"Significant (p < {pvalue_threshold}): {len(sig)}")

    if len(sig) > 0:
        print("\nTop hits:")
        print(sig[['gene_symbol', 'annotation', 'pvalue', 'beta']].head(10))

    return df

# Burden test categories
# pLoF: predicted loss of function
# missense: missense variants
# synonymous: synonymous (used as negative control)
def compare_burden_tests(results_df, gene):
    """Compare burden test results across variant categories."""
    gene_data = results_df[results_df['gene_symbol'] == gene]
    for ann in ['pLoF', 'missense|LC', 'synonymous']:
        subset = gene_data[gene_data['annotation'] == ann]
        if len(subset) > 0:
            row = subset.iloc[0]
            print(f"{ann}: p={row['pvalue']:.2e}, beta={row.get('beta', 'N/A')}")

5. Protein Interaction Networks

Build and analyze networks from BioGRID data.

import pandas as pd
import networkx as nx

# Download: https://thebiogrid.org/downloads/archives/Latest%20Release
# File: BIOGRID-ALL-LATEST.tab3.zip

def load_biogrid(biogrid_path, organism=9606, experiment_types=None):
    """Load BioGRID protein-protein interaction data.

    Args:
        biogrid_path: path to BioGRID tab3 file
        organism: NCBI taxonomy ID (9606 = human)
        experiment_types: list of experiment types to include
    """
    cols = ['BioGRID Interaction ID', 'Official Symbol Interactor A',
            'Official Symbol Interactor B', 'Organism ID Interactor A',
            'Organism ID Interactor B', 'Experimental System',
            'Experimental System Type', 'Throughput']

    df = pd.read_csv(biogrid_path, sep='\t', usecols=cols, low_memory=False)

    # Filter by organism
    df = df[(df['Organism ID Interactor A'] == organism) &
            (df['Organism ID Interactor B'] == organism)]

    # Filter by experiment type
    if experiment_types:
        df = df[df['Experimental System'].isin(experiment_types)]

    print(f"Interactions: {len(df)}")
    print(f"Unique proteins: {pd.concat([df['Official Symbol Interactor A'],
                                          df['Official Symbol Interactor B']]).nunique()}")
    print(f"\nExperiment types:\n{df['Experimental System'].value_counts().head()}")
    return df

def build_ppi_network(biogrid_df, genes_of_interest=None):
    """Build NetworkX graph from BioGRID interactions."""
    G = nx.Graph()

    for _, row in biogrid_df.iterrows():
        a = row['Official Symbol Interactor A']
        b = row['Official Symbol Interactor B']
        if a != b:  # Exclude self-interactions
            if G.has_edge(a, b):
                G[a][b]['weight'] += 1
            else:
                G.add_edge(a, b, weight=1)

    print(f"Network: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges")

    # Subnetwork for genes of interest
    if genes_of_interest:
        # Include 1st-degree neighbors
        neighbors = set()
        for gene in genes_of_interest:
            if gene in G:
                neighbors.update(G.neighbors(gene))
        subgraph_nodes = set(genes_of_interest) | neighbors
        subG = G.subgraph(subgraph_nodes)
        print(f"Subnetwork: {subG.number_of_nodes()} nodes, {subG.number_of_edges()} edges")

        # Hub analysis
        degrees = dict(subG.degree())
        hubs = sorted(degrees.items(), key=lambda x: x[1], reverse=True)[:10]
        print("Top hubs:")
        for gene, deg in hubs:
            print(f"  {gene}: degree={deg}")

        return subG

    return G

6. MSigDB Gene Sets

Load and use MSigDB gene set collections.

import pandas as pd

# Download GMT files from: https://www.gsea-msigdb.org/gsea/msigdb/collections.jsp
# Collections: H (hallmark), C1-C8

def parse_gmt(gmt_path):
    """Parse GMT (Gene Matrix Transposed) file format."""
    gene_sets = {}
    with open(gmt_path) as f:
        for line in f:
            parts = line.strip().split('\t')
            name = parts[0]
            description = parts[1]
            genes = [g for g in parts[2:] if g]
            gene_sets[name] = {'description': description, 'genes': genes}

    print(f"Loaded {len(gene_sets)} gene sets from {gmt_path}")
    sizes = [len(gs['genes']) for gs in gene_sets.values()]
    print(f"Gene set sizes: {min(sizes)}-{max(sizes)} (median: {sorted(sizes)[len(sizes)//2]})")
    return gene_sets

def enrichment_with_msigdb(gene_list, collection='MSigDB_Hallmark_2020'):
    """Run enrichment analysis using MSigDB via gseapy."""
    import gseapy as gp

    enr = gp.enrichr(gene_list=gene_list, gene_sets=collection, outdir=None)
    results = enr.results[enr.results['Adjusted P-value'] < 0.05]

    print(f"Significant pathways (padj < 0.05): {len(results)}")
    if len(results) > 0:
        print(results[['Term', 'Adjusted P-value', 'Overlap', 'Combined Score']].head(10))
    return enr

# Available collections via gseapy
MSIGDB_COLLECTIONS = {
    'H': 'MSigDB_Hallmark_2020',          # 50 hallmark gene sets
    'C2_CP': 'KEGG_2021_Human',           # Canonical pathways
    'C5_GO_BP': 'GO_Biological_Process_2021',  # GO biological process
    'C5_GO_MF': 'GO_Molecular_Function_2021',  # GO molecular function
    'C5_GO_CC': 'GO_Cellular_Component_2021',  # GO cellular component
    'C7': 'Immunologic_Signature',        # Immunologic signatures
}

7. DisGeNET & OMIM

Access disease-gene association databases.

import pandas as pd
import requests

# DisGeNET: https://www.disgenet.org/downloads
# Download: curated_gene_disease_associations.tsv.gz

def load_disgenet(disgenet_path):
    """Load DisGeNET disease-gene associations."""
    df = pd.read_csv(disgenet_path, sep='\t')

    print(f"Total associations: {len(df)}")
    print(f"Unique genes: {df['geneSymbol'].nunique()}")
    print(f"Unique diseases: {df['diseaseName'].nunique()}")
    return df

def search_disgenet_by_gene(disgenet_df, gene_symbol, min_score=0.3):
    """Find diseases associated with a gene."""
    results = disgenet_df[disgenet_df['geneSymbol'] == gene_symbol]
    results = results[results['score'] >= min_score]
    results = results.sort_values('score', ascending=False)

    print(f"Diseases associated with {gene_symbol} (score >= {min_score}): {len(results)}")
    if len(results) > 0:
        print(results[['diseaseName', 'score', 'NofPmids']].head(10))
    return results

def search_disgenet_by_disease(disgenet_df, disease_keyword, min_score=0.3):
    """Find genes associated with a disease."""
    mask = disgenet_df['diseaseName'].str.contains(disease_keyword, case=False, na=False)
    results = disgenet_df[mask]
    results = results[results['score'] >= min_score]
    results = results.sort_values('score', ascending=False)

    print(f"Genes associated with '{disease_keyword}': {len(results)}")
    if len(results) > 0:
        print(results[['geneSymbol', 'diseaseName', 'score']].head(15))
    return results

8. Gene Ontology

Work with GO terms and hierarchies.

import json
import requests

# GO OBO file: http://purl.obolibrary.org/obo/go.obo
# GO JSON: https://purl.obolibrary.org/obo/go.json

def parse_go_obo(obo_path):
    """Parse Gene Ontology OBO file."""
    terms = {}
    current = None

    with open(obo_path) as f:
        for line in f:
            line = line.strip()
            if line == '[Term]':
                current = {}
            elif line == '' and current is not None:
                if 'id' in current:
                    terms[current['id']] = current
                current = None
            elif current is not None and ':' in line:
                key, value = line.split(': ', 1)
                if key in ('id', 'name', 'namespace', 'def'):
                    current[key] = value
                elif key == 'is_a':
                    current.setdefault('parents', []).append(value.split(' ! ')[0])
                elif key == 'is_obsolete' and value == 'true':
                    current = None

    print(f"GO terms: {len(terms)}")
    namespaces = {}
    for t in terms.values():
        ns = t.get('namespace', 'unknown')
        namespaces[ns] = namespaces.get(ns, 0) + 1
    print(f"Namespaces: {namespaces}")
    return terms

def get_ancestors(terms, go_id, max_depth=10):
    """Get all ancestor terms of a GO term."""
    ancestors = set()
    queue = [go_id]
    depth = 0

    while queue and depth < max_depth:
        next_queue = []
        for term_id in queue:
            if term_id in terms:
                parents = terms[term_id].get('parents', [])
                for parent in parents:
                    if parent not in ancestors:
                        ancestors.add(parent)
                        next_queue.append(parent)
        queue = next_queue
        depth += 1

    return ancestors

def lookup_go_term(go_id):
    """Look up a GO term using the QuickGO API."""
    url = f"https://www.ebi.ac.uk/QuickGO/services/ontology/go/terms/{go_id}"
    response = requests.get(url, headers={'Accept': 'application/json'})

    if response.status_code == 200:
        data = response.json()
        if 'results' in data and len(data['results']) > 0:
            term = data['results'][0]
            print(f"GO:{go_id}")
            print(f"  Name: {term.get('name')}")
            print(f"  Namespace: {term.get('aspect')}")
            print(f"  Definition: {term.get('definition', {}).get('text', 'N/A')[:200]}")
            return term
    return None

Typical Workflows

Workflow 1: Download and Filter COSMIC Cancer Gene Census

census = load_cosmic_census('cancer_gene_census.csv')
breast_genes = filter_census_by_cancer(census, 'breast')
print(f"\nBreast cancer genes: {list(breast_genes['Gene Symbol'].head(20))}")

Workflow 2: Build Protein Interaction Network from BioGRID

biogrid = load_biogrid('BIOGRID-ALL-LATEST.tab3.txt',
                        experiment_types=['Affinity Capture-MS', 'Two-hybrid'])
G = build_ppi_network(biogrid, genes_of_interest=['TP53', 'BRCA1', 'ATM'])

Workflow 3: Run Pathway Enrichment with MSigDB Gene Sets

gene_list = ['TP53', 'BRCA1', 'ATM', 'CHEK2', 'PTEN', 'RB1', 'CDKN2A']
enr = enrichment_with_msigdb(gene_list, collection='MSigDB_Hallmark_2020')

Best Practices

1. COSMIC access — requires free registration; respect data license terms; use COSMIC CGC for curated cancer genes, full dataset for comprehensive analysis
2. GTEx — use median TPM for tissue comparisons; raw counts for differential analysis; eQTL data for variant interpretation
3. GWAS Catalog — apply genome-wide significance threshold (5e-8); check LD with lead SNP; use mapped genes, not just reported genes
4. BioGRID — filter by experiment type for quality; "Affinity Capture-MS" and "Two-hybrid" are highest confidence; weight edges by evidence count
5. MSigDB — Hallmark gene sets are most interpretable; use C2 (curated) for canonical pathways; C5 (GO) for biological process analysis
6. DisGeNET — filter by score (>0.3 for moderate confidence); prioritize curated sources; cross-reference with OMIM for Mendelian diseases
7. File sizes — COSMIC full mutation export is ~30GB; GTEx TPM matrix is ~2GB; download and process locally, don't reload repeatedly

Troubleshooting

Problem: COSMIC download requires authentication Solution: Register for free account at cancer.sanger.ac.uk. Use SFTP or download from the web portal. Academic license is free.

Problem: GTEx API returns empty results Solution: Check gene symbol is HUGO-approved. Try ENSG ID instead. GTEx v8 uses GRCh38 coordinates.

Problem: BioGRID file too large to load Solution: Use organism filter during loading. Read in chunks with chunksize parameter. Pre-filter with command-line tools (grep).

Problem: gseapy enrichment returns no results Solution: Ensure gene symbols match the gene set database (HUGO symbols). Check that gene list has >5 genes. Try different collections.

Resources

• COSMIC
• GTEx Portal
• GWAS Catalog
• GeneBass
• BioGRID
• MSigDB
• DisGeNET
• Gene Ontology

deepTools: NGS Data Analysis Toolkit

Overview

deepTools is a comprehensive suite of Python command-line tools designed for processing and analyzing high-throughput sequencing data. Use deepTools to perform quality control, normalize data, compare samples, and generate publication-quality visualizations for ChIP-seq, RNA-seq, ATAC-seq, MNase-seq, and other NGS experiments.

Core capabilities:

• Convert BAM alignments to normalized coverage tracks (bigWig/bedGraph)
• Quality control assessment (fingerprint, correlation, coverage)
• Sample comparison and correlation analysis
• Heatmap and profile plot generation around genomic features
• Enrichment analysis and peak region visualization

When to Use This Skill

This skill should be used when:

• File conversion: "Convert BAM to bigWig", "generate coverage tracks", "normalize ChIP-seq data"
• Quality control: "check ChIP quality", "compare replicates", "assess sequencing depth", "QC analysis"
• Visualization: "create heatmap around TSS", "plot ChIP signal", "visualize enrichment", "generate profile plot"
• Sample comparison: "compare treatment vs control", "correlate samples", "PCA analysis"
• Analysis workflows: "analyze ChIP-seq data", "RNA-seq coverage", "ATAC-seq analysis", "complete workflow"
• Working with specific file types: BAM files, bigWig files, BED region files in genomics context

Quick Start

For users new to deepTools, start with file validation and common workflows:

1. Validate Input Files

Before running any analysis, validate BAM, bigWig, and BED files using the validation script:

python scripts/validate_files.py --bam sample1.bam sample2.bam --bed regions.bed

This checks file existence, BAM indices, and format correctness.

2. Generate Workflow Template

For standard analyses, use the workflow generator to create customized scripts:

# List available workflows
python scripts/workflow_generator.py --list

# Generate ChIP-seq QC workflow
python scripts/workflow_generator.py chipseq_qc -o qc_workflow.sh \
    --input-bam Input.bam --chip-bams "ChIP1.bam ChIP2.bam" \
    --genome-size 2913022398

# Make executable and run
chmod +x qc_workflow.sh
./qc_workflow.sh

3. Most Common Operations

See assets/quick_reference.md for frequently used commands and parameters.

Installation

uv pip install deeptools

Core Workflows

deepTools workflows typically follow this pattern: QC → Normalization → Comparison/Visualization

ChIP-seq Quality Control Workflow

When users request ChIP-seq QC or quality assessment:

1. Generate workflow script using scripts/workflow_generator.py chipseq_qc
2. Key QC steps:
   • Sample correlation (multiBamSummary + plotCorrelation)
   • PCA analysis (plotPCA)
   • Coverage assessment (plotCoverage)
   • Fragment size validation (bamPEFragmentSize)
   • ChIP enrichment strength (plotFingerprint)

Interpreting results:

• Correlation: Replicates should cluster together with high correlation (>0.9)
• Fingerprint: Strong ChIP shows steep rise; flat diagonal indicates poor enrichment
• Coverage: Assess if sequencing depth is adequate for analysis

Full workflow details in references/workflows.md → "ChIP-seq Quality Control Workflow"

ChIP-seq Complete Analysis Workflow

For full ChIP-seq analysis from BAM to visualizations:

1. Generate coverage tracks with normalization (bamCoverage)
2. Create comparison tracks (bamCompare for log2 ratio)
3. Compute signal matrices around features (computeMatrix)
4. Generate visualizations (plotHeatmap, plotProfile)
5. Enrichment analysis at peaks (plotEnrichment)

Use scripts/workflow_generator.py chipseq_analysis to generate template.

Complete command sequences in references/workflows.md → "ChIP-seq Analysis Workflow"

RNA-seq Coverage Workflow

For strand-specific RNA-seq coverage tracks:

Use bamCoverage with --filterRNAstrand to separate forward and reverse strands.

Important: NEVER use --extendReads for RNA-seq (would extend over splice junctions).

Use normalization: CPM for fixed bins, RPKM for gene-level analysis.

Template available: scripts/workflow_generator.py rnaseq_coverage

Details in references/workflows.md → "RNA-seq Coverage Workflow"

ATAC-seq Analysis Workflow

ATAC-seq requires Tn5 offset correction:

1. Shift reads using alignmentSieve with --ATACshift
2. Generate coverage with bamCoverage
3. Analyze fragment sizes (expect nucleosome ladder pattern)
4. Visualize at peaks if available

Template: scripts/workflow_generator.py atacseq

Full workflow in references/workflows.md → "ATAC-seq Workflow"

Tool Categories and Common Tasks

BAM/bigWig Processing

Convert BAM to normalized coverage:

bamCoverage --bam input.bam --outFileName output.bw \
    --normalizeUsing RPGC --effectiveGenomeSize 2913022398 \
    --binSize 10 --numberOfProcessors 8

Compare two samples (log2 ratio):

bamCompare -b1 treatment.bam -b2 control.bam -o ratio.bw \
    --operation log2 --scaleFactorsMethod readCount

Key tools: bamCoverage, bamCompare, multiBamSummary, multiBigwigSummary, correctGCBias, alignmentSieve

Complete reference: references/tools_reference.md → "BAM and bigWig File Processing Tools"

Quality Control

Check ChIP enrichment:

plotFingerprint -b input.bam chip.bam -o fingerprint.png \
    --extendReads 200 --ignoreDuplicates

Sample correlation:

multiBamSummary bins --bamfiles *.bam -o counts.npz
plotCorrelation -in counts.npz --corMethod pearson \
    --whatToShow heatmap -o correlation.png

Key tools: plotFingerprint, plotCoverage, plotCorrelation, plotPCA, bamPEFragmentSize

Complete reference: references/tools_reference.md → "Quality Control Tools"

Visualization

Create heatmap around TSS:

# Compute matrix
computeMatrix reference-point -S signal.bw -R genes.bed \
    -b 3000 -a 3000 --referencePoint TSS -o matrix.gz

# Generate heatmap
plotHeatmap -m matrix.gz -o heatmap.png \
    --colorMap RdBu --kmeans 3

Create profile plot:

plotProfile -m matrix.gz -o profile.png \
    --plotType lines --colors blue red

Key tools: computeMatrix, plotHeatmap, plotProfile, plotEnrichment

Complete reference: references/tools_reference.md → "Visualization Tools"

Normalization Methods

Choosing the correct normalization is critical for valid comparisons. Consult references/normalization_methods.md for comprehensive guidance.

Quick selection guide:

• ChIP-seq coverage: Use RPGC or CPM
• ChIP-seq comparison: Use bamCompare with log2 and readCount
• RNA-seq bins: Use CPM
• RNA-seq genes: Use RPKM (accounts for gene length)
• ATAC-seq: Use RPGC or CPM

Normalization methods:

• RPGC: 1× genome coverage (requires --effectiveGenomeSize)
• CPM: Counts per million mapped reads
• RPKM: Reads per kb per million (accounts for region length)
• BPM: Bins per million
• None: Raw counts (not recommended for comparisons)

Full explanation: references/normalization_methods.md

Effective Genome Sizes

RPGC normalization requires effective genome size. Common values:

Complete table with read-length-specific values: references/effective_genome_sizes.md

Common Parameters Across Tools

Many deepTools commands share these options:

Performance:

• --numberOfProcessors, -p: Enable parallel processing (always use available cores)
• --region: Process specific regions for testing (e.g., chr1:1-1000000)

Read Filtering:

• --ignoreDuplicates: Remove PCR duplicates (recommended for most analyses)
• --minMappingQuality: Filter by alignment quality (e.g., --minMappingQuality 10)
• --minFragmentLength / --maxFragmentLength: Fragment length bounds
• --samFlagInclude / --samFlagExclude: SAM flag filtering

Read Processing:

• --extendReads: Extend to fragment length (ChIP-seq: YES, RNA-seq: NO)
• --centerReads: Center at fragment midpoint for sharper signals

Best Practices

File Validation

Always validate files first using scripts/validate_files.py to check:

• File existence and readability
• BAM indices present (.bai files)
• BED format correctness
• File sizes reasonable

Analysis Strategy

1. Start with QC: Run correlation, coverage, and fingerprint analysis before proceeding
2. Test on small regions: Use --region chr1:1-10000000 for parameter testing
3. Document commands: Save full command lines for reproducibility
4. Use consistent normalization: Apply same method across samples in comparisons
5. Verify genome assembly: Ensure BAM and BED files use matching genome builds

ChIP-seq Specific

• Always extend reads for ChIP-seq: --extendReads 200
• Remove duplicates: Use --ignoreDuplicates in most cases
• Check enrichment first: Run plotFingerprint before detailed analysis
• GC correction: Only apply if significant bias detected; never use --ignoreDuplicates after GC correction

RNA-seq Specific

• Never extend reads for RNA-seq (would span splice junctions)
• Strand-specific: Use --filterRNAstrand forward/reverse for stranded libraries
• Normalization: CPM for bins, RPKM for genes

ATAC-seq Specific

• Apply Tn5 correction: Use alignmentSieve with --ATACshift
• Fragment filtering: Set appropriate min/max fragment lengths
• Check nucleosome pattern: Fragment size plot should show ladder pattern

Performance Optimization

1. Use multiple processors: --numberOfProcessors 8 (or available cores)
2. Increase bin size for faster processing and smaller files
3. Process chromosomes separately for memory-limited systems
4. Pre-filter BAM files using alignmentSieve to create reusable filtered files
5. Use bigWig over bedGraph: Compressed and faster to process

Troubleshooting

Common Issues

BAM index missing:

samtools index input.bam

Out of memory: Process chromosomes individually using --region:

bamCoverage --bam input.bam -o chr1.bw --region chr1

Slow processing: Increase --numberOfProcessors and/or increase --binSize

bigWig files too large: Increase bin size: --binSize 50 or larger

Validation Errors

Run validation script to identify issues:

python scripts/validate_files.py --bam *.bam --bed regions.bed

Common errors and solutions explained in script output.

Reference Documentation

This skill includes comprehensive reference documentation:

references/tools_reference.md

Complete documentation of all deepTools commands organized by category:

• BAM and bigWig processing tools (9 tools)
• Quality control tools (6 tools)
• Visualization tools (3 tools)
• Miscellaneous tools (2 tools)

Each tool includes:

• Purpose and overview
• Key parameters with explanations
• Usage examples
• Important notes and best practices

Use this reference when: Users ask about specific tools, parameters, or detailed usage.

references/workflows.md

Complete workflow examples for common analyses:

• ChIP-seq quality control workflow
• ChIP-seq complete analysis workflow
• RNA-seq coverage workflow
• ATAC-seq analysis workflow
• Multi-sample comparison workflow
• Peak region analysis workflow
• Troubleshooting and performance tips

Use this reference when: Users need complete analysis pipelines or workflow examples.

references/normalization_methods.md

Comprehensive guide to normalization methods:

• Detailed explanation of each method (RPGC, CPM, RPKM, BPM, etc.)
• When to use each method
• Formulas and interpretation
• Selection guide by experiment type
• Common pitfalls and solutions
• Quick reference table

Use this reference when: Users ask about normalization, comparing samples, or which method to use.

references/effective_genome_sizes.md

Effective genome size values and usage:

• Common organism values (human, mouse, fly, worm, zebrafish)
• Read-length-specific values
• Calculation methods
• When and how to use in commands
• Custom genome calculation instructions

Use this reference when: Users need genome size for RPGC normalization or GC bias correction.

Helper Scripts

scripts/validate_files.py

Validates BAM, bigWig, and BED files for deepTools analysis. Checks file existence, indices, and format.

Usage:

python scripts/validate_files.py --bam sample1.bam sample2.bam \
    --bed peaks.bed --bigwig signal.bw

When to use: Before starting any analysis, or when troubleshooting errors.

scripts/workflow_generator.py

Generates customizable bash script templates for common deepTools workflows.

Available workflows:

• chipseq_qc: ChIP-seq quality control
• chipseq_analysis: Complete ChIP-seq analysis
• rnaseq_coverage: Strand-specific RNA-seq coverage
• atacseq: ATAC-seq with Tn5 correction

Usage:

# List workflows
python scripts/workflow_generator.py --list

# Generate workflow
python scripts/workflow_generator.py chipseq_qc -o qc.sh \
    --input-bam Input.bam --chip-bams "ChIP1.bam ChIP2.bam" \
    --genome-size 2913022398 --threads 8

# Run generated workflow
chmod +x qc.sh
./qc.sh

When to use: Users request standard workflows or need template scripts to customize.

Assets

assets/quick_reference.md

Quick reference card with most common commands, effective genome sizes, and typical workflow pattern.

When to use: Users need quick command examples without detailed documentation.

Handling User Requests

For New Users

1. Start with installation verification
2. Validate input files using scripts/validate_files.py
3. Recommend appropriate workflow based on experiment type
4. Generate workflow template using scripts/workflow_generator.py
5. Guide through customization and execution

For Experienced Users

1. Provide specific tool commands for requested operations
2. Reference appropriate sections in references/tools_reference.md
3. Suggest optimizations and best practices
4. Offer troubleshooting for issues

For Specific Tasks

"Convert BAM to bigWig":

• Use bamCoverage with appropriate normalization
• Recommend RPGC or CPM based on use case
• Provide effective genome size for organism
• Suggest relevant parameters (extendReads, ignoreDuplicates, binSize)

"Check ChIP quality":

• Run full QC workflow or use plotFingerprint specifically
• Explain interpretation of results
• Suggest follow-up actions based on results

"Create heatmap":

• Guide through two-step process: computeMatrix → plotHeatmap
• Help choose appropriate matrix mode (reference-point vs scale-regions)
• Suggest visualization parameters and clustering options

"Compare samples":

• Recommend bamCompare for two-sample comparison
• Suggest multiBamSummary + plotCorrelation for multiple samples
• Guide normalization method selection

Referencing Documentation

When users need detailed information:

• Tool details: Direct to specific sections in references/tools_reference.md
• Workflows: Use references/workflows.md for complete analysis pipelines
• Normalization: Consult references/normalization_methods.md for method selection
• Genome sizes: Reference references/effective_genome_sizes.md

Search references using grep patterns:

# Find tool documentation
grep -A 20 "^### toolname" references/tools_reference.md

# Find workflow
grep -A 50 "^## Workflow Name" references/workflows.md

# Find normalization method
grep -A 15 "^### Method Name" references/normalization_methods.md

Example Interactions

User: "I need to analyze my ChIP-seq data"

Response approach:

1. Ask about files available (BAM files, peaks, genes)
2. Validate files using validation script
3. Generate chipseq_analysis workflow template
4. Customize for their specific files and organism
5. Explain each step as script runs

User: "Which normalization should I use?"

Response approach:

1. Ask about experiment type (ChIP-seq, RNA-seq, etc.)
2. Ask about comparison goal (within-sample or between-sample)
3. Consult references/normalization_methods.md selection guide
4. Recommend appropriate method with justification
5. Provide command example with parameters

User: "Create a heatmap around TSS"

Response approach:

1. Verify bigWig and gene BED files available
2. Use computeMatrix with reference-point mode at TSS
3. Generate plotHeatmap with appropriate visualization parameters
4. Suggest clustering if dataset is large
5. Offer profile plot as complement

Key Reminders

• File validation first: Always validate input files before analysis
• Normalization matters: Choose appropriate method for comparison type
• Extend reads carefully: YES for ChIP-seq, NO for RNA-seq
• Use all cores: Set --numberOfProcessors to available cores
• Test on regions: Use --region for parameter testing
• Check QC first: Run quality control before detailed analysis
• Document everything: Save commands for reproducibility
• Reference documentation: Use comprehensive references for detailed guidance

DNAnexus Integration

Overview

DNAnexus is a cloud platform for biomedical data analysis and genomics. Build and deploy apps/applets, manage data objects, run workflows, and use the dxpy Python SDK for genomics pipeline development and execution.

When to Use This Skill

This skill should be used when:

• Creating, building, or modifying DNAnexus apps/applets
• Uploading, downloading, searching, or organizing files and records
• Running analyses, monitoring jobs, creating workflows
• Writing scripts using dxpy to interact with the platform
• Setting up dxapp.json, managing dependencies, using Docker
• Processing FASTQ, BAM, VCF, or other bioinformatics files
• Managing projects, permissions, or platform resources

Core Capabilities

The skill is organized into five main areas, each with detailed reference documentation:

1. App Development

Purpose: Create executable programs (apps/applets) that run on the DNAnexus platform.

Key Operations:

• Generate app skeleton with dx-app-wizard
• Write Python or Bash apps with proper entry points
• Handle input/output data objects
• Deploy with dx build or dx build --app
• Test apps on the platform

Common Use Cases:

• Bioinformatics pipelines (alignment, variant calling)
• Data processing workflows
• Quality control and filtering
• Format conversion tools

Reference: See references/app-development.md for:

• Complete app structure and patterns
• Python entry point decorators
• Input/output handling with dxpy
• Development best practices
• Common issues and solutions

2. Data Operations

Purpose: Manage files, records, and other data objects on the platform.

Key Operations:

• Upload/download files with dxpy.upload_local_file() and dxpy.download_dxfile()
• Create and manage records with metadata
• Search for data objects by name, properties, or type
• Clone data between projects
• Manage project folders and permissions

Common Use Cases:

• Uploading sequencing data (FASTQ files)
• Organizing analysis results
• Searching for specific samples or experiments
• Backing up data across projects
• Managing reference genomes and annotations

Reference: See references/data-operations.md for:

• Complete file and record operations
• Data object lifecycle (open/closed states)
• Search and discovery patterns
• Project management
• Batch operations

3. Job Execution

Purpose: Run analyses, monitor execution, and orchestrate workflows.

Key Operations:

• Launch jobs with applet.run() or app.run()
• Monitor job status and logs
• Create subjobs for parallel processing
• Build and run multi-step workflows
• Chain jobs with output references

Common Use Cases:

• Running genomics analyses on sequencing data
• Parallel processing of multiple samples
• Multi-step analysis pipelines
• Monitoring long-running computations
• Debugging failed jobs

Reference: See references/job-execution.md for:

• Complete job lifecycle and states
• Workflow creation and orchestration
• Parallel execution patterns
• Job monitoring and debugging
• Resource management

4. Python SDK (dxpy)

Purpose: Programmatic access to DNAnexus platform through Python.

Key Operations:

• Work with data object handlers (DXFile, DXRecord, DXApplet, etc.)
• Use high-level functions for common tasks
• Make direct API calls for advanced operations
• Create links and references between objects
• Search and discover platform resources

Common Use Cases:

• Automation scripts for data management
• Custom analysis pipelines
• Batch processing workflows
• Integration with external tools
• Data migration and organization

Reference: See references/python-sdk.md for:

• Complete dxpy class reference
• High-level utility functions
• API method documentation
• Error handling patterns
• Common code patterns

5. Configuration and Dependencies

Purpose: Configure app metadata and manage dependencies.

Key Operations:

• Write dxapp.json with inputs, outputs, and run specs
• Install system packages (execDepends)
• Bundle custom tools and resources
• Use assets for shared dependencies
• Integrate Docker containers
• Configure instance types and timeouts

Common Use Cases:

• Defining app input/output specifications
• Installing bioinformatics tools (samtools, bwa, etc.)
• Managing Python package dependencies
• Using Docker images for complex environments
• Selecting computational resources

Reference: See references/configuration.md for:

• Complete dxapp.json specification
• Dependency management strategies
• Docker integration patterns
• Regional and resource configuration
• Example configurations

Quick Start Examples

Upload and Analyze Data

import dxpy

# Upload input file
input_file = dxpy.upload_local_file("sample.fastq", project="project-xxxx")

# Run analysis
job = dxpy.DXApplet("applet-xxxx").run({
    "reads": dxpy.dxlink(input_file.get_id())
})

# Wait for completion
job.wait_on_done()

# Download results
output_id = job.describe()["output"]["aligned_reads"]["$dnanexus_link"]
dxpy.download_dxfile(output_id, "aligned.bam")

Search and Download Files

import dxpy

# Find BAM files from a specific experiment
files = dxpy.find_data_objects(
    classname="file",
    name="*.bam",
    properties={"experiment": "exp001"},
    project="project-xxxx"
)

# Download each file
for file_result in files:
    file_obj = dxpy.DXFile(file_result["id"])
    filename = file_obj.describe()["name"]
    dxpy.download_dxfile(file_result["id"], filename)

Create Simple App

# src/my-app.py
import dxpy
import subprocess

@dxpy.entry_point('main')
def main(input_file, quality_threshold=30):
    # Download input
    dxpy.download_dxfile(input_file["$dnanexus_link"], "input.fastq")

    # Process
    subprocess.check_call([
        "quality_filter",
        "--input", "input.fastq",
        "--output", "filtered.fastq",
        "--threshold", str(quality_threshold)
    ])

    # Upload output
    output_file = dxpy.upload_local_file("filtered.fastq")

    return {
        "filtered_reads": dxpy.dxlink(output_file)
    }

dxpy.run()

Workflow Decision Tree

When working with DNAnexus, follow this decision tree:

1. Need to create a new executable?

   • Yes → Use App Development (references/app-development.md)
   • No → Continue to step 2

2. Need to manage files or data?

   • Yes → Use Data Operations (references/data-operations.md)
   • No → Continue to step 3

3. Need to run an analysis or workflow?

   • Yes → Use Job Execution (references/job-execution.md)
   • No → Continue to step 4

4. Writing Python scripts for automation?

   • Yes → Use Python SDK (references/python-sdk.md)
   • No → Continue to step 5

5. Configuring app settings or dependencies?

   • Yes → Use Configuration (references/configuration.md)

Often you'll need multiple capabilities together (e.g., app development + configuration, or data operations + job execution).

Installation and Authentication

Install dxpy

uv pip install dxpy

Login to DNAnexus

dx login

This authenticates your session and sets up access to projects and data.

Verify Installation

dx --version
dx whoami

Common Patterns

Pattern 1: Batch Processing

Process multiple files with the same analysis:

# Find all FASTQ files
files = dxpy.find_data_objects(
    classname="file",
    name="*.fastq",
    project="project-xxxx"
)

# Launch parallel jobs
jobs = []
for file_result in files:
    job = dxpy.DXApplet("applet-xxxx").run({
        "input": dxpy.dxlink(file_result["id"])
    })
    jobs.append(job)

# Wait for all completions
for job in jobs:
    job.wait_on_done()

Pattern 2: Multi-Step Pipeline

Chain multiple analyses together:

# Step 1: Quality control
qc_job = qc_applet.run({"reads": input_file})

# Step 2: Alignment (uses QC output)
align_job = align_applet.run({
    "reads": qc_job.get_output_ref("filtered_reads")
})

# Step 3: Variant calling (uses alignment output)
variant_job = variant_applet.run({
    "bam": align_job.get_output_ref("aligned_bam")
})

Pattern 3: Data Organization

Organize analysis results systematically:

# Create organized folder structure
dxpy.api.project_new_folder(
    "project-xxxx",
    {"folder": "/experiments/exp001/results", "parents": True}
)

# Upload with metadata
result_file = dxpy.upload_local_file(
    "results.txt",
    project="project-xxxx",
    folder="/experiments/exp001/results",
    properties={
        "experiment": "exp001",
        "sample": "sample1",
        "analysis_date": "2025-10-20"
    },
    tags=["validated", "published"]
)

Best Practices

1. Error Handling: Always wrap API calls in try-except blocks
2. Resource Management: Choose appropriate instance types for workloads
3. Data Organization: Use consistent folder structures and metadata
4. Cost Optimization: Archive old data, use appropriate storage classes
5. Documentation: Include clear descriptions in dxapp.json
6. Testing: Test apps with various input types before production use
7. Version Control: Use semantic versioning for apps
8. Security: Never hardcode credentials in source code
9. Logging: Include informative log messages for debugging
10. Cleanup: Remove temporary files and failed jobs

Resources

This skill includes detailed reference documentation:

references/

• app-development.md - Complete guide to building and deploying apps/applets
• data-operations.md - File management, records, search, and project operations
• job-execution.md - Running jobs, workflows, monitoring, and parallel processing
• python-sdk.md - Comprehensive dxpy library reference with all classes and functions
• configuration.md - dxapp.json specification and dependency management

Load these references when you need detailed information about specific operations or when working on complex tasks.

Getting Help

• Official documentation: https://documentation.dnanexus.com/
• API reference: http://autodoc.dnanexus.com/
• GitHub repository: https://github.com/dnanexus/dx-toolkit
• Support: support@dnanexus.com

ETE Toolkit Skill

Overview

ETE (Environment for Tree Exploration) is a toolkit for phylogenetic and hierarchical tree analysis. Manipulate trees, analyze evolutionary events, visualize results, and integrate with biological databases for phylogenomic research and clustering analysis.

Core Capabilities

1. Tree Manipulation and Analysis

Load, manipulate, and analyze hierarchical tree structures with support for:

• Tree I/O: Read and write Newick, NHX, PhyloXML, and NeXML formats
• Tree traversal: Navigate trees using preorder, postorder, or levelorder strategies
• Topology modification: Prune, root, collapse nodes, resolve polytomies
• Distance calculations: Compute branch lengths and topological distances between nodes
• Tree comparison: Calculate Robinson-Foulds distances and identify topological differences

Common patterns:

from ete3 import Tree

# Load tree from file
tree = Tree("tree.nw", format=1)

# Basic statistics
print(f"Leaves: {len(tree)}")
print(f"Total nodes: {len(list(tree.traverse()))}")

# Prune to taxa of interest
taxa_to_keep = ["species1", "species2", "species3"]
tree.prune(taxa_to_keep, preserve_branch_length=True)

# Midpoint root
midpoint = tree.get_midpoint_outgroup()
tree.set_outgroup(midpoint)

# Save modified tree
tree.write(outfile="rooted_tree.nw")

Use scripts/tree_operations.py for command-line tree manipulation:

# Display tree statistics
python scripts/tree_operations.py stats tree.nw

# Convert format
python scripts/tree_operations.py convert tree.nw output.nw --in-format 0 --out-format 1

# Reroot tree
python scripts/tree_operations.py reroot tree.nw rooted.nw --midpoint

# Prune to specific taxa
python scripts/tree_operations.py prune tree.nw pruned.nw --keep-taxa "sp1,sp2,sp3"

# Show ASCII visualization
python scripts/tree_operations.py ascii tree.nw

2. Phylogenetic Analysis

Analyze gene trees with evolutionary event detection:

• Sequence alignment integration: Link trees to multiple sequence alignments (FASTA, Phylip)
• Species naming: Automatic or custom species extraction from gene names
• Evolutionary events: Detect duplication and speciation events using Species Overlap or tree reconciliation
• Orthology detection: Identify orthologs and paralogs based on evolutionary events
• Gene family analysis: Split trees by duplications, collapse lineage-specific expansions

Workflow for gene tree analysis:

from ete3 import PhyloTree

# Load gene tree with alignment
tree = PhyloTree("gene_tree.nw", alignment="alignment.fasta")

# Set species naming function
def get_species(gene_name):
    return gene_name.split("_")[0]

tree.set_species_naming_function(get_species)

# Detect evolutionary events
events = tree.get_descendant_evol_events()

# Analyze events
for node in tree.traverse():
    if hasattr(node, "evoltype"):
        if node.evoltype == "D":
            print(f"Duplication at {node.name}")
        elif node.evoltype == "S":
            print(f"Speciation at {node.name}")

# Extract ortholog groups
ortho_groups = tree.get_speciation_trees()
for i, ortho_tree in enumerate(ortho_groups):
    ortho_tree.write(outfile=f"ortholog_group_{i}.nw")

Finding orthologs and paralogs:

# Find orthologs to query gene
query = tree & "species1_gene1"

orthologs = []
paralogs = []

for event in events:
    if query in event.in_seqs:
        if event.etype == "S":
            orthologs.extend([s for s in event.out_seqs if s != query])
        elif event.etype == "D":
            paralogs.extend([s for s in event.out_seqs if s != query])

3. NCBI Taxonomy Integration

Integrate taxonomic information from NCBI Taxonomy database:

• Database access: Automatic download and local caching of NCBI taxonomy (~300MB)
• Taxid/name translation: Convert between taxonomic IDs and scientific names
• Lineage retrieval: Get complete evolutionary lineages
• Taxonomy trees: Build species trees connecting specified taxa
• Tree annotation: Automatically annotate trees with taxonomic information

Building taxonomy-based trees:

from ete3 import NCBITaxa

ncbi = NCBITaxa()

# Build tree from species names
species = ["Homo sapiens", "Pan troglodytes", "Mus musculus"]
name2taxid = ncbi.get_name_translator(species)
taxids = [name2taxid[sp][0] for sp in species]

# Get minimal tree connecting taxa
tree = ncbi.get_topology(taxids)

# Annotate nodes with taxonomy info
for node in tree.traverse():
    if hasattr(node, "sci_name"):
        print(f"{node.sci_name} - Rank: {node.rank} - TaxID: {node.taxid}")

Annotating existing trees:

# Get taxonomy info for tree leaves
for leaf in tree:
    species = extract_species_from_name(leaf.name)
    taxid = ncbi.get_name_translator([species])[species][0]

    # Get lineage
    lineage = ncbi.get_lineage(taxid)
    ranks = ncbi.get_rank(lineage)
    names = ncbi.get_taxid_translator(lineage)

    # Add to node
    leaf.add_feature("taxid", taxid)
    leaf.add_feature("lineage", [names[t] for t in lineage])

4. Tree Visualization

Create publication-quality tree visualizations:

• Output formats: PNG (raster), PDF, and SVG (vector) for publications
• Layout modes: Rectangular and circular tree layouts
• Interactive GUI: Explore trees interactively with zoom, pan, and search
• Custom styling: NodeStyle for node appearance (colors, shapes, sizes)
• Faces: Add graphical elements (text, images, charts, heatmaps) to nodes
• Layout functions: Dynamic styling based on node properties

Basic visualization workflow:

from ete3 import Tree, TreeStyle, NodeStyle

tree = Tree("tree.nw")

# Configure tree style
ts = TreeStyle()
ts.show_leaf_name = True
ts.show_branch_support = True
ts.scale = 50  # pixels per branch length unit

# Style nodes
for node in tree.traverse():
    nstyle = NodeStyle()

    if node.is_leaf():
        nstyle["fgcolor"] = "blue"
        nstyle["size"] = 8
    else:
        # Color by support
        if node.support > 0.9:
            nstyle["fgcolor"] = "darkgreen"
        else:
            nstyle["fgcolor"] = "red"
        nstyle["size"] = 5

    node.set_style(nstyle)

# Render to file
tree.render("tree.pdf", tree_style=ts)
tree.render("tree.png", w=800, h=600, units="px", dpi=300)

Use scripts/quick_visualize.py for rapid visualization:

# Basic visualization
python scripts/quick_visualize.py tree.nw output.pdf

# Circular layout with custom styling
python scripts/quick_visualize.py tree.nw output.pdf --mode c --color-by-support

# High-resolution PNG
python scripts/quick_visualize.py tree.nw output.png --width 1200 --height 800 --units px --dpi 300

# Custom title and styling
python scripts/quick_visualize.py tree.nw output.pdf --title "Species Phylogeny" --show-support

Advanced visualization with faces:

from ete3 import Tree, TreeStyle, TextFace, CircleFace

tree = Tree("tree.nw")

# Add features to nodes
for leaf in tree:
    leaf.add_feature("habitat", "marine" if "fish" in leaf.name else "land")

# Layout function
def layout(node):
    if node.is_leaf():
        # Add colored circle
        color = "blue" if node.habitat == "marine" else "green"
        circle = CircleFace(radius=5, color=color)
        node.add_face(circle, column=0, position="aligned")

        # Add label
        label = TextFace(node.name, fsize=10)
        node.add_face(label, column=1, position="aligned")

ts = TreeStyle()
ts.layout_fn = layout
ts.show_leaf_name = False

tree.render("annotated_tree.pdf", tree_style=ts)

5. Clustering Analysis

Analyze hierarchical clustering results with data integration:

• ClusterTree: Specialized class for clustering dendrograms
• Data matrix linking: Connect tree leaves to numerical profiles
• Cluster metrics: Silhouette coefficient, Dunn index, inter/intra-cluster distances
• Validation: Test cluster quality with different distance metrics
• Heatmap visualization: Display data matrices alongside trees

Clustering workflow:

from ete3 import ClusterTree

# Load tree with data matrix
matrix = """#Names\tSample1\tSample2\tSample3
Gene1\t1.5\t2.3\t0.8
Gene2\t0.9\t1.1\t1.8
Gene3\t2.1\t2.5\t0.5"""

tree = ClusterTree("((Gene1,Gene2),Gene3);", text_array=matrix)

# Evaluate cluster quality
for node in tree.traverse():
    if not node.is_leaf():
        silhouette = node.get_silhouette()
        dunn = node.get_dunn()

        print(f"Cluster: {node.name}")
        print(f"  Silhouette: {silhouette:.3f}")
        print(f"  Dunn index: {dunn:.3f}")

# Visualize with heatmap
tree.show("heatmap")

6. Tree Comparison

Quantify topological differences between trees:

• Robinson-Foulds distance: Standard metric for tree comparison
• Normalized RF: Scale-invariant distance (0.0 to 1.0)
• Partition analysis: Identify unique and shared bipartitions
• Consensus trees: Analyze support across multiple trees
• Batch comparison: Compare multiple trees pairwise

Compare two trees:

from ete3 import Tree

tree1 = Tree("tree1.nw")
tree2 = Tree("tree2.nw")

# Calculate RF distance
rf, max_rf, common_leaves, parts_t1, parts_t2 = tree1.robinson_foulds(tree2)

print(f"RF distance: {rf}/{max_rf}")
print(f"Normalized RF: {rf/max_rf:.3f}")
print(f"Common leaves: {len(common_leaves)}")

# Find unique partitions
unique_t1 = parts_t1 - parts_t2
unique_t2 = parts_t2 - parts_t1

print(f"Unique to tree1: {len(unique_t1)}")
print(f"Unique to tree2: {len(unique_t2)}")

Compare multiple trees:

import numpy as np

trees = [Tree(f"tree{i}.nw") for i in range(4)]

# Create distance matrix
n = len(trees)
dist_matrix = np.zeros((n, n))

for i in range(n):
    for j in range(i+1, n):
        rf, max_rf, _, _, _ = trees[i].robinson_foulds(trees[j])
        norm_rf = rf / max_rf if max_rf > 0 else 0
        dist_matrix[i, j] = norm_rf
        dist_matrix[j, i] = norm_rf

Installation and Setup

Install ETE toolkit:

# Basic installation
uv pip install ete3

# With external dependencies for rendering (optional but recommended)
# On macOS:
brew install qt@5

# On Ubuntu/Debian:
sudo apt-get install python3-pyqt5 python3-pyqt5.qtsvg

# For full features including GUI
uv pip install ete3[gui]

First-time NCBI Taxonomy setup:

The first time NCBITaxa is instantiated, it automatically downloads the NCBI taxonomy database (~300MB) to ~/.etetoolkit/taxa.sqlite. This happens only once:

from ete3 import NCBITaxa
ncbi = NCBITaxa()  # Downloads database on first run

Update taxonomy database:

ncbi.update_taxonomy_database()  # Download latest NCBI data

Common Use Cases

Use Case 1: Phylogenomic Pipeline

Complete workflow from gene tree to ortholog identification:

from ete3 import PhyloTree, NCBITaxa

# 1. Load gene tree with alignment
tree = PhyloTree("gene_tree.nw", alignment="alignment.fasta")

# 2. Configure species naming
tree.set_species_naming_function(lambda x: x.split("_")[0])

# 3. Detect evolutionary events
tree.get_descendant_evol_events()

# 4. Annotate with taxonomy
ncbi = NCBITaxa()
for leaf in tree:
    if leaf.species in species_to_taxid:
        taxid = species_to_taxid[leaf.species]
        lineage = ncbi.get_lineage(taxid)
        leaf.add_feature("lineage", lineage)

# 5. Extract ortholog groups
ortho_groups = tree.get_speciation_trees()

# 6. Save and visualize
for i, ortho in enumerate(ortho_groups):
    ortho.write(outfile=f"ortho_{i}.nw")

Use Case 2: Tree Preprocessing and Formatting

Batch process trees for analysis:

# Convert format
python scripts/tree_operations.py convert input.nw output.nw --in-format 0 --out-format 1

# Root at midpoint
python scripts/tree_operations.py reroot input.nw rooted.nw --midpoint

# Prune to focal taxa
python scripts/tree_operations.py prune rooted.nw pruned.nw --keep-taxa taxa_list.txt

# Get statistics
python scripts/tree_operations.py stats pruned.nw

Use Case 3: Publication-Quality Figures

Create styled visualizations:

from ete3 import Tree, TreeStyle, NodeStyle, TextFace

tree = Tree("tree.nw")

# Define clade colors
clade_colors = {
    "Mammals": "red",
    "Birds": "blue",
    "Fish": "green"
}

def layout(node):
    # Highlight clades
    if node.is_leaf():
        for clade, color in clade_colors.items():
            if clade in node.name:
                nstyle = NodeStyle()
                nstyle["fgcolor"] = color
                nstyle["size"] = 8
                node.set_style(nstyle)
    else:
        # Add support values
        if node.support > 0.95:
            support = TextFace(f"{node.support:.2f}", fsize=8)
            node.add_face(support, column=0, position="branch-top")

ts = TreeStyle()
ts.layout_fn = layout
ts.show_scale = True

# Render for publication
tree.render("figure.pdf", w=200, units="mm", tree_style=ts)
tree.render("figure.svg", tree_style=ts)  # Editable vector

Use Case 4: Automated Tree Analysis

Process multiple trees systematically:

from ete3 import Tree
import os

input_dir = "trees"
output_dir = "processed"

for filename in os.listdir(input_dir):
    if filename.endswith(".nw"):
        tree = Tree(os.path.join(input_dir, filename))

        # Standardize: midpoint root, resolve polytomies
        midpoint = tree.get_midpoint_outgroup()
        tree.set_outgroup(midpoint)
        tree.resolve_polytomy(recursive=True)

        # Filter low support branches
        for node in tree.traverse():
            if hasattr(node, 'support') and node.support < 0.5:
                if not node.is_leaf() and not node.is_root():
                    node.delete()

        # Save processed tree
        output_file = os.path.join(output_dir, f"processed_{filename}")
        tree.write(outfile=output_file)

Reference Documentation

For comprehensive API documentation, code examples, and detailed guides, refer to the following resources in the references/ directory:

• api_reference.md: Complete API documentation for all ETE classes and methods (Tree, PhyloTree, ClusterTree, NCBITaxa), including parameters, return types, and code examples
• workflows.md: Common workflow patterns organized by task (tree operations, phylogenetic analysis, tree comparison, taxonomy integration, clustering analysis)
• visualization.md: Comprehensive visualization guide covering TreeStyle, NodeStyle, Faces, layout functions, and advanced visualization techniques

Load these references when detailed information is needed:

# To use API reference
# Read references/api_reference.md for complete method signatures and parameters

# To implement workflows
# Read references/workflows.md for step-by-step workflow examples

# To create visualizations
# Read references/visualization.md for styling and rendering options

Troubleshooting

Import errors:

# If "ModuleNotFoundError: No module named 'ete3'"
uv pip install ete3

# For GUI and rendering issues
uv pip install ete3[gui]

Rendering issues:

If tree.render() or tree.show() fails with Qt-related errors, install system dependencies:

# macOS
brew install qt@5

# Ubuntu/Debian
sudo apt-get install python3-pyqt5 python3-pyqt5.qtsvg

NCBI Taxonomy database:

If database download fails or becomes corrupted:

from ete3 import NCBITaxa
ncbi = NCBITaxa()
ncbi.update_taxonomy_database()  # Redownload database

Memory issues with large trees:

For very large trees (>10,000 leaves), use iterators instead of list comprehensions:

# Memory-efficient iteration
for leaf in tree.iter_leaves():
    process(leaf)

# Instead of
for leaf in tree.get_leaves():  # Loads all into memory
    process(leaf)

Newick Format Reference

ETE supports multiple Newick format specifications (0-100):

• Format 0: Flexible with branch lengths (default)
• Format 1: With internal node names
• Format 2: With bootstrap/support values
• Format 5: Internal node names + branch lengths
• Format 8: All features (names, distances, support)
• Format 9: Leaf names only
• Format 100: Topology only

Specify format when reading/writing:

tree = Tree("tree.nw", format=1)
tree.write(outfile="output.nw", format=5)

NHX (New Hampshire eXtended) format preserves custom features:

tree.write(outfile="tree.nhx", features=["habitat", "temperature", "depth"])

Best Practices

1. Preserve branch lengths: Use preserve_branch_length=True when pruning for phylogenetic analysis
2. Cache content: Use get_cached_content() for repeated access to node contents on large trees
3. Use iterators: Employ iter_* methods for memory-efficient processing of large trees
4. Choose appropriate traversal: Postorder for bottom-up analysis, preorder for top-down
5. Validate monophyly: Always check returned clade type (monophyletic/paraphyletic/polyphyletic)
6. Vector formats for publication: Use PDF or SVG for publication figures (scalable, editable)
7. Interactive testing: Use tree.show() to test visualizations before rendering to file
8. PhyloTree for phylogenetics: Use PhyloTree class for gene trees and evolutionary analysis
9. Copy method selection: "newick" for speed, "cpickle" for full fidelity, "deepcopy" for complex objects
10. NCBI query caching: Store NCBI taxonomy query results to avoid repeated database access

Flow Cytometry Analysis: Complete Analysis Pipeline

Overview

Flow Cytometry Analysis provides end-to-end computational workflows for analyzing flow cytometry data. Starting from FCS file parsing, through compensation matrix application, gating strategies (manual rectangular/polygon gates and automated Gaussian mixture model gating), to downstream analyses including immunophenotyping, CFSE proliferation tracking, cell cycle phase quantification (Dean-Jett-Fox model), and apoptosis assays (Annexin V/PI). This skill extends basic FCS file handling (flowio) with full analytical pipelines.

When to Use This Skill

• Analyzing multi-color flow cytometry experiments
• Applying compensation matrices to correct spectral overlap
• Building sequential gating hierarchies (debris exclusion, singlet gating, live/dead, marker gating)
• Immunophenotyping with multi-marker panels (CD3, CD4, CD8, etc.)
• Quantifying cell proliferation from CFSE/CellTrace dilution
• Determining cell cycle phase distribution from DNA content histograms
• Analyzing apoptosis from Annexin V / propidium iodide staining
• Creating density plots, histograms, and overlay visualizations
• Automated gating for high-throughput cytometry experiments

Related Skills: For raw FCS file parsing and creation use flowio. For single-cell RNA-seq analysis use scanpy.

Installation

uv pip install flowio fcsparser scipy numpy pandas matplotlib scikit-learn

Quick Start

from flowio import FlowData
import numpy as np

# Read FCS file
fcs = FlowData('sample.fcs')
events = fcs.as_array()
channels = fcs.pnn_labels

print(f"Events: {events.shape[0]}, Channels: {len(channels)}")
print(f"Channels: {channels}")

# Simple FSC/SSC gate (remove debris)
fsc_idx = channels.index('FSC-A')
ssc_idx = channels.index('SSC-A')
mask = (events[:, fsc_idx] > 50000) & (events[:, ssc_idx] > 10000)
gated = events[mask]
print(f"After gating: {gated.shape[0]} events ({100*mask.sum()/len(mask):.1f}%)")

Core Capabilities

1. FCS File Handling

Read FCS files and extract channel information.

from flowio import FlowData
import fcsparser
import numpy as np

# Method 1: flowio
fcs = FlowData('sample.fcs')
events = fcs.as_array()
channel_names = fcs.pnn_labels
stain_names = fcs.pns_labels

# Method 2: fcsparser (returns metadata + DataFrame)
meta, data = fcsparser.parse('sample.fcs', reformat_meta=True)
print(f"Channels: {list(data.columns)}")

# Extract compensation matrix from metadata
if '$SPILLOVER' in fcs.text or 'SPILL' in fcs.text:
    spill_str = fcs.text.get('$SPILLOVER', fcs.text.get('SPILL', ''))
    print("Compensation matrix found in metadata")

2. Compensation

Apply spectral overlap correction.

import numpy as np

def parse_spillover_matrix(spill_string):
    """Parse $SPILLOVER or SPILL keyword from FCS metadata."""
    parts = spill_string.split(',')
    n = int(parts[0])
    channel_names = parts[1:n+1]
    values = [float(x) for x in parts[n+1:]]
    matrix = np.array(values).reshape(n, n)
    return channel_names, matrix

def compensate(events, fluoro_indices, spillover_matrix):
    """Apply compensation to fluorescence channels."""
    inv_spill = np.linalg.inv(spillover_matrix)
    compensated = events.copy()
    fluoro_data = events[:, fluoro_indices]
    compensated[:, fluoro_indices] = fluoro_data @ inv_spill.T
    return compensated

# Apply compensation
spill_str = fcs.text.get('$SPILLOVER', fcs.text.get('SPILL', ''))
if spill_str:
    comp_channels, spill_matrix = parse_spillover_matrix(spill_str)
    fluoro_idx = [channel_names.index(ch) for ch in comp_channels]
    events_comp = compensate(events, fluoro_idx, spill_matrix)
    print(f"Compensated {len(fluoro_idx)} fluorescence channels")

3. Gating Strategies

Apply sequential gates to identify cell populations.

Manual rectangular gates:

import numpy as np

def rect_gate(events, ch_idx, lo, hi):
    """Apply rectangular gate on one channel."""
    return (events[:, ch_idx] >= lo) & (events[:, ch_idx] <= hi)

def polygon_gate(events, x_idx, y_idx, vertices):
    """Apply polygon gate using ray casting."""
    from matplotlib.path import Path
    points = np.column_stack([events[:, x_idx], events[:, y_idx]])
    poly = Path(vertices)
    return poly.contains_points(points)

# Sequential gating tree
# Step 1: FSC/SSC - remove debris
scatter_gate = rect_gate(events, fsc_idx, 30000, 250000) & \
               rect_gate(events, ssc_idx, 5000, 200000)

# Step 2: Singlet gate (FSC-A vs FSC-H)
fsch_idx = channel_names.index('FSC-H')
singlet_gate = scatter_gate.copy()
ratio = events[:, fsc_idx] / (events[:, fsch_idx] + 1)
singlet_gate &= (ratio > 0.8) & (ratio < 1.2)

# Step 3: Live gate (exclude viability dye positive)
# live_gate = singlet_gate & rect_gate(events, viability_idx, 0, threshold)

print(f"Debris exclusion: {scatter_gate.sum()} events")
print(f"Singlets: {singlet_gate.sum()} events")

Automated gating with Gaussian Mixture Models:

from sklearn.mixture import GaussianMixture
import numpy as np

def auto_gate_gmm(events, channels_idx, n_components=2):
    """Automated gating using Gaussian Mixture Model."""
    data = events[:, channels_idx]
    # Log transform for better separation
    data_log = np.log1p(np.clip(data, 0, None))

    gmm = GaussianMixture(n_components=n_components, random_state=42)
    labels = gmm.fit_predict(data_log)

    # Identify populations by mean intensity
    means = gmm.means_
    pop_order = np.argsort(means.sum(axis=1))

    return labels, gmm, pop_order

# Auto-gate lymphocytes vs debris
labels, gmm, order = auto_gate_gmm(events, [fsc_idx, ssc_idx], n_components=3)
lymphocyte_mask = labels == order[1]  # Middle population is typically lymphocytes
print(f"Lymphocyte gate: {lymphocyte_mask.sum()} events")

4. Immunophenotyping

Multi-marker panel analysis for population identification.

import numpy as np
import pandas as pd

def immunophenotype(events, channel_names, gates, parent_mask=None):
    """Calculate population frequencies from marker gates."""
    if parent_mask is None:
        parent_mask = np.ones(len(events), dtype=bool)

    results = {}
    parent_count = parent_mask.sum()

    for pop_name, marker_gates in gates.items():
        pop_mask = parent_mask.copy()
        for ch_name, threshold, direction in marker_gates:
            ch_idx = channel_names.index(ch_name)
            if direction == '+':
                pop_mask &= events[:, ch_idx] > threshold
            else:
                pop_mask &= events[:, ch_idx] <= threshold

        count = pop_mask.sum()
        freq = 100 * count / parent_count if parent_count > 0 else 0
        results[pop_name] = {'count': count, 'frequency': freq, 'mask': pop_mask}

    return results

# Define immunophenotyping panel
phenotype_gates = {
    'CD3+ T cells': [('CD3', 500, '+')],
    'CD4+ T helper': [('CD3', 500, '+'), ('CD4', 300, '+'), ('CD8', 300, '-')],
    'CD8+ T cytotoxic': [('CD3', 500, '+'), ('CD4', 300, '-'), ('CD8', 300, '+')],
    'B cells (CD19+)': [('CD3', 500, '-'), ('CD19', 200, '+')],
    'NK cells': [('CD3', 500, '-'), ('CD56', 200, '+')],
}

# Calculate (assuming live-gated events)
results = immunophenotype(events, channel_names, phenotype_gates, parent_mask=singlet_gate)
for pop, data in results.items():
    print(f"{pop}: {data['count']} cells ({data['frequency']:.1f}%)")

5. CFSE Proliferation Analysis

Quantify cell division from dye dilution.

import numpy as np
from scipy.signal import find_peaks
from scipy.optimize import curve_fit

def analyze_cfse_proliferation(cfse_values, n_generations=8):
    """Analyze CFSE dilution to quantify proliferation."""
    # Log-transform CFSE values
    log_cfse = np.log2(cfse_values[cfse_values > 0])

    # Build histogram
    hist, bin_edges = np.histogram(log_cfse, bins=200)
    bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2

    # Find peaks (each peak = one generation)
    peaks, properties = find_peaks(hist, height=len(cfse_values)*0.005,
                                    distance=10, prominence=50)

    # Peaks should be ~1 log2 unit apart (halving of dye)
    peak_positions = bin_centers[peaks]
    peak_heights = hist[peaks]

    # Calculate division index
    # DI = sum(cells in gen i) / sum(cells in gen 0 equivalents)
    total_cells = sum(peak_heights)
    undivided_equiv = sum(h / (2**i) for i, h in enumerate(peak_heights))
    division_index = total_cells / undivided_equiv if undivided_equiv > 0 else 0

    # Proliferation index (only among divided cells)
    if len(peak_heights) > 1:
        divided_cells = sum(peak_heights[1:])
        divided_precursors = sum(h / (2**i) for i, h in enumerate(peak_heights[1:], 1))
        prolif_index = divided_cells / divided_precursors if divided_precursors > 0 else 0
    else:
        prolif_index = 1.0

    return {
        'n_peaks': len(peaks),
        'division_index': division_index,
        'proliferation_index': prolif_index,
        'peak_positions': peak_positions,
        'peak_heights': peak_heights
    }

# Analyze
cfse_idx = channel_names.index('CFSE') if 'CFSE' in channel_names else 0
cfse_data = events[singlet_gate, cfse_idx]
prolif = analyze_cfse_proliferation(cfse_data)
print(f"Generations detected: {prolif['n_peaks']}")
print(f"Division index: {prolif['division_index']:.2f}")
print(f"Proliferation index: {prolif['proliferation_index']:.2f}")

6. Cell Cycle Analysis

Fit DNA content histograms to determine cell cycle phase distribution.

import numpy as np
from scipy.optimize import curve_fit

def dean_jett_fox(x, g1_mean, g1_std, g1_amp,
                  s_amp, s_slope,
                  g2_amp, g2_std):
    """Dean-Jett-Fox cell cycle model.
    G1 and G2/M as Gaussians, S-phase as polynomial."""
    g2_mean = 2 * g1_mean  # G2/M DNA content is 2x G1

    # G1 peak
    g1 = g1_amp * np.exp(-0.5 * ((x - g1_mean) / g1_std) ** 2)

    # G2/M peak
    g2 = g2_amp * np.exp(-0.5 * ((x - g2_mean) / g2_std) ** 2)

    # S-phase (linear between G1 and G2)
    s = np.zeros_like(x)
    s_mask = (x > g1_mean + g1_std) & (x < g2_mean - g2_std)
    s[s_mask] = s_amp + s_slope * (x[s_mask] - g1_mean)

    return g1 + g2 + np.clip(s, 0, None)

def cell_cycle_analysis(dna_values):
    """Determine G0/G1, S, G2/M percentages from PI staining."""
    hist, bin_edges = np.histogram(dna_values, bins=256, range=(0, dna_values.max()))
    bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2

    # Initial parameter estimates
    g1_peak = bin_centers[np.argmax(hist)]
    g1_amp = hist.max()

    p0 = [g1_peak, g1_peak*0.05, g1_amp, g1_amp*0.1, 0, g1_amp*0.3, g1_peak*0.07]
    bounds = ([0, 0, 0, 0, -np.inf, 0, 0],
              [np.inf, np.inf, np.inf, np.inf, np.inf, np.inf, np.inf])

    try:
        popt, pcov = curve_fit(dean_jett_fox, bin_centers, hist, p0=p0,
                                bounds=bounds, maxfev=10000)
        fitted = dean_jett_fox(bin_centers, *popt)

        # Calculate phase fractions
        g1_mean, g1_std = popt[0], popt[1]
        g2_mean = 2 * g1_mean

        g1_mask = bin_centers < (g1_mean + 2*g1_std)
        g2_mask = bin_centers > (g2_mean - 2*popt[5])
        s_mask = ~g1_mask & ~g2_mask

        total = hist.sum()
        g1_pct = 100 * hist[g1_mask].sum() / total
        s_pct = 100 * hist[s_mask].sum() / total
        g2m_pct = 100 * hist[g2_mask].sum() / total

        return {'G0/G1': g1_pct, 'S': s_pct, 'G2/M': g2m_pct}
    except RuntimeError:
        return {'error': 'Curve fitting failed — check DNA histogram quality'}

# Analyze
pi_idx = channel_names.index('PI') if 'PI' in channel_names else 0
dna_data = events[singlet_gate, pi_idx]
phases = cell_cycle_analysis(dna_data)
print(f"G0/G1: {phases.get('G0/G1', 'N/A'):.1f}%")
print(f"S: {phases.get('S', 'N/A'):.1f}%")
print(f"G2/M: {phases.get('G2/M', 'N/A'):.1f}%")

7. Apoptosis Assays

Analyze Annexin V / Propidium Iodide staining.

import numpy as np

def annexin_v_pi_analysis(events, annexin_idx, pi_idx,
                          annexin_threshold, pi_threshold, parent_mask=None):
    """Quadrant analysis for apoptosis assay."""
    if parent_mask is None:
        parent_mask = np.ones(len(events), dtype=bool)

    gated = events[parent_mask]
    total = len(gated)

    annexin_pos = gated[:, annexin_idx] > annexin_threshold
    pi_pos = gated[:, pi_idx] > pi_threshold

    viable = (~annexin_pos) & (~pi_pos)
    early_apoptotic = annexin_pos & (~pi_pos)
    late_apoptotic = annexin_pos & pi_pos
    necrotic = (~annexin_pos) & pi_pos

    return {
        'viable': 100 * viable.sum() / total,
        'early_apoptotic': 100 * early_apoptotic.sum() / total,
        'late_apoptotic': 100 * late_apoptotic.sum() / total,
        'necrotic': 100 * necrotic.sum() / total
    }

# Analyze
results = annexin_v_pi_analysis(events, annexin_idx=3, pi_idx=4,
                                 annexin_threshold=200, pi_threshold=200,
                                 parent_mask=singlet_gate)
for state, pct in results.items():
    print(f"{state}: {pct:.1f}%")

8. Visualization

Create publication-quality flow cytometry plots.

import matplotlib.pyplot as plt
import numpy as np

def density_plot(events, x_idx, y_idx, channel_names, ax=None, gate_mask=None):
    """2D density (pseudo-color) plot."""
    if ax is None:
        fig, ax = plt.subplots(figsize=(6, 6))
    data = events if gate_mask is None else events[gate_mask]
    ax.hist2d(data[:, x_idx], data[:, y_idx], bins=256,
              cmap='jet', norm=plt.matplotlib.colors.LogNorm())
    ax.set_xlabel(channel_names[x_idx])
    ax.set_ylabel(channel_names[y_idx])
    return ax

def histogram_overlay(datasets, channel_idx, labels, channel_name, ax=None):
    """Overlay histograms for comparing conditions."""
    if ax is None:
        fig, ax = plt.subplots(figsize=(8, 5))
    for data, label in zip(datasets, labels):
        ax.hist(data[:, channel_idx], bins=256, alpha=0.5, label=label, density=True)
    ax.set_xlabel(channel_name)
    ax.set_ylabel('Density')
    ax.legend()
    return ax

Typical Workflows

Workflow 1: Full Immunophenotyping from Multi-Color FCS Files

from flowio import FlowData
import numpy as np

fcs = FlowData('pbmc_panel.fcs')
events = fcs.as_array()
ch = fcs.pnn_labels

# Compensate
spill_str = fcs.text.get('$SPILLOVER', fcs.text.get('SPILL', ''))
if spill_str:
    comp_ch, spill = parse_spillover_matrix(spill_str)
    fidx = [ch.index(c) for c in comp_ch]
    events = compensate(events, fidx, spill)

# Gate: debris → singlets → live
fsc, ssc = ch.index('FSC-A'), ch.index('SSC-A')
gate = (events[:, fsc] > 30000) & (events[:, ssc] > 5000) & (events[:, ssc] < 200000)

# Immunophenotype
panels = {
    'T cells (CD3+)': [('CD3', 500, '+')],
    'Helper T (CD3+CD4+)': [('CD3', 500, '+'), ('CD4', 300, '+')],
    'Cytotoxic T (CD3+CD8+)': [('CD3', 500, '+'), ('CD8', 300, '+')],
}
results = immunophenotype(events, ch, panels, parent_mask=gate)
for pop, data in results.items():
    print(f"{pop}: {data['frequency']:.1f}%")

Workflow 2: CFSE Proliferation Analysis with Division Tracking

from flowio import FlowData

fcs = FlowData('cfse_stimulated.fcs')
events = fcs.as_array()
ch = fcs.pnn_labels

# Gate live cells
fsc = ch.index('FSC-A')
live = events[:, fsc] > 20000

# CFSE analysis
cfse_idx = ch.index('CFSE')
prolif = analyze_cfse_proliferation(events[live, cfse_idx])
print(f"Divisions detected: {prolif['n_peaks']}")
print(f"Division index: {prolif['division_index']:.2f}")
print(f"Proliferation index: {prolif['proliferation_index']:.2f}")

Workflow 3: Cell Cycle Distribution from PI-Stained Samples

from flowio import FlowData

fcs = FlowData('pi_stained.fcs')
events = fcs.as_array()
ch = fcs.pnn_labels

# Gate singlets (PI-area vs PI-width)
pi_a = ch.index('PI-A')
pi_w = ch.index('PI-W') if 'PI-W' in ch else None

if pi_w:
    singlets = (events[:, pi_w] > 50000) & (events[:, pi_w] < 150000)
else:
    singlets = np.ones(len(events), dtype=bool)

phases = cell_cycle_analysis(events[singlets, pi_a])
for phase, pct in phases.items():
    print(f"{phase}: {pct:.1f}%")

Best Practices

1. Always compensate before gating on fluorescence channels — uncompensated data leads to incorrect population identification
2. Gate sequentially — debris exclusion first, then singlets, then viability, then markers
3. Use FMO controls (Fluorescence Minus One) to set accurate positive/negative thresholds
4. Log or biexponential transform fluorescence data for visualization and gating
5. Back-gate to verify gated populations appear in expected scatter positions
6. Include isotype controls or unstained controls for threshold determination
7. Report parent gate denominators — frequencies must reference the correct parent population
8. Quality check — verify event counts are sufficient for rare populations (>100 events minimum)

Troubleshooting

Problem: Compensation produces negative values Solution: This is normal for properly compensated data. Do not zero-clip — negative values carry information. Use biexponential display for visualization.

Problem: GMM auto-gating splits one population into two Solution: Reduce n_components. Pre-gate to remove obvious debris first. Try log-transforming data before fitting.

Problem: Cell cycle fitting fails Solution: Ensure DNA histogram has clear G1 peak. Filter for singlets using DNA-area vs DNA-width. Check that PI staining is optimal (no under/over-staining).

Problem: CFSE peaks not resolved Solution: Increase histogram bins. Ensure cells were labeled at correct CFSE concentration. Later generations may be unresolvable — focus on first 4-5 divisions.

Problem: FCS file channels have unexpected names Solution: Check both pnn_labels (short names) and pns_labels (descriptive stain names). Different instruments use different naming conventions.

Resources

• FlowIO Documentation
• ISAC Data Standards
• Flow Cytometry Bioinformatics (Bioconductor)
• Practical Flow Cytometry by Howard Shapiro

FlowIO: Flow Cytometry Standard File Handler

Overview

FlowIO is a lightweight Python library for reading and writing Flow Cytometry Standard (FCS) files. Parse FCS metadata, extract event data, and create new FCS files with minimal dependencies. The library supports FCS versions 2.0, 3.0, and 3.1, making it ideal for backend services, data pipelines, and basic cytometry file operations.

When to Use This Skill

This skill should be used when:

• FCS files requiring parsing or metadata extraction
• Flow cytometry data needing conversion to NumPy arrays
• Event data requiring export to FCS format
• Multi-dataset FCS files needing separation
• Channel information extraction (scatter, fluorescence, time)
• Cytometry file validation or inspection
• Pre-processing workflows before advanced analysis

Related Tools: For advanced flow cytometry analysis including compensation, gating, and FlowJo/GatingML support, recommend FlowKit library as a companion to FlowIO.

Installation

uv pip install flowio

Requires Python 3.9 or later.

Quick Start

Basic File Reading

from flowio import FlowData

# Read FCS file
flow_data = FlowData('experiment.fcs')

# Access basic information
print(f"FCS Version: {flow_data.version}")
print(f"Events: {flow_data.event_count}")
print(f"Channels: {flow_data.pnn_labels}")

# Get event data as NumPy array
events = flow_data.as_array()  # Shape: (events, channels)

Creating FCS Files

import numpy as np
from flowio import create_fcs

# Prepare data
data = np.array([[100, 200, 50], [150, 180, 60]])  # 2 events, 3 channels
channels = ['FSC-A', 'SSC-A', 'FL1-A']

# Create FCS file
create_fcs('output.fcs', data, channels)

Core Workflows

Reading and Parsing FCS Files

The FlowData class provides the primary interface for reading FCS files.

Standard Reading:

from flowio import FlowData

# Basic reading
flow = FlowData('sample.fcs')

# Access attributes
version = flow.version              # '3.0', '3.1', etc.
event_count = flow.event_count      # Number of events
channel_count = flow.channel_count  # Number of channels
pnn_labels = flow.pnn_labels        # Short channel names
pns_labels = flow.pns_labels        # Descriptive stain names

# Get event data
events = flow.as_array()            # Preprocessed (gain, log scaling applied)
raw_events = flow.as_array(preprocess=False)  # Raw data

Memory-Efficient Metadata Reading:

When only metadata is needed (no event data):

# Only parse TEXT segment, skip DATA and ANALYSIS
flow = FlowData('sample.fcs', only_text=True)

# Access metadata
metadata = flow.text  # Dictionary of TEXT segment keywords
print(metadata.get('$DATE'))  # Acquisition date
print(metadata.get('$CYT'))   # Instrument name

Handling Problematic Files:

Some FCS files have offset discrepancies or errors:

# Ignore offset discrepancies between HEADER and TEXT sections
flow = FlowData('problematic.fcs', ignore_offset_discrepancy=True)

# Use HEADER offsets instead of TEXT offsets
flow = FlowData('problematic.fcs', use_header_offsets=True)

# Ignore offset errors entirely
flow = FlowData('problematic.fcs', ignore_offset_error=True)

Excluding Null Channels:

# Exclude specific channels during parsing
flow = FlowData('sample.fcs', null_channel_list=['Time', 'Null'])

Extracting Metadata and Channel Information

FCS files contain rich metadata in the TEXT segment.

Common Metadata Keywords:

flow = FlowData('sample.fcs')

# File-level metadata
text_dict = flow.text
acquisition_date = text_dict.get('$DATE', 'Unknown')
instrument = text_dict.get('$CYT', 'Unknown')
data_type = flow.data_type  # 'I', 'F', 'D', 'A'

# Channel metadata
for i in range(flow.channel_count):
    pnn = flow.pnn_labels[i]      # Short name (e.g., 'FSC-A')
    pns = flow.pns_labels[i]      # Descriptive name (e.g., 'Forward Scatter')
    pnr = flow.pnr_values[i]      # Range/max value
    print(f"Channel {i}: {pnn} ({pns}), Range: {pnr}")

Channel Type Identification:

FlowIO automatically categorizes channels:

# Get indices by channel type
scatter_idx = flow.scatter_indices    # [0, 1] for FSC, SSC
fluoro_idx = flow.fluoro_indices      # [2, 3, 4] for FL channels
time_idx = flow.time_index            # Index of time channel (or None)

# Access specific channel types
events = flow.as_array()
scatter_data = events[:, scatter_idx]
fluorescence_data = events[:, fluoro_idx]

ANALYSIS Segment:

If present, access processed results:

if flow.analysis:
    analysis_keywords = flow.analysis  # Dictionary of ANALYSIS keywords
    print(analysis_keywords)

Creating New FCS Files

Generate FCS files from NumPy arrays or other data sources.

Basic Creation:

import numpy as np
from flowio import create_fcs

# Create event data (rows=events, columns=channels)
events = np.random.rand(10000, 5) * 1000

# Define channel names
channel_names = ['FSC-A', 'SSC-A', 'FL1-A', 'FL2-A', 'Time']

# Create FCS file
create_fcs('output.fcs', events, channel_names)

With Descriptive Channel Names:

# Add optional descriptive names (PnS)
channel_names = ['FSC-A', 'SSC-A', 'FL1-A', 'FL2-A', 'Time']
descriptive_names = ['Forward Scatter', 'Side Scatter', 'FITC', 'PE', 'Time']

create_fcs('output.fcs',
           events,
           channel_names,
           opt_channel_names=descriptive_names)

With Custom Metadata:

# Add TEXT segment metadata
metadata = {
    '$SRC': 'Python script',
    '$DATE': '19-OCT-2025',
    '$CYT': 'Synthetic Instrument',
    '$INST': 'Laboratory A'
}

create_fcs('output.fcs',
           events,
           channel_names,
           opt_channel_names=descriptive_names,
           metadata=metadata)

Note: FlowIO exports as FCS 3.1 with single-precision floating-point data.

Exporting Modified Data

Modify existing FCS files and re-export them.

Approach 1: Using write_fcs() Method:

from flowio import FlowData

# Read original file
flow = FlowData('original.fcs')

# Write with updated metadata
flow.write_fcs('modified.fcs', metadata={'$SRC': 'Modified data'})

Approach 2: Extract, Modify, and Recreate:

For modifying event data:

from flowio import FlowData, create_fcs

# Read and extract data
flow = FlowData('original.fcs')
events = flow.as_array(preprocess=False)

# Modify event data
events[:, 0] = events[:, 0] * 1.5  # Scale first channel

# Create new FCS file with modified data
create_fcs('modified.fcs',
           events,
           flow.pnn_labels,
           opt_channel_names=flow.pns_labels,
           metadata=flow.text)

Handling Multi-Dataset FCS Files

Some FCS files contain multiple datasets in a single file.

Detecting Multi-Dataset Files:

from flowio import FlowData, MultipleDataSetsError

try:
    flow = FlowData('sample.fcs')
except MultipleDataSetsError:
    print("File contains multiple datasets")
    # Use read_multiple_data_sets() instead

Reading All Datasets:

from flowio import read_multiple_data_sets

# Read all datasets from file
datasets = read_multiple_data_sets('multi_dataset.fcs')

print(f"Found {len(datasets)} datasets")

# Process each dataset
for i, dataset in enumerate(datasets):
    print(f"\nDataset {i}:")
    print(f"  Events: {dataset.event_count}")
    print(f"  Channels: {dataset.pnn_labels}")

    # Get event data for this dataset
    events = dataset.as_array()
    print(f"  Shape: {events.shape}")
    print(f"  Mean values: {events.mean(axis=0)}")

Reading Specific Dataset:

from flowio import FlowData

# Read first dataset (nextdata_offset=0)
first_dataset = FlowData('multi.fcs', nextdata_offset=0)

# Read second dataset using NEXTDATA offset from first
next_offset = int(first_dataset.text['$NEXTDATA'])
if next_offset > 0:
    second_dataset = FlowData('multi.fcs', nextdata_offset=next_offset)

Data Preprocessing

FlowIO applies standard FCS preprocessing transformations when preprocess=True.

Preprocessing Steps:

1. Gain Scaling: Multiply values by PnG (gain) keyword
2. Logarithmic Transformation: Apply PnE exponential transformation if present
   • Formula: value = a * 10^(b * raw_value) where PnE = "a,b"
3. Time Scaling: Convert time values to appropriate units

Controlling Preprocessing:

# Preprocessed data (default)
preprocessed = flow.as_array(preprocess=True)

# Raw data (no transformations)
raw = flow.as_array(preprocess=False)

Error Handling

Handle common FlowIO exceptions appropriately.

from flowio import (
    FlowData,
    FCSParsingError,
    DataOffsetDiscrepancyError,
    MultipleDataSetsError
)

try:
    flow = FlowData('sample.fcs')
    events = flow.as_array()

except FCSParsingError as e:
    print(f"Failed to parse FCS file: {e}")
    # Try with relaxed parsing
    flow = FlowData('sample.fcs', ignore_offset_error=True)

except DataOffsetDiscrepancyError as e:
    print(f"Offset discrepancy detected: {e}")
    # Use ignore_offset_discrepancy parameter
    flow = FlowData('sample.fcs', ignore_offset_discrepancy=True)

except MultipleDataSetsError as e:
    print(f"Multiple datasets detected: {e}")
    # Use read_multiple_data_sets instead
    from flowio import read_multiple_data_sets
    datasets = read_multiple_data_sets('sample.fcs')

except Exception as e:
    print(f"Unexpected error: {e}")

Common Use Cases

Inspecting FCS File Contents

Quick exploration of FCS file structure:

from flowio import FlowData

flow = FlowData('unknown.fcs')

print("=" * 50)
print(f"File: {flow.name}")
print(f"Version: {flow.version}")
print(f"Size: {flow.file_size:,} bytes")
print("=" * 50)

print(f"\nEvents: {flow.event_count:,}")
print(f"Channels: {flow.channel_count}")

print("\nChannel Information:")
for i, (pnn, pns) in enumerate(zip(flow.pnn_labels, flow.pns_labels)):
    ch_type = "scatter" if i in flow.scatter_indices else \
              "fluoro" if i in flow.fluoro_indices else \
              "time" if i == flow.time_index else "other"
    print(f"  [{i}] {pnn:10s} | {pns:30s} | {ch_type}")

print("\nKey Metadata:")
for key in ['$DATE', '$BTIM', '$ETIM', '$CYT', '$INST', '$SRC']:
    value = flow.text.get(key, 'N/A')
    print(f"  {key:15s}: {value}")

Batch Processing Multiple Files

Process a directory of FCS files:

from pathlib import Path
from flowio import FlowData
import pandas as pd

# Find all FCS files
fcs_files = list(Path('data/').glob('*.fcs'))

# Extract summary information
summaries = []
for fcs_path in fcs_files:
    try:
        flow = FlowData(str(fcs_path), only_text=True)
        summaries.append({
            'filename': fcs_path.name,
            'version': flow.version,
            'events': flow.event_count,
            'channels': flow.channel_count,
            'date': flow.text.get('$DATE', 'N/A')
        })
    except Exception as e:
        print(f"Error processing {fcs_path.name}: {e}")

# Create summary DataFrame
df = pd.DataFrame(summaries)
print(df)

Converting FCS to CSV

Export event data to CSV format:

from flowio import FlowData
import pandas as pd

# Read FCS file
flow = FlowData('sample.fcs')

# Convert to DataFrame
df = pd.DataFrame(
    flow.as_array(),
    columns=flow.pnn_labels
)

# Add metadata as attributes
df.attrs['fcs_version'] = flow.version
df.attrs['instrument'] = flow.text.get('$CYT', 'Unknown')

# Export to CSV
df.to_csv('output.csv', index=False)
print(f"Exported {len(df)} events to CSV")

Filtering Events and Re-exporting

Apply filters and save filtered data:

from flowio import FlowData, create_fcs
import numpy as np

# Read original file
flow = FlowData('sample.fcs')
events = flow.as_array(preprocess=False)

# Apply filtering (example: threshold on first channel)
fsc_idx = 0
threshold = 500
mask = events[:, fsc_idx] > threshold
filtered_events = events[mask]

print(f"Original events: {len(events)}")
print(f"Filtered events: {len(filtered_events)}")

# Create new FCS file with filtered data
create_fcs('filtered.fcs',
           filtered_events,
           flow.pnn_labels,
           opt_channel_names=flow.pns_labels,
           metadata={**flow.text, '$SRC': 'Filtered data'})

Extracting Specific Channels

Extract and process specific channels:

from flowio import FlowData
import numpy as np

flow = FlowData('sample.fcs')
events = flow.as_array()

# Extract fluorescence channels only
fluoro_indices = flow.fluoro_indices
fluoro_data = events[:, fluoro_indices]
fluoro_names = [flow.pnn_labels[i] for i in fluoro_indices]

print(f"Fluorescence channels: {fluoro_names}")
print(f"Shape: {fluoro_data.shape}")

# Calculate statistics per channel
for i, name in enumerate(fluoro_names):
    channel_data = fluoro_data[:, i]
    print(f"\n{name}:")
    print(f"  Mean: {channel_data.mean():.2f}")
    print(f"  Median: {np.median(channel_data):.2f}")
    print(f"  Std Dev: {channel_data.std():.2f}")

Best Practices

1. Memory Efficiency: Use only_text=True when event data is not needed
2. Error Handling: Wrap file operations in try-except blocks for robust code
3. Multi-Dataset Detection: Check for MultipleDataSetsError and use appropriate function
4. Preprocessing Control: Explicitly set preprocess parameter based on analysis needs
5. Offset Issues: If parsing fails, try ignore_offset_discrepancy=True parameter
6. Channel Validation: Verify channel counts and names match expectations before processing
7. Metadata Preservation: When modifying files, preserve original TEXT segment keywords

Advanced Topics

Understanding FCS File Structure

FCS files consist of four segments:

1. HEADER: FCS version and byte offsets for other segments
2. TEXT: Key-value metadata pairs (delimiter-separated)
3. DATA: Raw event data (binary/float/ASCII format)
4. ANALYSIS (optional): Results from data processing

Access these segments via FlowData attributes:

• flow.header - HEADER segment
• flow.text - TEXT segment keywords
• flow.events - DATA segment (as bytes)
• flow.analysis - ANALYSIS segment keywords (if present)

Detailed API Reference

For comprehensive API documentation including all parameters, methods, exceptions, and FCS keyword reference, consult the detailed reference file:

Read: references/api_reference.md

The reference includes:

• Complete FlowData class documentation
• All utility functions (read_multiple_data_sets, create_fcs)
• Exception classes and handling
• FCS file structure details
• Common TEXT segment keywords
• Extended example workflows

When working with complex FCS operations or encountering unusual file formats, load this reference for detailed guidance.

Integration Notes

NumPy Arrays: All event data is returned as NumPy ndarrays with shape (events, channels)

Pandas DataFrames: Easily convert to DataFrames for analysis:

import pandas as pd
df = pd.DataFrame(flow.as_array(), columns=flow.pnn_labels)

FlowKit Integration: For advanced analysis (compensation, gating, FlowJo support), use FlowKit library which builds on FlowIO's parsing capabilities

Web Applications: FlowIO's minimal dependencies make it ideal for web backend services processing FCS uploads

Troubleshooting

Problem: "Offset discrepancy error" Solution: Use ignore_offset_discrepancy=True parameter

Problem: "Multiple datasets error" Solution: Use read_multiple_data_sets() function instead of FlowData constructor

Problem: Out of memory with large files Solution: Use only_text=True for metadata-only operations, or process events in chunks

Problem: Unexpected channel counts Solution: Check for null channels; use null_channel_list parameter to exclude them

Problem: Cannot modify event data in place Solution: FlowIO doesn't support direct modification; extract data, modify, then use create_fcs() to save

Summary

FlowIO provides essential FCS file handling capabilities for flow cytometry workflows. Use it for parsing, metadata extraction, and file creation. For simple file operations and data extraction, FlowIO is sufficient. For complex analysis including compensation and gating, integrate with FlowKit or other specialized tools.

gget

Overview

gget is a command-line bioinformatics tool and Python package providing unified access to 20+ genomic databases and analysis methods. Query gene information, sequence analysis, protein structures, expression data, and disease associations through a consistent interface. All gget modules work both as command-line tools and as Python functions.

Important: The databases queried by gget are continuously updated, which sometimes changes their structure. gget modules are tested automatically on a biweekly basis and updated to match new database structures when necessary.

Installation

Install gget in a clean virtual environment to avoid conflicts:

# Using uv (recommended)
uv uv pip install gget

# Or using pip
uv pip install --upgrade gget

# In Python/Jupyter
import gget

Quick Start

Basic usage pattern for all modules:

# Command-line
gget <module> [arguments] [options]

# Python
gget.module(arguments, options)

Most modules return:

• Command-line: JSON (default) or CSV with -csv flag
• Python: DataFrame or dictionary

Common flags across modules:

• -o/--out: Save results to file
• -q/--quiet: Suppress progress information
• -csv: Return CSV format (command-line only)

Module Categories

1. Reference & Gene Information

gget ref - Reference Genome Downloads

Retrieve download links and metadata for Ensembl reference genomes.

Parameters:

• species: Genus_species format (e.g., 'homo_sapiens', 'mus_musculus'). Shortcuts: 'human', 'mouse'
• -w/--which: Specify return types (gtf, cdna, dna, cds, cdrna, pep). Default: all
• -r/--release: Ensembl release number (default: latest)
• -l/--list_species: List available vertebrate species
• -liv/--list_iv_species: List available invertebrate species
• -ftp: Return only FTP links
• -d/--download: Download files (requires curl)

Examples:

# List available species
gget ref --list_species

# Get all reference files for human
gget ref homo_sapiens

# Download only GTF annotation for mouse
gget ref -w gtf -d mouse

# Python
gget.ref("homo_sapiens")
gget.ref("mus_musculus", which="gtf", download=True)

gget search - Gene Search

Locate genes by name or description across species.

Parameters:

• searchwords: One or more search terms (case-insensitive)
• -s/--species: Target species (e.g., 'homo_sapiens', 'mouse')
• -r/--release: Ensembl release number
• -t/--id_type: Return 'gene' (default) or 'transcript'
• -ao/--andor: 'or' (default) finds ANY searchword; 'and' requires ALL
• -l/--limit: Maximum results to return

Returns: ensembl_id, gene_name, ensembl_description, ext_ref_description, biotype, URL

Examples:

# Search for GABA-related genes in human
gget search -s human gaba gamma-aminobutyric

# Find specific gene, require all terms
gget search -s mouse -ao and pax7 transcription

# Python
gget.search(["gaba", "gamma-aminobutyric"], species="homo_sapiens")

gget info - Gene/Transcript Information

Retrieve comprehensive gene and transcript metadata from Ensembl, UniProt, and NCBI.

Parameters:

• ens_ids: One or more Ensembl IDs (also supports WormBase, Flybase IDs). Limit: ~1000 IDs
• -n/--ncbi: Disable NCBI data retrieval
• -u/--uniprot: Disable UniProt data retrieval
• -pdb: Include PDB identifiers (increases runtime)

Returns: UniProt ID, NCBI gene ID, primary gene name, synonyms, protein names, descriptions, biotype, canonical transcript

Examples:

# Get info for multiple genes
gget info ENSG00000034713 ENSG00000104853 ENSG00000170296

# Include PDB IDs
gget info ENSG00000034713 -pdb

# Python
gget.info(["ENSG00000034713", "ENSG00000104853"], pdb=True)

gget seq - Sequence Retrieval

Fetch nucleotide or amino acid sequences for genes and transcripts.

Parameters:

• ens_ids: One or more Ensembl identifiers
• -t/--translate: Fetch amino acid sequences instead of nucleotide
• -iso/--isoforms: Return all transcript variants (gene IDs only)

Returns: FASTA format sequences

Examples:

# Get nucleotide sequences
gget seq ENSG00000034713 ENSG00000104853

# Get all protein isoforms
gget seq -t -iso ENSG00000034713

# Python
gget.seq(["ENSG00000034713"], translate=True, isoforms=True)

2. Sequence Analysis & Alignment

gget blast - BLAST Searches

BLAST nucleotide or amino acid sequences against standard databases.

Parameters:

• sequence: Sequence string or path to FASTA/.txt file
• -p/--program: blastn, blastp, blastx, tblastn, tblastx (auto-detected)
• -db/--database:
  • Nucleotide: nt, refseq_rna, pdbnt
  • Protein: nr, swissprot, pdbaa, refseq_protein
• -l/--limit: Max hits (default: 50)
• -e/--expect: E-value cutoff (default: 10.0)
• -lcf/--low_comp_filt: Enable low complexity filtering
• -mbo/--megablast_off: Disable MegaBLAST (blastn only)

Examples:

# BLAST protein sequence
gget blast MKWMFKEDHSLEHRCVESAKIRAKYPDRVPVIVEKVSGSQIVDIDKRKYLVPSDITVAQFMWIIRKRIQLPSEKAIFLFVDKTVPQSR

# BLAST from file with specific database
gget blast sequence.fasta -db swissprot -l 10

# Python
gget.blast("MKWMFK...", database="swissprot", limit=10)

gget blat - BLAT Searches

Locate genomic positions of sequences using UCSC BLAT.

Parameters:

• sequence: Sequence string or path to FASTA/.txt file
• -st/--seqtype: 'DNA', 'protein', 'translated%20RNA', 'translated%20DNA' (auto-detected)
• -a/--assembly: Target assembly (default: 'human'/hg38; options: 'mouse'/mm39, 'zebrafinch'/taeGut2, etc.)

Returns: genome, query size, alignment positions, matches, mismatches, alignment percentage

Examples:

# Find genomic location in human
gget blat ATCGATCGATCGATCG

# Search in different assembly
gget blat -a mm39 ATCGATCGATCGATCG

# Python
gget.blat("ATCGATCGATCGATCG", assembly="mouse")

gget muscle - Multiple Sequence Alignment

Align multiple nucleotide or amino acid sequences using Muscle5.

Parameters:

• fasta: Sequences or path to FASTA/.txt file
• -s5/--super5: Use Super5 algorithm for faster processing (large datasets)

Returns: Aligned sequences in ClustalW format or aligned FASTA (.afa)

Examples:

# Align sequences from file
gget muscle sequences.fasta -o aligned.afa

# Use Super5 for large dataset
gget muscle large_dataset.fasta -s5

# Python
gget.muscle("sequences.fasta", save=True)

gget diamond - Local Sequence Alignment

Perform fast local protein or translated DNA alignment using DIAMOND.

Parameters:

• Query: Sequences (string/list) or FASTA file path
• --reference: Reference sequences (string/list) or FASTA file path (required)
• --sensitivity: fast, mid-sensitive, sensitive, more-sensitive, very-sensitive (default), ultra-sensitive
• --threads: CPU threads (default: 1)
• --diamond_db: Save database for reuse
• --translated: Enable nucleotide-to-amino acid alignment

Returns: Identity percentage, sequence lengths, match positions, gap openings, E-values, bit scores

Examples:

# Align against reference
gget diamond GGETISAWESQME -ref reference.fasta --threads 4

# Save database for reuse
gget diamond query.fasta -ref ref.fasta --diamond_db my_db.dmnd

# Python
gget.diamond("GGETISAWESQME", reference="reference.fasta", threads=4)

3. Structural & Protein Analysis

gget pdb - Protein Structures

Query RCSB Protein Data Bank for structure and metadata.

Parameters:

• pdb_id: PDB identifier (e.g., '7S7U')
• -r/--resource: Data type (pdb, entry, pubmed, assembly, entity types)
• -i/--identifier: Assembly, entity, or chain ID

Returns: PDB format (structures) or JSON (metadata)

Examples:

# Download PDB structure
gget pdb 7S7U -o 7S7U.pdb

# Get metadata
gget pdb 7S7U -r entry

# Python
gget.pdb("7S7U", save=True)

gget alphafold - Protein Structure Prediction

Predict 3D protein structures using simplified AlphaFold2.

Setup Required:

# Install OpenMM first
uv pip install openmm

# Then setup AlphaFold
gget setup alphafold

Parameters:

• sequence: Amino acid sequence (string), multiple sequences (list), or FASTA file. Multiple sequences trigger multimer modeling
• -mr/--multimer_recycles: Recycling iterations (default: 3; recommend 20 for accuracy)
• -mfm/--multimer_for_monomer: Apply multimer model to single proteins
• -r/--relax: AMBER relaxation for top-ranked model
• plot: Python-only; generate interactive 3D visualization (default: True)
• show_sidechains: Python-only; include side chains (default: True)

Returns: PDB structure file, JSON alignment error data, optional 3D visualization

Examples:

# Predict single protein structure
gget alphafold MKWMFKEDHSLEHRCVESAKIRAKYPDRVPVIVEKVSGSQIVDIDKRKYLVPSDITVAQFMWIIRKRIQLPSEKAIFLFVDKTVPQSR

# Predict multimer with higher accuracy
gget alphafold sequence1.fasta -mr 20 -r

# Python with visualization
gget.alphafold("MKWMFK...", plot=True, show_sidechains=True)

# Multimer prediction
gget.alphafold(["sequence1", "sequence2"], multimer_recycles=20)

gget elm - Eukaryotic Linear Motifs

Predict Eukaryotic Linear Motifs in protein sequences.

Setup Required:

gget setup elm

Parameters:

• sequence: Amino acid sequence or UniProt Acc
• -u/--uniprot: Indicates sequence is UniProt Acc
• -e/--expand: Include protein names, organisms, references
• -s/--sensitivity: DIAMOND alignment sensitivity (default: "very-sensitive")
• -t/--threads: Number of threads (default: 1)

Returns: Two outputs:

1. ortholog_df: Linear motifs from orthologous proteins
2. regex_df: Motifs directly matched in input sequence

Examples:

# Predict motifs from sequence
gget elm LIAQSIGQASFV -o results

# Use UniProt accession with expanded info
gget elm --uniprot Q02410 -e

# Python
ortholog_df, regex_df = gget.elm("LIAQSIGQASFV")

4. Expression & Disease Data

gget archs4 - Gene Correlation & Tissue Expression

Query ARCHS4 database for correlated genes or tissue expression data.

Parameters:

• gene: Gene symbol or Ensembl ID (with --ensembl flag)
• -w/--which: 'correlation' (default, returns 100 most correlated genes) or 'tissue' (expression atlas)
• -s/--species: 'human' (default) or 'mouse' (tissue data only)
• -e/--ensembl: Input is Ensembl ID

Returns:

• Correlation mode: Gene symbols, Pearson correlation coefficients
• Tissue mode: Tissue identifiers, min/Q1/median/Q3/max expression values

Examples:

# Get correlated genes
gget archs4 ACE2

# Get tissue expression
gget archs4 -w tissue ACE2

# Python
gget.archs4("ACE2", which="tissue")

gget cellxgene - Single-Cell RNA-seq Data

Query CZ CELLxGENE Discover Census for single-cell data.

Setup Required:

gget setup cellxgene

Parameters:

• --gene (-g): Gene names or Ensembl IDs (case-sensitive! 'PAX7' for human, 'Pax7' for mouse)
• --tissue: Tissue type(s)
• --cell_type: Specific cell type(s)
• --species (-s): 'homo_sapiens' (default) or 'mus_musculus'
• --census_version (-cv): Version ("stable", "latest", or dated)
• --ensembl (-e): Use Ensembl IDs
• --meta_only (-mo): Return metadata only
• Additional filters: disease, development_stage, sex, assay, dataset_id, donor_id, ethnicity, suspension_type

Returns: AnnData object with count matrices and metadata (or metadata-only dataframes)

Examples:

# Get single-cell data for specific genes and cell types
gget cellxgene --gene ACE2 ABCA1 --tissue lung --cell_type "mucus secreting cell" -o lung_data.h5ad

# Metadata only
gget cellxgene --gene PAX7 --tissue muscle --meta_only -o metadata.csv

# Python
adata = gget.cellxgene(gene=["ACE2", "ABCA1"], tissue="lung", cell_type="mucus secreting cell")

gget enrichr - Enrichment Analysis

Perform ontology enrichment analysis on gene lists using Enrichr.

Parameters:

• genes: Gene symbols or Ensembl IDs
• -db/--database: Reference database (supports shortcuts: 'pathway', 'transcription', 'ontology', 'diseases_drugs', 'celltypes')
• -s/--species: human (default), mouse, fly, yeast, worm, fish
• -bkg_l/--background_list: Background genes for comparison
• -ko/--kegg_out: Save KEGG pathway images with highlighted genes
• plot: Python-only; generate graphical results

Database Shortcuts:

• 'pathway' → KEGG_2021_Human
• 'transcription' → ChEA_2016
• 'ontology' → GO_Biological_Process_2021
• 'diseases_drugs' → GWAS_Catalog_2019
• 'celltypes' → PanglaoDB_Augmented_2021

Examples:

# Enrichment analysis for ontology
gget enrichr -db ontology ACE2 AGT AGTR1

# Save KEGG pathways
gget enrichr -db pathway ACE2 AGT AGTR1 -ko ./kegg_images/

# Python with plot
gget.enrichr(["ACE2", "AGT", "AGTR1"], database="ontology", plot=True)

gget bgee - Orthology & Expression

Retrieve orthology and gene expression data from Bgee database.

Parameters:

• ens_id: Ensembl gene ID or NCBI gene ID (for non-Ensembl species). Multiple IDs supported when type=expression
• -t/--type: 'orthologs' (default) or 'expression'

Returns:

• Orthologs mode: Matching genes across species with IDs, names, taxonomic info
• Expression mode: Anatomical entities, confidence scores, expression status

Examples:

# Get orthologs
gget bgee ENSG00000169194

# Get expression data
gget bgee ENSG00000169194 -t expression

# Multiple genes
gget bgee ENSBTAG00000047356 ENSBTAG00000018317 -t expression

# Python
gget.bgee("ENSG00000169194", type="orthologs")

gget opentargets - Disease & Drug Associations

Retrieve disease and drug associations from OpenTargets.

Parameters:

• Ensembl gene ID (required)
• -r/--resource: diseases (default), drugs, tractability, pharmacogenetics, expression, depmap, interactions
• -l/--limit: Cap results count
• Filter arguments (vary by resource):
  • drugs: --filter_disease
  • pharmacogenetics: --filter_drug
  • expression/depmap: --filter_tissue, --filter_anat_sys, --filter_organ
  • interactions: --filter_protein_a, --filter_protein_b, --filter_gene_b

Examples:

# Get associated diseases
gget opentargets ENSG00000169194 -r diseases -l 5

# Get associated drugs
gget opentargets ENSG00000169194 -r drugs -l 10

# Get tissue expression
gget opentargets ENSG00000169194 -r expression --filter_tissue brain

# Python
gget.opentargets("ENSG00000169194", resource="diseases", limit=5)

gget cbio - cBioPortal Cancer Genomics

Plot cancer genomics heatmaps using cBioPortal data.

Two subcommands:

search - Find study IDs:

gget cbio search breast lung

plot - Generate heatmaps:

Parameters:

• -s/--study_ids: Space-separated cBioPortal study IDs (required)
• -g/--genes: Space-separated gene names or Ensembl IDs (required)
• -st/--stratification: Column to organize data (tissue, cancer_type, cancer_type_detailed, study_id, sample)
• -vt/--variation_type: Data type (mutation_occurrences, cna_nonbinary, sv_occurrences, cna_occurrences, Consequence)
• -f/--filter: Filter by column value (e.g., 'study_id:msk_impact_2017')
• -dd/--data_dir: Cache directory (default: ./gget_cbio_cache)
• -fd/--figure_dir: Output directory (default: ./gget_cbio_figures)
• -dpi: Resolution (default: 100)
• -sh/--show: Display plot in window
• -nc/--no_confirm: Skip download confirmations

Examples:

# Search for studies
gget cbio search esophag ovary

# Create heatmap
gget cbio plot -s msk_impact_2017 -g AKT1 ALK BRAF -st tissue -vt mutation_occurrences

# Python
gget.cbio_search(["esophag", "ovary"])
gget.cbio_plot(["msk_impact_2017"], ["AKT1", "ALK"], stratification="tissue")

gget cosmic - COSMIC Database

Search COSMIC (Catalogue Of Somatic Mutations In Cancer) database.

Important: License fees apply for commercial use. Requires COSMIC account credentials.

Parameters:

• searchterm: Gene name, Ensembl ID, mutation notation, or sample ID
• -ctp/--cosmic_tsv_path: Path to downloaded COSMIC TSV file (required for querying)
• -l/--limit: Maximum results (default: 100)

Database download flags:

• -d/--download_cosmic: Activate download mode
• -gm/--gget_mutate: Create version for gget mutate
• -cp/--cosmic_project: Database type (cancer, census, cell_line, resistance, genome_screen, targeted_screen)
• -cv/--cosmic_version: COSMIC version
• -gv/--grch_version: Human reference genome (37 or 38)
• --email, --password: COSMIC credentials

Examples:

# First download database
gget cosmic -d --email user@example.com --password xxx -cp cancer

# Then query
gget cosmic EGFR -ctp cosmic_data.tsv -l 10

# Python
gget.cosmic("EGFR", cosmic_tsv_path="cosmic_data.tsv", limit=10)

5. Additional Tools

gget mutate - Generate Mutated Sequences

Generate mutated nucleotide sequences from mutation annotations.

Parameters:

• sequences: FASTA file path or direct sequence input (string/list)
• -m/--mutations: CSV/TSV file or DataFrame with mutation data (required)
• -mc/--mut_column: Mutation column name (default: 'mutation')
• -sic/--seq_id_column: Sequence ID column (default: 'seq_ID')
• -mic/--mut_id_column: Mutation ID column
• -k/--k: Length of flanking sequences (default: 30 nucleotides)

Returns: Mutated sequences in FASTA format

Examples:

# Single mutation
gget mutate ATCGCTAAGCT -m "c.4G>T"

# Multiple sequences with mutations from file
gget mutate sequences.fasta -m mutations.csv -o mutated.fasta

# Python
import pandas as pd
mutations_df = pd.DataFrame({"seq_ID": ["seq1"], "mutation": ["c.4G>T"]})
gget.mutate(["ATCGCTAAGCT"], mutations=mutations_df)

gget gpt - OpenAI Text Generation

Generate natural language text using OpenAI's API.

Setup Required:

gget setup gpt

Important: Free tier limited to 3 months after account creation. Set monthly billing limits.

Parameters:

• prompt: Text input for generation (required)
• api_key: OpenAI authentication (required)
• Model configuration: temperature, top_p, max_tokens, frequency_penalty, presence_penalty
• Default model: gpt-3.5-turbo (configurable)

Examples:

gget gpt "Explain CRISPR" --api_key your_key_here

# Python
gget.gpt("Explain CRISPR", api_key="your_key_here")

gget setup - Install Dependencies

Install/download third-party dependencies for specific modules.

Parameters:

• module: Module name requiring dependency installation
• -o/--out: Output folder path (elm module only)

Modules requiring setup:

• alphafold - Downloads ~4GB of model parameters
• cellxgene - Installs cellxgene-census (may not support latest Python)
• elm - Downloads local ELM database
• gpt - Configures OpenAI integration

Examples:

# Setup AlphaFold
gget setup alphafold

# Setup ELM with custom directory
gget setup elm -o /path/to/elm_data

# Python
gget.setup("alphafold")

Common Workflows

Workflow 1: Gene Discovery to Sequence Analysis

Find and analyze genes of interest:

# 1. Search for genes
results = gget.search(["GABA", "receptor"], species="homo_sapiens")

# 2. Get detailed information
gene_ids = results["ensembl_id"].tolist()
info = gget.info(gene_ids[:5])

# 3. Retrieve sequences
sequences = gget.seq(gene_ids[:5], translate=True)

Workflow 2: Sequence Alignment and Structure

Align sequences and predict structures:

# 1. Align multiple sequences
alignment = gget.muscle("sequences.fasta")

# 2. Find similar sequences
blast_results = gget.blast(my_sequence, database="swissprot", limit=10)

# 3. Predict structure
structure = gget.alphafold(my_sequence, plot=True)

# 4. Find linear motifs
ortholog_df, regex_df = gget.elm(my_sequence)

Workflow 3: Gene Expression and Enrichment

Analyze expression patterns and functional enrichment:

# 1. Get tissue expression
tissue_expr = gget.archs4("ACE2", which="tissue")

# 2. Find correlated genes
correlated = gget.archs4("ACE2", which="correlation")

# 3. Get single-cell data
adata = gget.cellxgene(gene=["ACE2"], tissue="lung", cell_type="epithelial cell")

# 4. Perform enrichment analysis
gene_list = correlated["gene_symbol"].tolist()[:50]
enrichment = gget.enrichr(gene_list, database="ontology", plot=True)

Workflow 4: Disease and Drug Analysis

Investigate disease associations and therapeutic targets:

# 1. Search for genes
genes = gget.search(["breast cancer"], species="homo_sapiens")

# 2. Get disease associations
diseases = gget.opentargets("ENSG00000169194", resource="diseases")

# 3. Get drug associations
drugs = gget.opentargets("ENSG00000169194", resource="drugs")

# 4. Query cancer genomics data
study_ids = gget.cbio_search(["breast"])
gget.cbio_plot(study_ids[:2], ["BRCA1", "BRCA2"], stratification="cancer_type")

# 5. Search COSMIC for mutations
cosmic_results = gget.cosmic("BRCA1", cosmic_tsv_path="cosmic.tsv")

Workflow 5: Comparative Genomics

Compare proteins across species:

# 1. Get orthologs
orthologs = gget.bgee("ENSG00000169194", type="orthologs")

# 2. Get sequences for comparison
human_seq = gget.seq("ENSG00000169194", translate=True)
mouse_seq = gget.seq("ENSMUSG00000026091", translate=True)

# 3. Align sequences
alignment = gget.muscle([human_seq, mouse_seq])

# 4. Compare structures
human_structure = gget.pdb("7S7U")
mouse_structure = gget.alphafold(mouse_seq)

Workflow 6: Building Reference Indices

Prepare reference data for downstream analysis (e.g., kallisto|bustools):

# 1. List available species
gget ref --list_species

# 2. Download reference files
gget ref -w gtf -w cdna -d homo_sapiens

# 3. Build kallisto index
kallisto index -i transcriptome.idx transcriptome.fasta

# 4. Download genome for alignment
gget ref -w dna -d homo_sapiens

Best Practices

Data Retrieval

• Use --limit to control result sizes for large queries
• Save results with -o/--out for reproducibility
• Check database versions/releases for consistency across analyses
• Use --quiet in production scripts to reduce output

Sequence Analysis

• For BLAST/BLAT, start with default parameters, then adjust sensitivity
• Use gget diamond with --threads for faster local alignment
• Save DIAMOND databases with --diamond_db for repeated queries
• For multiple sequence alignment, use -s5/--super5 for large datasets

Expression and Disease Data

• Gene symbols are case-sensitive in cellxgene (e.g., 'PAX7' vs 'Pax7')
• Run gget setup before first use of alphafold, cellxgene, elm, gpt
• For enrichment analysis, use database shortcuts for convenience
• Cache cBioPortal data with -dd to avoid repeated downloads

Structure Prediction

• AlphaFold multimer predictions: use -mr 20 for higher accuracy
• Use -r flag for AMBER relaxation of final structures
• Visualize results in Python with plot=True
• Check PDB database first before running AlphaFold predictions

Error Handling

• Database structures change; update gget regularly: uv pip install --upgrade gget
• Process max ~1000 Ensembl IDs at once with gget info
• For large-scale analyses, implement rate limiting for API queries
• Use virtual environments to avoid dependency conflicts

Output Formats

Command-line

• Default: JSON
• CSV: Add -csv flag
• FASTA: gget seq, gget mutate
• PDB: gget pdb, gget alphafold
• PNG: gget cbio plot

Python

• Default: DataFrame or dictionary
• JSON: Add json=True parameter
• Save to file: Add save=True or specify out="filename"
• AnnData: gget cellxgene

Resources

This skill includes reference documentation for detailed module information:

references/

• module_reference.md - Comprehensive parameter reference for all modules
• database_info.md - Information about queried databases and their update frequencies
• workflows.md - Extended workflow examples and use cases

For additional help:

• Official documentation: https://pachterlab.github.io/gget/
• GitHub issues: https://github.com/pachterlab/gget/issues
• Citation: Luebbert, L. & Pachter, L. (2023). Efficient querying of genomic reference databases with gget. Bioinformatics. https://doi.org/10.1093/bioinformatics/btac836

Glycobiology: Glycosylation Analysis

Overview

Glycobiology provides lightweight computational tools for predicting and analyzing glycosylation sites in protein sequences. This skill covers N-glycosylation sequon motif finding (N-X-S/T where X is not P), O-glycosylation hotspot prediction using a sliding window serine/threonine density heuristic, glycan structure tool references, and combined glycoprotein analysis with domain mapping. All analyses use pure Python with minimal dependencies (Biopython for sequence I/O, regex for pattern matching).

When to Use This Skill

• Predicting N-glycosylation sites from protein sequences
• Identifying O-glycosylation hotspot regions
• Planning glycosylation site mutagenesis experiments
• Comparing predicted vs experimentally determined glycosylation sites
• Annotating glycosylation in biotherapeutic protein design
• Surveying glycan analysis tools for downstream structural studies

Related Skills: For protein function and existing glycosylation annotations use uniprot-database. For protein 3D structure and site accessibility use alphafold-database. For sequence manipulation use biopython.

Installation

uv pip install biopython numpy

No additional dependencies required — this skill uses pure Python.

Quick Start

import re

def find_n_glycosylation_sites(sequence):
    """Find N-X-S/T sequons where X != P."""
    sites = []
    for i in range(len(sequence) - 2):
        if sequence[i] == 'N' and sequence[i+1] != 'P' and sequence[i+2] in ('S', 'T'):
            sites.append({
                'position': i + 1,  # 1-based
                'motif': sequence[i:i+3],
                'context': sequence[max(0,i-3):i+6]
            })
    return sites

# Example: human EPO
epo_seq = "MGVHECPAWLWLLLSLLSLPLGLPVLGAPPRLICDSRVLERYLLEAKEAENITTGCAEHCSLNENITVPDTKVNFYAWKRMEVGQQAVEVWQGLALLSEAVLRGQALLVNSSQPWEPLQLHVDKAVSGLRSLTTLLRALGAQKEAISPPDAASAAPLRTITADTFRKLFRVYSNFLRGKLKLYTGEACRTGDR"

sites = find_n_glycosylation_sites(epo_seq)
print(f"N-glycosylation sites: {len(sites)}")
for s in sites:
    print(f"  Position {s['position']}: {s['motif']} (context: ...{s['context']}...)")

Core Capabilities

1. N-Glycosylation Motif Finding

Identify N-X-S/T sequons in protein sequences.

import re
from Bio import SeqIO

def find_n_glycosylation_sites(sequence, exclude_proline=True):
    """Find N-linked glycosylation sequons (N-X-S/T).

    Args:
        sequence: protein sequence string
        exclude_proline: if True, exclude N-P-S/T motifs (standard rule)
    """
    sites = []
    seq = str(sequence).upper()

    for i in range(len(seq) - 2):
        if seq[i] != 'N':
            continue

        x_residue = seq[i + 1]
        st_residue = seq[i + 2]

        # Standard rule: X != P
        if exclude_proline and x_residue == 'P':
            continue

        if st_residue not in ('S', 'T'):
            continue

        # Context (5 residues each side)
        context_start = max(0, i - 5)
        context_end = min(len(seq), i + 8)
        context = seq[context_start:context_end]

        sites.append({
            'position': i + 1,  # 1-based position
            'residue': seq[i],
            'motif': seq[i:i+3],
            'x_residue': x_residue,
            'acceptor': st_residue,
            'context': context,
            'context_start': context_start + 1
        })

    return sites

def scan_fasta_for_n_glyc(fasta_path):
    """Scan all sequences in a FASTA file for N-glycosylation sites."""
    results = []
    for record in SeqIO.parse(fasta_path, 'fasta'):
        sites = find_n_glycosylation_sites(str(record.seq))
        results.append({
            'id': record.id,
            'description': record.description,
            'length': len(record.seq),
            'n_sites': len(sites),
            'sites': sites,
            'density': len(sites) / len(record.seq) * 1000  # per 1000 residues
        })

    import pandas as pd
    df = pd.DataFrame([{k: v for k, v in r.items() if k != 'sites'} for r in results])
    print(f"Scanned {len(df)} sequences")
    print(f"Total N-glyc sites: {df['n_sites'].sum()}")
    print(f"Mean density: {df['density'].mean():.1f} per 1000 residues")
    return results

# Example with multiple sequences
sites = find_n_glycosylation_sites("MANLQTSPNGTLKNVTSITDA")
for s in sites:
    print(f"N{s['position']}: {s['motif']} (X={s['x_residue']}, acceptor={s['acceptor']})")

2. O-Glycosylation Hotspot Prediction

Identify regions enriched in serine/threonine that may be O-glycosylated.

import numpy as np

def predict_o_glycosylation_hotspots(sequence, window_size=20, threshold=0.4,
                                       exclude_near_proline=True):
    """Predict O-glycosylation hotspots using S/T density heuristic.

    O-glycosylation preferentially occurs in S/T-rich regions, often near
    proline residues. This heuristic identifies candidate regions.

    Args:
        sequence: protein sequence string
        window_size: sliding window size
        threshold: minimum S/T fraction to call a hotspot
        exclude_near_proline: if True, downweight S/T not near P
    """
    seq = str(sequence).upper()
    n = len(seq)

    if n < window_size:
        return []

    # Calculate S/T density in sliding windows
    densities = []
    for i in range(n - window_size + 1):
        window = seq[i:i + window_size]
        st_count = window.count('S') + window.count('T')
        density = st_count / window_size
        densities.append(density)

    densities = np.array(densities)

    # Find hotspot regions
    hotspots = []
    in_hotspot = False
    start = 0

    for i, d in enumerate(densities):
        if d >= threshold and not in_hotspot:
            start = i
            in_hotspot = True
        elif d < threshold and in_hotspot:
            hotspots.append({
                'start': start + 1,  # 1-based
                'end': i + window_size,
                'length': i + window_size - start,
                'max_density': float(densities[start:i].max()),
                'mean_density': float(densities[start:i].mean()),
                'region': seq[start:i + window_size]
            })
            in_hotspot = False

    if in_hotspot:
        hotspots.append({
            'start': start + 1,
            'end': n,
            'length': n - start,
            'max_density': float(densities[start:].max()),
            'mean_density': float(densities[start:].mean()),
            'region': seq[start:]
        })

    # Individual S/T residues in hotspots
    all_st_sites = []
    for hs in hotspots:
        region_start = hs['start'] - 1
        region_seq = hs['region']
        for j, aa in enumerate(region_seq):
            if aa in ('S', 'T'):
                pos = region_start + j + 1
                # Check proline neighbors
                has_nearby_P = False
                for k in range(-3, 4):
                    check_pos = region_start + j + k
                    if 0 <= check_pos < n and seq[check_pos] == 'P':
                        has_nearby_P = True
                        break

                all_st_sites.append({
                    'position': pos,
                    'residue': aa,
                    'in_hotspot': True,
                    'near_proline': has_nearby_P
                })

    print(f"O-glycosylation hotspots: {len(hotspots)}")
    for hs in hotspots:
        print(f"  Pos {hs['start']}-{hs['end']}: density={hs['max_density']:.2f}, "
              f"{hs['region'][:30]}{'...' if len(hs['region'])>30 else ''}")
    print(f"Candidate S/T residues: {len(all_st_sites)}")

    return hotspots, all_st_sites

3. Glycan Structure Resources

Curated list of tools for glycan structure analysis and prediction.

GLYCAN_TOOLS = {
    'NetNGlyc': {
        'url': 'https://services.healthtech.dtu.dk/services/NetNGlyc-1.0/',
        'description': 'Neural network prediction of N-glycosylation sites',
        'input': 'Protein sequence (FASTA)',
        'output': 'N-glycosylation probability per Asn',
        'note': 'More accurate than motif-only scanning; considers sequence context'
    },
    'NetOGlyc': {
        'url': 'https://services.healthtech.dtu.dk/services/NetOGlyc-4.0/',
        'description': 'Neural network prediction of mucin-type O-glycosylation',
        'input': 'Protein sequence (FASTA)',
        'output': 'O-glycosylation probability per Ser/Thr',
        'note': 'Version 4.0 trained on O-GalNAc glycoproteomics data'
    },
    'GlycoWorkbench': {
        'url': 'https://code.google.com/archive/p/glycoworkbench/',
        'description': 'Glycan structure drawing and MS annotation',
        'input': 'Mass spectrometry data',
        'output': 'Glycan structure assignments'
    },
    'GLYCAM-Web': {
        'url': 'https://glycam.org/',
        'description': 'Glycan 3D structure building and MD simulation',
        'input': 'Glycan sequence (IUPAC condensed)',
        'output': '3D coordinates (PDB), AMBER parameters'
    },
    'Glycoshield-MD': {
        'url': 'https://github.com/GlycoSHIELD-MD/GlycoSHIELD-MD',
        'description': 'Model glycan shields on protein structures for MD',
        'input': 'Protein PDB + glycosylation sites',
        'output': 'Glycosylated protein structure'
    },
    'SweetTalk': {
        'url': 'https://glycoproteome.expasy.org/sweettalk/',
        'description': 'Glycoprotein structure visualization',
        'input': 'UniProt ID',
        'output': 'Interactive glycoprotein 3D viewer'
    }
}

def list_glycan_tools():
    """Print available glycan analysis tools."""
    for name, info in GLYCAN_TOOLS.items():
        print(f"\n{name}")
        print(f"  URL: {info['url']}")
        print(f"  Description: {info['description']}")
        if 'note' in info:
            print(f"  Note: {info['note']}")

4. Glycoprotein Analysis

Combined N/O glycosylation analysis with domain mapping.

def analyze_glycoprotein(sequence, protein_name='protein',
                          uniprot_sites=None):
    """Comprehensive glycosylation analysis.

    Args:
        sequence: protein sequence
        protein_name: name for reporting
        uniprot_sites: dict of known sites from UniProt
                       {'N_glyc': [positions], 'O_glyc': [positions]}
    """
    seq = str(sequence).upper()
    print(f"=== Glycoprotein Analysis: {protein_name} ===")
    print(f"Length: {len(seq)} residues")

    # N-glycosylation
    n_sites = find_n_glycosylation_sites(seq)
    print(f"\nN-glycosylation sites (sequon motif): {len(n_sites)}")
    for s in n_sites:
        print(f"  N{s['position']}: {s['motif']}")

    # O-glycosylation hotspots
    o_hotspots, o_sites = predict_o_glycosylation_hotspots(seq)

    # Comparison with UniProt annotations
    if uniprot_sites:
        known_n = set(uniprot_sites.get('N_glyc', []))
        predicted_n = set(s['position'] for s in n_sites)

        tp = known_n & predicted_n
        fp = predicted_n - known_n
        fn = known_n - predicted_n

        print(f"\nComparison with UniProt annotations:")
        print(f"  True positives (predicted & annotated): {len(tp)}")
        print(f"  False positives (predicted, not annotated): {len(fp)}")
        print(f"  False negatives (annotated, not predicted): {len(fn)}")

        if fn:
            print(f"  Missed sites: {sorted(fn)}")
            for pos in fn:
                context = seq[max(0,pos-4):pos+5]
                print(f"    N{pos}: ...{context}... (may have N-P-S/T or non-standard motif)")

    # Summary
    print(f"\nSummary:")
    print(f"  N-glyc sites: {len(n_sites)}")
    print(f"  O-glyc hotspots: {len(o_hotspots)}")
    print(f"  Total candidate O-glyc S/T: {len(o_sites)}")
    print(f"  Glycosylation density: {(len(n_sites) + len(o_sites)) / len(seq) * 100:.1f}%")

    return {
        'n_glyc_sites': n_sites,
        'o_glyc_hotspots': o_hotspots,
        'o_glyc_sites': o_sites
    }

Typical Workflows

Workflow 1: Predict All Glycosylation Sites in a Protein

sequence = "MANLQTSPNGTLKNVTSITDANLTQSPNNFTLRNITSANDTVTQSTPNVTQLNVSQSTPNVTQLNVSQST"
results = analyze_glycoprotein(sequence, protein_name='MyProtein')

Workflow 2: Compare Predicted vs UniProt-Annotated Sites

sequence = "MGVHECPAWLWLLLSLLSLPLGLPVLGAPPRLICDSRVLERYLLEAKEAENITTGCAEHCSLNENITVPDT"
known = {'N_glyc': [24, 38, 83], 'O_glyc': []}  # Known from UniProt
results = analyze_glycoprotein(sequence, protein_name='EPO', uniprot_sites=known)

Best Practices

1. N-glycosylation prediction — sequon motif (N-X-S/T, X!=P) is necessary but not sufficient; not all sequons are glycosylated; use NetNGlyc for probability-based prediction
2. O-glycosylation — much harder to predict than N-glyc; the S/T density heuristic identifies candidate regions but has high false positive rate; use NetOGlyc for better accuracy
3. Validate with UniProt — always cross-reference predictions with experimentally verified glycosylation annotations in UniProt
4. Signal peptide — glycosylation occurs in the ER/Golgi; ensure the protein has a signal peptide (positions are relative to mature protein in databases)
5. Structural context — glycosylation requires surface accessibility; use AlphaFold to check if predicted sites are surface-exposed
6. Biotherapeutics — for antibody/protein drug design, N-glycosylation in Fc regions (N297 in IgG) is critical for effector function

Troubleshooting

Problem: Sequon search finds sites in cytoplasmic proteins Solution: N-glycosylation only occurs in secretory pathway proteins. Filter predictions by subcellular localization (check UniProt).

Problem: Known glycosylation site not detected Solution: Some rare N-glycosylation occurs at non-canonical motifs (N-X-C). Check if the site has N-P-S/T which is excluded by default.

Problem: Too many O-glycosylation hotspots Solution: Increase the density threshold (e.g., 0.5). Focus on regions that are both S/T-rich and in extracellular domains.

Resources

• UniProt Glycosylation Annotations
• NetNGlyc Server
• NetOGlyc Server
• GlyConnect Database
• Essentials of Glycobiology (NCBI)

Histolab

Overview

Histolab is a Python library for processing whole slide images (WSI) in digital pathology. It automates tissue detection, extracts informative tiles from gigapixel images, and prepares datasets for deep learning pipelines. The library handles multiple WSI formats, implements sophisticated tissue segmentation, and provides flexible tile extraction strategies.

Installation

uv pip install histolab

Quick Start

Basic workflow for extracting tiles from a whole slide image:

from histolab.slide import Slide
from histolab.tiler import RandomTiler

# Load slide
slide = Slide("slide.svs", processed_path="output/")

# Configure tiler
tiler = RandomTiler(
    tile_size=(512, 512),
    n_tiles=100,
    level=0,
    seed=42
)

# Preview tile locations
tiler.locate_tiles(slide, n_tiles=20)

# Extract tiles
tiler.extract(slide)

Core Capabilities

1. Slide Management

Load, inspect, and work with whole slide images in various formats.

Common operations:

• Loading WSI files (SVS, TIFF, NDPI, etc.)
• Accessing slide metadata (dimensions, magnification, properties)
• Generating thumbnails for visualization
• Working with pyramidal image structures
• Extracting regions at specific coordinates

Key classes: Slide

Reference: references/slide_management.md contains comprehensive documentation on:

• Slide initialization and configuration
• Built-in sample datasets (prostate, ovarian, breast, heart, kidney tissues)
• Accessing slide properties and metadata
• Thumbnail generation and visualization
• Working with pyramid levels
• Multi-slide processing workflows

Example workflow:

from histolab.slide import Slide
from histolab.data import prostate_tissue

# Load sample data
prostate_svs, prostate_path = prostate_tissue()

# Initialize slide
slide = Slide(prostate_path, processed_path="output/")

# Inspect properties
print(f"Dimensions: {slide.dimensions}")
print(f"Levels: {slide.levels}")
print(f"Magnification: {slide.properties.get('openslide.objective-power')}")

# Save thumbnail
slide.save_thumbnail()

2. Tissue Detection and Masks

Automatically identify tissue regions and filter background/artifacts.

Common operations:

• Creating binary tissue masks
• Detecting largest tissue region
• Excluding background and artifacts
• Custom tissue segmentation
• Removing pen annotations

Key classes: TissueMask, BiggestTissueBoxMask, BinaryMask

Reference: references/tissue_masks.md contains comprehensive documentation on:

• TissueMask: Segments all tissue regions using automated filters
• BiggestTissueBoxMask: Returns bounding box of largest tissue region (default)
• BinaryMask: Base class for custom mask implementations
• Visualizing masks with locate_mask()
• Creating custom rectangular and annotation-exclusion masks
• Mask integration with tile extraction
• Best practices and troubleshooting

Example workflow:

from histolab.masks import TissueMask, BiggestTissueBoxMask

# Create tissue mask for all tissue regions
tissue_mask = TissueMask()

# Visualize mask on slide
slide.locate_mask(tissue_mask)

# Get mask array
mask_array = tissue_mask(slide)

# Use largest tissue region (default for most extractors)
biggest_mask = BiggestTissueBoxMask()

When to use each mask:

• TissueMask: Multiple tissue sections, comprehensive analysis
• BiggestTissueBoxMask: Single main tissue section, exclude artifacts (default)
• Custom BinaryMask: Specific ROI, exclude annotations, custom segmentation

3. Tile Extraction

Extract smaller regions from large WSI using different strategies.

Three extraction strategies:

RandomTiler: Extract fixed number of randomly positioned tiles

• Best for: Sampling diverse regions, exploratory analysis, training data
• Key parameters: n_tiles, seed for reproducibility

GridTiler: Systematically extract tiles across tissue in grid pattern

• Best for: Complete coverage, spatial analysis, reconstruction
• Key parameters: pixel_overlap for sliding windows

ScoreTiler: Extract top-ranked tiles based on scoring functions

• Best for: Most informative regions, quality-driven selection
• Key parameters: scorer (NucleiScorer, CellularityScorer, custom)

Common parameters:

• tile_size: Tile dimensions (e.g., (512, 512))
• level: Pyramid level for extraction (0 = highest resolution)
• check_tissue: Filter tiles by tissue content
• tissue_percent: Minimum tissue coverage (default 80%)
• extraction_mask: Mask defining extraction region

Reference: references/tile_extraction.md contains comprehensive documentation on:

• Detailed explanation of each tiler strategy
• Available scorers (NucleiScorer, CellularityScorer, custom)
• Tile preview with locate_tiles()
• Extraction workflows and reporting
• Advanced patterns (multi-level, hierarchical extraction)
• Performance optimization and troubleshooting

Example workflows:

from histolab.tiler import RandomTiler, GridTiler, ScoreTiler
from histolab.scorer import NucleiScorer

# Random sampling (fast, diverse)
random_tiler = RandomTiler(
    tile_size=(512, 512),
    n_tiles=100,
    level=0,
    seed=42,
    check_tissue=True,
    tissue_percent=80.0
)
random_tiler.extract(slide)

# Grid coverage (comprehensive)
grid_tiler = GridTiler(
    tile_size=(512, 512),
    level=0,
    pixel_overlap=0,
    check_tissue=True
)
grid_tiler.extract(slide)

# Score-based selection (most informative)
score_tiler = ScoreTiler(
    tile_size=(512, 512),
    n_tiles=50,
    scorer=NucleiScorer(),
    level=0
)
score_tiler.extract(slide, report_path="tiles_report.csv")

Always preview before extracting:

# Preview tile locations on thumbnail
tiler.locate_tiles(slide, n_tiles=20)

4. Filters and Preprocessing

Apply image processing filters for tissue detection, quality control, and preprocessing.

Filter categories:

Image Filters: Color space conversions, thresholding, contrast enhancement

• RgbToGrayscale, RgbToHsv, RgbToHed
• OtsuThreshold, AdaptiveThreshold
• StretchContrast, HistogramEqualization

Morphological Filters: Structural operations on binary images

• BinaryDilation, BinaryErosion
• BinaryOpening, BinaryClosing
• RemoveSmallObjects, RemoveSmallHoles

Composition: Chain multiple filters together

• Compose: Create filter pipelines

Reference: references/filters_preprocessing.md contains comprehensive documentation on:

• Detailed explanation of each filter type
• Filter composition and chaining
• Common preprocessing pipelines (tissue detection, pen removal, nuclei enhancement)
• Applying filters to tiles
• Custom mask filters
• Quality control filters (blur detection, tissue coverage)
• Best practices and troubleshooting

Example workflows:

from histolab.filters.compositions import Compose
from histolab.filters.image_filters import RgbToGrayscale, OtsuThreshold
from histolab.filters.morphological_filters import (
    BinaryDilation, RemoveSmallHoles, RemoveSmallObjects
)

# Standard tissue detection pipeline
tissue_detection = Compose([
    RgbToGrayscale(),
    OtsuThreshold(),
    BinaryDilation(disk_size=5),
    RemoveSmallHoles(area_threshold=1000),
    RemoveSmallObjects(area_threshold=500)
])

# Use with custom mask
from histolab.masks import TissueMask
custom_mask = TissueMask(filters=tissue_detection)

# Apply filters to tile
from histolab.tile import Tile
filtered_tile = tile.apply_filters(tissue_detection)

5. Visualization

Visualize slides, masks, tile locations, and extraction quality.

Common visualization tasks:

• Displaying slide thumbnails
• Visualizing tissue masks
• Previewing tile locations
• Assessing tile quality
• Creating reports and figures

Reference: references/visualization.md contains comprehensive documentation on:

• Slide thumbnail display and saving
• Mask visualization with locate_mask()
• Tile location preview with locate_tiles()
• Displaying extracted tiles and mosaics
• Quality assessment (score distributions, top vs bottom tiles)
• Multi-slide visualization
• Filter effect visualization
• Exporting high-resolution figures and PDF reports
• Interactive visualization in Jupyter notebooks

Example workflows:

import matplotlib.pyplot as plt
from histolab.masks import TissueMask

# Display slide thumbnail
plt.figure(figsize=(10, 10))
plt.imshow(slide.thumbnail)
plt.title(f"Slide: {slide.name}")
plt.axis('off')
plt.show()

# Visualize tissue mask
tissue_mask = TissueMask()
slide.locate_mask(tissue_mask)

# Preview tile locations
tiler = RandomTiler(tile_size=(512, 512), n_tiles=50)
tiler.locate_tiles(slide, n_tiles=20)

# Display extracted tiles in grid
from pathlib import Path
from PIL import Image

tile_paths = list(Path("output/tiles/").glob("*.png"))[:16]
fig, axes = plt.subplots(4, 4, figsize=(12, 12))
axes = axes.ravel()

for idx, tile_path in enumerate(tile_paths):
    tile_img = Image.open(tile_path)
    axes[idx].imshow(tile_img)
    axes[idx].set_title(tile_path.stem, fontsize=8)
    axes[idx].axis('off')

plt.tight_layout()
plt.show()

Typical Workflows

Workflow 1: Exploratory Tile Extraction

Quick sampling of diverse tissue regions for initial analysis.

from histolab.slide import Slide
from histolab.tiler import RandomTiler
import logging

# Enable logging for progress tracking
logging.basicConfig(level=logging.INFO)

# Load slide
slide = Slide("slide.svs", processed_path="output/random_tiles/")

# Inspect slide
print(f"Dimensions: {slide.dimensions}")
print(f"Levels: {slide.levels}")
slide.save_thumbnail()

# Configure random tiler
random_tiler = RandomTiler(
    tile_size=(512, 512),
    n_tiles=100,
    level=0,
    seed=42,
    check_tissue=True,
    tissue_percent=80.0
)

# Preview locations
random_tiler.locate_tiles(slide, n_tiles=20)

# Extract tiles
random_tiler.extract(slide)

Workflow 2: Comprehensive Grid Extraction

Complete tissue coverage for whole-slide analysis.

from histolab.slide import Slide
from histolab.tiler import GridTiler
from histolab.masks import TissueMask

# Load slide
slide = Slide("slide.svs", processed_path="output/grid_tiles/")

# Use TissueMask for all tissue sections
tissue_mask = TissueMask()
slide.locate_mask(tissue_mask)

# Configure grid tiler
grid_tiler = GridTiler(
    tile_size=(512, 512),
    level=1,  # Use level 1 for faster extraction
    pixel_overlap=0,
    check_tissue=True,
    tissue_percent=70.0
)

# Preview grid
grid_tiler.locate_tiles(slide)

# Extract all tiles
grid_tiler.extract(slide, extraction_mask=tissue_mask)

Workflow 3: Quality-Driven Tile Selection

Extract most informative tiles based on nuclei density.

from histolab.slide import Slide
from histolab.tiler import ScoreTiler
from histolab.scorer import NucleiScorer
import pandas as pd
import matplotlib.pyplot as plt

# Load slide
slide = Slide("slide.svs", processed_path="output/scored_tiles/")

# Configure score tiler
score_tiler = ScoreTiler(
    tile_size=(512, 512),
    n_tiles=50,
    level=0,
    scorer=NucleiScorer(),
    check_tissue=True
)

# Preview top tiles
score_tiler.locate_tiles(slide, n_tiles=15)

# Extract with report
score_tiler.extract(slide, report_path="tiles_report.csv")

# Analyze scores
report_df = pd.read_csv("tiles_report.csv")
plt.hist(report_df['score'], bins=20, edgecolor='black')
plt.xlabel('Tile Score')
plt.ylabel('Frequency')
plt.title('Distribution of Tile Scores')
plt.show()

Workflow 4: Multi-Slide Processing Pipeline

Process entire slide collection with consistent parameters.

from pathlib import Path
from histolab.slide import Slide
from histolab.tiler import RandomTiler
import logging

logging.basicConfig(level=logging.INFO)

# Configure tiler once
tiler = RandomTiler(
    tile_size=(512, 512),
    n_tiles=50,
    level=0,
    seed=42,
    check_tissue=True
)

# Process all slides
slide_dir = Path("slides/")
output_base = Path("output/")

for slide_path in slide_dir.glob("*.svs"):
    print(f"\nProcessing: {slide_path.name}")

    # Create slide-specific output directory
    output_dir = output_base / slide_path.stem
    output_dir.mkdir(parents=True, exist_ok=True)

    # Load and process slide
    slide = Slide(slide_path, processed_path=output_dir)

    # Save thumbnail for review
    slide.save_thumbnail()

    # Extract tiles
    tiler.extract(slide)

    print(f"Completed: {slide_path.name}")

Workflow 5: Custom Tissue Detection and Filtering

Handle slides with artifacts, annotations, or unusual staining.

from histolab.slide import Slide
from histolab.masks import TissueMask
from histolab.tiler import RandomTiler
from histolab.filters.compositions import Compose
from histolab.filters.image_filters import RgbToGrayscale, OtsuThreshold
from histolab.filters.morphological_filters import (
    BinaryDilation, RemoveSmallObjects, RemoveSmallHoles
)

# Define custom filter pipeline for aggressive artifact removal
aggressive_filters = Compose([
    RgbToGrayscale(),
    OtsuThreshold(),
    BinaryDilation(disk_size=10),
    RemoveSmallHoles(area_threshold=5000),
    RemoveSmallObjects(area_threshold=3000)  # Remove larger artifacts
])

# Create custom mask
custom_mask = TissueMask(filters=aggressive_filters)

# Load slide and visualize mask
slide = Slide("slide.svs", processed_path="output/")
slide.locate_mask(custom_mask)

# Extract with custom mask
tiler = RandomTiler(tile_size=(512, 512), n_tiles=100)
tiler.extract(slide, extraction_mask=custom_mask)

Best Practices

Slide Loading and Inspection

1. Always inspect slide properties before processing
2. Save thumbnails for quick visual review
3. Check pyramid levels and dimensions
4. Verify tissue is present using thumbnails

Tissue Detection

1. Preview masks with locate_mask() before extraction
2. Use TissueMask for multiple sections, BiggestTissueBoxMask for single sections
3. Customize filters for specific stains (H&E vs IHC)
4. Handle pen annotations with custom masks
5. Test masks on diverse slides

Tile Extraction

1. Always preview with locate_tiles() before extracting
2. Choose appropriate tiler:
   • RandomTiler: Sampling and exploration
   • GridTiler: Complete coverage
   • ScoreTiler: Quality-driven selection
3. Set appropriate tissue_percent threshold (70-90% typical)
4. Use seeds for reproducibility in RandomTiler
5. Extract at appropriate pyramid level for analysis resolution
6. Enable logging for large datasets

Performance

1. Extract at lower levels (1, 2) for faster processing
2. Use BiggestTissueBoxMask over TissueMask when appropriate
3. Adjust tissue_percent to reduce invalid tile attempts
4. Limit n_tiles for initial exploration
5. Use pixel_overlap=0 for non-overlapping grids

Quality Control

1. Validate tile quality (check for blur, artifacts, focus)
2. Review score distributions for ScoreTiler
3. Inspect top and bottom scoring tiles
4. Monitor tissue coverage statistics
5. Filter extracted tiles by additional quality metrics if needed

Common Use Cases

Training Deep Learning Models

• Extract balanced datasets using RandomTiler across multiple slides
• Use ScoreTiler with NucleiScorer to focus on cell-rich regions
• Extract at consistent resolution (level 0 or level 1)
• Generate CSV reports for tracking tile metadata

Whole Slide Analysis

• Use GridTiler for complete tissue coverage
• Extract at multiple pyramid levels for hierarchical analysis
• Maintain spatial relationships with grid positions
• Use pixel_overlap for sliding window approaches

Tissue Characterization

• Sample diverse regions with RandomTiler
• Quantify tissue coverage with masks
• Extract stain-specific information with HED decomposition
• Compare tissue patterns across slides

Quality Assessment

• Identify optimal focus regions with ScoreTiler
• Detect artifacts using custom masks and filters
• Assess staining quality across slide collection
• Flag problematic slides for manual review

Dataset Curation

• Use ScoreTiler to prioritize informative tiles
• Filter tiles by tissue percentage
• Generate reports with tile scores and metadata
• Create stratified datasets across slides and tissue types

Troubleshooting

No tiles extracted

• Lower tissue_percent threshold
• Verify slide contains tissue (check thumbnail)
• Ensure extraction_mask captures tissue regions
• Check tile_size is appropriate for slide resolution

Many background tiles

• Enable check_tissue=True
• Increase tissue_percent threshold
• Use appropriate mask (TissueMask vs BiggestTissueBoxMask)
• Customize mask filters to better detect tissue

Extraction very slow

• Extract at lower pyramid level (level=1 or 2)
• Reduce n_tiles for RandomTiler/ScoreTiler
• Use RandomTiler instead of GridTiler for sampling
• Use BiggestTissueBoxMask instead of TissueMask

Tiles have artifacts

• Implement custom annotation-exclusion masks
• Adjust filter parameters for artifact removal
• Increase small object removal threshold
• Apply post-extraction quality filtering

Inconsistent results across slides

• Use same seed for RandomTiler
• Normalize staining with preprocessing filters
• Adjust tissue_percent per staining quality
• Implement slide-specific mask customization

Resources

This skill includes detailed reference documentation in the references/ directory:

references/slide_management.md

Comprehensive guide to loading, inspecting, and working with whole slide images:

• Slide initialization and configuration
• Built-in sample datasets
• Slide properties and metadata
• Thumbnail generation and visualization
• Working with pyramid levels
• Multi-slide processing workflows
• Best practices and common patterns

references/tissue_masks.md

Complete documentation on tissue detection and masking:

• TissueMask, BiggestTissueBoxMask, BinaryMask classes
• How tissue detection filters work
• Customizing masks with filter chains
• Visualizing masks
• Creating custom rectangular and annotation-exclusion masks
• Integration with tile extraction
• Best practices and troubleshooting

references/tile_extraction.md

Detailed explanation of tile extraction strategies:

• RandomTiler, GridTiler, ScoreTiler comparison
• Available scorers (NucleiScorer, CellularityScorer, custom)
• Common and strategy-specific parameters
• Tile preview with locate_tiles()
• Extraction workflows and CSV reporting
• Advanced patterns (multi-level, hierarchical)
• Performance optimization
• Troubleshooting common issues

references/filters_preprocessing.md

Complete filter reference and preprocessing guide:

• Image filters (color conversion, thresholding, contrast)
• Morphological filters (dilation, erosion, opening, closing)
• Filter composition and chaining
• Common preprocessing pipelines
• Applying filters to tiles
• Custom mask filters
• Quality control filters
• Best practices and troubleshooting

references/visualization.md

Comprehensive visualization guide:

• Slide thumbnail display and saving
• Mask visualization techniques
• Tile location preview
• Displaying extracted tiles and creating mosaics
• Quality assessment visualizations
• Multi-slide comparison
• Filter effect visualization
• Exporting high-resolution figures and PDFs
• Interactive visualization in Jupyter notebooks

Usage pattern: Reference files contain in-depth information to support workflows described in this main skill document. Load specific reference files as needed for detailed implementation guidance, troubleshooting, or advanced features.

Immunology Assays: Experimental Data Analysis

Overview

Immunology Assays provides computational tools for analyzing data from common immunology experiments. This skill covers ATAC-seq differential accessibility analysis (MACS2 peak calling, motif enrichment), ELISA data processing with 4-parameter logistic standard curve fitting, immune cell tracking from time-lapse microscopy, immunohistochemistry (IHC) quantification with H-score calculation, antibody titer determination from serial dilution ELISA, cell cycle phase duration estimation from dual-nucleoside labeling, and multiplex cytokine assay data processing.

When to Use This Skill

• Processing ELISA plate data with standard curve fitting (4PL)
• Quantifying IHC staining intensity (H-score, positive pixel percentage)
• Analyzing ATAC-seq peaks and differential chromatin accessibility
• Tracking immune cell migration from microscopy time-lapse data
• Determining antibody titers from serial dilution experiments
• Estimating cell cycle phase durations from pulse-labeling data
• Processing multiplex cytokine/chemokine assay data (Luminex, MSD)

Related Skills: For flow cytometry analysis use flow-cytometry-analysis. For single-cell RNA-seq use scanpy. For bioimage analysis use bioimage-analysis.

Installation

uv pip install scipy scikit-image opencv-python numpy pandas matplotlib

For ATAC-seq (optional):

# conda install -c bioconda macs2 homer

Quick Start

import numpy as np
from scipy.optimize import curve_fit

# 4-Parameter Logistic (4PL) for ELISA standard curve
def four_pl(x, a, b, c, d):
    """a=min, b=Hill slope, c=EC50, d=max"""
    return d + (a - d) / (1 + (x / c) ** b)

# Standard curve data
concentrations = np.array([0, 15.6, 31.25, 62.5, 125, 250, 500, 1000])
od_values = np.array([0.05, 0.12, 0.22, 0.45, 0.82, 1.35, 1.85, 2.15])

popt, pcov = curve_fit(four_pl, concentrations[1:], od_values[1:],
                       p0=[0.05, 1.0, 200, 2.2], maxfev=10000)
print(f"EC50: {popt[2]:.1f} pg/mL")
print(f"Dynamic range: {popt[0]:.3f} - {popt[3]:.3f} OD")

Core Capabilities

1. ATAC-seq Accessibility Analysis

Peak calling and differential accessibility.

import subprocess
import pandas as pd

def run_macs2_atacseq(bam_path, output_prefix, genome_size='hs'):
    """Call ATAC-seq peaks with MACS2."""
    cmd = [
        'macs2', 'callpeak',
        '-t', bam_path,
        '-f', 'BAMPE',           # Paired-end
        '-g', genome_size,
        '--nomodel',
        '--shift', '-100',
        '--extsize', '200',
        '--broad',               # Broad peaks for open chromatin
        '-n', output_prefix,
        '--outdir', 'macs2_output'
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"MACS2 failed: {result.stderr}")
    return f'macs2_output/{output_prefix}_peaks.broadPeak'

def parse_broadpeak(peak_file):
    """Parse MACS2 broadPeak output."""
    cols = ['chrom', 'start', 'end', 'name', 'score', 'strand',
            'signal', 'pvalue', 'qvalue']
    df = pd.read_csv(peak_file, sep='\t', header=None, names=cols)
    print(f"Total peaks: {len(df)}")
    print(f"Mean peak width: {(df['end'] - df['start']).mean():.0f} bp")
    return df

def differential_accessibility(count_matrix, conditions, padj_threshold=0.05):
    """DESeq2-style differential accessibility from peak count matrix.

    Args:
        count_matrix: peaks x samples DataFrame (integer counts)
        conditions: list of condition labels per sample
    """
    from scipy.stats import mannwhitneyu
    from statsmodels.stats.multitest import multipletests

    results = []
    groups = list(set(conditions))
    g1_idx = [i for i, c in enumerate(conditions) if c == groups[0]]
    g2_idx = [i for i, c in enumerate(conditions) if c == groups[1]]

    for peak in count_matrix.index:
        vals1 = count_matrix.loc[peak].iloc[g1_idx].values
        vals2 = count_matrix.loc[peak].iloc[g2_idx].values

        fc = (vals2.mean() + 1) / (vals1.mean() + 1)
        log2fc = np.log2(fc)

        stat, pval = mannwhitneyu(vals1, vals2, alternative='two-sided')
        results.append({'peak': peak, 'log2FC': log2fc, 'pvalue': pval})

    df = pd.DataFrame(results)
    _, df['padj'], _, _ = multipletests(df['pvalue'], method='fdr_bh')

    sig = df[df['padj'] < padj_threshold]
    print(f"Significant DA peaks: {len(sig)} / {len(df)}")
    print(f"  More accessible: {(sig['log2FC'] > 0).sum()}")
    print(f"  Less accessible: {(sig['log2FC'] < 0).sum()}")
    return df

def run_homer_motif(peak_bed, genome, output_dir):
    """Run HOMER motif enrichment on peaks."""
    cmd = [
        'findMotifsGenome.pl', peak_bed, genome, output_dir,
        '-size', '200', '-mask', '-p', '4'
    ]
    subprocess.run(cmd, capture_output=True, text=True, check=True)
    return output_dir

2. ELISA Data Processing

4-parameter logistic standard curve fitting and unknown interpolation.

import numpy as np
from scipy.optimize import curve_fit
import pandas as pd

def four_pl(x, a, b, c, d):
    """4-Parameter Logistic curve.
    a = minimum asymptote, b = Hill slope, c = EC50, d = maximum asymptote"""
    return d + (a - d) / (1 + (x / c) ** b)

def inverse_four_pl(y, a, b, c, d):
    """Inverse 4PL to interpolate concentration from OD."""
    return c * ((a - d) / (y - d) - 1) ** (1 / b)

def process_elisa_plate(standards, unknowns, blank_od=None):
    """Process ELISA plate with 4PL standard curve.

    Args:
        standards: dict of concentration -> list of OD replicates
        unknowns: dict of sample_name -> list of OD replicates
        blank_od: blank well OD (subtracted from all values)
    """
    # Prepare standard curve data
    conc_list, od_list = [], []
    for conc, ods in sorted(standards.items()):
        for od in ods:
            if conc > 0:  # Skip zero for fitting
                conc_list.append(conc)
                od_list.append(od - (blank_od or 0))

    conc_arr = np.array(conc_list)
    od_arr = np.array(od_list)

    # Fit 4PL
    p0 = [min(od_arr), 1.0, np.median(conc_arr), max(od_arr)]
    popt, pcov = curve_fit(four_pl, conc_arr, od_arr, p0=p0, maxfev=10000)
    a, b, c, d = popt

    # R-squared
    predicted = four_pl(conc_arr, *popt)
    ss_res = np.sum((od_arr - predicted) ** 2)
    ss_tot = np.sum((od_arr - np.mean(od_arr)) ** 2)
    r_squared = 1 - ss_res / ss_tot

    # Detection limits
    lod = inverse_four_pl(a + 2 * np.std(od_arr[:3]), *popt) if a + 2 * np.std(od_arr[:3]) < d else None

    print(f"4PL fit: R² = {r_squared:.4f}")
    print(f"EC50: {c:.2f}")
    print(f"Dynamic range: {a:.3f} - {d:.3f} OD")
    if lod:
        print(f"LOD: {lod:.2f}")

    # Interpolate unknowns
    results = []
    for name, ods in unknowns.items():
        od_corrected = [od - (blank_od or 0) for od in ods]
        concentrations = []
        for od in od_corrected:
            if a < od < d:  # Within curve range
                conc = inverse_four_pl(od, *popt)
                concentrations.append(conc)
            else:
                concentrations.append(np.nan)

        results.append({
            'sample': name,
            'mean_od': np.mean(od_corrected),
            'mean_conc': np.nanmean(concentrations),
            'std_conc': np.nanstd(concentrations),
            'n': len(concentrations),
            'in_range': sum(~np.isnan(c) for c in concentrations)
        })

    return pd.DataFrame(results), popt, r_squared

# Example
standards = {
    0: [0.05, 0.06], 15.6: [0.11, 0.13], 31.25: [0.21, 0.23],
    62.5: [0.44, 0.46], 125: [0.80, 0.84], 250: [1.32, 1.38],
    500: [1.82, 1.88], 1000: [2.12, 2.18]
}
unknowns = {
    'Patient_1': [0.95, 0.98], 'Patient_2': [0.35, 0.38],
    'Patient_3': [1.55, 1.60], 'Control': [0.08, 0.09]
}

results, params, r2 = process_elisa_plate(standards, unknowns, blank_od=0.05)
print(results[['sample', 'mean_conc', 'std_conc']])

3. Immune Cell Tracking

Track immune cells in time-lapse microscopy.

import numpy as np
import pandas as pd

def track_immune_cells(tracks_df, pixel_size_um=0.65, frame_interval_min=1):
    """Analyze immune cell migration from tracking data.

    Args:
        tracks_df: DataFrame with columns [particle, frame, x, y]
        pixel_size_um: microns per pixel
        frame_interval_min: minutes between frames
    """
    results = []

    for pid, track in tracks_df.groupby('particle'):
        track = track.sort_values('frame')
        x = track['x'].values * pixel_size_um
        y = track['y'].values * pixel_size_um
        t = track['frame'].values * frame_interval_min

        # Instantaneous velocity
        dx = np.diff(x)
        dy = np.diff(y)
        dt = np.diff(t)
        speeds = np.sqrt(dx**2 + dy**2) / dt

        # Displacement (start to end)
        displacement = np.sqrt((x[-1] - x[0])**2 + (y[-1] - y[0])**2)

        # Total path length
        path_length = np.sum(np.sqrt(dx**2 + dy**2))

        # Confinement ratio (displacement / path_length)
        confinement = displacement / path_length if path_length > 0 else 0

        # Mean squared displacement
        msd_values = []
        for lag in range(1, min(len(x), 20)):
            displacements = (x[lag:] - x[:-lag])**2 + (y[lag:] - y[:-lag])**2
            msd_values.append(np.mean(displacements))

        results.append({
            'particle': pid,
            'mean_speed': np.mean(speeds),
            'max_speed': np.max(speeds),
            'displacement': displacement,
            'path_length': path_length,
            'confinement_ratio': confinement,
            'duration_min': t[-1] - t[0],
            'n_frames': len(track)
        })

    df = pd.DataFrame(results)

    # Classify migratory phenotype
    df['phenotype'] = 'confined'
    df.loc[df['confinement_ratio'] > 0.5, 'phenotype'] = 'directed'
    df.loc[(df['confinement_ratio'] > 0.2) & (df['confinement_ratio'] <= 0.5), 'phenotype'] = 'random_walk'

    print(f"Tracked {len(df)} cells")
    print(f"Mean speed: {df['mean_speed'].mean():.2f} um/min")
    print(f"Phenotypes:\n{df['phenotype'].value_counts()}")
    return df

4. IHC Quantification

Quantify immunohistochemistry staining intensity.

import numpy as np
from skimage.color import rgb2hed
import skimage.io

def quantify_ihc(image_path, method='h_score'):
    """Quantify IHC staining using color deconvolution.

    Separates DAB (brown) from hematoxylin (blue) using
    Ruifrok & Johnston color deconvolution.
    """
    image = skimage.io.imread(image_path)

    # Color deconvolution: RGB → HED (Hematoxylin, Eosin, DAB)
    hed = rgb2hed(image)
    dab_channel = hed[:, :, 2]  # DAB is channel 2
    hematoxylin = hed[:, :, 0]

    # Threshold to identify tissue
    tissue_mask = hematoxylin > 0.05

    # DAB intensity classification for H-score
    dab_tissue = dab_channel[tissue_mask]

    # Classify: negative (0), weak (1+), moderate (2+), strong (3+)
    negative = (dab_tissue < 0.1).sum()
    weak = ((dab_tissue >= 0.1) & (dab_tissue < 0.2)).sum()
    moderate = ((dab_tissue >= 0.2) & (dab_tissue < 0.4)).sum()
    strong = (dab_tissue >= 0.4).sum()
    total = len(dab_tissue)

    # H-score: 1*(% weak) + 2*(% moderate) + 3*(% strong)
    h_score = (1 * weak + 2 * moderate + 3 * strong) / total * 100

    # Positive pixel percentage
    positive_pct = 100 * (weak + moderate + strong) / total

    print(f"H-score: {h_score:.1f} (range 0-300)")
    print(f"Positive pixels: {positive_pct:.1f}%")
    print(f"  Weak (1+): {100*weak/total:.1f}%")
    print(f"  Moderate (2+): {100*moderate/total:.1f}%")
    print(f"  Strong (3+): {100*strong/total:.1f}%")

    return {
        'h_score': h_score,
        'positive_pct': positive_pct,
        'negative_pct': 100 * negative / total,
        'weak_pct': 100 * weak / total,
        'moderate_pct': 100 * moderate / total,
        'strong_pct': 100 * strong / total
    }

5. Antibody Titer Analysis

Determine endpoint titers from serial dilution ELISA.

import numpy as np
from scipy import stats

def calculate_endpoint_titer(dilutions, od_values, cutoff_method='mean_plus_3sd',
                              negative_ods=None):
    """Determine endpoint antibody titer from serial dilution ELISA.

    Args:
        dilutions: list of dilution factors (e.g., [100, 200, 400, ...])
        od_values: list of OD values at each dilution
        negative_ods: list of negative control OD values
    """
    if negative_ods is None:
        cutoff = 0.1  # Default cutoff
    elif cutoff_method == 'mean_plus_3sd':
        cutoff = np.mean(negative_ods) + 3 * np.std(negative_ods)
    elif cutoff_method == 'mean_plus_2sd':
        cutoff = np.mean(negative_ods) + 2 * np.std(negative_ods)
    else:
        cutoff = 0.1

    # Find last dilution above cutoff
    endpoint = None
    for dil, od in zip(dilutions, od_values):
        if od > cutoff:
            endpoint = dil

    if endpoint is None:
        return {'titer': '<' + str(min(dilutions)), 'cutoff': cutoff}

    return {
        'titer': endpoint,
        'log2_titer': np.log2(endpoint),
        'cutoff': cutoff
    }

def geometric_mean_titer(titers):
    """Calculate geometric mean titer from multiple samples."""
    log_titers = np.log2([t for t in titers if t > 0])
    gmt = 2 ** np.mean(log_titers)
    ci = stats.t.interval(0.95, df=len(log_titers)-1,
                          loc=np.mean(log_titers),
                          scale=stats.sem(log_titers))
    return {
        'gmt': gmt,
        'ci_lower': 2 ** ci[0],
        'ci_upper': 2 ** ci[1],
        'n': len(log_titers)
    }

# Example
dilutions = [100, 200, 400, 800, 1600, 3200, 6400, 12800]
sample_ods = [2.1, 1.8, 1.4, 0.9, 0.45, 0.18, 0.08, 0.05]
neg_ods = [0.06, 0.05, 0.07, 0.04]

result = calculate_endpoint_titer(dilutions, sample_ods, negative_ods=neg_ods)
print(f"Endpoint titer: 1:{result['titer']}")
print(f"Cutoff OD: {result['cutoff']:.3f}")

6. Cell Cycle Phase Duration

Estimate phase durations from dual-nucleoside pulse labeling.

import numpy as np

def estimate_phase_durations(labeled_fractions, pulse_interval_hours,
                              total_cycle_time=None):
    """Estimate cell cycle phase durations from dual-nucleoside labeling.

    Args:
        labeled_fractions: dict of timepoints -> fraction labeled
        pulse_interval_hours: time between pulses
        total_cycle_time: if known, constrains the estimates
    """
    times = sorted(labeled_fractions.keys())
    fractions = [labeled_fractions[t] for t in times]

    # S-phase duration estimate: fraction labeled at first timepoint * total cycle time
    # If total cycle time unknown, estimate from growth rate
    if total_cycle_time is None:
        # Assume exponential growth, estimate from labeling kinetics
        # Rate of increase in labeled fraction approximates 1/Tc
        if len(fractions) > 1:
            rate = (fractions[-1] - fractions[0]) / (times[-1] - times[0])
            total_cycle_time = 1 / rate if rate > 0 else 24
        else:
            total_cycle_time = 24  # Default

    s_phase = fractions[0] * total_cycle_time
    g2m_phase = pulse_interval_hours  # Time for labeled cells to reach mitosis

    # G1 = total - S - G2/M
    g1_phase = total_cycle_time - s_phase - g2m_phase
    g1_phase = max(g1_phase, 0)

    return {
        'total_cycle': total_cycle_time,
        'G1': g1_phase,
        'S': s_phase,
        'G2_M': g2m_phase
    }

7. Cytokine Analysis

Process multiplex cytokine assay data.

import numpy as np
import pandas as pd
from scipy.optimize import curve_fit

def process_multiplex_cytokines(plate_data, analytes, standard_curves):
    """Process multiplex cytokine assay (Luminex/MSD) data.

    Args:
        plate_data: DataFrame with well, sample, analyte, MFI columns
        analytes: list of analyte names
        standard_curves: dict of analyte -> (concentrations, MFI_values)
    """
    def five_pl(x, a, b, c, d, g):
        return d + (a - d) / (1 + (x / c) ** b) ** g

    results = []
    for analyte in analytes:
        # Fit standard curve
        conc, mfi = standard_curves[analyte]
        try:
            popt, _ = curve_fit(five_pl, conc[conc > 0], mfi[conc > 0],
                               p0=[min(mfi), 1, np.median(conc), max(mfi), 1],
                               maxfev=10000)
        except RuntimeError:
            # Fall back to 4PL
            popt = None

        # Interpolate unknowns
        analyte_data = plate_data[plate_data['analyte'] == analyte]
        for _, row in analyte_data.iterrows():
            if popt is not None:
                # Inverse 5PL
                try:
                    from scipy.optimize import brentq
                    conc_val = brentq(lambda x: five_pl(x, *popt) - row['MFI'],
                                     0.01, 100000)
                except ValueError:
                    conc_val = np.nan
            else:
                conc_val = np.nan

            results.append({
                'sample': row['sample'],
                'analyte': analyte,
                'MFI': row['MFI'],
                'concentration': conc_val
            })

    return pd.DataFrame(results)

Typical Workflows

Workflow 1: Process ELISA Plate Data with 4PL Standard Curve

standards = {0: [0.05], 15.6: [0.12, 0.13], 31.25: [0.22, 0.24],
             62.5: [0.45, 0.47], 125: [0.82, 0.85], 250: [1.35, 1.38],
             500: [1.85, 1.88], 1000: [2.15, 2.18]}
unknowns = {'Sample_A': [0.68, 0.72], 'Sample_B': [1.45, 1.50]}
results, params, r2 = process_elisa_plate(standards, unknowns, blank_od=0.05)
print(results)

Workflow 2: Quantify IHC Staining Intensity (H-Score)

result = quantify_ihc('ihc_slide.tif')
print(f"H-score: {result['h_score']:.1f}")
print(f"Positive: {result['positive_pct']:.1f}%")

Workflow 3: ATAC-seq Peak Calling and Differential Accessibility

# Call peaks for each condition
peaks_ctrl = run_macs2_atacseq('control.bam', 'control')
peaks_treat = run_macs2_atacseq('treated.bam', 'treated')

# Parse and analyze
ctrl_df = parse_broadpeak(peaks_ctrl)
treat_df = parse_broadpeak(peaks_treat)
print(f"Control peaks: {len(ctrl_df)}, Treatment peaks: {len(treat_df)}")

Best Practices

1. ELISA standard curves — always run standards in duplicate; verify R² > 0.99 for 4PL fit; samples outside curve range should be re-run at appropriate dilutions
2. IHC quantification — use color deconvolution (not simple thresholding) for DAB separation; calibrate thresholds for weak/moderate/strong on control tissue
3. ATAC-seq — use paired-end mode (-f BAMPE); shift reads to account for Tn5 insertion; call narrow peaks for TF footprinting, broad peaks for open chromatin
4. Antibody titers — use geometric mean for group statistics (titers are log-distributed); calculate seroconversion as 4-fold rise
5. Cell tracking — require minimum track length (>10 frames) to exclude artifacts; validate tracking by visual inspection of overlaid tracks
6. Multiplex assays — check for cross-reactivity; flag analytes with CV >15% between replicates

Troubleshooting

Problem: 4PL fit fails to converge Solution: Check that standard curve spans the expected range. Adjust initial parameters (p0). Ensure blank subtraction is correct. Try 5PL for asymmetric curves.

Problem: IHC color deconvolution gives poor separation Solution: Verify image is RGB (not grayscale). Check that staining uses standard DAB/hematoxylin. Adjust HED matrix if using non-standard chromogens.

Problem: ATAC-seq produces too many peaks Solution: Increase -q (q-value) threshold. Filter for peaks with fold enrichment >2. Check for high mitochondrial read fraction (indicates poor quality).

Problem: Antibody titer is below lowest dilution Solution: Use lower starting dilution. Report as "< [lowest dilution]". Check that positive control shows expected titer.

Resources

• ELISA Technical Guide (Thermo Fisher)
• MACS2 Documentation
• HOMER Motif Analysis
• H-score Calculation Guidelines

LatchBio Integration

Overview

Latch is a Python framework for building and deploying bioinformatics workflows as serverless pipelines. Built on Flyte, create workflows with @workflow/@task decorators, manage cloud data with LatchFile/LatchDir, configure resources, and integrate Nextflow/Snakemake pipelines.

Core Capabilities

The Latch platform provides four main areas of functionality:

1. Workflow Creation and Deployment

• Define serverless workflows using Python decorators
• Support for native Python, Nextflow, and Snakemake pipelines
• Automatic containerization with Docker
• Auto-generated no-code user interfaces
• Version control and reproducibility

2. Data Management

• Cloud storage abstractions (LatchFile, LatchDir)
• Structured data organization with Registry (Projects → Tables → Records)
• Type-safe data operations with links and enums
• Automatic file transfer between local and cloud
• Glob pattern matching for file selection

3. Resource Configuration

• Pre-configured task decorators (@small_task, @large_task, @small_gpu_task, @large_gpu_task)
• Custom resource specifications (CPU, memory, GPU, storage)
• GPU support (K80, V100, A100)
• Timeout and storage configuration
• Cost optimization strategies

4. Verified Workflows

• Production-ready pre-built pipelines
• Bulk RNA-seq, DESeq2, pathway analysis
• AlphaFold and ColabFold for protein structure prediction
• Single-cell tools (ArchR, scVelo, emptyDropsR)
• CRISPR analysis, phylogenetics, and more

Quick Start

Installation and Setup

# Install Latch SDK
python3 -m uv pip install latch

# Login to Latch
latch login

# Initialize a new workflow
latch init my-workflow

# Register workflow to platform
latch register my-workflow

Prerequisites:

• Docker installed and running
• Latch account credentials
• Python 3.8+

Basic Workflow Example

from latch import workflow, small_task
from latch.types import LatchFile

@small_task
def process_file(input_file: LatchFile) -> LatchFile:
    """Process a single file"""
    # Processing logic
    return output_file

@workflow
def my_workflow(input_file: LatchFile) -> LatchFile:
    """
    My bioinformatics workflow

    Args:
        input_file: Input data file
    """
    return process_file(input_file=input_file)

When to Use This Skill

This skill should be used when encountering any of the following scenarios:

Workflow Development:

• "Create a Latch workflow for RNA-seq analysis"
• "Deploy my pipeline to Latch"
• "Convert my Nextflow pipeline to Latch"
• "Add GPU support to my workflow"
• Working with @workflow, @task decorators

Data Management:

• "Organize my sequencing data in Latch Registry"
• "How do I use LatchFile and LatchDir?"
• "Set up sample tracking in Latch"
• Working with latch:/// paths

Resource Configuration:

• "Configure GPU for AlphaFold on Latch"
• "My task is running out of memory"
• "How do I optimize workflow costs?"
• Working with task decorators

Verified Workflows:

• "Run AlphaFold on Latch"
• "Use DESeq2 for differential expression"
• "Available pre-built workflows"
• Using latch.verified module

Detailed Documentation

This skill includes comprehensive reference documentation organized by capability:

references/workflow-creation.md

Read this for:

• Creating and registering workflows
• Task definition and decorators
• Supporting Python, Nextflow, Snakemake
• Launch plans and conditional sections
• Workflow execution (CLI and programmatic)
• Multi-step and parallel pipelines
• Troubleshooting registration issues

Key topics:

• latch init and latch register commands
• @workflow and @task decorators
• LatchFile and LatchDir basics
• Type annotations and docstrings
• Launch plans with preset parameters
• Conditional UI sections

references/data-management.md

Read this for:

• Cloud storage with LatchFile and LatchDir
• Registry system (Projects, Tables, Records)
• Linked records and relationships
• Enum and typed columns
• Bulk operations and transactions
• Integration with workflows
• Account and workspace management

Key topics:

• latch:/// path format
• File transfer and glob patterns
• Creating and querying Registry tables
• Column types (string, number, file, link, enum)
• Record CRUD operations
• Workflow-Registry integration

references/resource-configuration.md

Read this for:

• Task resource decorators
• Custom CPU, memory, GPU configuration
• GPU types (K80, V100, A100)
• Timeout and storage settings
• Resource optimization strategies
• Cost-effective workflow design
• Monitoring and debugging

Key topics:

• @small_task, @large_task, @small_gpu_task, @large_gpu_task
• @custom_task with precise specifications
• Multi-GPU configuration
• Resource selection by workload type
• Platform limits and quotas

references/verified-workflows.md

Read this for:

• Pre-built production workflows
• Bulk RNA-seq and DESeq2
• AlphaFold and ColabFold
• Single-cell analysis (ArchR, scVelo)
• CRISPR editing analysis
• Pathway enrichment
• Integration with custom workflows

Key topics:

• latch.verified module imports
• Available verified workflows
• Workflow parameters and options
• Combining verified and custom steps
• Version management

Common Workflow Patterns

Complete RNA-seq Pipeline

from latch import workflow, small_task, large_task
from latch.types import LatchFile, LatchDir

@small_task
def quality_control(fastq: LatchFile) -> LatchFile:
    """Run FastQC"""
    return qc_output

@large_task
def alignment(fastq: LatchFile, genome: str) -> LatchFile:
    """STAR alignment"""
    return bam_output

@small_task
def quantification(bam: LatchFile) -> LatchFile:
    """featureCounts"""
    return counts

@workflow
def rnaseq_pipeline(
    input_fastq: LatchFile,
    genome: str,
    output_dir: LatchDir
) -> LatchFile:
    """RNA-seq analysis pipeline"""
    qc = quality_control(fastq=input_fastq)
    aligned = alignment(fastq=qc, genome=genome)
    return quantification(bam=aligned)

GPU-Accelerated Workflow

from latch import workflow, small_task, large_gpu_task
from latch.types import LatchFile

@small_task
def preprocess(input_file: LatchFile) -> LatchFile:
    """Prepare data"""
    return processed

@large_gpu_task
def gpu_computation(data: LatchFile) -> LatchFile:
    """GPU-accelerated analysis"""
    return results

@workflow
def gpu_pipeline(input_file: LatchFile) -> LatchFile:
    """Pipeline with GPU tasks"""
    preprocessed = preprocess(input_file=input_file)
    return gpu_computation(data=preprocessed)

Registry-Integrated Workflow

from latch import workflow, small_task
from latch.registry.table import Table
from latch.registry.record import Record
from latch.types import LatchFile

@small_task
def process_and_track(sample_id: str, table_id: str) -> str:
    """Process sample and update Registry"""
    # Get sample from registry
    table = Table.get(table_id=table_id)
    records = Record.list(table_id=table_id, filter={"sample_id": sample_id})
    sample = records[0]

    # Process
    input_file = sample.values["fastq_file"]
    output = process(input_file)

    # Update registry
    sample.update(values={"status": "completed", "result": output})
    return "Success"

@workflow
def registry_workflow(sample_id: str, table_id: str):
    """Workflow integrated with Registry"""
    return process_and_track(sample_id=sample_id, table_id=table_id)

Best Practices

Workflow Design

1. Use type annotations for all parameters
2. Write clear docstrings (appear in UI)
3. Start with standard task decorators, scale up if needed
4. Break complex workflows into modular tasks
5. Implement proper error handling

Data Management

6. Use consistent folder structures
7. Define Registry schemas before bulk entry
8. Use linked records for relationships
9. Store metadata in Registry for traceability

Resource Configuration

10. Right-size resources (don't over-allocate)
11. Use GPU only when algorithms support it
12. Monitor execution metrics and optimize
13. Design for parallel execution when possible

Development Workflow

14. Test locally with Docker before registration
15. Use version control for workflow code
16. Document resource requirements
17. Profile workflows to determine actual needs

Troubleshooting

Common Issues

Registration Failures:

• Ensure Docker is running
• Check authentication with latch login
• Verify all dependencies in Dockerfile
• Use --verbose flag for detailed logs

Resource Problems:

• Out of memory: Increase memory in task decorator
• Timeouts: Increase timeout parameter
• Storage issues: Increase ephemeral storage_gib

Data Access:

• Use correct latch:/// path format
• Verify file exists in workspace
• Check permissions for shared workspaces

Type Errors:

• Add type annotations to all parameters
• Use LatchFile/LatchDir for file/directory parameters
• Ensure workflow return type matches actual return

Additional Resources

• Official Documentation: https://docs.latch.bio
• GitHub Repository: https://github.com/latchbio/latch
• Slack Community: Join Latch SDK workspace
• API Reference: https://docs.latch.bio/api/latch.html
• Blog: https://blog.latch.bio

Support

For issues or questions:

1. Check documentation links above
2. Search GitHub issues
3. Ask in Slack community
4. Contact support@latch.bio

Microbial Dynamics: Population Dynamics & Modeling

Overview

Microbial Dynamics provides computational tools for modeling and analyzing microbial populations. This skill covers bacterial growth curve fitting using standard models (logistic, Gompertz, Baranyi), multi-species community dynamics via Lotka-Volterra equations, stochastic population simulation using the Gillespie algorithm, biofilm quantification from crystal violet assays, colony-forming unit enumeration with statistical analysis, bacterial genome annotation via Prokka, and simplified anaerobic digestion modeling.

When to Use This Skill

• Fitting bacterial growth curves from OD600 time-series data
• Extracting growth parameters: lag phase duration, maximum growth rate (mu_max), carrying capacity (K)
• Modeling multi-species microbial community interactions
• Running stochastic simulations of gene expression or population dynamics
• Processing crystal violet biofilm assay data
• Calculating CFU/mL from serial dilution plating
• Annotating bacterial genomes and extracting gene statistics
• Modeling biogas production from anaerobic digestion

Related Skills: For constraint-based metabolic modeling use cobrapy. For sequence manipulation and BLAST use biopython. For statistical analysis use statistical-analysis.

Installation

uv pip install scipy numpy pandas matplotlib

For genome annotation (optional):

# conda install -c bioconda prokka

Quick Start

from scipy.optimize import curve_fit
import numpy as np

# Logistic growth model
def logistic(t, y0, K, r, lag):
    return K / (1 + ((K - y0) / y0) * np.exp(-r * (t - lag)))

# Example OD600 data
time = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 20, 24])
od600 = np.array([0.02, 0.02, 0.03, 0.06, 0.15, 0.38, 0.72, 1.05, 1.25, 1.42, 1.48, 1.50, 1.51, 1.51, 1.52])

popt, pcov = curve_fit(logistic, time, od600, p0=[0.02, 1.5, 0.5, 2.0], maxfev=10000)
print(f"y0={popt[0]:.4f}, K={popt[1]:.3f}, r={popt[2]:.3f} h^-1, lag={popt[3]:.2f} h")

Core Capabilities

1. Growth Curve Modeling

Fit OD600 data to standard microbial growth models.

import numpy as np
from scipy.optimize import curve_fit

def logistic(t, y0, K, r, lag):
    """Logistic growth model."""
    return K / (1 + ((K - y0) / y0) * np.exp(-r * (t - lag)))

def gompertz(t, y0, K, mu_max, lag):
    """Modified Gompertz growth model."""
    return y0 + (K - y0) * np.exp(-np.exp((mu_max * np.e / (K - y0)) * (lag - t) + 1))

def baranyi(t, y0, K, mu_max, lag):
    """Baranyi growth model."""
    A_t = t + (1 / mu_max) * np.log(np.exp(-mu_max * t) +
           np.exp(-mu_max * lag) - np.exp(-mu_max * (t + lag)))
    return K - np.log(1 + (np.exp(K) - np.exp(y0)) / np.exp(y0) * np.exp(-mu_max * A_t))

def fit_growth_curve(time, od600, model='logistic'):
    """Fit growth curve to OD data and return parameters."""
    models = {'logistic': logistic, 'gompertz': gompertz, 'baranyi': baranyi}
    func = models[model]

    # Initial guesses
    y0_guess = od600[0]
    K_guess = od600.max()
    r_guess = 0.5
    lag_guess = time[np.argmax(np.gradient(od600))] - 1

    p0 = [y0_guess, K_guess, r_guess, max(lag_guess, 0)]
    bounds = ([0, 0, 0, 0], [np.inf, np.inf, 10, time.max()])

    popt, pcov = curve_fit(func, time, od600, p0=p0, bounds=bounds, maxfev=10000)
    perr = np.sqrt(np.diag(pcov))

    # R-squared
    residuals = od600 - func(time, *popt)
    ss_res = np.sum(residuals**2)
    ss_tot = np.sum((od600 - np.mean(od600))**2)
    r_squared = 1 - (ss_res / ss_tot)

    param_names = ['y0', 'K', 'mu_max', 'lag']
    result = {name: {'value': val, 'std': err}
              for name, val, err in zip(param_names, popt, perr)}
    result['r_squared'] = r_squared
    result['model'] = model

    return result, popt

# Compare models
for model_name in ['logistic', 'gompertz', 'baranyi']:
    try:
        result, _ = fit_growth_curve(time, od600, model=model_name)
        print(f"{model_name}: R²={result['r_squared']:.4f}, "
              f"mu_max={result['mu_max']['value']:.3f} ± {result['mu_max']['std']:.3f} h⁻¹, "
              f"lag={result['lag']['value']:.2f} h")
    except RuntimeError:
        print(f"{model_name}: fitting failed")

2. Lotka-Volterra Community Dynamics

Simulate multi-species interactions.

import numpy as np
from scipy.integrate import solve_ivp

def lotka_volterra(t, N, r, K, alpha):
    """Generalized Lotka-Volterra for n species.

    Args:
        N: array of population sizes
        r: array of intrinsic growth rates
        K: array of carrying capacities
        alpha: interaction matrix (alpha[i,j] = effect of j on i)
    """
    n = len(N)
    dNdt = np.zeros(n)
    for i in range(n):
        interaction = sum(alpha[i, j] * N[j] for j in range(n))
        dNdt[i] = r[i] * N[i] * (1 - interaction / K[i])
    return dNdt

# 3-species community
r = np.array([0.5, 0.4, 0.3])      # Growth rates
K = np.array([1000, 800, 600])       # Carrying capacities
alpha = np.array([                    # Interaction matrix
    [1.0, 0.5, 0.1],   # Species 1: self + competition from 2 and 3
    [0.3, 1.0, 0.4],   # Species 2
    [0.2, 0.6, 1.0],   # Species 3
])
N0 = np.array([10, 10, 10])  # Initial populations

sol = solve_ivp(
    lotka_volterra, [0, 100], N0,
    args=(r, K, alpha),
    t_eval=np.linspace(0, 100, 500),
    method='RK45'
)

print("Final populations:")
for i in range(3):
    print(f"  Species {i+1}: {sol.y[i, -1]:.1f}")

# Stability analysis: check if coexistence is stable
A = np.diag(1/K) @ alpha
try:
    eigenvalues = np.linalg.eigvals(A)
    stable = all(ev.real > 0 for ev in eigenvalues)
    print(f"Coexistence equilibrium is {'stable' if stable else 'unstable'}")
except np.linalg.LinAlgError:
    print("Stability analysis failed")

3. Stochastic Population Simulation

Gillespie SSA for exact stochastic simulation.

import numpy as np

def gillespie_ssa(propensity_func, stoich_matrix, x0, t_end, max_steps=100000):
    """Gillespie Stochastic Simulation Algorithm.

    Args:
        propensity_func: function(x) -> array of reaction propensities
        stoich_matrix: reactions x species stoichiometry matrix
        x0: initial state vector
        t_end: simulation end time
    """
    t = 0
    x = np.array(x0, dtype=float)
    times = [t]
    states = [x.copy()]

    for step in range(max_steps):
        props = propensity_func(x)
        total_prop = np.sum(props)

        if total_prop == 0 or t >= t_end:
            break

        # Time to next reaction
        dt = np.random.exponential(1 / total_prop)
        t += dt

        if t > t_end:
            break

        # Choose reaction
        reaction = np.searchsorted(np.cumsum(props), np.random.uniform(0, total_prop))
        reaction = min(reaction, len(props) - 1)

        # Update state
        x += stoich_matrix[reaction]
        x = np.clip(x, 0, None)

        times.append(t)
        states.append(x.copy())

    return np.array(times), np.array(states)

# Birth-death-immigration model
def propensities(x):
    N = x[0]
    birth_rate = 0.5 * N
    death_rate = 0.01 * N * (N - 1)  # Density-dependent death
    immigration = 5.0
    return np.array([birth_rate, death_rate, immigration])

stoich = np.array([[1], [-1], [1]])  # Birth: +1, Death: -1, Immigration: +1

# Run ensemble
n_runs = 50
results = []
for i in range(n_runs):
    times, states = gillespie_ssa(propensities, stoich, [10], t_end=50)
    results.append((times, states))

# Summary statistics
final_pops = [states[-1, 0] for _, states in results]
print(f"Mean final population: {np.mean(final_pops):.1f} ± {np.std(final_pops):.1f}")

4. Biofilm Quantification

Process crystal violet biofilm assay data.

import numpy as np
import pandas as pd

def analyze_biofilm_cv(od_data, blank_od=0.05, conditions=None):
    """Analyze crystal violet biofilm assay.

    Args:
        od_data: dict of condition -> list of OD570 replicates
        blank_od: blank well OD for background subtraction
    """
    results = []
    for condition, replicates in od_data.items():
        corrected = np.array(replicates) - blank_od
        corrected = np.clip(corrected, 0, None)
        results.append({
            'condition': condition,
            'mean_od': np.mean(corrected),
            'std_od': np.std(corrected, ddof=1),
            'n': len(corrected),
            'sem': np.std(corrected, ddof=1) / np.sqrt(len(corrected))
        })

    df = pd.DataFrame(results)
    # Normalize to control
    control_mean = df.iloc[0]['mean_od']
    df['fold_change'] = df['mean_od'] / control_mean
    return df

# Example: biofilm inhibition dose-response
od_data = {
    'Control': [0.85, 0.92, 0.88, 0.90],
    '1 uM': [0.80, 0.78, 0.82, 0.79],
    '10 uM': [0.55, 0.52, 0.58, 0.50],
    '100 uM': [0.20, 0.18, 0.22, 0.19],
}

df = analyze_biofilm_cv(od_data)
print(df[['condition', 'mean_od', 'std_od', 'fold_change']])

5. CFU Enumeration

Calculate colony-forming units from serial dilution plating.

import numpy as np
from scipy import stats

def calculate_cfu(counts, dilution_factor, volume_plated_ml=0.1):
    """Calculate CFU/mL from plate counts.

    Args:
        counts: list of colony counts per plate
        dilution_factor: dilution used (e.g., 1e-6 for 10^-6)
        volume_plated_ml: volume plated in mL
    """
    counts = np.array(counts)
    cfu_per_ml = counts / (dilution_factor * volume_plated_ml)

    mean_cfu = np.mean(cfu_per_ml)
    std_cfu = np.std(cfu_per_ml, ddof=1)
    sem = std_cfu / np.sqrt(len(counts))

    # 95% confidence interval
    ci = stats.t.interval(0.95, df=len(counts)-1, loc=mean_cfu, scale=sem)

    return {
        'mean_cfu_per_ml': mean_cfu,
        'std': std_cfu,
        'sem': sem,
        'ci_95': ci,
        'n': len(counts),
        'log10_cfu': np.log10(mean_cfu)
    }

# Example
result = calculate_cfu(counts=[42, 38, 45], dilution_factor=1e-6, volume_plated_ml=0.1)
print(f"CFU/mL: {result['mean_cfu_per_ml']:.2e}")
print(f"Log10 CFU/mL: {result['log10_cfu']:.2f}")
print(f"95% CI: ({result['ci_95'][0]:.2e}, {result['ci_95'][1]:.2e})")

6. Bacterial Genome Annotation

Run Prokka and parse results.

import subprocess
import pandas as pd

def run_prokka(fasta_path, output_dir, prefix='genome', genus=None, species=None):
    """Annotate bacterial genome with Prokka."""
    cmd = [
        'prokka', fasta_path,
        '--outdir', output_dir,
        '--prefix', prefix,
        '--cpus', '4',
        '--force'
    ]
    if genus:
        cmd.extend(['--genus', genus])
    if species:
        cmd.extend(['--species', species])

    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"Prokka failed: {result.stderr}")
    return f'{output_dir}/{prefix}'

def parse_prokka_gff(gff_path):
    """Parse Prokka GFF3 output to extract gene statistics."""
    genes = []
    with open(gff_path) as f:
        for line in f:
            if line.startswith('#') or line.startswith('>'):
                continue
            if '\t' not in line:
                continue
            parts = line.strip().split('\t')
            if len(parts) < 9:
                continue

            feature_type = parts[2]
            if feature_type in ('CDS', 'tRNA', 'rRNA', 'tmRNA'):
                attrs = dict(kv.split('=', 1) for kv in parts[8].split(';') if '=' in kv)
                genes.append({
                    'type': feature_type,
                    'start': int(parts[3]),
                    'end': int(parts[4]),
                    'strand': parts[6],
                    'gene': attrs.get('gene', ''),
                    'product': attrs.get('product', ''),
                    'length': int(parts[4]) - int(parts[3]) + 1
                })

    df = pd.DataFrame(genes)
    print(f"Total features: {len(df)}")
    print(f"Feature types:\n{df['type'].value_counts()}")
    print(f"Mean CDS length: {df[df['type']=='CDS']['length'].mean():.0f} bp")

    # Coding density
    total_coding = df[df['type'] == 'CDS']['length'].sum()
    genome_size = df['end'].max()
    coding_density = 100 * total_coding / genome_size
    print(f"Coding density: {coding_density:.1f}%")

    return df

7. Anaerobic Digestion Modeling

Simplified ADM1 for biogas prediction.

import numpy as np
from scipy.integrate import solve_ivp

def adm1_simplified(t, y, params):
    """Simplified anaerobic digestion model.
    State: [S_substrate, X_acidogens, X_methanogens, S_VFA, CH4]
    """
    S, Xa, Xm, VFA, CH4 = y
    k_hyd = params['k_hyd']       # Hydrolysis rate
    mu_a = params['mu_a']         # Acidogen max growth rate
    Ks_a = params['Ks_a']         # Acidogen half-saturation
    mu_m = params['mu_m']         # Methanogen max growth rate
    Ks_m = params['Ks_m']         # Methanogen half-saturation
    Y_a = params['Y_a']           # Acidogen yield
    Y_m = params['Y_m']           # Methanogen yield
    kd = params['kd']             # Decay rate
    Ki = params['Ki']             # VFA inhibition constant

    # Hydrolysis
    r_hyd = k_hyd * S

    # Acidogenesis (Monod kinetics)
    r_acid = mu_a * (S / (Ks_a + S)) * Xa

    # Methanogenesis (Monod with VFA inhibition)
    inhibition = Ki / (Ki + VFA)
    r_meth = mu_m * (VFA / (Ks_m + VFA)) * Xm * inhibition

    dSdt = -r_hyd - r_acid / Y_a
    dXa = Y_a * r_acid - kd * Xa
    dXm = Y_m * r_meth - kd * Xm
    dVFA = r_acid - r_meth / Y_m
    dCH4 = r_meth

    return [dSdt, dXa, dXm, dVFA, dCH4]

params = {
    'k_hyd': 0.25, 'mu_a': 0.5, 'Ks_a': 200,
    'mu_m': 0.2, 'Ks_m': 50, 'Y_a': 0.1,
    'Y_m': 0.05, 'kd': 0.02, 'Ki': 3000
}

y0 = [5000, 100, 50, 100, 0]  # Initial conditions
sol = solve_ivp(adm1_simplified, [0, 60], y0, args=(params,),
                t_eval=np.linspace(0, 60, 300), method='RK45')

print(f"Final substrate: {sol.y[0, -1]:.0f} mg/L")
print(f"Total CH4 produced: {sol.y[4, -1]:.0f} mg/L")

Typical Workflows

Workflow 1: Fit Growth Curves and Compare Conditions

import pandas as pd
import numpy as np

# Load multi-condition OD600 data
data = pd.read_csv('growth_data.csv')  # columns: time, condition, od600

results = []
for condition, group in data.groupby('condition'):
    time = group['time'].values
    od = group['od600'].values
    fit, popt = fit_growth_curve(time, od, model='gompertz')
    fit['condition'] = condition
    results.append(fit)
    print(f"{condition}: mu_max={fit['mu_max']['value']:.3f} h⁻¹, "
          f"lag={fit['lag']['value']:.1f} h, K={fit['K']['value']:.3f}")

Workflow 2: Simulate 3-Species Lotka-Volterra Community

import numpy as np
from scipy.integrate import solve_ivp

r = np.array([0.5, 0.4, 0.3])
K = np.array([1000, 800, 600])
alpha = np.array([[1.0, 0.5, 0.1], [0.3, 1.0, 0.4], [0.2, 0.6, 1.0]])
N0 = [10, 10, 10]

sol = solve_ivp(lotka_volterra, [0, 200], N0, args=(r, K, alpha),
                t_eval=np.linspace(0, 200, 1000), method='RK45')

for i in range(3):
    print(f"Species {i+1}: equilibrium = {sol.y[i, -1]:.0f}")

Workflow 3: Annotate Bacterial Genome and Extract Statistics

prefix = run_prokka('assembly.fasta', 'prokka_output', genus='Escherichia', species='coli')
genes_df = parse_prokka_gff(f'{prefix}.gff')
print(f"\nCDS count: {len(genes_df[genes_df['type'] == 'CDS'])}")
print(f"tRNA count: {len(genes_df[genes_df['type'] == 'tRNA'])}")
print(f"rRNA count: {len(genes_df[genes_df['type'] == 'rRNA'])}")

Best Practices

1. Growth curve replicates — fit each replicate individually, then report mean ± SEM of parameters; do not average curves before fitting
2. Model selection — compare logistic, Gompertz, and Baranyi by AIC/BIC; Baranyi is most mechanistically justified but needs more data points during lag phase
3. Gillespie SSA — run sufficient ensemble size (>100 trajectories) for reliable statistics; check that propensities remain finite
4. CFU statistics — count plates with 30-300 colonies only; below 30 is unreliable, above 300 is too dense
5. Biofilm normalization — normalize to planktonic growth (OD600) to distinguish biofilm-specific effects from growth differences
6. ODE integration — use RK45 for non-stiff systems, BDF or Radau for stiff systems (common in multi-species models)

Troubleshooting

Problem: Growth curve fit fails to converge Solution: Adjust initial parameter guesses closer to expected values. Increase maxfev. Check that data has sufficient points during lag and exponential phases.

Problem: Lotka-Volterra simulation diverges Solution: Reduce step size or use adaptive solver. Check that interaction matrix doesn't produce negative populations — use events parameter in solve_ivp to stop at zero.

Problem: Gillespie SSA runs too slowly Solution: For large populations (>10000), switch to tau-leaping approximation. Or use ODE mean-field approximation and add noise analytically.

Problem: Prokka fails with "no genes found" Solution: Check FASTA file is properly formatted (no extra whitespace). Verify sequences are bacterial. Use --kingdom Bacteria flag explicitly.

Resources

• Baranyi Growth Model Paper
• Prokka Documentation
• scipy.integrate.solve_ivp
• Gillespie Algorithm Tutorial

Molecular Cloning: Sequence Engineering & Cloning Design

Overview

Molecular Cloning provides computational tools for simulating and designing molecular cloning workflows. This skill covers PCR amplicon prediction with primer binding analysis, restriction enzyme digestion simulation, Golden Gate and Gibson assembly design and verification, primer design with thermodynamic calculations, CRISPR sgRNA design with off-target scoring, and plasmid feature annotation. All simulations use Biopython's Bio.Restriction and Bio.SeqUtils for accurate enzyme and sequence handling.

When to Use This Skill

• Predicting PCR amplicons from primer sequences and templates
• Simulating restriction enzyme digestions and predicting fragment sizes
• Designing Golden Gate assembly with 4bp overhang compatibility
• Planning Gibson assembly with overlap design
• Designing PCR primers with Tm and specificity constraints
• Designing CRISPR sgRNAs and scoring off-target potential
• Annotating plasmid features (promoters, CDS, terminators, origins)
• Generating plasmid maps in GenBank format

Related Skills: For protein-level sequence analysis use biopython or esm. For gene/transcript lookups use gene-database or ensembl-database. For synthetic biology circuit design use synthetic-biology.

Installation

uv pip install biopython primer3-py numpy

Quick Start

from Bio.Seq import Seq
from Bio.Restriction import BamHI, EcoRI
from Bio.SeqUtils import MeltingTemp as mt

# Restriction digestion
sequence = Seq("ATCGATCGGGATCCATCGATCGAATTCATCGATCG")
print(f"BamHI cuts at: {BamHI.search(sequence)}")
print(f"EcoRI cuts at: {EcoRI.search(sequence)}")

# Primer Tm calculation
primer = Seq("ATCGATCGGATCCATCGATCG")
tm = mt.Tm_NN(primer)
print(f"Primer Tm: {tm:.1f} C")

Core Capabilities

1. PCR Simulation

Predict amplicon from primer binding on template.

from Bio.Seq import Seq
from Bio.SeqUtils import MeltingTemp as mt
import re

def find_primer_binding(template, primer, max_mismatches=2):
    """Find primer binding sites on template (both strands).

    Returns list of (position, strand, mismatches) tuples.
    """
    template_str = str(template).upper()
    primer_str = str(primer).upper()
    rc_template = str(template.reverse_complement()).upper()

    sites = []

    # Search forward strand
    for i in range(len(template_str) - len(primer_str) + 1):
        region = template_str[i:i+len(primer_str)]
        mismatches = sum(a != b for a, b in zip(primer_str, region))
        if mismatches <= max_mismatches:
            sites.append((i, '+', mismatches))

    # Search reverse strand
    for i in range(len(rc_template) - len(primer_str) + 1):
        region = rc_template[i:i+len(primer_str)]
        mismatches = sum(a != b for a, b in zip(primer_str, region))
        if mismatches <= max_mismatches:
            pos = len(template_str) - i - len(primer_str)
            sites.append((pos, '-', mismatches))

    return sites

def simulate_pcr(template, fwd_primer, rev_primer, max_mismatches=2):
    """Simulate PCR and predict amplicon.

    Args:
        template: Bio.Seq template sequence (can be circular)
        fwd_primer: forward primer sequence
        rev_primer: reverse primer sequence (as ordered, 5'->3')
    """
    fwd_sites = find_primer_binding(template, fwd_primer, max_mismatches)
    rev_rc = Seq(str(rev_primer)).reverse_complement()
    rev_sites = find_primer_binding(template, rev_rc, max_mismatches)

    amplicons = []
    for f_pos, f_strand, f_mm in fwd_sites:
        if f_strand != '+':
            continue
        for r_pos, r_strand, r_mm in rev_sites:
            if r_strand != '-':
                continue
            if r_pos > f_pos:
                amp_len = r_pos + len(str(rev_primer)) - f_pos
                if 50 < amp_len < 10000:  # Reasonable amplicon size
                    amplicon = template[f_pos:r_pos + len(str(rev_primer))]
                    amplicons.append({
                        'start': f_pos,
                        'end': r_pos + len(str(rev_primer)),
                        'length': amp_len,
                        'fwd_mismatches': f_mm,
                        'rev_mismatches': r_mm,
                        'sequence': str(amplicon)
                    })

    # Calculate primer Tm
    fwd_tm = mt.Tm_NN(fwd_primer)
    rev_tm = mt.Tm_NN(rev_primer)

    print(f"Forward primer Tm: {fwd_tm:.1f} C")
    print(f"Reverse primer Tm: {rev_tm:.1f} C")
    print(f"Tm difference: {abs(fwd_tm - rev_tm):.1f} C")
    print(f"Predicted amplicons: {len(amplicons)}")

    for i, amp in enumerate(amplicons):
        print(f"  Amplicon {i+1}: {amp['length']} bp "
              f"(pos {amp['start']}-{amp['end']}, "
              f"mismatches: fwd={amp['fwd_mismatches']}, rev={amp['rev_mismatches']})")

    return amplicons

2. Restriction Digestion

Simulate enzyme digestion and predict fragments.

from Bio.Seq import Seq
from Bio.Restriction import *
from Bio.Restriction import RestrictionBatch, Analysis

def restriction_digest(sequence, enzymes, is_linear=True):
    """Simulate restriction enzyme digestion.

    Args:
        sequence: Bio.Seq DNA sequence
        enzymes: list of enzyme names (e.g., ['EcoRI', 'BamHI'])
        is_linear: True for linear DNA, False for circular
    """
    # Create restriction batch
    rb = RestrictionBatch()
    for enz_name in enzymes:
        rb.add(eval(enz_name))

    # Run analysis
    analysis = Analysis(rb, sequence, linear=is_linear)
    results = analysis.full()

    all_cut_sites = []
    for enzyme, sites in results.items():
        if sites:
            print(f"{enzyme}: cuts at positions {sites}")
            all_cut_sites.extend(sites)
        else:
            print(f"{enzyme}: no cut sites")

    # Calculate fragment sizes
    if not all_cut_sites:
        print(f"No cuts — single fragment: {len(sequence)} bp")
        return [len(sequence)]

    all_cut_sites = sorted(set(all_cut_sites))

    if is_linear:
        positions = [0] + all_cut_sites + [len(sequence)]
    else:
        positions = all_cut_sites

    fragments = []
    if is_linear:
        for i in range(len(positions) - 1):
            fragments.append(positions[i+1] - positions[i])
    else:
        for i in range(len(positions)):
            next_i = (i + 1) % len(positions)
            if next_i == 0:
                frag = len(sequence) - positions[i] + positions[0]
            else:
                frag = positions[next_i] - positions[i]
            fragments.append(frag)

    fragments.sort(reverse=True)
    print(f"\nFragments ({len(fragments)}): {fragments}")
    print(f"Total: {sum(fragments)} bp")

    return fragments

# Example: double digest
seq = Seq("ATCG" * 100 + "GGATCC" + "ATCG" * 50 + "GAATTC" + "ATCG" * 75)
frags = restriction_digest(seq, ['BamHI', 'EcoRI'], is_linear=True)

3. Golden Gate Assembly

Design and simulate Golden Gate cloning.

from Bio.Seq import Seq

def design_golden_gate(parts, enzyme='BsaI'):
    """Design Golden Gate assembly with verified overhang compatibility.

    Args:
        parts: list of dicts with 'name', 'sequence' keys
        enzyme: Type IIS enzyme ('BsaI' or 'BpiI')
    """
    # Standard overhangs for 4-part Golden Gate
    standard_overhangs = [
        'AATG',  # Start codon region
        'AGGT',
        'TTCG',
        'GCTT',
        'CGCT',  # After last part (return to vector)
    ]

    if len(parts) + 1 > len(standard_overhangs):
        raise ValueError(f"Too many parts ({len(parts)}) for standard overhang set")

    # Check overhang compatibility (no palindromes, no near-matches)
    overhangs_used = standard_overhangs[:len(parts) + 1]
    for i, oh in enumerate(overhangs_used):
        rc = str(Seq(oh).reverse_complement())
        if oh == rc:
            print(f"WARNING: Overhang {oh} is palindromic — may self-ligate")
        for j, oh2 in enumerate(overhangs_used):
            if i != j and oh == oh2:
                raise ValueError(f"Duplicate overhangs: position {i} and {j}")

    # Build assembly plan
    assembly = []
    for i, part in enumerate(parts):
        left_oh = overhangs_used[i]
        right_oh = overhangs_used[i + 1]

        # Part with enzyme sites added
        if enzyme == 'BsaI':
            recognition = 'GGTCTC'
            spacer = 'N'
        else:  # BpiI
            recognition = 'GAAGAC'
            spacer = 'NN'

        assembled_part = f"{recognition}{spacer}{left_oh}{part['sequence']}{right_oh}"

        assembly.append({
            'name': part['name'],
            'left_overhang': left_oh,
            'right_overhang': right_oh,
            'part_length': len(part['sequence']),
            'total_length': len(assembled_part)
        })

        print(f"Part {i+1} ({part['name']}): [{left_oh}]--{len(part['sequence'])}bp--[{right_oh}]")

    # Predict assembled product
    total_insert = sum(len(p['sequence']) for p in parts) + \
                   len(overhangs_used) * 4  # Overhangs contribute 4bp each
    print(f"\nAssembly: {len(parts)} parts")
    print(f"Total insert: ~{total_insert} bp (excluding vector)")
    print(f"Overhang set: {' -> '.join(overhangs_used)}")

    return assembly

# Example
parts = [
    {'name': 'Promoter', 'sequence': 'TTGACAATTAATCATCGGCTCG' * 5},
    {'name': 'RBS', 'sequence': 'AAGGAGATATACAT'},
    {'name': 'GFP_CDS', 'sequence': 'ATGGTGAGCAAGGGCGAG' * 40},
    {'name': 'Terminator', 'sequence': 'TTTTTTTTTTT' * 4},
]

assembly = design_golden_gate(parts)

4. Gibson Assembly

Design overlapping fragments for Gibson assembly.

from Bio.Seq import Seq
from Bio.SeqUtils import MeltingTemp as mt

def design_gibson_assembly(fragments, overlap_length=30):
    """Design Gibson assembly with overlap primers.

    Args:
        fragments: list of dicts with 'name', 'sequence' keys (in assembly order)
        overlap_length: overlap length in bp (20-40 recommended)
    """
    assembly_plan = []

    for i in range(len(fragments)):
        current = fragments[i]
        next_frag = fragments[(i + 1) % len(fragments)]

        # Forward primer: end of previous fragment + start of current
        if i == 0:
            fwd_primer = current['sequence'][:20]
        else:
            prev = fragments[i - 1]
            overlap_5 = prev['sequence'][-overlap_length:]
            fwd_primer = overlap_5 + current['sequence'][:20]

        # Reverse primer: RC of (end of current + start of next)
        overlap_3 = next_frag['sequence'][:overlap_length]
        rev_binding = str(Seq(current['sequence'][-20:]).reverse_complement())
        rev_primer = str(Seq(overlap_3).reverse_complement()) + rev_binding

        # Tm of binding region
        fwd_tm = mt.Tm_NN(Seq(current['sequence'][:20]))
        rev_tm = mt.Tm_NN(Seq(current['sequence'][-20:]))

        assembly_plan.append({
            'fragment': current['name'],
            'fragment_length': len(current['sequence']),
            'fwd_primer': fwd_primer,
            'rev_primer': rev_primer,
            'fwd_tm': fwd_tm,
            'rev_tm': rev_tm,
            'overlap_length': overlap_length
        })

        print(f"Fragment {i+1} ({current['name']}): {len(current['sequence'])} bp")
        print(f"  Fwd primer: {fwd_primer[:40]}... ({len(fwd_primer)} nt, Tm={fwd_tm:.1f} C)")
        print(f"  Rev primer: {rev_primer[:40]}... ({len(rev_primer)} nt, Tm={rev_tm:.1f} C)")

    total = sum(len(f['sequence']) for f in fragments)
    print(f"\nTotal assembled length: ~{total} bp")

    return assembly_plan

5. Primer Design

Design primers with thermodynamic constraints.

import primer3

def design_primers(template_seq, target_start, target_length,
                   product_size_range=(200, 500), tm_target=60):
    """Design PCR primers using primer3.

    Args:
        template_seq: template DNA sequence (string)
        target_start: start position of target region
        target_length: length of target region
        product_size_range: (min, max) product size
        tm_target: target melting temperature
    """
    result = primer3.bindings.design_primers(
        seq_args={
            'SEQUENCE_TEMPLATE': template_seq,
            'SEQUENCE_TARGET': [target_start, target_length],
        },
        global_args={
            'PRIMER_NUM_RETURN': 5,
            'PRIMER_OPT_SIZE': 20,
            'PRIMER_MIN_SIZE': 18,
            'PRIMER_MAX_SIZE': 25,
            'PRIMER_OPT_TM': tm_target,
            'PRIMER_MIN_TM': tm_target - 5,
            'PRIMER_MAX_TM': tm_target + 5,
            'PRIMER_MIN_GC': 40,
            'PRIMER_MAX_GC': 60,
            'PRIMER_PRODUCT_SIZE_RANGE': [list(product_size_range)],
            'PRIMER_MAX_SELF_COMPLEMENT': 6,
            'PRIMER_MAX_SELF_END': 3,
            'PRIMER_MAX_HAIRPIN_TH': 47,
        }
    )

    primers = []
    for i in range(result.get('PRIMER_PAIR_NUM_RETURNED', 0)):
        pair = {
            'left_seq': result[f'PRIMER_LEFT_{i}_SEQUENCE'],
            'right_seq': result[f'PRIMER_RIGHT_{i}_SEQUENCE'],
            'left_tm': result[f'PRIMER_LEFT_{i}_TM'],
            'right_tm': result[f'PRIMER_RIGHT_{i}_TM'],
            'left_gc': result[f'PRIMER_LEFT_{i}_GC_PERCENT'],
            'right_gc': result[f'PRIMER_RIGHT_{i}_GC_PERCENT'],
            'product_size': result[f'PRIMER_PAIR_{i}_PRODUCT_SIZE'],
            'penalty': result[f'PRIMER_PAIR_{i}_PENALTY'],
        }
        primers.append(pair)
        print(f"Pair {i+1}: product={pair['product_size']}bp, penalty={pair['penalty']:.2f}")
        print(f"  Fwd: {pair['left_seq']} (Tm={pair['left_tm']:.1f}, GC={pair['left_gc']:.0f}%)")
        print(f"  Rev: {pair['right_seq']} (Tm={pair['right_tm']:.1f}, GC={pair['right_gc']:.0f}%)")

    return primers

6. CRISPR sgRNA Design

Design guide RNAs for CRISPR-Cas9 knockout.

from Bio.Seq import Seq
import re

def design_crispr_guides(target_seq, pam='NGG', guide_length=20, top_n=10):
    """Design CRISPR sgRNAs for SpCas9.

    Args:
        target_seq: target gene/region sequence (string)
        pam: PAM sequence ('NGG' for SpCas9)
        guide_length: guide RNA length (typically 20)
        top_n: number of top guides to return
    """
    seq = str(target_seq).upper()
    rc_seq = str(Seq(seq).reverse_complement())

    guides = []

    # Search forward strand for PAM (NGG at 3' end of target)
    for i in range(len(seq) - guide_length - len(pam) + 1):
        pam_site = seq[i + guide_length:i + guide_length + 3]
        if re.match(pam.replace('N', '.'), pam_site):
            guide_seq = seq[i:i + guide_length]
            score = score_guide(guide_seq)
            guides.append({
                'sequence': guide_seq,
                'pam': pam_site,
                'position': i,
                'strand': '+',
                'score': score,
                'full_target': guide_seq + pam_site
            })

    # Search reverse strand
    for i in range(len(rc_seq) - guide_length - len(pam) + 1):
        pam_site = rc_seq[i + guide_length:i + guide_length + 3]
        if re.match(pam.replace('N', '.'), pam_site):
            guide_seq = rc_seq[i:i + guide_length]
            score = score_guide(guide_seq)
            guides.append({
                'sequence': guide_seq,
                'pam': pam_site,
                'position': len(seq) - i - guide_length,
                'strand': '-',
                'score': score,
                'full_target': guide_seq + pam_site
            })

    # Sort by score (higher is better)
    guides.sort(key=lambda x: x['score'], reverse=True)

    print(f"Found {len(guides)} potential guide RNAs")
    print(f"\nTop {min(top_n, len(guides))} guides:")
    for i, g in enumerate(guides[:top_n]):
        print(f"  {i+1}. {g['sequence']} {g['pam']} "
              f"(strand={g['strand']}, pos={g['position']}, score={g['score']:.2f})")

    return guides[:top_n]

def score_guide(guide_seq):
    """Heuristic guide scoring based on known design rules."""
    score = 50.0  # Base score

    # GC content (40-70% preferred)
    gc = (guide_seq.count('G') + guide_seq.count('C')) / len(guide_seq)
    if 0.4 <= gc <= 0.7:
        score += 10
    elif gc < 0.3 or gc > 0.8:
        score -= 20

    # Avoid poly-T (>4 consecutive T = pol III terminator)
    if 'TTTT' in guide_seq:
        score -= 30

    # G at position 20 (adjacent to PAM) preferred
    if guide_seq[-1] == 'G':
        score += 5

    # Avoid GG at position 19-20 (can cause off-target)
    if guide_seq[-2:] == 'GG':
        score -= 5

    # Self-complementarity penalty
    rc = str(Seq(guide_seq).reverse_complement())
    matches = sum(a == b for a, b in zip(guide_seq, rc))
    if matches > 12:
        score -= 15

    return max(score, 0)

7. Plasmid Annotation

Identify and annotate features in plasmid sequences.

from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
from Bio.SeqFeature import SeqFeature, FeatureLocation
from Bio import SeqIO
from Bio.Restriction import RestrictionBatch, Analysis, CommOnly

def annotate_plasmid(sequence, name='plasmid'):
    """Annotate plasmid features and restriction sites.

    Args:
        sequence: plasmid DNA sequence (string)
        name: plasmid name
    """
    seq = Seq(sequence)
    record = SeqRecord(seq, id=name, name=name,
                       description=f'{name} annotated plasmid',
                       annotations={'molecule_type': 'DNA', 'topology': 'circular'})

    # Common feature patterns
    feature_patterns = {
        'T7_promoter': 'TAATACGACTCACTATAG',
        'lac_operator': 'AATTGTGAGCGGATAACAATT',
        'RBS_consensus': 'AAGGAG',
        'T7_terminator': 'CTAGCATAACCCCTTGGGGCCTCTAAACGGGTCTTGAGG',
        'ColE1_origin': 'CCTGTTTTGGCGGATGAGAGAAG',
    }

    for feat_name, pattern in feature_patterns.items():
        pos = str(seq).find(pattern)
        if pos >= 0:
            record.features.append(SeqFeature(
                FeatureLocation(pos, pos + len(pattern)),
                type='misc_feature',
                qualifiers={'label': feat_name}
            ))
            print(f"Found {feat_name} at position {pos}")

    # Find ORFs (>300bp)
    for strand, nuc in [(1, seq), (-1, seq.reverse_complement())]:
        for frame in range(3):
            trans = str(nuc[frame:].translate())
            start = 0
            while True:
                start = trans.find('M', start)
                if start == -1:
                    break
                stop = trans.find('*', start)
                if stop == -1:
                    stop = len(trans)
                orf_len = (stop - start) * 3

                if orf_len >= 300:
                    if strand == 1:
                        dna_start = frame + start * 3
                        dna_end = frame + stop * 3 + 3
                    else:
                        dna_end = len(seq) - frame - start * 3
                        dna_start = len(seq) - frame - stop * 3 - 3

                    record.features.append(SeqFeature(
                        FeatureLocation(min(dna_start, dna_end),
                                       max(dna_start, dna_end), strand),
                        type='CDS',
                        qualifiers={'label': f'ORF_{orf_len}bp'}
                    ))
                    print(f"ORF: {orf_len}bp at {dna_start}-{dna_end} (strand {'+' if strand==1 else '-'})")

                start = stop + 1

    # Map restriction sites
    rb = CommOnly
    analysis = Analysis(rb, seq, linear=False)
    unique_sites = {str(enz): sites for enz, sites in analysis.full().items()
                    if len(sites) == 1}

    print(f"\nUnique restriction sites: {len(unique_sites)}")
    for enz, sites in sorted(unique_sites.items()):
        print(f"  {enz}: {sites[0]}")

    return record

# Write annotated plasmid
# record = annotate_plasmid(plasmid_seq, name='pMyPlasmid')
# SeqIO.write(record, 'pMyPlasmid.gb', 'genbank')

Typical Workflows

Workflow 1: Design PCR Primers and Predict Amplicon

from Bio.Seq import Seq

template = Seq("ATCGATCGATCGATCGATCGATCGATCG" * 50)  # 1400 bp template
fwd = Seq("ATCGATCGATCGATCGATCG")
rev = Seq("CGATCGATCGATCGATCGAT")

amplicons = simulate_pcr(template, fwd, rev)
if amplicons:
    print(f"Amplicon size: {amplicons[0]['length']} bp")

Workflow 2: Plan Golden Gate Assembly with 4 Parts

parts = [
    {'name': 'J23100_promoter', 'sequence': 'TTGACGGCTAGCTCAGTCCTAGGTACAGTGCTAGC'},
    {'name': 'B0034_RBS', 'sequence': 'AAAGAGGAGAAA'},
    {'name': 'GFP', 'sequence': 'ATGGTGAGCAAGGGCGAGGAG' + 'NNN' * 230 + 'TAA'},
    {'name': 'B0015_terminator', 'sequence': 'CCAGGCATCAAATAAAACGAAAGGCTCAGTCGAAAG'},
]
assembly = design_golden_gate(parts)

Workflow 3: Design CRISPR Knockout sgRNAs

# Target: exon 3 of a gene
target_exon = "ATGCGATCGATCGATCGATCGATCGAGGCGATCGATCGATCGATCGATCGATCGATCGATCG"
guides = design_crispr_guides(target_exon, pam='NGG', top_n=5)

Best Practices

1. Primer design — aim for Tm within 2 C between forward and reverse primers; 40-60% GC content; avoid 3' complementarity
2. Golden Gate — verify 4bp overhangs are unique and non-palindromic; use standardized overhang sets from published studies
3. Gibson assembly — use 20-40bp overlaps; ensure fragments have similar Tm at overlap junctions; minimum 2 fragments
4. CRISPR guides — avoid poly-T (>4 T) which terminates U6/H1 transcription; prefer guides in early exons; validate against off-target databases
5. Restriction digestion — always check both orientations; verify compatible cohesive ends for ligation; methylation sensitivity can block cutting
6. Plasmid annotation — verify ORFs are in correct reading frame; check for cryptic promoters; map all unique restriction sites for future cloning

Troubleshooting

Problem: PCR simulation finds no amplicons Solution: Check primer orientation (forward should match sense strand 5'->3'). Increase max_mismatches. Verify template contains the target region.

Problem: Restriction enzyme doesn't cut expected site Solution: Check for CpG methylation sensitivity (dam/dcm methylation). Verify site isn't in a modified context. Use isoschizomers() to find alternatives.

Problem: Golden Gate assembly produces wrong product Solution: Verify all overhangs are unique. Check enzyme is BsaI (not BsmBI) — different cut distances. Ensure parts are in correct orientation.

Problem: primer3 returns no primers Solution: Relax constraints: widen Tm range, increase max size, lower GC requirement. Ensure target region has sufficient flanking sequence.

Problem: CRISPR guide design returns few candidates Solution: Expand search region (use full gene, not just one exon). Try alternative PAM sequences (NAG for relaxed SpCas9). Consider Cas12a (TTTV PAM) for AT-rich regions.

Resources

• Biopython Restriction Module
• primer3-py Documentation
• CRISPR Design Rules
• Golden Gate Assembly (Engler et al.)
• Gibson Assembly (Gibson et al.)

NeuroKit2

Overview

NeuroKit2 is a comprehensive Python toolkit for processing and analyzing physiological signals (biosignals). Use this skill to process cardiovascular, neural, autonomic, respiratory, and muscular signals for psychophysiology research, clinical applications, and human-computer interaction studies.

When to Use This Skill

Apply this skill when working with:

• Cardiac signals: ECG, PPG, heart rate variability (HRV), pulse analysis
• Brain signals: EEG frequency bands, microstates, complexity, source localization
• Autonomic signals: Electrodermal activity (EDA/GSR), skin conductance responses (SCR)
• Respiratory signals: Breathing rate, respiratory variability (RRV), volume per time
• Muscular signals: EMG amplitude, muscle activation detection
• Eye tracking: EOG, blink detection and analysis
• Multi-modal integration: Processing multiple physiological signals simultaneously
• Complexity analysis: Entropy measures, fractal dimensions, nonlinear dynamics

Core Capabilities

1. Cardiac Signal Processing (ECG/PPG)

Process electrocardiogram and photoplethysmography signals for cardiovascular analysis. See references/ecg_cardiac.md for detailed workflows.

Primary workflows:

• ECG processing pipeline: cleaning → R-peak detection → delineation → quality assessment
• HRV analysis across time, frequency, and nonlinear domains
• PPG pulse analysis and quality assessment
• ECG-derived respiration extraction

Key functions:

import neurokit2 as nk

# Complete ECG processing pipeline
signals, info = nk.ecg_process(ecg_signal, sampling_rate=1000)

# Analyze ECG data (event-related or interval-related)
analysis = nk.ecg_analyze(signals, sampling_rate=1000)

# Comprehensive HRV analysis
hrv = nk.hrv(peaks, sampling_rate=1000)  # Time, frequency, nonlinear domains

2. Heart Rate Variability Analysis

Compute comprehensive HRV metrics from cardiac signals. See references/hrv.md for all indices and domain-specific analysis.

Supported domains:

• Time domain: SDNN, RMSSD, pNN50, SDSD, and derived metrics
• Frequency domain: ULF, VLF, LF, HF, VHF power and ratios
• Nonlinear domain: Poincaré plot (SD1/SD2), entropy measures, fractal dimensions
• Specialized: Respiratory sinus arrhythmia (RSA), recurrence quantification analysis (RQA)

Key functions:

# All HRV indices at once
hrv_indices = nk.hrv(peaks, sampling_rate=1000)

# Domain-specific analysis
hrv_time = nk.hrv_time(peaks)
hrv_freq = nk.hrv_frequency(peaks, sampling_rate=1000)
hrv_nonlinear = nk.hrv_nonlinear(peaks, sampling_rate=1000)
hrv_rsa = nk.hrv_rsa(peaks, rsp_signal, sampling_rate=1000)

3. Brain Signal Analysis (EEG)

Analyze electroencephalography signals for frequency power, complexity, and microstate patterns. See references/eeg.md for detailed workflows and MNE integration.

Primary capabilities:

• Frequency band power analysis (Delta, Theta, Alpha, Beta, Gamma)
• Channel quality assessment and re-referencing
• Source localization (sLORETA, MNE)
• Microstate segmentation and transition dynamics
• Global field power and dissimilarity measures

Key functions:

# Power analysis across frequency bands
power = nk.eeg_power(eeg_data, sampling_rate=250, channels=['Fz', 'Cz', 'Pz'])

# Microstate analysis
microstates = nk.microstates_segment(eeg_data, n_microstates=4, method='kmod')
static = nk.microstates_static(microstates)
dynamic = nk.microstates_dynamic(microstates)

4. Electrodermal Activity (EDA)

Process skin conductance signals for autonomic nervous system assessment. See references/eda.md for detailed workflows.

Primary workflows:

• Signal decomposition into tonic and phasic components
• Skin conductance response (SCR) detection and analysis
• Sympathetic nervous system index calculation
• Autocorrelation and changepoint detection

Key functions:

# Complete EDA processing
signals, info = nk.eda_process(eda_signal, sampling_rate=100)

# Analyze EDA data
analysis = nk.eda_analyze(signals, sampling_rate=100)

# Sympathetic nervous system activity
sympathetic = nk.eda_sympathetic(signals, sampling_rate=100)

5. Respiratory Signal Processing (RSP)

Analyze breathing patterns and respiratory variability. See references/rsp.md for detailed workflows.

Primary capabilities:

• Respiratory rate calculation and variability analysis
• Breathing amplitude and symmetry assessment
• Respiratory volume per time (fMRI applications)
• Respiratory amplitude variability (RAV)

Key functions:

# Complete RSP processing
signals, info = nk.rsp_process(rsp_signal, sampling_rate=100)

# Respiratory rate variability
rrv = nk.rsp_rrv(signals, sampling_rate=100)

# Respiratory volume per time
rvt = nk.rsp_rvt(signals, sampling_rate=100)

6. Electromyography (EMG)

Process muscle activity signals for activation detection and amplitude analysis. See references/emg.md for workflows.

Key functions:

# Complete EMG processing
signals, info = nk.emg_process(emg_signal, sampling_rate=1000)

# Muscle activation detection
activation = nk.emg_activation(signals, sampling_rate=1000, method='threshold')

7. Electrooculography (EOG)

Analyze eye movement and blink patterns. See references/eog.md for workflows.

Key functions:

# Complete EOG processing
signals, info = nk.eog_process(eog_signal, sampling_rate=500)

# Extract blink features
features = nk.eog_features(signals, sampling_rate=500)

8. General Signal Processing

Apply filtering, decomposition, and transformation operations to any signal. See references/signal_processing.md for comprehensive utilities.

Key operations:

• Filtering (lowpass, highpass, bandpass, bandstop)
• Decomposition (EMD, SSA, wavelet)
• Peak detection and correction
• Power spectral density estimation
• Signal interpolation and resampling
• Autocorrelation and synchrony analysis

Key functions:

# Filtering
filtered = nk.signal_filter(signal, sampling_rate=1000, lowcut=0.5, highcut=40)

# Peak detection
peaks = nk.signal_findpeaks(signal)

# Power spectral density
psd = nk.signal_psd(signal, sampling_rate=1000)

9. Complexity and Entropy Analysis

Compute nonlinear dynamics, fractal dimensions, and information-theoretic measures. See references/complexity.md for all available metrics.

Available measures:

• Entropy: Shannon, approximate, sample, permutation, spectral, fuzzy, multiscale
• Fractal dimensions: Katz, Higuchi, Petrosian, Sevcik, correlation dimension
• Nonlinear dynamics: Lyapunov exponents, Lempel-Ziv complexity, recurrence quantification
• DFA: Detrended fluctuation analysis, multifractal DFA
• Information theory: Fisher information, mutual information

Key functions:

# Multiple complexity metrics at once
complexity_indices = nk.complexity(signal, sampling_rate=1000)

# Specific measures
apen = nk.entropy_approximate(signal)
dfa = nk.fractal_dfa(signal)
lyap = nk.complexity_lyapunov(signal, sampling_rate=1000)

10. Event-Related Analysis

Create epochs around stimulus events and analyze physiological responses. See references/epochs_events.md for workflows.

Primary capabilities:

• Epoch creation from event markers
• Event-related averaging and visualization
• Baseline correction options
• Grand average computation with confidence intervals

Key functions:

# Find events in signal
events = nk.events_find(trigger_signal, threshold=0.5)

# Create epochs around events
epochs = nk.epochs_create(signals, events, sampling_rate=1000,
                          epochs_start=-0.5, epochs_end=2.0)

# Average across epochs
grand_average = nk.epochs_average(epochs)

11. Multi-Signal Integration

Process multiple physiological signals simultaneously with unified output. See references/bio_module.md for integration workflows.

Key functions:

# Process multiple signals at once
bio_signals, bio_info = nk.bio_process(
    ecg=ecg_signal,
    rsp=rsp_signal,
    eda=eda_signal,
    emg=emg_signal,
    sampling_rate=1000
)

# Analyze all processed signals
bio_analysis = nk.bio_analyze(bio_signals, sampling_rate=1000)

Analysis Modes

NeuroKit2 automatically selects between two analysis modes based on data duration:

Event-related analysis (< 10 seconds):

• Analyzes stimulus-locked responses
• Epoch-based segmentation
• Suitable for experimental paradigms with discrete trials

Interval-related analysis (≥ 10 seconds):

• Characterizes physiological patterns over extended periods
• Resting state or continuous activities
• Suitable for baseline measurements and long-term monitoring

Most *_analyze() functions automatically choose the appropriate mode.

Installation

uv pip install neurokit2

For development version:

uv pip install https://github.com/neuropsychology/NeuroKit/zipball/dev

Common Workflows

Quick Start: ECG Analysis

import neurokit2 as nk

# Load example data
ecg = nk.ecg_simulate(duration=60, sampling_rate=1000)

# Process ECG
signals, info = nk.ecg_process(ecg, sampling_rate=1000)

# Analyze HRV
hrv = nk.hrv(info['ECG_R_Peaks'], sampling_rate=1000)

# Visualize
nk.ecg_plot(signals, info)

Multi-Modal Analysis

# Process multiple signals
bio_signals, bio_info = nk.bio_process(
    ecg=ecg_signal,
    rsp=rsp_signal,
    eda=eda_signal,
    sampling_rate=1000
)

# Analyze all signals
results = nk.bio_analyze(bio_signals, sampling_rate=1000)

Event-Related Potential

# Find events
events = nk.events_find(trigger_channel, threshold=0.5)

# Create epochs
epochs = nk.epochs_create(processed_signals, events,
                          sampling_rate=1000,
                          epochs_start=-0.5, epochs_end=2.0)

# Event-related analysis for each signal type
ecg_epochs = nk.ecg_eventrelated(epochs)
eda_epochs = nk.eda_eventrelated(epochs)

References

This skill includes comprehensive reference documentation organized by signal type and analysis method:

• ecg_cardiac.md: ECG/PPG processing, R-peak detection, delineation, quality assessment
• hrv.md: Heart rate variability indices across all domains
• eeg.md: EEG analysis, frequency bands, microstates, source localization
• eda.md: Electrodermal activity processing and SCR analysis
• rsp.md: Respiratory signal processing and variability
• ppg.md: Photoplethysmography signal analysis
• emg.md: Electromyography processing and activation detection
• eog.md: Electrooculography and blink analysis
• signal_processing.md: General signal utilities and transformations
• complexity.md: Entropy, fractal, and nonlinear measures
• epochs_events.md: Event-related analysis and epoch creation
• bio_module.md: Multi-signal integration workflows

Load specific reference files as needed using the Read tool to access detailed function documentation and parameters.

Additional Resources

• Official Documentation: https://neuropsychology.github.io/NeuroKit/
• GitHub Repository: https://github.com/neuropsychology/NeuroKit
• Publication: Makowski et al. (2021). NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods. https://doi.org/10.3758/s13428-020-01516-y

Neuropixels Data Analysis

Overview

Comprehensive toolkit for analyzing Neuropixels high-density neural recordings using current best practices from SpikeInterface, Allen Institute, and International Brain Laboratory (IBL). Supports the full workflow from raw data to publication-ready curated units.

When to Use This Skill

This skill should be used when:

• Working with Neuropixels recordings (.ap.bin, .lf.bin, .meta files)
• Loading data from SpikeGLX, Open Ephys, or NWB formats
• Preprocessing neural recordings (filtering, CAR, bad channel detection)
• Detecting and correcting motion/drift in recordings
• Running spike sorting (Kilosort4, SpykingCircus2, Mountainsort5)
• Computing quality metrics (SNR, ISI violations, presence ratio)
• Curating units using Allen/IBL criteria
• Creating visualizations of neural data
• Exporting results to Phy or NWB

Supported Hardware & Formats

Quick Start

Basic Import and Setup

import spikeinterface.full as si
import neuropixels_analysis as npa

# Configure parallel processing
job_kwargs = dict(n_jobs=-1, chunk_duration='1s', progress_bar=True)

Loading Data

# SpikeGLX (most common)
recording = si.read_spikeglx('/path/to/data', stream_id='imec0.ap')

# Open Ephys (common for many labs)
recording = si.read_openephys('/path/to/Record_Node_101/')

# Check available streams
streams, ids = si.get_neo_streams('spikeglx', '/path/to/data')
print(streams)  # ['imec0.ap', 'imec0.lf', 'nidq']

# For testing with subset of data
recording = recording.frame_slice(0, int(60 * recording.get_sampling_frequency()))

Complete Pipeline (One Command)

# Run full analysis pipeline
results = npa.run_pipeline(
    recording,
    output_dir='output/',
    sorter='kilosort4',
    curation_method='allen',
)

# Access results
sorting = results['sorting']
metrics = results['metrics']
labels = results['labels']

Standard Analysis Workflow

1. Preprocessing

# Recommended preprocessing chain
rec = si.highpass_filter(recording, freq_min=400)
rec = si.phase_shift(rec)  # Required for Neuropixels 1.0
bad_ids, _ = si.detect_bad_channels(rec)
rec = rec.remove_channels(bad_ids)
rec = si.common_reference(rec, operator='median')

# Or use our wrapper
rec = npa.preprocess(recording)

2. Check and Correct Drift

# Check for drift (always do this!)
motion_info = npa.estimate_motion(rec, preset='kilosort_like')
npa.plot_drift(rec, motion_info, output='drift_map.png')

# Apply correction if needed
if motion_info['motion'].max() > 10:  # microns
    rec = npa.correct_motion(rec, preset='nonrigid_accurate')

3. Spike Sorting

# Kilosort4 (recommended, requires GPU)
sorting = si.run_sorter('kilosort4', rec, folder='ks4_output')

# CPU alternatives
sorting = si.run_sorter('tridesclous2', rec, folder='tdc2_output')
sorting = si.run_sorter('spykingcircus2', rec, folder='sc2_output')
sorting = si.run_sorter('mountainsort5', rec, folder='ms5_output')

# Check available sorters
print(si.installed_sorters())

4. Postprocessing

# Create analyzer and compute all extensions
analyzer = si.create_sorting_analyzer(sorting, rec, sparse=True)

analyzer.compute('random_spikes', max_spikes_per_unit=500)
analyzer.compute('waveforms', ms_before=1.0, ms_after=2.0)
analyzer.compute('templates', operators=['average', 'std'])
analyzer.compute('spike_amplitudes')
analyzer.compute('correlograms', window_ms=50.0, bin_ms=1.0)
analyzer.compute('unit_locations', method='monopolar_triangulation')
analyzer.compute('quality_metrics')

metrics = analyzer.get_extension('quality_metrics').get_data()

5. Curation

# Allen Institute criteria (conservative)
good_units = metrics.query("""
    presence_ratio > 0.9 and
    isi_violations_ratio < 0.5 and
    amplitude_cutoff < 0.1
""").index.tolist()

# Or use automated curation
labels = npa.curate(metrics, method='allen')  # 'allen', 'ibl', 'strict'

6. AI-Assisted Curation (For Uncertain Units)

When using this skill with Claude Code, Claude can directly analyze waveform plots and provide expert curation decisions. For programmatic API access:

from anthropic import Anthropic

# Setup API client
client = Anthropic()

# Analyze uncertain units visually
uncertain = metrics.query('snr > 3 and snr < 8').index.tolist()

for unit_id in uncertain:
    result = npa.analyze_unit_visually(analyzer, unit_id, api_client=client)
    print(f"Unit {unit_id}: {result['classification']}")
    print(f"  Reasoning: {result['reasoning'][:100]}...")

Claude Code Integration: When running within Claude Code, ask Claude to examine waveform/correlogram plots directly - no API setup required.

7. Generate Analysis Report

# Generate comprehensive HTML report with visualizations
report_dir = npa.generate_analysis_report(results, 'output/')
# Opens report.html with summary stats, figures, and unit table

# Print formatted summary to console
npa.print_analysis_summary(results)

8. Export Results

# Export to Phy for manual review
si.export_to_phy(analyzer, output_folder='phy_export/',
                 compute_pc_features=True, compute_amplitudes=True)

# Export to NWB
from spikeinterface.exporters import export_to_nwb
export_to_nwb(rec, sorting, 'output.nwb')

# Save quality metrics
metrics.to_csv('quality_metrics.csv')

Common Pitfalls and Best Practices

1. Always check drift before spike sorting - drift > 10μm significantly impacts quality
2. Use phase_shift for Neuropixels 1.0 probes (not needed for 2.0)
3. Save preprocessed data to avoid recomputing - use rec.save(folder='preprocessed/')
4. Use GPU for Kilosort4 - it's 10-50x faster than CPU alternatives
5. Review uncertain units manually - automated curation is a starting point
6. Combine metrics with AI - use metrics for clear cases, AI for borderline units
7. Document your thresholds - different analyses may need different criteria
8. Export to Phy for critical experiments - human oversight is valuable

Key Parameters to Adjust

Preprocessing

• freq_min: Highpass cutoff (300-400 Hz typical)
• detect_threshold: Bad channel detection sensitivity

Motion Correction

• preset: 'kilosort_like' (fast) or 'nonrigid_accurate' (better for severe drift)

Spike Sorting (Kilosort4)

• batch_size: Samples per batch (30000 default)
• nblocks: Number of drift blocks (increase for long recordings)
• Th_learned: Detection threshold (lower = more spikes)

Quality Metrics

• snr_threshold: Signal-to-noise cutoff (3-5 typical)
• isi_violations_ratio: Refractory violations (0.01-0.5)
• presence_ratio: Recording coverage (0.5-0.95)

Bundled Resources

scripts/preprocess_recording.py

Automated preprocessing script:

python scripts/preprocess_recording.py /path/to/data --output preprocessed/

scripts/run_sorting.py

Run spike sorting:

python scripts/run_sorting.py preprocessed/ --sorter kilosort4 --output sorting/

scripts/compute_metrics.py

Compute quality metrics and apply curation:

python scripts/compute_metrics.py sorting/ preprocessed/ --output metrics/ --curation allen

scripts/export_to_phy.py

Export to Phy for manual curation:

python scripts/export_to_phy.py metrics/analyzer --output phy_export/

assets/analysis_template.py

Complete analysis template. Copy and customize:

cp assets/analysis_template.py my_analysis.py
# Edit parameters and run
python my_analysis.py

reference/standard_workflow.md

Detailed step-by-step workflow with explanations for each stage.

reference/api_reference.md

Quick function reference organized by module.

reference/plotting_guide.md

Comprehensive visualization guide for publication-quality figures.

Detailed Reference Guides

Installation

# Core packages
pip install spikeinterface[full] probeinterface neo

# Spike sorters
pip install kilosort          # Kilosort4 (GPU required)
pip install spykingcircus     # SpykingCircus2 (CPU)
pip install mountainsort5     # Mountainsort5 (CPU)

# Our toolkit
pip install neuropixels-analysis

# Optional: AI curation
pip install anthropic

# Optional: IBL tools
pip install ibl-neuropixel ibllib

Project Structure

project/
├── raw_data/
│   └── recording_g0/
│       └── recording_g0_imec0/
│           ├── recording_g0_t0.imec0.ap.bin
│           └── recording_g0_t0.imec0.ap.meta
├── preprocessed/           # Saved preprocessed recording
├── motion/                 # Motion estimation results
├── sorting_output/         # Spike sorter output
├── analyzer/               # SortingAnalyzer (waveforms, metrics)
├── phy_export/             # For manual curation
├── ai_curation/            # AI analysis reports
└── results/
    ├── quality_metrics.csv
    ├── curation_labels.json
    └── output.nwb

Additional Resources

• SpikeInterface Docs: https://spikeinterface.readthedocs.io/
• Neuropixels Tutorial: https://spikeinterface.readthedocs.io/en/stable/how_to/analyze_neuropixels.html
• Kilosort4 GitHub: https://github.com/MouseLand/Kilosort
• IBL Neuropixel Tools: https://github.com/int-brain-lab/ibl-neuropixel
• Allen Institute ecephys: https://github.com/AllenInstitute/ecephys_spike_sorting
• Bombcell (Automated QC): https://github.com/Julie-Fabre/bombcell
• SpikeAgent (AI Curation): https://github.com/SpikeAgent/SpikeAgent

OMERO Integration

Overview

OMERO is an open-source platform for managing, visualizing, and analyzing microscopy images and metadata. Access images via Python API, retrieve datasets, analyze pixels, manage ROIs and annotations, for high-content screening and microscopy workflows.

When to Use This Skill

This skill should be used when:

• Working with OMERO Python API (omero-py) to access microscopy data
• Retrieving images, datasets, projects, or screening data programmatically
• Analyzing pixel data and creating derived images
• Creating or managing ROIs (regions of interest) on microscopy images
• Adding annotations, tags, or metadata to OMERO objects
• Storing measurement results in OMERO tables
• Creating server-side scripts for batch processing
• Performing high-content screening analysis

Core Capabilities

This skill covers eight major capability areas. Each is documented in detail in the references/ directory:

1. Connection & Session Management

File: references/connection.md

Establish secure connections to OMERO servers, manage sessions, handle authentication, and work with group contexts. Use this for initial setup and connection patterns.

Common scenarios:

• Connect to OMERO server with credentials
• Use existing session IDs
• Switch between group contexts
• Manage connection lifecycle with context managers

2. Data Access & Retrieval

File: references/data_access.md

Navigate OMERO's hierarchical data structure (Projects → Datasets → Images) and screening data (Screens → Plates → Wells). Retrieve objects, query by attributes, and access metadata.

Common scenarios:

• List all projects and datasets for a user
• Retrieve images by ID or dataset
• Access screening plate data
• Query objects with filters

3. Metadata & Annotations

File: references/metadata.md

Create and manage annotations including tags, key-value pairs, file attachments, and comments. Link annotations to images, datasets, or other objects.

Common scenarios:

• Add tags to images
• Attach analysis results as files
• Create custom key-value metadata
• Query annotations by namespace

4. Image Processing & Rendering

File: references/image_processing.md

Access raw pixel data as NumPy arrays, manipulate rendering settings, create derived images, and manage physical dimensions.

Common scenarios:

• Extract pixel data for computational analysis
• Generate thumbnail images
• Create maximum intensity projections
• Modify channel rendering settings

5. Regions of Interest (ROIs)

File: references/rois.md

Create, retrieve, and analyze ROIs with various shapes (rectangles, ellipses, polygons, masks, points, lines). Extract intensity statistics from ROI regions.

Common scenarios:

• Draw rectangular ROIs on images
• Create polygon masks for segmentation
• Analyze pixel intensities within ROIs
• Export ROI coordinates

6. OMERO Tables

File: references/tables.md

Store and query structured tabular data associated with OMERO objects. Useful for analysis results, measurements, and metadata.

Common scenarios:

• Store quantitative measurements for images
• Create tables with multiple column types
• Query table data with conditions
• Link tables to specific images or datasets

7. Scripts & Batch Operations

File: references/scripts.md

Create OMERO.scripts that run server-side for batch processing, automated workflows, and integration with OMERO clients.

Common scenarios:

• Process multiple images in batch
• Create automated analysis pipelines
• Generate summary statistics across datasets
• Export data in custom formats

8. Advanced Features

File: references/advanced.md

Covers permissions, filesets, cross-group queries, delete operations, and other advanced functionality.

Common scenarios:

• Handle group permissions
• Access original imported files
• Perform cross-group queries
• Delete objects with callbacks

Installation

uv pip install omero-py

Requirements:

• Python 3.7+
• Zeroc Ice 3.6+
• Access to an OMERO server (host, port, credentials)

Quick Start

Basic connection pattern:

from omero.gateway import BlitzGateway

# Connect to OMERO server
conn = BlitzGateway(username, password, host=host, port=port)
connected = conn.connect()

if connected:
    # Perform operations
    for project in conn.listProjects():
        print(project.getName())

    # Always close connection
    conn.close()
else:
    print("Connection failed")

Recommended pattern with context manager:

from omero.gateway import BlitzGateway

with BlitzGateway(username, password, host=host, port=port) as conn:
    # Connection automatically managed
    for project in conn.listProjects():
        print(project.getName())
    # Automatically closed on exit

Selecting the Right Capability

For data exploration:

• Start with references/connection.md to establish connection
• Use references/data_access.md to navigate hierarchy
• Check references/metadata.md for annotation details

For image analysis:

• Use references/image_processing.md for pixel data access
• Use references/rois.md for region-based analysis
• Use references/tables.md to store results

For automation:

• Use references/scripts.md for server-side processing
• Use references/data_access.md for batch data retrieval

For advanced operations:

• Use references/advanced.md for permissions and deletion
• Check references/connection.md for cross-group queries

Common Workflows

Workflow 1: Retrieve and Analyze Images

1. Connect to OMERO server (references/connection.md)
2. Navigate to dataset (references/data_access.md)
3. Retrieve images from dataset (references/data_access.md)
4. Access pixel data as NumPy array (references/image_processing.md)
5. Perform analysis
6. Store results as table or file annotation (references/tables.md or references/metadata.md)

Workflow 2: Batch ROI Analysis

1. Connect to OMERO server
2. Retrieve images with existing ROIs (references/rois.md)
3. For each image, get ROI shapes
4. Extract pixel intensities within ROIs (references/rois.md)
5. Store measurements in OMERO table (references/tables.md)

Workflow 3: Create Analysis Script

1. Design analysis workflow
2. Use OMERO.scripts framework (references/scripts.md)
3. Access data through script parameters
4. Process images in batch
5. Generate outputs (new images, tables, files)

Error Handling

Always wrap OMERO operations in try-except blocks and ensure connections are properly closed:

from omero.gateway import BlitzGateway
import traceback

try:
    conn = BlitzGateway(username, password, host=host, port=port)
    if not conn.connect():
        raise Exception("Connection failed")

    # Perform operations

except Exception as e:
    print(f"Error: {e}")
    traceback.print_exc()
finally:
    if conn:
        conn.close()

Additional Resources

• Official Documentation: https://omero.readthedocs.io/en/stable/developers/Python.html
• BlitzGateway API: https://omero.readthedocs.io/en/stable/developers/Python.html#omero-blitzgateway
• OMERO Model: https://omero.readthedocs.io/en/stable/developers/Model.html
• Community Forum: https://forum.image.sc/tag/omero

Notes

• OMERO uses group-based permissions (READ-ONLY, READ-ANNOTATE, READ-WRITE)
• Images in OMERO are organized hierarchically: Project > Dataset > Image
• Screening data uses: Screen > Plate > Well > WellSample > Image
• Always close connections to free server resources
• Use context managers for automatic resource management
• Pixel data is returned as NumPy arrays for analysis

Opentrons Integration

Overview

Opentrons is a Python-based lab automation platform for Flex and OT-2 robots. Write Protocol API v2 protocols for liquid handling, control hardware modules (heater-shaker, thermocycler), manage labware, for automated pipetting workflows.

When to Use This Skill

This skill should be used when:

• Writing Opentrons Protocol API v2 protocols in Python
• Automating liquid handling workflows on Flex or OT-2 robots
• Controlling hardware modules (temperature, magnetic, heater-shaker, thermocycler)
• Setting up labware configurations and deck layouts
• Implementing complex pipetting operations (serial dilutions, plate replication, PCR setup)
• Managing tip usage and optimizing protocol efficiency
• Working with multi-channel pipettes for 96-well plate operations
• Simulating and testing protocols before robot execution

Core Capabilities

1. Protocol Structure and Metadata

Every Opentrons protocol follows a standard structure:

from opentrons import protocol_api

# Metadata
metadata = {
    'protocolName': 'My Protocol',
    'author': 'Name <email@example.com>',
    'description': 'Protocol description',
    'apiLevel': '2.19'  # Use latest available API version
}

# Requirements (optional)
requirements = {
    'robotType': 'Flex',  # or 'OT-2'
    'apiLevel': '2.19'
}

# Run function
def run(protocol: protocol_api.ProtocolContext):
    # Protocol commands go here
    pass

Key elements:

• Import protocol_api from opentrons
• Define metadata dict with protocolName, author, description, apiLevel
• Optional requirements dict for robot type and API version
• Implement run() function receiving ProtocolContext as parameter
• All protocol logic goes inside the run() function

2. Loading Hardware

Loading Instruments (Pipettes):

def run(protocol: protocol_api.ProtocolContext):
    # Load pipette on specific mount
    left_pipette = protocol.load_instrument(
        'p1000_single_flex',  # Instrument name
        'left',               # Mount: 'left' or 'right'
        tip_racks=[tip_rack]  # List of tip rack labware objects
    )

Common pipette names:

• Flex: p50_single_flex, p1000_single_flex, p50_multi_flex, p1000_multi_flex
• OT-2: p20_single_gen2, p300_single_gen2, p1000_single_gen2, p20_multi_gen2, p300_multi_gen2

Loading Labware:

# Load labware directly on deck
plate = protocol.load_labware(
    'corning_96_wellplate_360ul_flat',  # Labware API name
    'D1',                                # Deck slot (Flex: A1-D3, OT-2: 1-11)
    label='Sample Plate'                 # Optional display label
)

# Load tip rack
tip_rack = protocol.load_labware('opentrons_flex_96_tiprack_1000ul', 'C1')

# Load labware on adapter
adapter = protocol.load_adapter('opentrons_flex_96_tiprack_adapter', 'B1')
tips = adapter.load_labware('opentrons_flex_96_tiprack_200ul')

Loading Modules:

# Temperature module
temp_module = protocol.load_module('temperature module gen2', 'D3')
temp_plate = temp_module.load_labware('corning_96_wellplate_360ul_flat')

# Magnetic module
mag_module = protocol.load_module('magnetic module gen2', 'C2')
mag_plate = mag_module.load_labware('nest_96_wellplate_100ul_pcr_full_skirt')

# Heater-Shaker module
hs_module = protocol.load_module('heaterShakerModuleV1', 'D1')
hs_plate = hs_module.load_labware('corning_96_wellplate_360ul_flat')

# Thermocycler module (takes up specific slots automatically)
tc_module = protocol.load_module('thermocyclerModuleV2')
tc_plate = tc_module.load_labware('nest_96_wellplate_100ul_pcr_full_skirt')

3. Liquid Handling Operations

Basic Operations:

# Pick up tip
pipette.pick_up_tip()

# Aspirate (draw liquid in)
pipette.aspirate(
    volume=100,           # Volume in µL
    location=source['A1'] # Well or location object
)

# Dispense (expel liquid)
pipette.dispense(
    volume=100,
    location=dest['B1']
)

# Drop tip
pipette.drop_tip()

# Return tip to rack
pipette.return_tip()

Complex Operations:

# Transfer (combines pick_up, aspirate, dispense, drop_tip)
pipette.transfer(
    volume=100,
    source=source_plate['A1'],
    dest=dest_plate['B1'],
    new_tip='always'  # 'always', 'once', or 'never'
)

# Distribute (one source to multiple destinations)
pipette.distribute(
    volume=50,
    source=reservoir['A1'],
    dest=[plate['A1'], plate['A2'], plate['A3']],
    new_tip='once'
)

# Consolidate (multiple sources to one destination)
pipette.consolidate(
    volume=50,
    source=[plate['A1'], plate['A2'], plate['A3']],
    dest=reservoir['A1'],
    new_tip='once'
)

Advanced Techniques:

# Mix (aspirate and dispense in same location)
pipette.mix(
    repetitions=3,
    volume=50,
    location=plate['A1']
)

# Air gap (prevent dripping)
pipette.aspirate(100, source['A1'])
pipette.air_gap(20)  # 20µL air gap
pipette.dispense(120, dest['A1'])

# Blow out (expel remaining liquid)
pipette.blow_out(location=dest['A1'].top())

# Touch tip (remove droplets on tip exterior)
pipette.touch_tip(location=plate['A1'])

Flow Rate Control:

# Set flow rates (µL/s)
pipette.flow_rate.aspirate = 150
pipette.flow_rate.dispense = 300
pipette.flow_rate.blow_out = 400

4. Accessing Wells and Locations

Well Access Methods:

# By name
well_a1 = plate['A1']

# By index
first_well = plate.wells()[0]

# All wells
all_wells = plate.wells()  # Returns list

# By rows
rows = plate.rows()  # Returns list of lists
row_a = plate.rows()[0]  # All wells in row A

# By columns
columns = plate.columns()  # Returns list of lists
column_1 = plate.columns()[0]  # All wells in column 1

# Wells by name (dictionary)
wells_dict = plate.wells_by_name()  # {'A1': Well, 'A2': Well, ...}

Location Methods:

# Top of well (default: 1mm below top)
pipette.aspirate(100, well.top())
pipette.aspirate(100, well.top(z=5))  # 5mm above top

# Bottom of well (default: 1mm above bottom)
pipette.aspirate(100, well.bottom())
pipette.aspirate(100, well.bottom(z=2))  # 2mm above bottom

# Center of well
pipette.aspirate(100, well.center())

5. Hardware Module Control

Temperature Module:

# Set temperature
temp_module.set_temperature(celsius=4)

# Wait for temperature
temp_module.await_temperature(celsius=4)

# Deactivate
temp_module.deactivate()

# Check status
current_temp = temp_module.temperature  # Current temperature
target_temp = temp_module.target  # Target temperature

Magnetic Module:

# Engage (raise magnets)
mag_module.engage(height_from_base=10)  # mm from labware base

# Disengage (lower magnets)
mag_module.disengage()

# Check status
is_engaged = mag_module.status  # 'engaged' or 'disengaged'

Heater-Shaker Module:

# Set temperature
hs_module.set_target_temperature(celsius=37)

# Wait for temperature
hs_module.wait_for_temperature()

# Set shake speed
hs_module.set_and_wait_for_shake_speed(rpm=500)

# Close labware latch
hs_module.close_labware_latch()

# Open labware latch
hs_module.open_labware_latch()

# Deactivate heater
hs_module.deactivate_heater()

# Deactivate shaker
hs_module.deactivate_shaker()

Thermocycler Module:

# Open lid
tc_module.open_lid()

# Close lid
tc_module.close_lid()

# Set lid temperature
tc_module.set_lid_temperature(celsius=105)

# Set block temperature
tc_module.set_block_temperature(
    temperature=95,
    hold_time_seconds=30,
    hold_time_minutes=0.5,
    block_max_volume=50  # µL per well
)

# Execute profile (PCR cycling)
profile = [
    {'temperature': 95, 'hold_time_seconds': 30},
    {'temperature': 57, 'hold_time_seconds': 30},
    {'temperature': 72, 'hold_time_seconds': 60}
]
tc_module.execute_profile(
    steps=profile,
    repetitions=30,
    block_max_volume=50
)

# Deactivate
tc_module.deactivate_lid()
tc_module.deactivate_block()

Absorbance Plate Reader:

# Initialize and read
result = plate_reader.read(wavelengths=[450, 650])

# Access readings
absorbance_data = result  # Dict with wavelength keys

6. Liquid Tracking and Labeling

Define Liquids:

# Define liquid types
water = protocol.define_liquid(
    name='Water',
    description='Ultrapure water',
    display_color='#0000FF'  # Hex color code
)

sample = protocol.define_liquid(
    name='Sample',
    description='Cell lysate sample',
    display_color='#FF0000'
)

Load Liquids into Wells:

# Load liquid into specific wells
reservoir['A1'].load_liquid(liquid=water, volume=50000)  # µL
plate['A1'].load_liquid(liquid=sample, volume=100)

# Mark wells as empty
plate['B1'].load_empty()

7. Protocol Control and Utilities

Execution Control:

# Pause protocol
protocol.pause(msg='Replace tip box and resume')

# Delay
protocol.delay(seconds=60)
protocol.delay(minutes=5)

# Comment (appears in logs)
protocol.comment('Starting serial dilution')

# Home robot
protocol.home()

Conditional Logic:

# Check if simulating
if protocol.is_simulating():
    protocol.comment('Running in simulation mode')
else:
    protocol.comment('Running on actual robot')

Rail Lights (Flex only):

# Turn lights on
protocol.set_rail_lights(on=True)

# Turn lights off
protocol.set_rail_lights(on=False)

8. Multi-Channel and 8-Channel Pipetting

When using multi-channel pipettes:

# Load 8-channel pipette
multi_pipette = protocol.load_instrument(
    'p300_multi_gen2',
    'left',
    tip_racks=[tips]
)

# Access entire column with single well reference
multi_pipette.transfer(
    volume=100,
    source=source_plate['A1'],  # Accesses entire column 1
    dest=dest_plate['A1']       # Dispenses to entire column 1
)

# Use rows() for row-wise operations
for row in plate.rows():
    multi_pipette.transfer(100, reservoir['A1'], row[0])

9. Common Protocol Patterns

Serial Dilution:

def run(protocol: protocol_api.ProtocolContext):
    # Load labware
    tips = protocol.load_labware('opentrons_flex_96_tiprack_200ul', 'D1')
    reservoir = protocol.load_labware('nest_12_reservoir_15ml', 'D2')
    plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 'D3')

    # Load pipette
    p300 = protocol.load_instrument('p300_single_flex', 'left', tip_racks=[tips])

    # Add diluent to all wells except first
    p300.transfer(100, reservoir['A1'], plate.rows()[0][1:])

    # Serial dilution across row
    p300.transfer(
        100,
        plate.rows()[0][:11],  # Source: wells 0-10
        plate.rows()[0][1:],   # Dest: wells 1-11
        mix_after=(3, 50),     # Mix 3x with 50µL after dispense
        new_tip='always'
    )

Plate Replication:

def run(protocol: protocol_api.ProtocolContext):
    # Load labware
    tips = protocol.load_labware('opentrons_flex_96_tiprack_1000ul', 'C1')
    source = protocol.load_labware('corning_96_wellplate_360ul_flat', 'D1')
    dest = protocol.load_labware('corning_96_wellplate_360ul_flat', 'D2')

    # Load pipette
    p1000 = protocol.load_instrument('p1000_single_flex', 'left', tip_racks=[tips])

    # Transfer from all wells in source to dest
    p1000.transfer(
        100,
        source.wells(),
        dest.wells(),
        new_tip='always'
    )

PCR Setup:

def run(protocol: protocol_api.ProtocolContext):
    # Load thermocycler
    tc_mod = protocol.load_module('thermocyclerModuleV2')
    tc_plate = tc_mod.load_labware('nest_96_wellplate_100ul_pcr_full_skirt')

    # Load tips and reagents
    tips = protocol.load_labware('opentrons_flex_96_tiprack_200ul', 'C1')
    reagents = protocol.load_labware('opentrons_24_tuberack_nest_1.5ml_snapcap', 'D1')

    # Load pipette
    p300 = protocol.load_instrument('p300_single_flex', 'left', tip_racks=[tips])

    # Open thermocycler lid
    tc_mod.open_lid()

    # Distribute master mix
    p300.distribute(
        20,
        reagents['A1'],
        tc_plate.wells(),
        new_tip='once'
    )

    # Add samples (example for first 8 wells)
    for i, well in enumerate(tc_plate.wells()[:8]):
        p300.transfer(5, reagents.wells()[i+1], well, new_tip='always')

    # Run PCR
    tc_mod.close_lid()
    tc_mod.set_lid_temperature(105)

    # PCR profile
    tc_mod.set_block_temperature(95, hold_time_seconds=180)

    profile = [
        {'temperature': 95, 'hold_time_seconds': 15},
        {'temperature': 60, 'hold_time_seconds': 30},
        {'temperature': 72, 'hold_time_seconds': 30}
    ]
    tc_mod.execute_profile(steps=profile, repetitions=35, block_max_volume=25)

    tc_mod.set_block_temperature(72, hold_time_minutes=5)
    tc_mod.set_block_temperature(4)

    tc_mod.deactivate_lid()
    tc_mod.open_lid()

Best Practices

1. Always specify API level: Use the latest stable API version in metadata
2. Use meaningful labels: Label labware for easier identification in logs
3. Check tip availability: Ensure sufficient tips for protocol completion
4. Add comments: Use protocol.comment() for debugging and logging
5. Simulate first: Always test protocols in simulation before running on robot
6. Handle errors gracefully: Add pauses for manual intervention when needed
7. Consider timing: Use delays when protocols require incubation periods
8. Track liquids: Use liquid tracking for better setup validation
9. Optimize tip usage: Use new_tip='once' when appropriate to save tips
10. Control flow rates: Adjust flow rates for viscous or volatile liquids

Troubleshooting

Common Issues:

• Out of tips: Verify tip rack capacity matches protocol requirements
• Labware collisions: Check deck layout for spatial conflicts
• Volume errors: Ensure volumes don't exceed well or pipette capacities
• Module not responding: Verify module is properly connected and firmware is updated
• Inaccurate volumes: Calibrate pipettes and check for air bubbles
• Protocol fails in simulation: Check API version compatibility and labware definitions

Resources

For detailed API documentation, see references/api_reference.md in this skill directory.

For example protocol templates, see scripts/ directory.

PathML

Overview

PathML is a comprehensive Python toolkit for computational pathology workflows, designed to facilitate machine learning and image analysis for whole-slide pathology images. The framework provides modular, composable tools for loading diverse slide formats, preprocessing images, constructing spatial graphs, training deep learning models, and analyzing multiparametric imaging data from technologies like CODEX and multiplex immunofluorescence.

When to Use This Skill

Apply this skill for:

• Loading and processing whole-slide images (WSI) in various proprietary formats
• Preprocessing H&E stained tissue images with stain normalization
• Nucleus detection, segmentation, and classification workflows
• Building cell and tissue graphs for spatial analysis
• Training or deploying machine learning models (HoVer-Net, HACTNet) on pathology data
• Analyzing multiparametric imaging (CODEX, Vectra, MERFISH) for spatial proteomics
• Quantifying marker expression from multiplex immunofluorescence
• Managing large-scale pathology datasets with HDF5 storage
• Tile-based analysis and stitching operations

Core Capabilities

PathML provides six major capability areas documented in detail within reference files:

1. Image Loading & Formats

Load whole-slide images from 160+ proprietary formats including Aperio SVS, Hamamatsu NDPI, Leica SCN, Zeiss ZVI, DICOM, and OME-TIFF. PathML automatically handles vendor-specific formats and provides unified interfaces for accessing image pyramids, metadata, and regions of interest.

See: references/image_loading.md for supported formats, loading strategies, and working with different slide types.

2. Preprocessing Pipelines

Build modular preprocessing pipelines by composing transforms for image manipulation, quality control, stain normalization, tissue detection, and mask operations. PathML's Pipeline architecture enables reproducible, scalable preprocessing across large datasets.

Key transforms:

• StainNormalizationHE - Macenko/Vahadane stain normalization
• TissueDetectionHE, NucleusDetectionHE - Tissue/nucleus segmentation
• MedianBlur, GaussianBlur - Noise reduction
• LabelArtifactTileHE - Quality control for artifacts

See: references/preprocessing.md for complete transform catalog, pipeline construction, and preprocessing workflows.

3. Graph Construction

Construct spatial graphs representing cellular and tissue-level relationships. Extract features from segmented objects to create graph-based representations suitable for graph neural networks and spatial analysis.

See: references/graphs.md for graph construction methods, feature extraction, and spatial analysis workflows.

4. Machine Learning

Train and deploy deep learning models for nucleus detection, segmentation, and classification. PathML integrates PyTorch with pre-built models (HoVer-Net, HACTNet), custom DataLoaders, and ONNX support for inference.

Key models:

• HoVer-Net - Simultaneous nucleus segmentation and classification
• HACTNet - Hierarchical cell-type classification

See: references/machine_learning.md for model training, evaluation, inference workflows, and working with public datasets.

5. Multiparametric Imaging

Analyze spatial proteomics and gene expression data from CODEX, Vectra, MERFISH, and other multiplex imaging platforms. PathML provides specialized slide classes and transforms for processing multiparametric data, cell segmentation with Mesmer, and quantification workflows.

See: references/multiparametric.md for CODEX/Vectra workflows, cell segmentation, marker quantification, and integration with AnnData.

6. Data Management

Efficiently store and manage large pathology datasets using HDF5 format. PathML handles tiles, masks, metadata, and extracted features in unified storage structures optimized for machine learning workflows.

See: references/data_management.md for HDF5 integration, tile management, dataset organization, and batch processing strategies.

Quick Start

Installation

# Install PathML
uv pip install pathml

# With optional dependencies for all features
uv pip install pathml[all]

Basic Workflow Example

from pathml.core import SlideData
from pathml.preprocessing import Pipeline, StainNormalizationHE, TissueDetectionHE

# Load a whole-slide image
wsi = SlideData.from_slide("path/to/slide.svs")

# Create preprocessing pipeline
pipeline = Pipeline([
    TissueDetectionHE(),
    StainNormalizationHE(target='normalize', stain_estimation_method='macenko')
])

# Run pipeline
pipeline.run(wsi)

# Access processed tiles
for tile in wsi.tiles:
    processed_image = tile.image
    tissue_mask = tile.masks['tissue']

Common Workflows

H&E Image Analysis:

1. Load WSI with appropriate slide class
2. Apply tissue detection and stain normalization
3. Perform nucleus detection or train segmentation models
4. Extract features and build spatial graphs
5. Conduct downstream analysis

Multiparametric Imaging (CODEX):

1. Load CODEX slide with CODEXSlide
2. Collapse multi-run channel data
3. Segment cells using Mesmer model
4. Quantify marker expression
5. Export to AnnData for single-cell analysis

Training ML Models:

1. Prepare dataset with public pathology data
2. Create PyTorch DataLoader with PathML datasets
3. Train HoVer-Net or custom models
4. Evaluate on held-out test sets
5. Deploy with ONNX for inference

References to Detailed Documentation

When working on specific tasks, refer to the appropriate reference file for comprehensive information:

• Loading images: references/image_loading.md
• Preprocessing workflows: references/preprocessing.md
• Spatial analysis: references/graphs.md
• Model training: references/machine_learning.md
• CODEX/multiplex IF: references/multiparametric.md
• Data storage: references/data_management.md

Resources

This skill includes comprehensive reference documentation organized by capability area. Each reference file contains detailed API information, workflow examples, best practices, and troubleshooting guidance for specific PathML functionality.

references/

Documentation files providing in-depth coverage of PathML capabilities:

• image_loading.md - Whole-slide image formats, loading strategies, slide classes
• preprocessing.md - Complete transform catalog, pipeline construction, preprocessing workflows
• graphs.md - Graph construction methods, feature extraction, spatial analysis
• machine_learning.md - Model architectures, training workflows, evaluation, inference
• multiparametric.md - CODEX, Vectra, multiplex IF analysis, cell segmentation, quantification
• data_management.md - HDF5 storage, tile management, batch processing, dataset organization

Load these references as needed when working on specific computational pathology tasks.

Pharmacology Wet-Lab: Experimental Data Analysis

Overview

Pharmacology Wet-Lab provides computational tools for analyzing data from pharmacology experiments. This skill covers western blot densitometry and quantification, xenograft tumor growth inhibition analysis, pharmaceutical stability modeling using Arrhenius kinetics, radiolabeled antibody biodistribution calculations, MIRD-based dosimetry, adverse event grading against CTCAE criteria, and dose-response curve fitting for IC50/EC50 determination.

When to Use This Skill

• Quantifying protein expression from western blot images
• Analyzing xenograft tumor growth data and calculating TGI%
• Predicting pharmaceutical shelf life from accelerated stability data
• Processing radiolabeled antibody biodistribution data (%ID/g)
• Estimating absorbed radiation doses (MIRD dosimetry)
• Grading adverse events against CTCAE or VCOG-CTCAE scales
• Fitting dose-response curves for IC50/EC50 determination
• Calculating combination indices (Chou-Talalay method)

Related Skills: For drug database queries use chembl-database or fda-database. For molecular docking use diffdock. For survival analysis use scikit-survival.

Installation

uv pip install opencv-python scipy pandas numpy matplotlib lifelines

Quick Start

import numpy as np
from scipy.optimize import curve_fit

# 4-Parameter Logistic for dose-response (IC50)
def four_pl(x, bottom, top, ic50, hill):
    return bottom + (top - bottom) / (1 + (x / ic50) ** hill)

concentrations = np.array([0.001, 0.01, 0.1, 1, 10, 100])  # uM
viability = np.array([98, 95, 82, 45, 12, 3])  # % viability

popt, pcov = curve_fit(four_pl, concentrations, viability,
                       p0=[0, 100, 1, 1], maxfev=10000)
print(f"IC50: {popt[2]:.3f} uM")
print(f"Hill coefficient: {popt[3]:.2f}")

Core Capabilities

1. Western Blot Densitometry

Quantify protein bands from western blot images.

import cv2
import numpy as np

def quantify_western_blot(image_path, n_lanes, band_height=50):
    """Quantify western blot band intensities.

    Args:
        image_path: path to blot image
        n_lanes: number of lanes
        band_height: expected band height in pixels
    """
    image = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    if image is None:
        raise FileNotFoundError(f"Cannot load {image_path}")

    # Invert (bands are dark on light background)
    inverted = 255 - image
    h, w = inverted.shape

    # Divide into lanes
    lane_width = w // n_lanes
    lane_intensities = []

    for i in range(n_lanes):
        x_start = i * lane_width + int(lane_width * 0.1)
        x_end = (i + 1) * lane_width - int(lane_width * 0.1)
        lane = inverted[:, x_start:x_end]

        # Find band (peak in vertical intensity profile)
        profile = lane.mean(axis=1)

        # Background subtraction (rolling ball approximation)
        from scipy.ndimage import minimum_filter1d
        background = minimum_filter1d(profile, size=100)
        corrected = profile - background
        corrected = np.clip(corrected, 0, None)

        # Band detection
        from scipy.signal import find_peaks
        peaks, props = find_peaks(corrected, height=corrected.max()*0.1,
                                   distance=band_height)

        # Integrate band intensity (area under curve)
        total_intensity = 0
        for peak in peaks:
            start = max(0, peak - band_height // 2)
            end = min(len(corrected), peak + band_height // 2)
            band_area = corrected[start:end].sum()
            total_intensity += band_area

        lane_intensities.append({
            'lane': i + 1,
            'raw_intensity': total_intensity,
            'n_bands': len(peaks),
            'peak_positions': list(peaks)
        })

    # Normalize to loading control (first lane or specified)
    import pandas as pd
    df = pd.DataFrame(lane_intensities)
    control_intensity = df.iloc[0]['raw_intensity']
    df['normalized'] = df['raw_intensity'] / control_intensity
    df['fold_change'] = df['normalized']

    print("Lane intensities:")
    print(df[['lane', 'raw_intensity', 'normalized', 'fold_change']])

    return df

def calculate_fold_change(target_intensities, loading_control_intensities):
    """Calculate normalized fold change with loading control.

    Args:
        target_intensities: list of target protein band intensities
        loading_control_intensities: list of loading control (e.g., actin) intensities
    """
    target = np.array(target_intensities)
    control = np.array(loading_control_intensities)

    # Normalize target to loading control
    normalized = target / control

    # Fold change relative to first sample
    fold_change = normalized / normalized[0]

    return fold_change

2. Xenograft Tumor Growth Analysis

Analyze in vivo tumor growth and treatment efficacy.

import numpy as np
import pandas as pd
from scipy import stats

def calculate_tgi(tumor_volumes, groups, time_points, control_group='Vehicle'):
    """Calculate Tumor Growth Inhibition (TGI%).

    TGI% = (1 - (Tt - T0) / (Ct - C0)) * 100
    where T=treatment, C=control, t=final, 0=initial

    Args:
        tumor_volumes: DataFrame with columns [animal_id, group, time, volume]
        control_group: name of control group
    """
    results = []
    final_time = max(time_points)

    # Control group growth
    ctrl = tumor_volumes[tumor_volumes['group'] == control_group]
    ctrl_initial = ctrl[ctrl['time'] == 0]['volume'].mean()
    ctrl_final = ctrl[ctrl['time'] == final_time]['volume'].mean()
    ctrl_growth = ctrl_final - ctrl_initial

    for group in tumor_volumes['group'].unique():
        if group == control_group:
            continue

        treat = tumor_volumes[tumor_volumes['group'] == group]
        treat_initial = treat[treat['time'] == 0]['volume'].mean()
        treat_final = treat[treat['time'] == final_time]['volume'].mean()
        treat_growth = treat_final - treat_initial

        tgi = (1 - treat_growth / ctrl_growth) * 100

        # Statistical comparison at final timepoint
        ctrl_final_vals = ctrl[ctrl['time'] == final_time]['volume'].values
        treat_final_vals = treat[treat['time'] == final_time]['volume'].values
        stat, pval = stats.mannwhitneyu(ctrl_final_vals, treat_final_vals,
                                         alternative='two-sided')

        results.append({
            'group': group,
            'TGI%': tgi,
            'mean_volume_final': treat_final,
            'control_volume_final': ctrl_final,
            'p_value': pval,
            'n_animals': len(treat['animal_id'].unique())
        })

    df = pd.DataFrame(results)
    print("Tumor Growth Inhibition:")
    print(df[['group', 'TGI%', 'mean_volume_final', 'p_value']])
    return df

def growth_delay(tumor_volumes, groups, doubling_volume):
    """Calculate tumor growth delay (time to reach doubling volume).

    Args:
        doubling_volume: target volume for comparison (e.g., 2x initial)
    """
    results = []
    for group in tumor_volumes['group'].unique():
        grp = tumor_volumes[tumor_volumes['group'] == group]
        times_to_double = []

        for animal in grp['animal_id'].unique():
            animal_data = grp[grp['animal_id'] == animal].sort_values('time')
            reached = animal_data[animal_data['volume'] >= doubling_volume]
            if len(reached) > 0:
                times_to_double.append(reached.iloc[0]['time'])

        if times_to_double:
            results.append({
                'group': group,
                'median_time_to_double': np.median(times_to_double),
                'mean_time_to_double': np.mean(times_to_double),
                'n_reached': len(times_to_double)
            })

    return pd.DataFrame(results)

3. Pharmaceutical Stability

Predict shelf life from accelerated stability data.

import numpy as np
from scipy.optimize import curve_fit

def arrhenius_stability(temperatures_C, rate_constants):
    """Fit Arrhenius equation to predict shelf life.

    k = A * exp(-Ea / RT)

    Args:
        temperatures_C: storage temperatures in Celsius
        rate_constants: degradation rate constants at each temperature
    """
    T_kelvin = np.array(temperatures_C) + 273.15
    k = np.array(rate_constants)

    # Linearize: ln(k) = ln(A) - Ea/(R*T)
    ln_k = np.log(k)
    inv_T = 1 / T_kelvin

    # Linear regression
    slope, intercept, r_value, p_value, std_err = stats.linregress(inv_T, ln_k)

    R = 8.314  # J/(mol*K)
    Ea = -slope * R / 1000  # kJ/mol
    A = np.exp(intercept)

    print(f"Activation energy: {Ea:.1f} kJ/mol")
    print(f"Pre-exponential factor: {A:.4e}")
    print(f"R²: {r_value**2:.4f}")

    # Predict rate at storage temperature (25°C)
    T_storage = 25 + 273.15
    k_25 = A * np.exp(-Ea * 1000 / (R * T_storage))

    # Shelf life (time to 10% degradation, first-order kinetics)
    # C/C0 = exp(-k*t) → t = -ln(0.9) / k
    shelf_life_days = -np.log(0.9) / k_25
    shelf_life_months = shelf_life_days / 30.44

    print(f"\nPredicted at 25°C:")
    print(f"  Rate constant: {k_25:.6f} day⁻¹")
    print(f"  Shelf life (to 90%): {shelf_life_months:.0f} months ({shelf_life_days:.0f} days)")

    return {
        'Ea_kJ_mol': Ea,
        'A': A,
        'k_25C': k_25,
        'shelf_life_months': shelf_life_months,
        'shelf_life_days': shelf_life_days
    }

def fit_degradation_kinetics(time_points, concentrations, order=1):
    """Fit degradation kinetics (zero, first, or second order)."""
    time = np.array(time_points)
    conc = np.array(concentrations)

    if order == 0:
        # C = C0 - k*t
        slope, intercept, r, p, se = stats.linregress(time, conc)
        k = -slope
        r_sq = r**2
    elif order == 1:
        # ln(C) = ln(C0) - k*t
        slope, intercept, r, p, se = stats.linregress(time, np.log(conc))
        k = -slope
        r_sq = r**2
    elif order == 2:
        # 1/C = 1/C0 + k*t
        slope, intercept, r, p, se = stats.linregress(time, 1/conc)
        k = slope
        r_sq = r**2

    print(f"Order {order}: k = {k:.6f}, R² = {r_sq:.4f}")
    return {'k': k, 'r_squared': r_sq, 'order': order}

4. Radiolabeled Antibody Biodistribution

Process biodistribution data from radiolabeled compounds.

import numpy as np
import pandas as pd

def calculate_biodistribution(tissue_counts, tissue_weights_g, injection_dose,
                               decay_correction_factor=1.0):
    """Calculate %ID/g from tissue radioactivity counts.

    Args:
        tissue_counts: dict of tissue_name -> counts per minute (CPM)
        tissue_weights_g: dict of tissue_name -> weight in grams
        injection_dose: total injected dose in CPM
        decay_correction_factor: factor to correct for radioactive decay
    """
    results = []
    for tissue in tissue_counts:
        cpm = tissue_counts[tissue] * decay_correction_factor
        weight = tissue_weights_g[tissue]

        id_per_g = (cpm / weight) / injection_dose * 100  # %ID/g

        results.append({
            'tissue': tissue,
            'CPM': tissue_counts[tissue],
            'weight_g': weight,
            'percent_ID_per_g': id_per_g
        })

    df = pd.DataFrame(results).sort_values('percent_ID_per_g', ascending=False)

    # Tumor-to-blood ratio (if both present)
    if 'tumor' in tissue_counts and 'blood' in tissue_counts:
        tumor_uptake = df[df['tissue'] == 'tumor']['percent_ID_per_g'].values[0]
        blood_uptake = df[df['tissue'] == 'blood']['percent_ID_per_g'].values[0]
        ratio = tumor_uptake / blood_uptake if blood_uptake > 0 else float('inf')
        print(f"Tumor-to-blood ratio: {ratio:.2f}")

    print(df[['tissue', 'percent_ID_per_g']].to_string(index=False))
    return df

5. MIRD Dosimetry

Estimate absorbed radiation dose using MIRD formalism.

import numpy as np
from scipy.integrate import trapezoid

def mird_dosimetry(time_points_h, activity_values_MBq, s_values, organ_masses_kg):
    """Estimate absorbed dose using MIRD formalism.

    Dose = cumulative_activity * S_value

    Args:
        time_points_h: time points in hours
        activity_values_MBq: dict of organ -> list of activity values at each time
        s_values: dict of (source, target) -> S-value in mGy/(MBq*h)
        organ_masses_kg: dict of organ -> mass in kg
    """
    results = []
    time = np.array(time_points_h)

    for organ, activities in activity_values_MBq.items():
        act = np.array(activities)

        # Time-integrated activity (cumulative activity, A_tilde)
        cumulative_activity = trapezoid(act, time)  # MBq*h

        # Self-dose
        s_self = s_values.get((organ, organ), 0)
        dose_self = cumulative_activity * s_self  # mGy

        # Cross-dose from other organs
        dose_cross = 0
        for source_organ, source_activities in activity_values_MBq.items():
            if source_organ != organ:
                s_cross = s_values.get((source_organ, organ), 0)
                source_cumulative = trapezoid(np.array(source_activities), time)
                dose_cross += source_cumulative * s_cross

        total_dose = dose_self + dose_cross

        results.append({
            'organ': organ,
            'cumulative_activity_MBq_h': cumulative_activity,
            'self_dose_mGy': dose_self,
            'cross_dose_mGy': dose_cross,
            'total_dose_mGy': total_dose
        })

    import pandas as pd
    df = pd.DataFrame(results)
    print(df[['organ', 'cumulative_activity_MBq_h', 'total_dose_mGy']])
    return df

6. Adverse Event Grading

Grade adverse events against CTCAE v5.0 criteria.

import pandas as pd

# CTCAE v5.0 grading examples (subset)
CTCAE_CRITERIA = {
    'neutrophil_count': {
        1: lambda x: 1500 <= x < 2000,  # Grade 1: LLN - 1500/mm3
        2: lambda x: 1000 <= x < 1500,  # Grade 2: 1000-1500
        3: lambda x: 500 <= x < 1000,   # Grade 3: 500-1000
        4: lambda x: x < 500,           # Grade 4: <500
    },
    'platelet_count': {
        1: lambda x: 75000 <= x < 150000,
        2: lambda x: 50000 <= x < 75000,
        3: lambda x: 25000 <= x < 50000,
        4: lambda x: x < 25000,
    },
    'hemoglobin': {
        1: lambda x: 10 <= x < 12,      # g/dL
        2: lambda x: 8 <= x < 10,
        3: lambda x: x < 8,
        4: lambda x: False,  # Life-threatening
    },
    'ALT': {
        1: lambda x: 1 <= x/40 < 3,     # x ULN (ULN=40)
        2: lambda x: 3 <= x/40 < 5,
        3: lambda x: 5 <= x/40 < 20,
        4: lambda x: x/40 >= 20,
    }
}

def grade_adverse_event(parameter, value):
    """Grade an adverse event by CTCAE v5.0."""
    if parameter not in CTCAE_CRITERIA:
        return {'grade': None, 'error': f'Unknown parameter: {parameter}'}

    criteria = CTCAE_CRITERIA[parameter]
    for grade in [4, 3, 2, 1]:  # Check highest grade first
        if criteria[grade](value):
            return {'parameter': parameter, 'value': value, 'grade': grade}

    return {'parameter': parameter, 'value': value, 'grade': 0}

def assess_dlt(adverse_events, dlt_grades={'hematologic': 4, 'non_hematologic': 3}):
    """Assess dose-limiting toxicity from adverse event list."""
    dlts = []
    for ae in adverse_events:
        if ae['grade'] >= dlt_grades.get('hematologic', 4):
            dlts.append(ae)
    return dlts

7. Dose-Response Curves

Fit dose-response data for IC50/EC50 determination.

import numpy as np
from scipy.optimize import curve_fit

def four_pl(x, bottom, top, ec50, hill):
    """4-Parameter Logistic dose-response."""
    return bottom + (top - bottom) / (1 + (x / ec50) ** hill)

def fit_dose_response(concentrations, responses, response_type='inhibition'):
    """Fit dose-response curve and calculate IC50/EC50.

    Args:
        concentrations: drug concentrations
        responses: measured responses (viability %, activity, etc.)
        response_type: 'inhibition' (response decreases) or 'activation'
    """
    conc = np.array(concentrations)
    resp = np.array(responses)

    if response_type == 'inhibition':
        p0 = [resp.min(), resp.max(), np.median(conc), 1.0]
    else:
        p0 = [resp.min(), resp.max(), np.median(conc), 1.0]

    bounds = ([0, 0, 0, 0.1], [200, 200, conc.max() * 10, 10])

    popt, pcov = curve_fit(four_pl, conc, resp, p0=p0, bounds=bounds, maxfev=10000)
    perr = np.sqrt(np.diag(pcov))

    bottom, top, ec50, hill = popt

    # R-squared
    predicted = four_pl(conc, *popt)
    ss_res = np.sum((resp - predicted) ** 2)
    ss_tot = np.sum((resp - np.mean(resp)) ** 2)
    r_squared = 1 - ss_res / ss_tot

    label = 'IC50' if response_type == 'inhibition' else 'EC50'
    print(f"{label}: {ec50:.4f} ± {perr[2]:.4f}")
    print(f"Hill coefficient: {hill:.2f}")
    print(f"Bottom: {bottom:.1f}%, Top: {top:.1f}%")
    print(f"R²: {r_squared:.4f}")

    return {label: ec50, 'hill': hill, 'bottom': bottom, 'top': top,
            'r_squared': r_squared, 'params': popt}

def combination_index(drug1_ic50, drug2_ic50, combo_ic50_d1, combo_ic50_d2):
    """Calculate Chou-Talalay Combination Index.

    CI < 1: synergistic, CI = 1: additive, CI > 1: antagonistic
    """
    ci = (combo_ic50_d1 / drug1_ic50) + (combo_ic50_d2 / drug2_ic50)
    interpretation = 'synergistic' if ci < 0.9 else ('antagonistic' if ci > 1.1 else 'additive')
    print(f"CI = {ci:.3f} ({interpretation})")
    return {'CI': ci, 'interpretation': interpretation}

Typical Workflows

Workflow 1: Quantify Western Blot Bands and Calculate Fold Change

# Target protein bands
target = [1500, 3200, 4800, 2100]  # Arbitrary units per lane
actin = [2000, 1950, 2100, 1900]   # Loading control

fc = calculate_fold_change(target, actin)
for i, f in enumerate(fc):
    print(f"Lane {i+1}: fold change = {f:.2f}")

Workflow 2: Analyze Xenograft Tumor Growth and Calculate TGI%

import pandas as pd

data = pd.DataFrame({
    'animal_id': ['V1','V1','V2','V2','T1','T1','T2','T2'],
    'group': ['Vehicle','Vehicle','Vehicle','Vehicle','Drug','Drug','Drug','Drug'],
    'time': [0, 14, 0, 14, 0, 14, 0, 14],
    'volume': [100, 800, 120, 750, 110, 350, 105, 380]
})

tgi = calculate_tgi(data, data['group'].unique(), [0, 14])

Workflow 3: Predict Shelf Life from Accelerated Stability Data

from scipy import stats

temps = [40, 50, 60, 70]  # Celsius
rates = [0.001, 0.003, 0.01, 0.03]  # day^-1

result = arrhenius_stability(temps, rates)
print(f"Predicted shelf life at 25°C: {result['shelf_life_months']:.0f} months")

Best Practices

1. Western blots — always normalize to loading control (beta-actin, GAPDH, total protein); report fold change, not raw intensity
2. Xenograft TGI — require minimum 8-10 animals per group; report both TGI% and statistical significance; use caliper measurements consistently
3. Stability studies — use at least 3 temperatures for Arrhenius fitting; verify linearity of ln(k) vs 1/T; ICH guidelines require 25°C/60%RH long-term
4. Dose-response — use at least 6 concentrations spanning 3 log units; include 0 and saturating dose; fit with Hill slope unconstrained
5. Combination index — measure IC50 of each drug alone and in fixed-ratio combination; CI only valid at Fa (fraction affected) near 0.5
6. Adverse events — use CTCAE v5.0 for human clinical trials; VCOG-CTCAE for veterinary studies

Troubleshooting

Problem: Dose-response fit gives unreasonable IC50 Solution: Ensure concentrations span the IC50 (responses should range from ~10% to ~90% effect). Add more points around the inflection. Check if response truly follows sigmoidal pattern.

Problem: Western blot quantification inconsistent Solution: Ensure uniform exposure across lanes. Use ECL with linear dynamic range. Avoid saturated bands. Use total protein stain (Ponceau S) instead of housekeeping gene for normalization.

Problem: Arrhenius fit has low R² Solution: Verify degradation follows the assumed kinetic order. Check for multiple degradation mechanisms at different temperatures. Exclude temperatures where phase transitions occur.

Problem: TGI% is negative Solution: Treatment group grew faster than control. Verify group assignments. Check for handling errors. This can occur with growth factors or hormones.

Resources

• CTCAE v5.0 Grading Criteria
• ICH Stability Guidelines (Q1A-Q1E)
• Chou-Talalay Method
• MIRD Primer

Protocols.io Integration

Overview

Protocols.io is a comprehensive platform for developing, sharing, and managing scientific protocols. This skill provides complete integration with the protocols.io API v3, enabling programmatic access to protocols, workspaces, discussions, file management, and collaboration features.

When to Use This Skill

Use this skill when working with protocols.io in any of the following scenarios:

• Protocol Discovery: Searching for existing protocols by keywords, DOI, or category
• Protocol Management: Creating, updating, or publishing scientific protocols
• Step Management: Adding, editing, or organizing protocol steps and procedures
• Collaborative Development: Working with team members on shared protocols
• Workspace Organization: Managing lab or institutional protocol repositories
• Discussion & Feedback: Adding or responding to protocol comments
• File Management: Uploading data files, images, or documents to protocols
• Experiment Tracking: Documenting protocol executions and results
• Data Export: Backing up or migrating protocol collections
• Integration Projects: Building tools that interact with protocols.io

Core Capabilities

This skill provides comprehensive guidance across five major capability areas:

1. Authentication & Access

Manage API authentication using access tokens and OAuth flows. Includes both client access tokens (for personal content) and OAuth tokens (for multi-user applications).

Key operations:

• Generate authorization links for OAuth flow
• Exchange authorization codes for access tokens
• Refresh expired tokens
• Manage rate limits and permissions

Reference: Read references/authentication.md for detailed authentication procedures, OAuth implementation, and security best practices.

2. Protocol Operations

Complete protocol lifecycle management from creation to publication.

Key operations:

• Search and discover protocols by keywords, filters, or DOI
• Retrieve detailed protocol information with all steps
• Create new protocols with metadata and tags
• Update protocol information and settings
• Manage protocol steps (create, update, delete, reorder)
• Handle protocol materials and reagents
• Publish protocols with DOI issuance
• Bookmark protocols for quick access
• Generate protocol PDFs

Reference: Read references/protocols_api.md for comprehensive protocol management guidance, including API endpoints, parameters, common workflows, and examples.

3. Discussions & Collaboration

Enable community engagement through comments and discussions.

Key operations:

• View protocol-level and step-level comments
• Create new comments and threaded replies
• Edit or delete your own comments
• Analyze discussion patterns and feedback
• Respond to user questions and issues

Reference: Read references/discussions.md for discussion management, comment threading, and collaboration workflows.

4. Workspace Management

Organize protocols within team workspaces with role-based permissions.

Key operations:

• List and access user workspaces
• Retrieve workspace details and member lists
• Request access or join workspaces
• List workspace-specific protocols
• Create protocols within workspaces
• Manage workspace permissions and collaboration

Reference: Read references/workspaces.md for workspace organization, permission management, and team collaboration patterns.

5. File Operations

Upload, organize, and manage files associated with protocols.

Key operations:

• Search workspace files and folders
• Upload files with metadata and tags
• Download files and verify uploads
• Organize files into folder hierarchies
• Update file metadata
• Delete and restore files
• Manage storage and organization

Reference: Read references/file_manager.md for file upload procedures, organization strategies, and storage management.

6. Additional Features

Supplementary functionality including profiles, notifications, and exports.

Key operations:

• Manage user profiles and settings
• Query recently published protocols
• Create and track experiment records
• Receive and manage notifications
• Export organization data for archival

Reference: Read references/additional_features.md for profile management, publication discovery, experiment tracking, and data export.

Getting Started

Step 1: Authentication Setup

Before using any protocols.io API functionality:

1. Obtain an access token (CLIENT_ACCESS_TOKEN or OAUTH_ACCESS_TOKEN)
2. Read references/authentication.md for detailed authentication procedures
3. Store the token securely
4. Include in all requests as: Authorization: Bearer YOUR_TOKEN

Step 2: Identify Your Use Case

Determine which capability area addresses your needs:

• Working with protocols? → Read references/protocols_api.md
• Managing team protocols? → Read references/workspaces.md
• Handling comments/feedback? → Read references/discussions.md
• Uploading files/data? → Read references/file_manager.md
• Tracking experiments or profiles? → Read references/additional_features.md

Step 3: Implement Integration

Follow the guidance in the relevant reference files:

• Each reference includes detailed endpoint documentation
• API parameters and request/response formats are specified
• Common use cases and workflows are provided with examples
• Best practices and error handling guidance included

Base URL and Request Format

All API requests use the base URL:

https://protocols.io/api/v3

All requests require the Authorization header:

Authorization: Bearer YOUR_ACCESS_TOKEN

Most endpoints support JSON request/response format with Content-Type: application/json.

Content Format Options

Many endpoints support a content_format parameter to control how protocol content is returned:

• json: Draft.js JSON format (default)
• html: HTML format
• markdown: Markdown format