AI-Augmented Development for Science

From Petabytes to Pipelines: How AI Dev Tools Accelerate Research Software

James Westover | Westover Labs
westoverlabs.eu | james@westover.dev

GitHub
James Westover

James Westover

Physicist → 10M QPS* at Uber → AI & Applied ML → Ready to catalog the solar system

Uber HERE University of Rochester UCF

Petabyte-scale pipelines · Distributed systems · Python · HPC clusters · AI-augmented dev

NDS Westover Labs Founder — Westover Labs

The Problem We All Share

Science software is getting harder to build, not easier:

  • Datasets growing exponentially (LSST: 20 TB/night, 60 PB over 10 years)
  • Pipelines spanning multiple languages, frameworks, and institutions
  • Small teams, big ambitions, limited engineering bandwidth
  • "It works on my laptop" doesn't scale to production

What if your development tools were as smart as the science they support?

Glossary: AI Development Terms (1/2)

Term Definition
Agentic AI AI that can take actions — read files, run tests, call APIs — not just generate text
Sequential Reasoning AI technique where the model plans step-by-step before acting (think vs. autocomplete)
Claude Code Anthropic's CLI tool for AI-assisted development — the main tool demonstrated today
MCP Model Context Protocol — open standard that lets AI agents call your tools and access your data
Knowledge Graph A database that stores entities and their relationships; here used for persistent AI memory

Glossary: AI Development Terms (2/2)

Term Definition
pgvector PostgreSQL extension for vector similarity search — enables semantic memory
ONNX Runtime Cross-platform ML inference engine — runs AI models efficiently on CPU or GPU
CI/CD Continuous Integration / Continuous Deployment — automated testing and deployment pipelines
GitHub Actions GitHub's automation platform for CI/CD workflows — triggers on push, PR, schedule
Ansible Infrastructure-as-code tool — automates server setup and deployment repeatably

These terms will come up during the demo. Refer back to this slide anytime!

Resources & Links

Follow along with the demo:

This presentation: lsst.westoverlabs.eu

What is AI-Augmented Development?

Not "AI writes code for you." Instead:

AI as a force multiplier for the developer

  • Context-aware: Understands your codebase, not just syntax
  • Tool-integrated: Runs tests, reads docs, searches code, calls APIs
  • Conversational: Explain what you want in natural language
  • Persistent: Remembers decisions across sessions

Think: a senior colleague who has read every file in your repo.

Two Breakthroughs That Changed Everything

1. Sequential Reasoning (Chain of Thought)

AI that plans before it acts — explores your code, reads tests, checks docs, then writes.
Not autocomplete. A reasoning engine.

2. Persistent Memory (Knowledge Graphs)

Context that survives between sessions. Why you chose Healpix over R-trees.
What the team decided about coordinate systems. Architectural decisions preserved forever.

These aren't nice-to-haves. They're what make AI dev tools actually work for science.

The Stack: What You Can Build Today

Layer Tool Purpose
AI Engine Claude Code (CLI) Interactive dev agent with tool use
Memory Knowledge Graph (pgvector) Persistent context across sessions
Tool Access MCP (Model Context Protocol) Let AI call your APIs, read your data
Automation Ansible + GitHub Actions Infrastructure as code
Deployment Cloudflare + Tailscale Edge + secure access

All open, all self-hostable. No vendor lock-in.

What is MCP? (Model Context Protocol)

MCP lets you give your AI agent access to your tools and data.

┌─────────────┐     MCP      ┌──────────────────────┐
│  Claude Code │ ◄──────────► │  Your MCP Server     │
│  (AI Agent)  │   (JSON-RPC) │  ┌── Fink API        │
└─────────────┘              │  ├── Database access  │
                              │  ├── Telescope queue  │
                              │  └── Pipeline control │
                              └──────────────────────┘

Build one with Claude itself — describe what tools you need, Claude writes the MCP server. Your entire instrument suite becomes AI-accessible.

Why Deploy a Knowledge Graph?

Without memory, every session starts from zero:

Monday:   "We chose SQLite for storage because no server overhead."
Tuesday:  "Why are we using SQLite?"  ← AI has no idea

With a KG (PostgreSQL + pgvector):

Monday:   Decision stored → "SQLite: no server, WAL mode, fits HPC"
Tuesday:  AI queries KG → "We chose SQLite because [full rationale]"

Every architectural decision, every pipeline choice, every gotcha — preserved.

Cost: One PostgreSQL instance. Value: Never re-explain your project again.

Demo: LSST Alert Pipeline with Fink

A Real Project — github.com/fedorets/lsst-extendedness

Built in 2 days with AI-augmented dev:

Metric Value
Timeline Started yesterday, production-ready today
Commits 30+
Lines of Code 21,000 (11K source + 10K tests)
Test Functions 587
Source Modules 32
CI/CD GitHub Actions, Dependabot, auto-merge
Alert Sources 5 pluggable (ANTARES, Kafka, Fink, File, Mock)

Let's see how the prompts worked...

Step 1: Scaffolding from a Prompt

You: "Create a Python package that pulls astronomical alerts from
     the Fink broker REST API (api.fink-portal.org), filters for
     objects with high extendedness scores, and stores them in
     SQLite. Use Pydantic for data models and Click for the CLI."

Claude: [reads Fink API docs via web search]
        [creates src/pipeline/models.py with AlertRecord]
        [creates src/pipeline/fink_client.py with REST calls]
        [creates src/pipeline/storage.py with SQLite]
        [sets up pyproject.toml with PDM]
        [writes initial tests with fixture data]

Time: ~5 minutes. The AI knows Pydantic, Click, SQLite patterns — and it researched the Fink API structure.

Step 2: Domain Logic — The Hard Part

You: "The extendedness field maps from Fink's classtar:
     extendedness = 1 - classtar. Filter for extendedness > 0.7.
     Also detect SSO reassociations by tracking ss_object_id
     changes over time."

Claude: [reads Fink field documentation]
        [implements extendedness conversion with edge cases]
        [adds state tracking for reassociation detection]
        [creates SQL views for minimoon candidates]
        [writes parametric tests with real ZTF fixture data]

Key insight: The AI researches domain-specific formats — Fink field mappings, Julian date conversions, magnitude-to-flux — so you don't read RFCs for hours.

Step 3: Build an MCP Server for It

You: "Now wrap this pipeline as an MCP server so the AI agent
     can query alerts, run filters, and check pipeline status
     as tools."

Claude: [creates mcp_server.py with tool definitions]
        [exposes: query_alerts, run_pipeline, get_stats]
        [adds to Claude Code MCP config]
        [tests tool calling end-to-end]

Now the AI agent can use your pipeline as a tool:

"How many minimoon candidates did we find this week?"
→ AI calls query_alerts tool → returns structured data → answers naturally.

Step 4: Knowledge Graph Persistence

Session 1: "We decided to use Healpix for spatial indexing
           because it handles the celestial sphere natively."

  → KG: Entity "Decision: Healpix Spatial Index"
        Observation: "Chosen over R-tree for celestial data"
        Relation: depends_on → "astropy-healpix"

Session 2 (days later):
You: "Why did we use Healpix instead of a simple R-tree?"
Claude: [queries KG] → "We chose Healpix because it handles
        the celestial sphere natively, avoiding distortion at
        the poles that an R-tree in Cartesian coords would have."

No context lost between sessions.

Recommendations for Science Teams

  1. Deploy a Knowledge Graph — PostgreSQL + pgvector. Store all decisions.
    One CREATE EXTENSION vector; away from persistent AI memory.

  2. Build MCP servers for your instruments — Let Claude call your telescope
    queue, query your catalog, run your pipeline. Claude can write the MCP server for you.

  3. Use sequential reasoning — Don't use autocomplete-style AI. Use agents
    that plan, explore, test, then write. Claude Code's agentic mode.

  4. Start small, iterate fast — One pipeline. One MCP tool. One KG entity.
    The compound effect is enormous.

  5. Infrastructure as code from day one — Ansible + GitHub Actions.
    If you SSH to fix something, you've already lost.

Getting Started

  1. Install Claude Code: npm install -g @anthropic-ai/claude-code
  2. Point at your repo: cd your-project && claude
  3. Start talking: "Help me understand this codebase"
  4. Build up context: AI learns your patterns over time

For science teams specifically:

  • Deploy PostgreSQL + pgvector for a knowledge graph
  • Build an MCP server for your data sources (Claude can help!)
  • Use Claude Code's sequential reasoning — not autocomplete

Contact & Consulting

Westover Labs — AI development consulting for science teams:

  • AI dev workshops and hands-on training
  • Pipeline architecture review and Knowledge Graph deployment
  • MCP server development for instruments and data sources
  • CI/CD and infrastructure automation

James Westover | westoverlabs.eu | james@westover.dev

2x BS Rochester, MS + PhD (ABD) UCF, MBA Quantic
Ex-Uber/HERE/Postmates — I speak both science and engineering fluently.

Westover Labs · Westover Labs · westover.dev