Ssh Lab

# ssh-lab — Remote Server SSH Workbench

> Structured SSH operations for managing remote GPU servers.
> Standardizes what's universal (GPU/disk/process), delegates interpretation to the LLM.

## When to Use

- User asks about GPU status, server health, remote processes
- Need to check if training is running, disk is full, GPU is idle
- Running commands on remote servers
- Tailing remote log files, listing checkpoints
- File transfer between local and remote (rsync)
- Comparing hosts side-by-side (GPU, disk, processes)
- Setting up alerts for GPU idle, disk full, process died
- Any mention of: `ssh-lab`, `gpu status`, `server status`, `nvidia-smi`, remote server

## Setup

```bash
cd /path/to/ssh-lab && npm run build
```

After building, use the CLI via any of:
```bash
ssh-lab status all          # if npm link'd or installed globally
npx ssh-lab status all      # via npx
node dist/cli.js status all # direct invocation
```

The CLI reads `~/.ssh/config` automatically — zero configuration needed for existing SSH hosts.

## Commands

### List hosts
```bash
ssh-lab hosts [--json]
```

### Server status (GPU + disk + process)
```bash
ssh-lab status [host|all] [--json] [--timeout ms] [--heartbeat] [--quiet]
```
Returns structured probe data: GPU utilization/VRAM/temp, disk usage, active processes.
Includes automatic alerts: GPU idle, disk full (>90%), high temp (>85°C).
`--heartbeat` gives compact one-liner output for agent integration. `--quiet` suppresses all-clear output (for cron).

### Run remote command
```bash
ssh-lab run <host|all> <command...> [--json] [--timeout ms]
```
Supports `all` for parallel execution across all hosts.

### Tail remote log
```bash
ssh-lab tail <host> <path> [-n lines] [--json]
```
Default: last 50 lines. Smart truncation at 50KB.

### List remote directory
```bash
ssh-lab ls <host> <path> [--sort time|size|name] [--json]
```
Lists files with size, date, permissions. Great for checkpoint directories.

### Disk usage
```bash
ssh-lab df <host|...

# ssh-lab

Remote server SSH workbench for managing GPU training servers. Provides structured data collection (GPU/disk/process probes), declarative alerting, and side-by-side comparison — all through a zero-dependency CLI.

Built for AI agents: structured JSON output, semantic exit codes, heartbeat mode for cron/polling.

## Install

```bash
git clone <repo-url> && cd ssh-lab
npm install && npm run build
```

Requires: Node.js 18+, native `ssh` client.

## Quick Start

```bash
# List all SSH hosts (reads ~/.ssh/config automatically)
ssh-lab hosts

# GPU/disk/process status for all servers
ssh-lab status all

# Run a command on a specific server
ssh-lab run GMI4 "nvidia-smi"

# Compare two servers side-by-side
ssh-lab compare GMI4 GMI5

# Check if a server is healthy
ssh-lab doctor GMI4

# Set up alerts
ssh-lab alert add gpu_idle all
ssh-lab alert add process_died GMI4 --process-pattern "train.py"
ssh-lab alert add disk_full all --threshold 85

# Check alerts (exit code: 0=ok, 1=warn, 2=critical)
ssh-lab alert check all --quiet
```

## Commands

| Command | Description |
|---------|-------------|
| `hosts` | List all configured servers |
| `status [host\|all]` | GPU + disk + process overview |
| `run <host\|all> <cmd>` | Execute command on remote host(s) |
| `tail <host> <path>` | Tail a remote log file |
| `ls <host> <path>` | List remote directory |
| `df <host\|all> [path]` | Disk usage details |
| `sync <host> <src> <dst>` | rsync file transfer |
| `watch <host> <path>` | Snapshot file state for polling |
| `compare <hosts\|all>` | Side-by-side host comparison |
| `doctor <host>` | Connectivity & health diagnostics |
| `alert add\|list\|check\|remove` | Declarative alert rules |
| `add <name> <user@host>` | Add a custom host |

## Output Modes

All commands support `--json`, `--raw`, or default summary output.

```bash
ssh-lab status GMI4 --json    # Structured JSON
ssh-lab status all --raw      # Raw probe output
ssh-lab status all             # Human-readable...