Afrexai Data Engineering

# Data Engineering Command Center

Complete methodology for designing, building, operating, and scaling data pipelines and infrastructure. Zero dependencies — pure agent skill.

---

## Phase 1: Data Architecture Assessment

Before building anything, understand the landscape.

### Architecture Brief

```yaml
project_name: ""
business_context: ""
data_consumers:
  - team: ""
    use_case: ""          # analytics | ML | operational | reporting | reverse-ETL
    latency_requirement: ""  # real-time (<1s) | near-real-time (<5min) | batch (hourly+)
    query_pattern: ""     # ad-hoc | scheduled | API | dashboard

current_state:
  sources: []             # list every system producing data
  storage: []             # where data lives today
  pain_points: []         # what's broken, slow, unreliable
  data_volume:
    current_gb_per_day: 0
    growth_rate_percent: 0
    retention_months: 0

constraints:
  budget_monthly_usd: 0
  team_size: 0
  skill_level: ""         # junior | mid | senior | mixed
  compliance: []          # GDPR, HIPAA, SOX, PCI, none
  cloud_provider: ""      # AWS | GCP | Azure | multi | on-prem
```

### Architecture Pattern Decision Matrix

| Signal | Pattern | When to Use |
|--------|---------|-------------|
| All consumers need data hourly+ | **Batch ETL** | Reporting, warehousing, most analytics |
| Some need <5 min latency | **Micro-batch** | Dashboard freshness, near-real-time analytics |
| Events need <1s processing | **Streaming** | Fraud detection, real-time pricing, alerts |
| Need both batch + streaming | **Lambda** | When batch accuracy + real-time speed both matter |
| Want to simplify Lambda | **Kappa** | When you can reprocess from stream replay |
| Data lake + warehouse combined | **Lakehouse** | When you need both cheap storage + fast SQL |
| Sources change independently | **Data Mesh** | Large orgs, domain-owned data, >5 teams |
| ML is primary consumer | **Feature Store** | ML-heavy orgs with feature reuse needs |

### Technology Sele...

# AfrexAI Data Engineering Command Center

Turn your AI agent into a senior data engineer. Complete methodology for designing, building, and operating production data pipelines — no code dependencies, no API keys, pure knowledge.

## Install

```bash
clawhub install afrexai-data-engineering
```

## What You Get

- **Architecture decision frameworks** — batch vs streaming vs lakehouse vs data mesh
- **Technology selection guides** — Airflow, dbt, Spark, Flink, Snowflake, BigQuery, and more
- **Data modeling methodology** — Kimball, Data Vault, OBT with SCD type guides
- **Pipeline design patterns** — extraction strategies, load patterns, idempotency rules
- **Data quality framework** — 6-dimension quality checks, data contracts, severity levels
- **Performance optimization** — SQL tuning, Spark optimization, partitioning strategies
- **Data governance** — classification, PII handling, catalog templates, lineage
- **Monitoring & alerting** — dashboard template, structured logging, alert routing
- **Testing strategy** — full test pyramid from unit to E2E
- **Cost optimization** — cloud cost reduction, ROI tracking, red flags
- **Operational runbooks** — failure triage, schema change management, disaster recovery
- **Advanced patterns** — CDC, feature stores, SCD Type 2 SQL, data mesh

## Quick Start

Tell your agent:
- *"Design a data pipeline for Postgres to Snowflake"*
- *"Model our sales domain for analytics"*
- *"Audit our data infrastructure"*
- *"Set up data quality for the orders table"*

## ⚡ Level Up

Want industry-specific data engineering context? Our vertical packs include data architecture patterns tailored to your domain:

- **[SaaS Context Pack](https://afrexai-cto.github.io/context-packs/)** — product analytics, usage metering, multi-tenant data
- **[Fintech Context Pack](https://afrexai-cto.github.io/context-packs/)** — transaction pipelines, regulatory reporting
- **[Manufacturing Context Pack](https://afrexai-cto.github.io/context-packs/)** — IoT da...