# Quark - Neural Training Analysis Platform

Developer-facing product specification for a desktop application and Python SDK that provide deep analysis of neural network training behavior.

---

## Overview

The Neural Training Analysis Platform is a **training observability and diagnostics system** for neural networks.

It is composed of:

- a **Python SDK** embedded into training loops
- a **desktop UI** built with Electron, TypeScript, React, MobX, and Less

This product is **not** intended to be a shallow metrics logger.
It must go beyond loss curves and scalar dashboards to help developers and researchers understand:

- why training becomes unstable
- which layers are not learning effectively
- when collapse, saturation, or divergence begins
- which data patterns are contributing to failure modes
- how runtime bottlenecks correlate with model behavior
- which interventions are most likely to improve outcomes

The platform should behave more like an **observability stack for training systems** than a traditional experiment tracker.

---

## SDK Overview

The Python SDK is organized around a shared run context:

1. Start a `TrainingSession`.
2. Attach writers such as `MetricsLogger`, `EventLogger`, `SystemTelemetry`, `HookEngine`, and `ParameterDiagnostics`.
3. Log metrics, events, runtime logs, telemetry, layer summaries, parameter summaries, and artifacts against the same run.
4. Let the session and writer context managers flush buffered records when the run ends.

Important distinctions:

- `MetricsLogger` is the flexible scalar path. It supports user-defined custom metrics, namespaced metric keys, optional descriptions, and JSON-serializable metadata.
- `SystemTelemetry` is the built-in runtime path. Its first-class schema is fixed to runtime timings, CPU and RAM usage, GPU utilization and memory, overflow counters, skipped-step counters, and metadata.
- `SystemTelemetry` can mirror selected runtime fields into scalar metrics such as `system/runtime/step_time_ms`.
- `TrainingSession.add_log(...)` and `EventLogger.add_log(...)` create structured log events with `Info`, `Warning`, or `Error` identifiers.
- `ParameterDiagnostics` stores model parameter, gradient, and optimizer-update summaries separately from hook activation and gradient outputs.
- Custom scalar metrics are supported. Custom telemetry columns are not currently part of the first-class telemetry schema.
- The persistence client talks to the external service selected by `QUARK_PERSISTENCE_URL`, authenticates with `QUARK_USER_KEY`, and sends `X-Quark-Storage-Dir` so the service can resolve the run storage root.
- Lifecycle activity is persisted by `TrainingSession`; top-alert style diagnostics are persisted by `AnalyzerEngine` as alerts and health scores.
- Session context managers automatically log uncaught runtime exceptions as `Error` logs and keyboard/system exits as `Warning` interruption logs before updating lifecycle state.
- Built-in analyzers and phase detection look for conventional metric keys such as `train/loss`, `val/loss`, and runtime timing keys by default, though those key lists are configurable.

See [`USAGE.md`](./USAGE.md) for setup and instrumentation examples, [`sdk.md`](./sdk.md) for the SDK contract, and [`src/persistence/README.md`](./src/persistence/README.md) for the client-to-service persistence model.

---

## Product Goals

### Primary Goals

- Instrument training runs with minimal developer friction
- Collect meaningful low-cost and probe-based telemetry
- Analyze optimization, representation, data, and runtime behavior
- Surface alerts, health scores, and likely causes
- Provide an interactive UI for exploring runs and comparing them
- Support reproducibility through structured metadata and lineage

### Non-Goals for Initial Release

- Full cloud SaaS deployment
- Automated training orchestration
- Full hyperparameter optimization platform
- Support for every ML framework in v1
- Automatic run control or self-healing actions in production

---

## Product Architecture

```text
Training Script
   │
   │ uses
   ▼
Python SDK
   ├── Session Management
   ├── Metrics Logging
   ├── Hook-Based Instrumentation
   ├── Parameter Diagnostics
   ├── Event Logging
   ├── Artifact Persistence
   ├── Analyzer Engine
   ├── Phase Detection
   ├── Comparison Engine
   ├── Data Diagnostics
   ├── Recommendation Engine
   └── System Telemetry
   │
   ▼
Local Storage Layer
   ├── SQLite / DuckDB metadata
   ├── Parquet / file-based artifacts
   ├── Run schemas
   └── Analysis outputs
   │
   ▼
Desktop Application
   ├── Project & Run Explorer
   ├── Overview Dashboard
   ├── Layer Health View
   ├── Timeline View
   ├── Diagnostics View
   ├── Training Phases View
   ├── Compare Runs View
   ├── Data Diagnostics View
   ├── Recommendations Panel
   ├── Runtime View
   └── Run Details Panel
```

---

## Technology Stack

### SDK

- Python 3.11+
- PyTorch-first integration
- Pydantic for schemas
- NumPy
- SQLite for metadata persistence
- Parquet and filesystem storage for artifacts
- optional DuckDB for analytical queries
- optional psutil / pynvml for runtime telemetry

### UI

- Electron
- TypeScript
- React
- MobX
- Less
- IPC bridge between Electron main and renderer
- good charting library suitable for time-series and heatmaps

---

## Core Product Principles

### 1. Deep Observability

The platform must capture and expose:

- optimization dynamics
- layer behavior
- representation signals
- training events
- data diagnostics
- runtime/system telemetry
- inferred diagnostics and recommendations

### 2. Layered Cost Model

Instrumentation must support three cost levels:

- **always-on** for cheap scalar metrics
- **periodic probes** for moderate-cost layer and batch inspection
- **deep inspection** for heavy diagnostics triggered manually or by anomaly

### 3. Paired SDK ↔ UI Design

Every SDK module must have a corresponding UI module that makes its output explorable.
No telemetry should exist without a clear user-facing surface.

### 4. Local-First Workflow

The entire product must work on a single machine without any cloud dependency.

### 5. Reproducibility by Default

Each run must capture:

- configuration
- seed
- code version if available
- framework version
- hardware/environment info
- dataset version or user-defined dataset fingerprint
- parent-child run lineage

---

## Documentation Structure

- [`USAGE.md`](./USAGE.md) — Practical SDK setup and instrumentation examples
- [`sdk.md`](./sdk.md) — Python SDK specification
- [`ui.md`](./ui.md) — Desktop UI specification
- [`architecture.md`](./architecture.md) — Desktop UI specification
- [`guidelines.md`](./guidelines.md) — Coding guidelines


---

## SDK ↔ UI Module Mapping

| SDK Module | UI Module | Purpose |
|---|---|---|
| Session Manager | Project & Run Explorer | Manage runs, projects, lineage, status |
| Metrics Logger | Overview Dashboard | Scalar metrics and time-series views |
| Hook Engine | Layer Health View | Layer-wise activations, gradients, update stats |
| Parameter Diagnostics | Parameter Health View | Weight, bias, gradient, and optimizer-update summaries |
| Event Logger | Timeline View | Training lifecycle, structured events, and runtime logs |
| Artifact Store | Artifacts Panel | Histograms, snapshots, and stored probe outputs |
| Analyzer Engine | Diagnostics View | Alerts, health scores, explanations |
| Phase Analyzer | Training Phases View | Regime segmentation and transition evidence |
| Comparison Engine | Compare Runs View | Run-to-run comparison and diffs |
| Data Diagnostics Module | Data Diagnostics View | Batch difficulty and exposure analysis |
| Recommendation Engine | Recommendations Panel | Ranked intervention suggestions |
| System Telemetry Module | Runtime View | Machine and runtime performance diagnostics |
| Config / Schema Module | Run Details Panel | Reproducibility, config, environment metadata |

---

## Initial Delivery Plan

Implementation should proceed in paired SDK → UI task groups.

1. Run lifecycle and project explorer
2. Scalar metrics and overview dashboard
3. Layer instrumentation and layer health UI
4. Events and timeline
5. Artifacts and artifact browser
6. Diagnostics and health scoring
7. Phase detection and phase UX
8. Run comparison
9. Data diagnostics
10. Recommendations
11. Runtime telemetry
12. Config, schemas, and run details

Detailed requirements live in the spec files below.

---

## MVP Definition

### SDK MVP

- Session Manager
- Metrics Logger
- Hook Engine
- Event Logger
- Analyzer Engine
- basic System Telemetry
- Config / Schema Module

### UI MVP

- Project & Run Explorer
- Overview Dashboard
- Layer Health View
- Timeline View
- Diagnostics View
- Run Details Panel

### MVP Outcome

A developer can instrument a PyTorch run locally, inspect training beyond shallow metrics, view layer-level signals and alerts, and compare run health with reproducibility intact.

---

## Acceptance Criteria

The first usable version is complete when:

1. A PyTorch training loop can be instrumented in under 15 lines of setup code.
2. A run appears automatically in the desktop app.
3. Scalar metrics, events, and layer summaries render correctly.
4. Alerts and health scores work end-to-end.
5. Reproducibility metadata is stored and visible.
6. The UI stays usable on long-running experiments.
7. The product clearly feels like training analysis infrastructure rather than another chart viewer with a confidence problem.
