03 / Research infrastructure and model evaluation
AIchemy / Alchemy SDK — orchestration and reproducible loops for AI research
Code-first infrastructure connecting GPU and SLURM scheduling, training observability, experiment lineage, and research decisions.
Platform layer: AIchemy
- Schedules work across SLURM clusters and standalone GPU nodes, with priorities, DAG dependencies, grid search, automatic placement, and capacity reservations.
- Handles reconnection, rolling-upgrade compatibility, convergent task state, and backups.
- Collects GPU and training metrics, classifies OOM, NCCL, and CUDA failures, and records audit logs and notifications.
Developer layer: Alchemy SDK
- Records steps, losses, metrics, evaluations, checkpoints, and artefacts directly from training code.
- Uses should_stop() to save cleanly after SLURM pre-emption or an operator stop.
- Provides callbacks for PyTorch Lightning and Hugging Face.
- Records experiment lineage, summaries, diffs, manifests, timelines, and research bundles.
- Turns keep, drop, rerun, and fork decisions into a reproducible and auditable loop.
Architecture choice
The platform answers where, when, and with which resources a task runs. The SDK records what the experiment did, what it produced, and why the next research decision was made. Keeping those concerns separate avoids forcing research semantics into the cluster scheduler.
Use case
Task orchestration, experiment management, reliability, and observability for algorithm engineers, spanning infrastructure and developer-facing APIs.