Domain Neighborhood
Production ML
The engineering discipline around trustworthy model use: evaluation pipelines, dataset and model versioning, monitoring, drift, reproducibility, and operational tradeoffs.
Recommended Route
Start here, then follow the prerequisites forward.
This sequence is ordered for learning rather than inventory: lower difficulty, fewer prerequisites, and more central concepts come first.
- 01Evaluation Pipelines
Evaluation pipelines freeze data, splits, metrics, contamination checks, and report artifacts so model comparisons stay trustworthy.
19 mincodedemoafter Model Selection and Hyperparameter Search, Classification Metrics, Thresholds, and Calibration, Train/Dev/Test Splits, Cross-Validation, and LeakageCheck Model Selection and Hyperparameter Search first if the symbols feel slippery.
- 02Cost and Latency Observability
Cost and latency observability turns serving traces into pinned evidence: workload shape, token counts, tail latency, queue/cache signals, SLOs, pricing dates, and report artifacts.
16 mincodedemoafter LLM Serving at Scale: Prefill, Decode & Continuous Batching, Evaluation Pipelines, Tokenization & Vocabulary DesignWhy this follows: Cost and Latency Observability uses Evaluation Pipelines directly.
- 03Evaluation Harnesses and Benchmark Contamination
Evaluation harnesses turn scores into pinned protocol cards, while contamination audits ask what benchmark items the model may have already seen.
15 mincodedemoafter Train/Dev/Test Splits, Cross-Validation, and Leakage, Evaluation Pipelines, Dataset VersioningWhy this follows: both pages keep the production ml / evaluation thread active.
- 04Monitoring and Drift
Monitoring and drift separates input movement, label movement, concept change, and metric regression so alerts trigger the right investigation.
15 mincodedemoafter Evaluation Pipelines, Dataset Versioning, Classification Metrics, Thresholds, and CalibrationWhy this follows: both pages keep the production ml / evaluation thread active.
All Published Notebooks
Browse the territory.
Evaluation Pipelines
Evaluation pipelines freeze data, splits, metrics, contamination checks, and report artifacts so model comparisons stay trustworthy.
Cost and Latency Observability
Cost and latency observability turns serving traces into pinned evidence: workload shape, token counts, tail latency, queue/cache signals, SLOs, pricing dates, and report artifacts.
Evaluation Harnesses and Benchmark Contamination
Evaluation harnesses turn scores into pinned protocol cards, while contamination audits ask what benchmark items the model may have already seen.
Monitoring and Drift
Monitoring and drift separates input movement, label movement, concept change, and metric regression so alerts trigger the right investigation.
Advanced Bridges
Use these after the core path.
In Progress