EXHIBIT 20 OF 24 · MODULE 5: RED TEAMING & PRIVACY
DEFENSE
Differential Privacy in Fine-Tuning (DP-SGD)
Provable privacy guarantees via per-sample gradient clipping and calibrated Gaussian noise addition during fine-tuning.
INTERACTIVE EXHIBIT 20
Differential Privacy in Fine-Tuning (DP-SGD)
PRIVACY BUDGET: ε = 2.828 (δ = 1e-5)
PRIVACY EPSILON (ε)
ε = 2.828
Lower ε means stronger privacy. Production standard targets ε ≤ 3.0.
💡 Opacus / JAX: Train with ghost clipping to compute per-example gradients without allocating B full model backward graphs.
ARCHITECTURAL OVERVIEW
Differential Privacy in Fine-Tuning (DP-SGD)
Differentially Private Stochastic Gradient Descent (DP-SGD) provides mathematically provable privacy guarantees when fine-tuning models on sensitive data (medical records, legal contracts, private communications). By clipping per-example gradients and injecting calibrated Gaussian noise, DP-SGD guarantees that no single training example can significantly influence final model weights.
Per-Sample Gradient Clipping
Standard SGD computes gradients across a minibatch. DP-SGD computes individual gradient g_i for each sample and clips its L2 norm to threshold C: g_i' = g_i / max(1, ||g_i||_2 / C), bounding the sensitivity of any single record.
Calibrated Gaussian Noise Addition
Noise drawn from N(0, sigma^2 C^2 I) is added to the summed clipped gradients before the optimizer update step.
Privacy Budget Accounting (Epsilon, Delta)
Rényi Differential Privacy (RDP) accounts track cumulative privacy expenditure epsilon across training steps T. Lower epsilon means stronger mathematical privacy guarantees.
Where B is batch size, C is clipping norm, sigma is noise multiplier, q is subsampling ratio, and T is total step count. An (epsilon = 1.0, delta = 1e-5) guarantee mathematically bounds adversary privacy leakage.
REAL-WORLD PRODUCTION ENGINEERING
Use PyTorch Opacus or JAX private-gradient libraries with ghost clipping to compute per-sample gradients without memory explosion.
Fine-tune only parameter-efficient LoRA adapters with DP-SGD to minimize the parameter dimension over which noise is added.
Increase batch size B (e.g., B = 2048 to 8192): larger batch sizes improve the signal-to-noise ratio in private gradient estimation.