Batch Normalization & Layer Norm

Medium

Normalize activations within a mini-batch to keep training stable. BatchNorm lets you use higher learning rates and reduces sensitivity to initialization.

Learning Objectives

→Normalize a mini-batch to zero mean unit variance
→Apply learnable scale and shift parameters
→Explain why BatchNorm reduces internal covariate shift
→Compare BatchNorm to LayerNorm for sequences

Coding Problems (2)

Batch Normalization Forward

Pro

~15 min· Medium

Layer Normalization

Pro

~12 min· Easy

Concept Check

Optional multiple-choice questions on the ideas behind this section. Most useful after you have tried the coding problems above.