Batch Normalization & Layer Norm

Normalize activations within a mini-batch to keep training stable. BatchNorm lets you use higher learning rates and reduces sensitivity to initialization.

Learning Objectives

• →Normalize a mini-batch to zero mean unit variance
• →Apply learnable scale and shift parameters
• →Explain why BatchNorm reduces internal covariate shift
• →Compare BatchNorm to LayerNorm for sequences

Practice