Sinusoidal Positional Encoding
Sinusoidal Positional Encoding
Self-attention is permutation-invariant: without extra information, the model cannot tell token order.
Transformers inject position information by adding a deterministic encoding vector to each token embedding.
For position and model dimension :
Key intuition:
Your task: Implement positional_encoding(seq_len, d_model) returning a NumPy array of shape (seq_len, d_model).
Implementation requirements:
1. Positions are 0-indexed.
2. Use vectorized NumPy operations (no Python loop over positions).
3. Use a numerically stable expression for the denominator term.
Example Tests
Single token, d_model=2
Input: {"d_model":2,"seq_len":1}
Expected: [[0,1]]
Two tokens, d_model=2
Input: {"d_model":2,"seq_len":2}
Expected: [[0,1],[0.84147,0.5403]]
Shape check: seq_len=4, d_model=4
Input: {"d_model":4,"seq_len":4}
Expected: [4,4]