Attention & TransformersMedium
Sinusoidal Positional Encoding
~12 mincode completion
Implement positional_encoding(seq_len, d_model) returning a NumPy array of shape (seq_len, d_model).
Examples
Single token, d_model=2
- Input
- positional_encoding(1, 2)
- Output
- [[0, 1]]
Two tokens, d_model=2
- Input
- positional_encoding(2, 2)
- Output
- [[0, 1], [0.84147, 0.5403]]
Hints
Hint 1
applies elementwise, so negate the whole array and exponentiate it in one go.
Hint 2
A common slip here: Starting positions at 1 instead of 0.
Requirements
Positions are 0-indexed.
Use vectorized NumPy operations (no Python loop over positions).
Use a numerically stable expression for the denominator term.
seq_len: Number of token positions.d_model: Embedding width (assume even for this problem).
Constraints
Allowed library: NumPy only
Time limit: 200 ms, Memory: 64 MB
Where this shows up
~12 min
••••••••••••••••
8 employers weight this skill
4 frontier labs, 3 AI product companies, 1 enterprise vendor. Top match scores 92.
Python
import numpy as np
def positional_encoding(seq_len: int, d_model: int) -> np.ndarray:
"""
Build sinusoidal positional encodings used in Transformers.
Args:
seq_len: Number of token positions.
d_model: Embedding width (assume even for this problem).
Returns:
PE matrix of shape (seq_len, d_model) where:
- even columns use sin(...)
- odd columns use cos(...)
"""
# 1) Create an output array of zeros with shape (seq_len, d_model)
# 2) Create a column vector of positions [0, 1, ..., seq_len-1]
# 3) Build the even-dimension index vector: [0, 2, 4, ...]
# 4) Compute the scaling term 10000^(-2i/d_model) in a numerically stable way
# 5) Fill even columns with sin(...) and odd columns with cos(...)
# YOUR CODE HERE
pass