Vaswani et al., Attention Is All You Need, NeurIPS 2017
Every sequence in this dataset ends with a token that appeared exactly once earlier. The answer is whatever token came immediately after that earlier occurrence.
Nothing about position or frequency helps. A bag-of-tokens model, an n-gram model, a feedforward net on the flattened sequence: all of them sit at chance, which is . The only way to solve it is to look back to a specific earlier position, chosen by content.
That is precisely what attention does. Equation 1 of the paper:
You will build each piece, check it against values you can verify by hand, and then hand-set the projections to form an induction head: key on the previous token, query with the current token, read the value at whichever position matches. No gradient descent anywhere.
Give the model a sense of order without any learned parameters.
Attention is permutation-invariant on its own, so position has to be injected. Section 3.5 uses fixed sinusoids of geometrically spaced wavelengths:
Even dimensions get the sine, odd ones the cosine, and the pair at index 2i and 2i+1 shares one wavelength.
Implement positional_encoding(train_df) for 16 positions and dmodel=8, returning shape, row_0, row_1, row_15_first_four and norm_row_5, with every float rounded to 6 places. Ignore train_df.
Check row_0 first: at pos=0 every sine is 0 and every cosine is 1, so it should alternate exactly [0, 1, 0, 1, ...]. If it does not, your even and odd indices are swapped.
Evaluated server-side against a hidden test set.