Sebastian Raschka releases a tutorial on positional embeddings and RoPE in modern language models
Author Sebastian Raschka published a technical tutorial on his blog Ahead of AI about positional embeddings in LLMs – from classic absolute and relative methods to RoPE, with math, visuals, and a PyTorch implementation.
Sebastian Raschka published a detailed technical tutorial on his blog Ahead of AI explaining how language models encode and process information about token position. The text gradually covers classic methods (learned absolute positional embeddings as in GPT-2) as well as more modern approaches, primarily RoPE (relative positional encoding), using visual examples, mathematical derivation, and a sample implementation in PyTorch.
According to the author, the article is designed as a standalone building block for an upcoming series on trends in LLM architecture and complements the earlier material Build a Large Language Model (From Scratch). The author states that the tutorial assumes basic knowledge of LLM architecture and the attention mechanism (particularly query, key, and value vectors) and refers to his own earlier text on self-attention as an introduction for less experienced readers.
The text also includes an overview of which positional embedding methods are used by open-weight models released in 2026 – according to the author, a more detailed breakdown of these choices is planned for a follow-up part of the series. As a concrete architecture comparison, the article uses GPT-2 (the classic approach of adding positional information to the embedding before it enters the transformer block) and MiniMax-M3 (a newer model that inserts positional information via RoPE inside attention in each transformer block).
The available portion of the article ends with an analysis of an example involving two sentences that differ only in word order, through which the author demonstrates why explicit positional information is needed even in models with a causal (autoregressive) mask. For details, see the source article.
Why it matters
Understanding how RoPE and other positional methods work is useful for anyone who implements, fine-tunes, or selects LLMs for a specific deployment – positional encoding affects, for example, a model's ability to work with long context. The tutorial also summarizes which methods current open-weight models choose, which can serve as a reference point for navigating architectural trends in 2026.
Relevant practical impact
What this means
For individuals
For developers and ML practitioners, this is an accessible educational resource with math and PyTorch code that can be used to gain a deeper understanding of how attention and RoPE work and to get oriented in the architectural choices of current open-weight models.
Check the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.