Skip to content
context Coding

Sebastian Raschka releases a tutorial on positional embeddings and RoPE in modern language models

only one source so far

Author Sebastian Raschka published a technical tutorial on his blog Ahead of AI about positional embeddings in LLMs – from classic absolute and relative methods to RoPE, with math, visuals, and a PyTorch implementation.

Sebastian Raschka published a detailed technical tutorial on his blog Ahead of AI explaining how language models encode and process information about token position. The text gradually covers classic methods (learned absolute positional embeddings as in GPT-2) as well as more modern approaches, primarily RoPE (relative positional encoding), using visual examples, mathematical derivation, and a sample implementation in PyTorch.

According to the author, the article is designed as a standalone building block for an upcoming series on trends in LLM architecture and complements the earlier material Build a Large Language Model (From Scratch). The author states that the tutorial assumes basic knowledge of LLM architecture and the attention mechanism (particularly query, key, and value vectors) and refers to his own earlier text on self-attention as an introduction for less experienced readers.

The text also includes an overview of which positional embedding methods are used by open-weight models released in 2026 – according to the author, a more detailed breakdown of these choices is planned for a follow-up part of the series. As a concrete architecture comparison, the article uses GPT-2 (the classic approach of adding positional information to the embedding before it enters the transformer block) and MiniMax-M3 (a newer model that inserts positional information via RoPE inside attention in each transformer block).

The available portion of the article ends with an analysis of an example involving two sentences that differ only in word order, through which the author demonstrates why explicit positional information is needed even in models with a causal (autoregressive) mask. For details, see the source article.

What changed

Why it matters

Understanding how RoPE and other positional methods work is useful for anyone who implements, fine-tunes, or selects LLMs for a specific deployment – positional encoding affects, for example, a model's ability to work with long context. The tutorial also summarizes which methods current open-weight models choose, which can serve as a reference point for navigating architectural trends in 2026.

Relevant practical impact

What this means

01

For individuals

For developers and ML practitioners, this is an accessible educational resource with math and PyTorch code that can be used to gain a deeper understanding of how attention and RoPE work and to get oriented in the architectural choices of current open-weight models.

What to do Go through the tutorial, including the PyTorch implementation, to verify your own understanding of RoPE and positional embeddings.
More practical updates →
LLM architecture Positional embeddings PyTorch RoPE transformers Tutorial

Check the original

Event sources

only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
Ahead of AI (Sebastian Raschka) community signal · first detected Understanding Positional Embeddings in Modern LLMs