Grant Sanderson and Alok Puranik: Positional Encodings Through Group Theory

Grant Sanderson asks Alok Puranik why he used group theory to study positional encodings. Puranik noticed that RoPE rotates pairs of coordinates as position changes, while some linear-attention methods exponentially suppress older values. Both resemble representations of an additive time group by matrices.

Sanderson likes the change in question: instead of asking whether a particular encoding is natural, what forms must an encoding take under reasonable assumptions? Puranik says he wanted to know whether the common algebraic structure was a coincidence and whether it left useful possibilities unexplored.

Sanderson reviews attention with a sentence in which adjectives such as “fluffy” and “blue” should influence the noun “creature.” Queries and keys are compared through dot products, but the basic operation does not itself tell the model where the words occur.

Puranik writes queries and keys as q(t) and k(s), using time notation that also permits continuous time. Restricting t and s to integers gives the familiar sequence setting. Sanderson asks whether this framing reflects financial time series.

Puranik says real-world time series may be irregular, and continuous time can also be useful outside finance, including settings with data-dependent decays. One can regard time as continuous even when observations are sampled at particular points.

They focus on attention scores rather than values: a conventional score uses q(t)ᵀk(s). Even though the vectors are indexed by time, the dot product has no explicit way to distinguish identical keys appearing at different positions.

The proposed starting point modifies each query and key as a function of its own time. At this stage the transformations are arbitrary; the presenters mention original sinusoidal encodings and RoPE’s rotations as examples of different ways to incorporate position.

Why transform each vector once rather than separately for every query-key pair? Puranik says the restriction is motivated by efficiency: n queries and n keys would otherwise expand into potentially n² differently modified pairs.

The first substantive mathematical restriction is linearity in the query and key vectors. Puranik says this is difficult to justify as a universal law, but linear maps are natural for an inner-product problem, and he had not found practical examples using nonlinear ones.

With that assumption the transformations become time-indexed matrices F(t) and G(s). The score is q(t)ᵀF(t)ᵀG(s)k(s), so the product between query and key contains the positional effect. This is a narrower and more concrete space than arbitrary functions.

The central additional assumption is translation invariance: the positional interaction depends on the difference t−s, not on an absolute origin. Puranik writes A(t−s)=F(t)ᵀG(s). Sanderson compares looking for a symmetry and deriving its consequences to a physicist’s approach.

The matrix A can be thought of as translating a key’s influence across a time difference before pairing it with a query. In the language example, the relevance of an adjective may change as the distance to a later token increases.

Puranik also imposes A(0)=I: with zero time shift, the ordinary dot product is sufficient. He notes that an invertible, time-independent matrix at zero could instead be absorbed into the keys, so this normalization is reasonably motivated.

Sanderson describes translation invariance visually: all pairs of positions along the same diagonal share the same relative difference. Puranik notes that the product A contains the relevant pairing information, while F and G can be recovered up to a common transformation that cancels in the dot product.

Can A be an arbitrary time-dependent matrix? Puranik says the assumptions impose consistency. Since F(t)ᵀG(t)=A(0)=I, the two factors have an inverse relationship at equal times.

They ask what happens when two A matrices are multiplied. This product need not appear directly in an attention computation, but it has a natural interpretation: move by one time difference and then by another.

Expanding A(s)A(t) in terms of F and G, translation invariance lets Puranik shift the first pair of time arguments by t. The middle factors then cancel using the equal-time inverse relationship, leaving A(s+t).

Thus addition in the time argument corresponds to multiplication of matrices: A(s)A(t)=A(s+t). Puranik acknowledges that the assumptions were motivated by this kind of structure, but emphasizes that they give a concise description once accepted.

Sanderson observes that a scalar function turning addition into multiplication suggests an exponential. Puranik says he began with the observation that both ordinary exponentials and rotations, which can be expressed as complex exponentials, occur in familiar methods.

Related representation-theoretic reasoning also appears in image positional encodings, where positions have two directions. There the composition of the two spatial translations must be handled consistently, rather than considering a single time axis.

For continuous time, Puranik assumes A is differentiable and differentiates the composition law with respect to t at zero. For integer indices, he notes, one can instead reach positions by repeatedly adding one, making that case simpler in some respects.

Calling A′(0) the matrix X gives A′(s)=A(s)X. Together with A(0)=I, the differential equation yields A(s)=e^(sX). This describes the solutions under the stated assumptions, including differentiability in the continuous-time argument.

Sanderson explains the matrix exponential through its power series: take powers of the square matrix just as in the Taylor expansion of the scalar exponential. Its differential property is another way to understand why it solves the equation.

The next task is to characterize what different choices of X mean. For a scalar exponent, positive values grow, negative values decay and imaginary values rotate; matrix exponentials could contain these behaviors and potentially more.

Puranik makes another simplifying assumption: time-independent changes of basis in keys and queries do not matter for the positional encoding. Such a transformation can be handled by the surrounding linear layers, so the matrices can be studied up to change of basis.

When X is diagonalizable, choosing an eigenbasis splits it into independent diagonal components. Exponentiating the diagonal matrix exponentiates each eigenvalue separately, so different components of the query and key do not interact through off-diagonal terms.

Sanderson explains the intuition: in an eigenbasis the action on each subspace stays in that subspace, and the matrix powers no longer mix the components. Puranik notes that the eigenvalues can be complex even when the original matrix is real.

The score becomes a sum of contributions from individual components, each weighted by an exponential of its eigenvalue. This reduces the diagonalizable case to examining real one-dimensional pieces and pairs associated with complex eigenvalues.

For a real eigenvalue, the associated factor is e^(tx). A positive x grows with distance, zero leaves that component time-independent, and a negative x gives exponential decay. Sanderson notes why decay can seem natural for the influence of older information.

Puranik cautions that a zero eigenvalue is not necessarily useless: it may be just one time-independent component in a larger encoding. Positive real growth is potentially unstable, while negative exponential decay appears in linear attention.

For a real matrix, non-real eigenvalues occur in complex conjugate pairs. Puranik separates the real part, which supplies growth or decay, from the imaginary part. For the rotation discussion he sets the real part to zero.

Sanderson uses a two-dimensional rotation to connect complex exponentials to geometry. In a suitable basis the eigenvalues are complex conjugates; expressed in a real two-dimensional subspace, their action is rotation.

The rotation rate is controlled by the imaginary component. A nonzero real component adds exponential scaling: positive produces growth, negative decay, and zero leaves pure rotation. A visualization from Puranik’s post shows the combined motion.

For diagonalizable matrices, these one-dimensional real components and two-dimensional rotational components can be combined. Sanderson emphasizes the useful intuition: on each relevant subspace the dynamics reduce to growth, decay or rotation, subject to the diagonalizability qualification.

Puranik connects the rotational case to RoPE. It uses pairs of coordinates with pure rotations and no exponential scaling, represented by a block-diagonal A(t) whose blocks rotate at different frequencies.

RoPE’s frequency selection is a practical design choice rather than a consequence of the group-theoretic derivation. Puranik describes geometrically spaced frequencies spanning different time scales; fast rotations distinguish nearby positions and slow rotations distinguish much longer separations.

They briefly return to F and G, the actual query and key transformations. Puranik says a suitable pair can be constructed from A, but Sanderson suggests moving on to the more interesting question: does the framework reveal anything beyond the familiar cases?

Puranik turns to non-diagonalizable matrices, described through Jordan blocks. Although they are often set aside and can be approximated by diagonalizable matrices, they deserve consideration when looking for the full range of mathematical behavior.

He starts with the two-by-two matrix X=[[0,1],[0,0]]. It maps the first basis vector to zero and the second to the first, so the two directions are coupled rather than acting independently as in the diagonal case.

Applying X twice gives zero, so the exponential series terminates: e^(tX)=I+tX. A matrix exponential has produced a polynomial in time, rather than just the growth, decay or rotation seen in the preceding components.

To build intuition, Puranik considers a puck moving at constant velocity with no forces. Its state is position and velocity; after time t the position is x₀+tv and the velocity is still v. The matrix [[1,t],[0,1]] produces exactly this evolution.

A larger Jordan block can generate quadratic rather than linear behavior, analogous to adding an acceleration component. In positional encoding terms, the simple block can introduce influence that grows linearly with separation; Puranik says that generic growth may still be poorly behaved.

A colleague pointed out a connection to ALiBi. Unlike the original setup, ALiBi directly subtracts a distance-dependent linear penalty from the query-key score, rather than explicitly rotating or otherwise modifying the vectors.

ALiBi’s penalty depends on t−s and therefore has translation invariance. Puranik suggests relating its linear term to the linear behavior produced by a Jordan block, by augmenting the query and key with fixed extra components.

Working through the matrix product, they choose the added components so the extra term becomes −p(t−s), with the other added products contributing zero. Sanderson welcomes the visible process of trying and correcting a construction rather than presenting it as an effortless leap.

Puranik notes that fixing extra input components introduces a time-independent modification beyond the original framing. With those particular augmented inputs and a defective block in the generator, however, the construction exactly recovers ALiBi’s linear bias.

Sanderson asks about the research culture behind the work. Puranik says he first shared the result internally with Jane Street’s machine-learning community, then revised it into a public blog post; other informal talks can range further from day-to-day work.

Looking back, Puranik feels reassured that sensible possibilities under the framework appear fairly well studied. He regards larger Jordan blocks and nonzero eigenvalues as less explored corners, but is skeptical they would unlock much in practice; he stresses that he did not investigate all their implications thoroughly.