A solo developer has released PSSA, a small language model that abandons the transformer architecture powering nearly every mainstream AI system today, from GPT to Llama. Written entirely in Rust with no PyTorch, no TensorFlow, and no machine learning framework underneath, PSSA reportedly learns faster than a matched transformer on the same data and generates text about twelve times quicker on identical CPU hardware. The project is early and small-scale, but it is a rare from-scratch attempt to prove that transformers are not the only viable path to competent language modeling.

What makes PSSA different from a transformer

Transformers work by having every token in a sequence “attend to” every other token, computing a score for each pair. This gives transformers their strong in-context reasoning, but it means the computational cost grows with the square of the sequence length, and the entire context has to be re-read at every single step of generation. That is a major reason long-context inference is expensive.

PSSA takes a different approach. It reads text one token at a time through a recurrent state-space layer, carrying a single fixed-size internal state forward through the sequence in one left-to-right pass rather than re-scanning everything each step. Alongside that, it maintains a bank of episodic memories it can look things up in in place of re-reading the full context, and it modifies part of its own weights while running. Because the state size stays constant, the cost of processing new tokens grows linearly with length instead of quadratically, and generating a new token does not require reprocessing everything that came before it.

How the training comparison was run

To test the claim fairly, the developer trained two models under identical conditions: same corpus, same tokenizer, same optimizer schedule, same random seed, and the same parameter count. One was PSSA, the other a standard transformer. Both were trained over 12.7 million tokens of cleaned WikiText-103, processed in 64 (PSSA) and 43 (transformer) sequential “links” of 200,000 encoded tokens each, with the transformer’s first training session cut short at link 43 by the hosting notebook’s time limit.

• PSSA finished at 3.98 training cross-entropy versus 4.43 for the transformer, a gap of 0.45 nats, corresponding to a perplexity of 53.7 versus 83.7.
• The transformer used its entire 12.7-million-token budget to reach a loss level PSSA had already surpassed around the 2-million-token mark.
• Across every checkpoint of both runs, the two loss curves never crossed and never touched, with PSSA ahead from the very first comparison point.

Checking the results hold up on unseen data

Training loss alone only shows how well a model fits the exact data it was fed, so the developer also scored both models on a separate 198,939-token slice of the corpus that neither run had ever seen during training. On a bounded 9,934-token window of that held-out data, PSSA finished 0.51 nats lower than the transformer, and the held-out gap of 0.43 nats closely tracked the training gap. The author frames this as evidence that PSSA is not simply memorizing the training stream more aggressively, but genuinely generalizing better to new text.

On generation speed, testing 200 tokens of output from the same prompt, same sampler, and same CPU showed PSSA producing text roughly twelve times faster than the transformer. This tracks with the architectural difference: a recurrent model’s per-token cost stays flat regardless of how much text has already been generated, while a transformer must re-read its entire growing context at every step.

What this does and does not prove

The author is notably candid about the limits of these results. The entire experiment was run on a free hosted notebook with a single entry-level GPU, with training broken into 200,000-token links because sessions get cut after a few hours. That means the comparison, while carefully controlled, happened at a small parameter scale on a modest corpus and modest hardware. Whether PSSA’s advantage holds at 10x or 100x the parameter count, whether the episodic memory bank continues to matter as scale increases, and how it stacks up against other modern recurrent architectures (not just transformers) are all open questions the author explicitly says require more compute to answer, specifically a GPU with real VRAM and time measured in days rather than hours.

Trying it and contributing

PSSA is available on GitHub with direct runtime dependencies limited to ureq for dataset downloads and tokenizers for byte-level BPE tokenization. Running the binary with no arguments brings up a home screen listing every available command along with any checkpoints and corpora already present in the working directory. The project accepts issues and pull requests, with the biggest current needs being kernel performance work, building a modern recurrent baseline for comparison, and evaluation methods beyond simple next-token loss; contributors are asked to validate branches with cargo test --release before opening a PR. The author is also seeking compute sponsorship to run larger-scale experiments, noting that sponsorship funds compute only, with sponsors credited in the repository and any resulting write-ups.

Frequently asked questions

Is PSSA meant to replace transformers like GPT? Not at this stage. It is a small-scale research project demonstrating that a recurrent, memory-augmented, self-modifying architecture can out-learn a matched transformer at limited scale. Whether the approach scales competitively with production-size transformer models remains untested and unproven.

Why does it matter that it runs faster on CPU? Because PSSA’s cost per token does not grow with sequence length the way a transformer’s does, it can generate text without repeatedly reprocessing the full context, which is the main reason it showed roughly a twelve-times generation speed advantage in this test.

Can I run PSSA myself? Yes, the code is public on GitHub and runnable with only two lightweight Rust dependencies, no ML framework required, though reproducing or extending the training results needs a GPU and time to run the multi-link training process.

Source: PSSA: A non-transformer language model written from scratch in Rust