Most language models today are built around the Transformer paradigm.

That makes sense.

But I wanted to explore a different question:

What if language generation does not need to be modeled as attention over a context window?

What if a model could generate language by carrying an evolving latent state through a learned geometry?

That is the idea behind DRM Language Emitter.

Repository:

https://github.com/gnai-creator/drm-language-emitter

What is DRM Language Emitter?

DRM Language Emitter is an experimental, geometry-first language model lab.

It is not a Transformer.

Inside the DRM model, it does not use:

• Transformer blocks
• self-attention
• Q/K/V attention
• nn.MultiheadAttention
• KV cache

Instead, it treats language generation as controlled motion through a learned relational manifold.

The basic flow is:

token
  -> latent state z_t
  -> active directions
  -> learned relational metric
  -> controlled latent motion
  -> next latent state z_{t+1}
  -> token logits

The model is still autoregressive.

But its memory is not attention over a token sequence.

Its memory is the evolving latent state.

The core hypothesis

The working hypothesis is:

Language generation can be modeled as motion through a learned relational state space.

That means the model does not simply ask:

Which previous tokens should I attend to?

It asks something closer to:

Where am I in latent space?
Which directions are active?
How expensive is movement under the learned metric?
How should the state move before emitting the next token?

This is why I call it a geometry-first language emitter.

A simplified architecture

The architecture can be summarized as:

input_ids
   |
TokenEmbedding
   |
for each time step:
   |
   z_t
   |
DirectionField(z_t)
   -> directions V(z_t)
   -> gates a(z_t)
   -> effective active dimension dimD
   |
RelationalMetric(z_t)
   -> diag + U U^T
   |
DRMFlow(z_t, token_embedding, directions, gates)
   -> dz
   |
Metric action g_z(dz, dz)
   |
StateUpdater
   -> z_{t+1}
   |
LanguageEmitter(z_{t+1})
   -> logits

A minimal conceptual version looks like this:

for token in sequence:
    embedding = token_embedding(token)

    directions, gates = direction_field(z)
    metric = relational_metric(z)

    dz = drm_flow(z, embedding, directions, gates)
    action = metric_action(metric, dz)

    z = state_updater(z, dz)
    logits = language_emitter(z)

The important part is that the model has an explicit internal geometry.

It can log and measure:

• metric action
• active dimension
• gate entropy
• metric norm
• condition proxy
• recurrence
• stability
• low-action path diagnostics

This makes the model interesting not only as a generator, but also as an object of study.

Why not just use a Transformer?

A Transformer is the correct baseline.

That is why the repository includes tiny Transformer comparisons.

But the goal of DRM is not to replace Transformers by declaration.

The goal is to test whether a different computational primitive can be useful in small regimes.

The Transformer primitive is attention.

The DRM primitive is controlled latent motion under a learned metric.

These are very different assumptions.

A Transformer builds context by looking backward.

DRM carries context by evolving state forward.

A Transformer computes token-token interactions.

DRM computes state-motion-emission dynamics.

Why geometry?

Because geometry gives us measurable structure.

If language is treated as a trajectory, we can ask questions like:

• Does the latent state collapse?
• Does the metric become unstable?
• Which directions are active?
• Is the model moving through a narrow or broad region?
• Is generation smooth or chaotic?
• Do symbolic transitions correspond to stable latent movement?

This opens the door to diagnostics that are harder to express in a standard black-box token predictor.

The goal is not mystical geometry.

The goal is measurable geometry.

Repository structure

The repository contains:

src/drm_language_emitter/   DRM model package
transformer/                tiny Transformer baseline
world_model/                tiny symbolic world-model baseline
scripts/                    training, generation, evaluation, sweeps, dashboards
configs/                    DRM and benchmark configs
docs/                       math, limitations, competition notes, benchmark artifacts
tests/                      smoke and invariant tests

The project is CPU-runnable.

CUDA is optional.

Quick start

Install:

pip install -e .

Train a tiny DRM model:

python scripts/train_tiny.py \
  --config configs/tiny.yaml \
  --text data/tiny.txt

Generate text:

python scripts/generate.py \
  --checkpoint runs/tiny/drm_tiny.pt \
  --prompt "DRM "

Run geometry diagnostics:

python scripts/eval_geometry.py \
  --checkpoint runs/tiny/drm_tiny.pt

python scripts/eval_geodesic_paths.py \
  --checkpoint runs/tiny/drm_tiny.pt

Tiny benchmark: DRM vs Transformer vs World Model

The repository also includes a small symbolic benchmark.

This benchmark compares:

• DRM Language Emitter
• Tiny Transformer
• Tiny supervised symbolic world model

The task is a deterministic symbolic gridworld serialized as text.

The models need to predict symbolic transitions such as:

state + action -> next state + reward + done

This is not visual world modeling.

This is not a benchmark against large multimodal world models.

It is a tiny symbolic text-world designed to test whether models can learn discrete dynamics expressed as language.

Metrics

The benchmark reports:

• validation cross-entropy
• next-state exact match
• rollout exact match
• reward accuracy
• done accuracy
• invalid state rate
• parameter count
• elapsed time
• tokens seen
• throughput

This is important because low loss alone does not necessarily mean correct symbolic dynamics.

A model can learn token-level regularities while still failing to predict exact state transitions.

Latest local result

The completed benchmark produced:

runs: 72
aggregate rows: 24

Top results by next-state exact match:

The most interesting part is not that DRM “wins everything”.

It does not.

The result is more nuanced:

• DRM had the best next-state exact match in this tiny symbolic text-world.
• Transformer 220k had the lowest invalid-state rate.
• The tiny supervised world model reached low CE, but did not convert that into strong exact-match or rollout performance.
• Rollout exact match is still very low.

So the honest interpretation is:

DRM shows an early signal on symbolic next-state prediction, but the benchmark is still diagnostic, not decisive.

The main lesson

For me, the most important takeaway is:

Low token-level cross-entropy does not automatically imply correct symbolic transition modeling.

That matters for world-model-like tasks.

If a model is supposed to represent dynamics, then we should not only ask whether it predicts likely tokens.

We should also ask whether it predicts valid states, exact transitions, and coherent rollouts.

What I am not claiming

I am not claiming that DRM is better than Transformers in general.

I am not claiming that DRM is better than world models in general.

I am not claiming that this benchmark says anything about large multimodal world models.

I am not claiming robust long-horizon planning.

This is a small research scaffold.

The results are early.

The exact-match values are still low.

The model needs more work.

What I am claiming

DRM Language Emitter is a functional non-Transformer language model prototype.

It has explicit, measurable geometry.

It can be compared against Transformer and symbolic world-model baselines.

And in a tiny symbolic text-world benchmark, it showed an interesting signal on next-state exact match.

That is enough to keep investigating.

Reproducing the symbolic benchmark

Generate the dataset:

python scripts/make_tiny_world_dataset.py \
  --output-root data/tiny_world \
  --seed 1 \
  --grid-size 5 \
  --num-train 20000 \
  --num-val 2000 \
  --max-rollout-len 8

Run the sweep:

python scripts/sweep_world_model_competition.py \
  --steps 1000 2000 3000 \
  --seeds 1 2 3 \
  --dataset-root data/tiny_world \
  --output-root runs/world_model_competition

Generate the dashboard:

python scripts/make_world_model_dashboard.py \
  --root runs/world_model_competition \
  --title "DRM vs Transformer vs Tiny Symbolic World Model"

Next steps

The next things I want to improve are:

• constrained symbolic decoding
• stronger rollout evaluation
• CUDA and time-matched runs
• curriculum variants for symbolic worlds
• more baselines
• better ablations around metric, gates, and active dimension
• tests to isolate whether learned geometry helps beyond cross-entropy

Final thought

This project started from a simple intuition:

Maybe language generation can be treated as movement.

Not metaphorically.

Computationally.

A token enters.

A state moves.

A geometry shapes the motion.

A new token is emitted.

That is DRM Language Emitter.

Repository:

https://github.com/gnai-creator/drm-language-emitter

Feedback, criticism, reproduction attempts, and benchmark suggestions are welcome.