Optimising Memory in Von Neumann Architecture
Architecture

Optimising Memory in Von Neumann Architecture

Why Memory Optimisation Matters, and Why It Is Not Enough

August 20269 min read

Introduction: The Workaround Era

In Module 02, we explored the von Neumann bottleneck, the fundamental limit imposed by the separation of processor and memory. Data must move across a bus. The bus is slow. The CPU waits. The bottleneck is physical, not software.

But engineers have not been idle. For decades, they have developed sophisticated techniques to work around this bottleneck. Caching. Hardware acceleration. Memory compression. Retrieval-Augmented Generation (RAG). Summarisation. Model quantization.

Each of these techniques has delivered real, measurable improvements. They have allowed AI to scale far beyond what seemed possible. They have bought time.

But they have not solved the problem.

This module explores the most common memory optimisation techniques in use today, how they work, why they help, and ultimately why they cannot overcome the von Neumann bottleneck.

Understanding these limitations is essential. It is the key to understanding why a new architecture is required, and why Zeus is not an improvement on von Neumann, but a replacement.


1. The Memory Hierarchy Revisited: Why It Exists and Why It Fails AI

The Trade-Off: Speed vs. Capacity

In Module 02, we introduced the memory hierarchy:

LevelTechnologySpeedCapacity
RegistersOn-chip~1 cycleBytes
L1 CacheSRAM~1-2 nsKB
L2/L3 CacheSRAM~10 nsMB
DRAMCapacitor-based~100 nsGB
SSDFlash~100 µsTB

This hierarchy exists because of a fundamental trade-off: speed is expensive.

SRAM (cache) is fast but costly and power-hungry. DRAM is slower but cheaper. SSDs are slower still but offer vastly more capacity. The hierarchy is a compromise, an attempt to balance speed, cost, and capacity.

For traditional computing, this compromise works.

For AI, it fails.

Why the Hierarchy Fails AI

ProblemWhy It Matters for AI
AI models are massiveA single model can require 140GB+ of weights, far more than cache or even DRAM can hold at once
AI requires low-latency accessMemory latency directly impacts inference speed, the user waits for the AI to respond
AI models have massive working setsThe entire model, plus the context window, must be accessed continuously

In traditional computing, data access is localised. The CPU can cache frequently used data. In AI, the working set is the entire model, the cache cannot hold it.

The hierarchy that works for spreadsheets fails for large language models.


2. Common Optimisation Techniques: And Their Limitations

Over the years, engineers have developed a range of techniques to work around the memory wall. Each one helps, but each one also has fundamental limitations.

1. Caching

What It Is: Small, fast memory (SRAM) placed close to the CPU to store frequently accessed data.

How It Works: When the CPU needs data, it first checks the cache. If the data is there (a "cache hit"), it is retrieved instantly. If not (a "cache miss"), the CPU must fetch it from slower DRAM.

Why It Helps: Reduces the number of slow DRAM accesses. Typical cache hit rates are 90%+ for traditional workloads.

Why It Fails for AI:

  • Caches are small, typically MB, not GB
  • AI models are huge, tens or hundreds of GB
  • The working set does not fit in cache
  • Cache miss rates are high, the CPU still waits for DRAM

Limit: Caching is effective for predictable, localised access patterns. AI access patterns are neither.


2. Hardware Acceleration (GPUs, TPUs)

What It Is: Specialised processors designed to perform many operations in parallel.

How It Works: GPUs and TPUs contain thousands of cores that can perform operations simultaneously. They are designed for the massive parallelism of matrix multiplication and neural network inference.

Why It Helps: GPUs can perform thousands of operations in the time it takes a CPU to perform one. Training time drops from months to days.

Why It Fails for AI:

  • Memory bandwidth is still the bottleneck
  • Even with thousands of cores, they all wait for data to arrive from memory
  • GPU memory is expensive, 288GB per unit, and even that fills quickly
  • The CPU must still move data to and from the GPU

Limit: GPUs are better at hiding latency than CPUs, but they cannot eliminate it. Data must still move across a bus. The memory wall remains.


3. Memory Compression

What It Is: Compressing data in memory to fit more into the available space.

How It Works: Data is stored in a compressed format. When the processor needs it, it is decompressed on the fly.

Why It Helps: Reduces memory footprint. A model that would not fit in available memory can be compressed to fit.

Why It Fails for AI:

  • Compression and decompression add overhead
  • The CPU must spend time compressing and decompressing data, time it could spend computing
  • Lossy compression loses information, accuracy degrades
  • Decompression latency adds to access time

Limit: Compression trades compute time for memory space. It helps when memory is the constraint, but adds latency when compute is the constraint. For AI, both matter.


4. Retrieval-Augmented Generation (RAG)

What It Is: Instead of storing everything in the context window, the AI retrieves relevant information from an external database when needed.

How It Works: When the user asks a question, the AI searches an external database for relevant information, retrieves it, and includes it in the context window.

Why It Helps: Extends the effective memory of the system. The AI can access information beyond its context window.

Why It Fails for AI:

  • RAG is a retrieval mechanism, not memory
  • It treats all retrieved information uniformly; it cannot capture the nuanced context of a conversation
  • It introduces latency, the AI must wait for the retrieval to complete
  • It can cause hallucinations, if the retrieved information is wrong or taken out of context
  • It is structurally vulnerable to memory poisoning, an attacker can poison the RAG database to influence the AI's outputs

Limit: RAG extends memory, but it is not true memory. It is a lookup mechanism. It cannot recall the full context of a conversation.


5. Summarisation

What It Is: Compressing long conversations into shorter summaries to fit within the context window.

How It Works: When a conversation exceeds the context window limit, the AI summarises the earlier parts and discards the original tokens.

Why It Helps: Reduces the number of tokens in the context window. The AI can maintain a longer conversation without exceeding the limit.

Why It Fails for AI:

  • Summarisation loses nuance, detail, and context
  • The AI cannot recall the specifics of what was discussed
  • Summarisation errors compound, a mistake in a summary propagates through the rest of the conversation
  • The AI is forced to choose between forgetting the past (summarisation) or limiting the conversation

Limit: Summarisation is a lossy compression technique. It helps in the short term, but long-term accuracy and context are sacrificed.


6. Model Quantisation

What It Is: Reducing the precision of the model's weights, for example, from 32-bit floating point to 8-bit integer.

How It Works: The model weights are stored in a lower precision format. This reduces memory footprint and speeds up computation.

Why It Helps: A quantised model uses less memory and runs faster.

Why It Fails for AI:

  • Quantisation reduces accuracy
  • The model loses nuance, precision, and capability
  • Some models degrade significantly, tasks that require high precision (e.g., reasoning, mathematics) are affected first

Limit: Quantisation is a trade-off, speed for accuracy. It helps when memory and compute are constraints, but at the cost of capability.


3. The Fundamental Problem: Why These Techniques Cannot Solve It

TechniqueWorks Around The ProblemSolves The ProblemWhy It Cannot Solve It
Caching✅❌Caches are too small for AI working sets.
Hardware Acceleration✅❌Data must still move, the memory wall remains.
Memory Compression✅❌Adds overhead and latency, reduces accuracy.
RAG✅❌Retrieval is not memory, lacks nuance and context.
Summarisation✅❌Loses information and context.
Model Quantisation✅❌Reduces accuracy and capability.

None of these techniques address the fundamental issue: the separation of processor and memory.

The von Neumann architecture forces data to move. As long as data must move, there will be a bottleneck. As long as there is a bottleneck, performance will be limited.

You cannot fix a physical problem with software patches.


4. The Industry's Blind Spot

The industry celebrates each of these techniques as a breakthrough. But this celebration misses the point.

AnnouncementReality
"2 million token context window"Accuracy degrades at 10% of that
"10 million token context window"Cost grows exponentially
"Infinite context"Still limited by hardware

The industry is treating the symptom, not the cause.

ClaimReality
Context windows are growingMemory requirements are growing faster
AI is getting smarterAI is getting more expensive to run
The future is unlimitedThe architecture is fundamentally limited

The industry is building bigger versions of the same broken architecture. More data centres. More GPUs. More energy. More money.

But the fundamental problem remains.


5. What This Means for the Learner

By understanding these limitations, you are equipped to see through the hype. When a vendor announces a larger context window or a new optimisation technique, you can now ask the right questions:

QuestionWhy It Matters
"Does this solve the memory wall?"If not, it is a workaround, not a solution.
"What is the accuracy trade-off?"Many techniques trade accuracy for memory.
"What is the latency impact?"Many techniques add latency.
"What is the cost?"Many techniques are expensive to implement and maintain.

You now understand why the industry is stuck.

And in Module 04, you will see how Zeus escapes this trap entirely.


Summary Table

ConceptKey Takeaway
Memory HierarchySpeed and capacity trade off; you cannot have both.
CachingEffective for small, predictable access patterns, not for AI.
GPUs/TPUsHide latency but do not eliminate the memory wall.
CompressionReduces memory at the cost of compute and accuracy.
RAGExtends memory but is not true memory, lacks nuance and context.
SummarisationSaves memory at the cost of losing detail.
Model QuantisationSpeeds inference at the cost of accuracy.
Fundamental TruthThese are workarounds, not solutions.

Discussion Questions

  1. Why are caching and hardware acceleration insufficient for large language models?

  2. What are the trade-offs of RAG? When is it appropriate, and when does it fail?

  3. Why does summarisation fail to preserve the full context of a conversation?

  4. What is the fundamental limitation that all these techniques share?

  5. If you were designing a system from scratch, how would you avoid these limitations?


, The Zeus Quantum Federation


"You cannot fix a physical problem with software patches."

Explore the Full Whitepaper

The complete ten-part whitepaper covers the full architecture, photonic-quantum engines, the SME, Zeus OS, the 7 Pillars of existence, and the path to Human Teleportation.

Read the Whitepaper

Related Research

Share With Friends