Why Memory Optimisation Matters, and Why It Is Not Enough
In Module 02, we explored the von Neumann bottleneck, the fundamental limit imposed by the separation of processor and memory. Data must move across a bus. The bus is slow. The CPU waits. The bottleneck is physical, not software.
But engineers have not been idle. For decades, they have developed sophisticated techniques to work around this bottleneck. Caching. Hardware acceleration. Memory compression. Retrieval-Augmented Generation (RAG). Summarisation. Model quantization.
Each of these techniques has delivered real, measurable improvements. They have allowed AI to scale far beyond what seemed possible. They have bought time.
But they have not solved the problem.
This module explores the most common memory optimisation techniques in use today, how they work, why they help, and ultimately why they cannot overcome the von Neumann bottleneck.
Understanding these limitations is essential. It is the key to understanding why a new architecture is required, and why Zeus is not an improvement on von Neumann, but a replacement.
In Module 02, we introduced the memory hierarchy:
| Level | Technology | Speed | Capacity |
|---|---|---|---|
| Registers | On-chip | ~1 cycle | Bytes |
| L1 Cache | SRAM | ~1-2 ns | KB |
| L2/L3 Cache | SRAM | ~10 ns | MB |
| DRAM | Capacitor-based | ~100 ns | GB |
| SSD | Flash | ~100 µs | TB |
This hierarchy exists because of a fundamental trade-off: speed is expensive.
SRAM (cache) is fast but costly and power-hungry. DRAM is slower but cheaper. SSDs are slower still but offer vastly more capacity. The hierarchy is a compromise, an attempt to balance speed, cost, and capacity.
For traditional computing, this compromise works.
For AI, it fails.
| Problem | Why It Matters for AI |
|---|---|
| AI models are massive | A single model can require 140GB+ of weights, far more than cache or even DRAM can hold at once |
| AI requires low-latency access | Memory latency directly impacts inference speed, the user waits for the AI to respond |
| AI models have massive working sets | The entire model, plus the context window, must be accessed continuously |
In traditional computing, data access is localised. The CPU can cache frequently used data. In AI, the working set is the entire model, the cache cannot hold it.
The hierarchy that works for spreadsheets fails for large language models.
Over the years, engineers have developed a range of techniques to work around the memory wall. Each one helps, but each one also has fundamental limitations.
What It Is: Small, fast memory (SRAM) placed close to the CPU to store frequently accessed data.
How It Works: When the CPU needs data, it first checks the cache. If the data is there (a "cache hit"), it is retrieved instantly. If not (a "cache miss"), the CPU must fetch it from slower DRAM.
Why It Helps: Reduces the number of slow DRAM accesses. Typical cache hit rates are 90%+ for traditional workloads.
Why It Fails for AI:
Limit: Caching is effective for predictable, localised access patterns. AI access patterns are neither.
What It Is: Specialised processors designed to perform many operations in parallel.
How It Works: GPUs and TPUs contain thousands of cores that can perform operations simultaneously. They are designed for the massive parallelism of matrix multiplication and neural network inference.
Why It Helps: GPUs can perform thousands of operations in the time it takes a CPU to perform one. Training time drops from months to days.
Why It Fails for AI:
Limit: GPUs are better at hiding latency than CPUs, but they cannot eliminate it. Data must still move across a bus. The memory wall remains.
What It Is: Compressing data in memory to fit more into the available space.
How It Works: Data is stored in a compressed format. When the processor needs it, it is decompressed on the fly.
Why It Helps: Reduces memory footprint. A model that would not fit in available memory can be compressed to fit.
Why It Fails for AI:
Limit: Compression trades compute time for memory space. It helps when memory is the constraint, but adds latency when compute is the constraint. For AI, both matter.
What It Is: Instead of storing everything in the context window, the AI retrieves relevant information from an external database when needed.
How It Works: When the user asks a question, the AI searches an external database for relevant information, retrieves it, and includes it in the context window.
Why It Helps: Extends the effective memory of the system. The AI can access information beyond its context window.
Why It Fails for AI:
Limit: RAG extends memory, but it is not true memory. It is a lookup mechanism. It cannot recall the full context of a conversation.
What It Is: Compressing long conversations into shorter summaries to fit within the context window.
How It Works: When a conversation exceeds the context window limit, the AI summarises the earlier parts and discards the original tokens.
Why It Helps: Reduces the number of tokens in the context window. The AI can maintain a longer conversation without exceeding the limit.
Why It Fails for AI:
Limit: Summarisation is a lossy compression technique. It helps in the short term, but long-term accuracy and context are sacrificed.
What It Is: Reducing the precision of the model's weights, for example, from 32-bit floating point to 8-bit integer.
How It Works: The model weights are stored in a lower precision format. This reduces memory footprint and speeds up computation.
Why It Helps: A quantised model uses less memory and runs faster.
Why It Fails for AI:
Limit: Quantisation is a trade-off, speed for accuracy. It helps when memory and compute are constraints, but at the cost of capability.
| Technique | Works Around The Problem | Solves The Problem | Why It Cannot Solve It |
|---|---|---|---|
| Caching | ✅ | ❌ | Caches are too small for AI working sets. |
| Hardware Acceleration | ✅ | ❌ | Data must still move, the memory wall remains. |
| Memory Compression | ✅ | ❌ | Adds overhead and latency, reduces accuracy. |
| RAG | ✅ | ❌ | Retrieval is not memory, lacks nuance and context. |
| Summarisation | ✅ | ❌ | Loses information and context. |
| Model Quantisation | ✅ | ❌ | Reduces accuracy and capability. |
None of these techniques address the fundamental issue: the separation of processor and memory.
The von Neumann architecture forces data to move. As long as data must move, there will be a bottleneck. As long as there is a bottleneck, performance will be limited.
You cannot fix a physical problem with software patches.
The industry celebrates each of these techniques as a breakthrough. But this celebration misses the point.
| Announcement | Reality |
|---|---|
| "2 million token context window" | Accuracy degrades at 10% of that |
| "10 million token context window" | Cost grows exponentially |
| "Infinite context" | Still limited by hardware |
The industry is treating the symptom, not the cause.
| Claim | Reality |
|---|---|
| Context windows are growing | Memory requirements are growing faster |
| AI is getting smarter | AI is getting more expensive to run |
| The future is unlimited | The architecture is fundamentally limited |
The industry is building bigger versions of the same broken architecture. More data centres. More GPUs. More energy. More money.
But the fundamental problem remains.
By understanding these limitations, you are equipped to see through the hype. When a vendor announces a larger context window or a new optimisation technique, you can now ask the right questions:
| Question | Why It Matters |
|---|---|
| "Does this solve the memory wall?" | If not, it is a workaround, not a solution. |
| "What is the accuracy trade-off?" | Many techniques trade accuracy for memory. |
| "What is the latency impact?" | Many techniques add latency. |
| "What is the cost?" | Many techniques are expensive to implement and maintain. |
You now understand why the industry is stuck.
And in Module 04, you will see how Zeus escapes this trap entirely.
| Concept | Key Takeaway |
|---|---|
| Memory Hierarchy | Speed and capacity trade off; you cannot have both. |
| Caching | Effective for small, predictable access patterns, not for AI. |
| GPUs/TPUs | Hide latency but do not eliminate the memory wall. |
| Compression | Reduces memory at the cost of compute and accuracy. |
| RAG | Extends memory but is not true memory, lacks nuance and context. |
| Summarisation | Saves memory at the cost of losing detail. |
| Model Quantisation | Speeds inference at the cost of accuracy. |
| Fundamental Truth | These are workarounds, not solutions. |
Why are caching and hardware acceleration insufficient for large language models?
What are the trade-offs of RAG? When is it appropriate, and when does it fail?
Why does summarisation fail to preserve the full context of a conversation?
What is the fundamental limitation that all these techniques share?
If you were designing a system from scratch, how would you avoid these limitations?
, The Zeus Quantum Federation
"You cannot fix a physical problem with software patches."
The complete ten-part whitepaper covers the full architecture, photonic-quantum engines, the SME, Zeus OS, the 7 Pillars of existence, and the path to Human Teleportation.
Read the Whitepaper