Running modern LLMs on a 2014 Tesla K80

Running a recent large language model on modern hardware is relatively straightforward today. Tools are plentiful, graphics cards are well supported, and common configurations are extensively documented.

The challenge becomes far more interesting when the hardware in question is an NVIDIA Tesla K80, a card released in 2014, based on the Kepler architecture, no longer supported by modern software stacks and absent from the officially supported configurations of most current inference engines.

That is exactly the project I took on as part of my personal local AI lab.

The goal was not simply to get a few tokens printed in a terminal. I wanted to determine, in a measurable way, whether this card could still play a useful role in a modern local AI infrastructure, serve models with 27 to 35 billion parameters, outperform my laptop and integrate cleanly into a multi-machine environment.

The result is more nuanced and more interesting than a simple “yes, it works”.

A Tesla K80 can indeed run modern models, but doing so requires working below the usual tooling layer, compiling the right components, methodically testing several strategies and, above all, discarding optimizations that do not hold up under measurement.

Headline result: a 2014 graphics card, unrecognized by modern runtimes, successfully served recent Qwen models ranging from 27B to 35B. The best measured results reached about 13.5 tokens per second for a 35B MoE model with a 64k context, and up to 13 to 29 tokens per second in certain speculative decoding scenarios.

This project is not about training models or building a new inference engine. It is about what I do best: understanding an unusual constraint, integrating components that were never designed to work together, measuring the real behaviour of the system and pushing the hardware to a useful, defensible limit.

The context: building a local AI infrastructure with hardware already on hand

The K80 project is part of a broader lab dedicated to local AI.

The starting point was simple: several machines were already available on my network, but they were underused. Rather than concentrating the entire workload on an expensive new workstation or systematically depending on cloud services, I wanted to distribute the workloads according to each device's strengths.

The lab uses four types of resources:

• a laptop used as the control station and for some CPU inference;
• a workstation with an RTX 2070, mainly used for transcription;
• another workstation with an RTX 3060 for fast GPU tasks;
• a Dell R730 server running Proxmox, hosting various services including a virtual machine with the Tesla K80 in PCI passthrough.

The long-term goal is a local-first environment: private, modular and incremental. It is not a finished product, nor an automated cluster in the strict sense. LLM task distribution is mostly handled through explicit configuration, while some tasks, such as transcription, use automatic selection of an available machine.

That distinction matters: I am not trying to present a generalized orchestration that does not exist. The project instead demonstrates practical integration across several tools, several hardware generations and several types of workloads.

Guiding principles

Four principles guided the technical decisions:

1. Local-first — core operation must not depend on a cloud service.
2. Privacy — private code, documents, transcripts and working data must be able to remain within the local network.
3. Modularity — each machine must be able to play a role suited to its capabilities without forcing a monolithic architecture.
4. Measured progress — each new building block must be tested and documented before the next is added.

That last point became central. Several assumptions that seemed logical at the outset turned out to be false once measured.

Why the Tesla K80 poses a unique challenge

The Tesla K80 is an unusual card.

It contains two GK210 graphics processors on a single board, each with roughly 12 GB of memory. In practice, with ECC enabled, about 22.8 GB of VRAM is usable in total.

On paper, that amount of memory is attractive for local inference. It allows loading quantized models far larger than an 8 or 12 GB consumer card could normally accommodate.

So the problem is not memory capacity alone.

The real obstacle is the Kepler sm_37 architecture.

Modern versions of CUDA, PyTorch, Ollama and several inference libraries no longer support this generation. A standard installation is not enough. The card can be visible to the system and perfectly functional at the hardware level while remaining unusable by modern tooling.

Main constraints

• Kepler sm_37 is deprecated;
• recent CUDA releases no longer compile for this architecture;
• modern PyTorch does not recognize the card;
• Ollama provides no directly compatible execution path;
• some modern kernels assume instructions introduced after Kepler;
• the card has two separate dies, which adds a constraint on how the model is distributed;
• freeing VRAM when switching models can be slow;
• memory capacity is substantial, but bandwidth and architecture remain dated.

The only realistic path was therefore to use a compatible legacy stack, then compile the inference engine myself.

Recompiling llama.cpp for Kepler

The chosen solution relies on llama.cpp, compiled specifically for the sm_37 architecture.

The setup used:

• CUDA 11.8;
• gcc-11 as the host compiler;
• compilation with CMAKE_CUDA_ARCHITECTURES=37;
• a dedicated virtual machine running under Proxmox;
• PCI passthrough of both K80 dies.

CUDA 11.8 can still compile for sm_37, but it rejects GCC versions that are too recent. The build chain therefore had to be controlled precisely rather than simply installing the system's default packages.

Simplified example of the build configuration:

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=37 \
  -DCMAKE_C_COMPILER=gcc-11 \
  -DCMAKE_CXX_COMPILER=g++-11

cmake --build build --config Release -j

This step makes the engine compatible with the card, but it does not guarantee practical performance.

A card capable of loading a model may still produce tokens too slowly to be useful. The rest of the work therefore focused on benchmarking the various configurations and identifying changes with a measurable impact.

First result: the model runs, but the baseline configuration is limited

The first model used in the primary benchmark campaign was a dense 27B Qwen3.6 model quantized to Q4.

With a conventional split between the two dies, the starting configuration delivered roughly:

Prefill was already reasonable, but generation at 3.25 tokens per second remained too close to what my laptop CPU could reach.

The card worked. The challenge now was to make it genuinely useful.

Using both dies in parallel

The K80 contains two distinct graphics processors. How the model is split between them has a significant effect.

Two strategies were compared:

• a pipeline-style split, where layers are separated between the dies;
• a row split, where both dies work in parallel on each layer.

Switching to a row strategy produced a clear improvement:

Generation went from 3.25 to 4.87 tokens per second, an improvement of about 50%.

This result confirms that, on this hardware, using both dies in parallel is preferable to a sequential pipeline for this workload.

Requantizing the model for an older architecture

Modern K-quant formats are generally a good choice on recent hardware. They offer an attractive trade-off between quality, size and performance.

On Kepler, however, that choice was not optimal.

I therefore tested a local requantization to Q4_0, an older and simpler format. The associated dequantization kernels are less complex and better suited to this GPU generation.

The result:

Generation reached 6.30 tokens per second, about 29% more than the previous configuration and nearly double the starting configuration.

To be transparent, the Q4_0 file was produced by requantizing an already quantized model. This double quantization is valid for measuring speed, but its effect on quality was not systematically evaluated.

I therefore do not present this result as a universal production recommendation. Instead, it demonstrates that a simpler quantization can be much better suited to an older architecture.

Comparison with the laptop

To determine whether the K80 still had real value, I compared it with my laptop's CPU and iGPU using models of comparable size.

The K80 outperforms the laptop on both metrics:

• about 3.2 times faster than the CPU on prefill;
• about 2.1 times faster than the CPU on generation.

It obviously does not compete with a modern high-end card, but it outperforms the laptop enough to justify its role as a specialized worker in the lab.

A reasonable hypothesis that did not pan out: returning to legacy CUDA paths

Once the model was running at 6.30 tokens per second, the next question was obvious: could performance be improved further by re-enabling legacy optimizations designed for Kepler?

Several legacy flags and compute paths were examined.

The problem is that the DMMV path used in older versions of llama.cpp has been removed. Modern token generation uses MMVQ instead, which relies on strategies optimized for more recent architectures.

Some operations in particular use instructions that are not natively available on Kepler and must be emulated.

The initial hypothesis was therefore that this emulation was the main bottleneck.

Recompiling with various legacy settings produced no significant gain, however.

One tested configuration gave 6.32 tokens per second, essentially the same result as the reference 6.30 tokens per second.

The conclusion was clear: these flags were no longer a useful optimization path in the modern engine.

Writing a custom CUDA kernel — and measuring a useful failure

To test the expensive-emulation hypothesis more directly, an experimental CUDA kernel was written for the sm_37 architecture.

Its goal was to perform Q4_0 to FP32 dequantization followed by a matrix-vector multiplication, without relying on the modern path assumed to perform poorly on Kepler.

Technically, the kernel worked, but its performance was far worse.

The custom kernel was about 3.8 times slower.

This failure was particularly instructive.

Emulating certain modern instructions was not the main problem. The standard llama.cpp kernel remained extremely optimized: coalesced memory access, better GPU occupancy and more efficient work scheduling.

The custom kernel, despite simpler arithmetic logic, used less efficient memory accesses. On a memory-bound workload, that weakness cost more than the emulation I was trying to avoid.

Work on it was therefore discontinued.

That decision is an integral part of the project. An optimization only has value if it genuinely improves the system. A well-measured negative result is preferable to an elegant but false intuition.

Identifying the real bottleneck: memory bandwidth

Measurements gradually showed that generation was primarily memory-bound.

In other words, compute throughput was not the bottleneck. The GPU spent most of its time moving and re-reading the model weights.

Several indicators pointed to the same conclusion:

• profiling attributed about 78.6% of the time to the MUL_MAT[q4_0] operation;
• a significant increase in GPU clock produced a gain of only about 2.7%;
• targeted arithmetic optimizations changed almost nothing;
• the simpler quantization improved results by reducing read and dequantization costs.

Once that was established, continuing to look only at kernels became pointless.

Instead, the goal was to reduce how many times the model weights were re-read.

Speculative decoding: exploiting the memory wall instead of fighting it

Speculative decoding based on an n-gram cache became the next optimization to explore.

The idea is to propose several candidate tokens from the existing context, then verify them as a group with the main model.

On a machine limited by memory bandwidth, this approach is attractive: the verification pass can validate several tokens after a single large read of the weights.

Results on real-world content were highly variable, but clearly superior to standard generation when the context contained enough reusable patterns.

Observed effective throughput:

All six recorded runs exceeded 10 tokens per second, compared to about 6.47 tokens per second in the reference dense configuration used for this series.

In the best cases, throughput doubled or even quadrupled.

An important limitation

N-gram speculative decoding depends heavily on repetition in the context.

On creative or unpredictable text, the gain can shrink and throughput can fall back to around 7 to 9 tokens per second.

It is therefore not a universal multiplier. It is a workload-dependent optimization that is particularly useful for code, technical documentation, rewriting and other structured workloads.

The MoE trap: a short test can lie

The lab also uses Mixture of Experts, or MoE, models.

These models can offer a good balance between total capacity and the number of parameters activated per token. On the available hardware, they proved particularly interesting.

However, adding speculative decoding to MoE models introduced a misleading failure mode.

The model could:

• load correctly;
• pass the health check;
• process the prefill;
• start generating normally.

Then, after several hundred tokens, the compute buffer grew enough to trigger an out-of-memory error.

The problem therefore did not show up in a quick test.

Generation tests at 800 tokens made it possible to distinguish stable configurations from fragile ones.

The final decision was to serve MoE models without speculative decoding, with their full context.

This configuration is both more stable and faster than the only speculative configuration small enough to fit in memory.

For the 35B MoE models tested, the best measured trade-off is about 13.5 tokens per second with a 64k context, without speculative decoding.

This step is a reminder of an important rule: a benchmark of a few dozen tokens does not validate a service. A configuration can pass every superficial check and fail only during a long generation run.

Handling slow VRAM release

Another problem emerged when switching models.

After stopping a model, the K80's memory did not become available again immediately. The new server could start, pass its initial check, then crash on the first generation because the previous model's VRAM had not yet been fully released.

The behaviour seemed random, but it was reproducible.

The solution was to add a startup wrapper that:

1. monitors VRAM usage;
2. waits for it to drop below a defined threshold;
3. checks at regular intervals;
4. then launches the new server.

This small integration layer made model switching far more reliable.

It illustrates the nature of the project well: the challenge lay not only in compilation or model behaviour, but also in the operational details that separate a one-off benchmark from a genuinely usable service.

Consolidated results

Here is the main progression for the dense model:

And for MoE models:

These results do not turn the K80 into a modern GPU. It remains limited by its architecture, its bandwidth, its energy efficiency and its lack of official support.

They do, however, turn nominally obsolete hardware into a genuinely useful worker for certain local workloads.

Beyond the K80: a distributed local AI environment organized by role

The K80 project is the most technical part of the lab, but it does not exist in isolation.

The local environment uses several existing components, configured and integrated according to hardware capabilities.

LLM inference

• Ollama on several machines;
• llama.cpp compiled for Kepler on the K80;
• llama-swap to load and unload models;
• OpenAI-compatible APIs to provide consistent endpoints.

Development agents

The main interface is Continue.dev, used in Visual Studio Code and on the command line.

Continue.dev was not rewritten. It was integrated and configured to use the local models available on different machines.

This work surfaced several non-obvious behaviours:

• Continue's context length setting did not propagate to Ollama as expected;
• custom models had to explicitly declare tool-use support;
• too small a context caused loops;
• a model's name could incorrectly affect capability detection;
• some observed limitations came from the model's behaviour rather than from a bug in tool-call forwarding.

Custom Ollama model variants were therefore created with a fixed 16k context window, and behaviour rules were added to the configuration.

A real-world case that previously required about ten manual retries was brought down to a single pass after analyzing telemetry and adjusting the rules.

Local web search

The search tool built into Continue.dev depended on a cloud proxy that did not work in the configuration used.

A self-hosted SearXNG service was therefore integrated over MCP to provide locally controlled web search without requiring an external API key.

Local transcription

The lab also hosts a transcription pipeline based on WhisperX, CTranslate2, pyannote and ffmpeg.

A script automatically selects an available machine:

1. RTX 2070 worker;
2. RTX 3060 worker;
3. CPU fallback if no GPU is available.

The pipeline handles, among other things:

• machine selection;
• serialization of GPU tasks;
• temporary unloading of an Ollama model when it conflicts with transcription;
• audio normalization;
• channel extraction;
• timestamping;
• diarization;
• glossary injection;
• execution manifest generation.

In a real-world test, the pipeline processed 42 minutes of audio in 2 minutes 15 seconds, roughly 24 times faster than real time, with timestamps and speaker attribution.

The transcription pipeline is currently the component closest to a reusable deliverable for another project.

What I integrated, and what came from existing tools

To describe this project accurately, it is important to distinguish integration work from third-party components.

I did not create:

• Ollama;
• llama.cpp;
• Continue.dev;
• WhisperX;
• pyannote;
• SearXNG;
• the Qwen models;
• an inference engine;
• a new large language model.

My work covered:

• architecture design;
• machine role assignment;
• service configuration;
• targeted compilation for out-of-support hardware;
• network and API integration;
• automation script development;
• transcription routing;
• model and context configuration;
• tool troubleshooting;
• benchmarking;
• profiling;
• documentation;
• hypothesis validation;
• deliberate abandonment of ineffective approaches.

This is exactly what I aim to highlight in my work as a systems integration consultant: I do not need to reinvent every component to create a solution that did not exist in this form.

The value lies in understanding the need, choosing the building blocks, adapting and integrating them, and proving that the result works.

An AI-assisted workflow, validated through measurement

I built the project by directing AI assistants working across different machines and repositories.

That distinction is worth stating clearly.

The agents helped produce scripts, documentation, diagnostic commands and avenues for experimentation. My role was to define the objectives, organize the work, choose the direction, run the tests, compare the results, revise the hypotheses and decide what was worth keeping.

Using agents does not replace technical validation.

On the contrary, this project reinforced a simple discipline:

Make no claims about a system's behaviour before measuring it.

Several proposed or intuitively appealing ideas turned out to be false:

• increasing the CPU thread count could slow down inference;
• the apparent format of tool calls did not predict an agent's quality;
• emulating a modern instruction was not the K80's main bottleneck;
• a simpler CUDA kernel could be much slower;
• a configuration that passed a health check could fail after 700 tokens;
• speculative decoding could be excellent for a dense model and poor for an MoE;
• an old card could beat a modern CPU when used in the right role.

AI accelerated exploration. Measurements decided what was true.

Current limitations of the lab

The project remains a personal lab, not a production platform for public use.

Some limitations are acknowledged:

• LLM distribution is not generally automated;
• some workers are powered on only when needed;
• the K80 service still has to be monitored and restarted in certain scenarios;
• model response quality has not yet been evaluated as systematically as speed;
• some documented avenues are not implemented yet;
• advanced multi-agent setups and RAG are not currently in use;
• performance data is tied to the hardware, models and versions tested;
• measurements must be dated and must not be generalized to other environments without validation.

These limitations do not diminish the project's value. They simply define what was actually built and what remains to be explored.

What this project demonstrates

Beyond the numbers, this work demonstrates several capabilities that transfer directly to client projects.

Integrating heterogeneous systems

Making machines from different generations, distinct APIs, GPUs with incompatible constraints and services developed by several teams work together.

Working with out-of-support hardware

Finding a compatible software toolchain, compiling the necessary tools and determining whether the result actually justifies the effort.

Diagnosing through data

Setting up reproducible benchmarks, verifying hypotheses and isolating the real bottlenecks.

Knowing when to abandon a bad path

The experimental CUDA kernel was retained as a record, but removed from the solution because it was 3.8 times slower. A serious solution is not about defending sunk effort; it is about keeping what works.

Designing for controlled degradation

When a GPU worker is unavailable, transcription can be redirected to another machine or to the CPU rather than failing outright.

Documenting reusable interfaces

Another project uses the transcription pipeline through a documented interface contract. The caller does not need to know the hardware topology.

Directing and verifying AI agents

The assistants were used as development accelerators, but their work was inspected, tested and corrected. This capability is becoming increasingly relevant in modern projects.

What this work can de-risk for a business

The value for a client is not in reproducing a Tesla K80. It lies in reducing uncertainty before investing: establishing what the existing infrastructure can actually do, identifying the dominant bottleneck and validating an architecture through measurement. This lab is not a market-ready product, but its method transfers to real-world needs.

Local processing of sensitive data

• transcription of meetings or interviews;
• analysis of confidential documents;
• internal assistants;
• search across a private document repository;
• processing data that must not be sent to an external provider.

Reusing existing infrastructure

• assessing whether hardware already owned can be repurposed;
• distributing workloads according to available resources;
• avoiding a premature purchase before validating the requirements;
• objectively determining the limits of the hardware.

Integrating AI solutions

• connecting local models to existing tools;
• standardizing access through compatible APIs;
• integrating internal or web search;
• adding transcription;
• automating service selection based on availability.

Technical prototyping

• validating an architecture before a larger investment;
• comparing several models or engines;
• measuring speed, memory and quality;
• turning a vague idea into a measurable prototype.

Key takeaways

1. Old hardware is not necessarily useless

An unsupported card can still be relevant if it offers a useful advantage — here, a large amount of VRAM spread across two dies.

2. Compatibility is not enough

Getting a model to load is only the beginning. You have to measure generation, prefill, stability over long responses and behaviour during model switches.

3. The best optimization depends on the real bottleneck

When the system is memory-bound, raising the clock speed or simplifying an arithmetic operation can have very little effect.

4. An optimization can be specific to the model type

Speculative decoding proved very effective in certain dense scenarios, but problematic for the MoE models tested.

5. Failures are results

The slower kernel, the late OOMs and the recompilations with no effect all helped avoid pursuing unproductive avenues.

6. Integration is a skill in itself

Assembling, configuring, measuring and documenting existing tools can create far more value than rewriting an already mature component.

Possible next steps

This work points to several possible directions, without implying that they are already implemented:

• systematically measuring response quality on a set of real tasks;
• further stabilizing the K80 service;
• publishing the raw benchmark data and detailed protocol;
• comparing operating costs against various cloud options;
• adding background tasks;
• exploring local RAG;
• testing Continue.dev's native sub-agents;
• potentially integrating additional accelerators;
• publishing a cleaned-up selection of scripts and results;
• turning the transcription pipeline into a standalone demonstration.

The priority remains the same: progress one building block at a time and keep only what has been validated.

Conclusion

The most visible result of this project is easy to summarize:

A 2014 Tesla K80, no longer supported by modern tooling, was brought back into service to run recent language models with 27 to 35 billion parameters on local hardware.

But the project's real value lies in the approach.

It required:

• understanding the limits of the architecture;
• building a compatible build chain;
• using both dies correctly;
• comparing several quantizations;
• profiling the real bottlenecks;
• writing and then discarding an experimental kernel;
• testing speculative decoding;
• identifying late OOMs;
• stabilizing model switches;
• integrating the card into a broader environment;
• documenting the results honestly.

This project reflects the kind of engagement that interests me: an atypical problem, several systems that are not naturally compatible, few ready-made answers and a result that has to be demonstrated rather than simply announced.

Technical summary