FOREWORD: THE AUDACITY OF PUTTING A BRAIN IN A BOTTLE
There is something almost philosophically provocative about the idea of running a neural network on a microcontroller. These small chips — often no bigger than a fingernail and costing less than a cup of coffee — were originally designed to blink LEDs, read temperature sensors, and tell motors when to spin. Nobody designed them with intelligence in mind. And yet, here we are.
In the summer of 2026, microcontrollers are running inference on compressed language models, classifying images from cameras, detecting anomalies in industrial machinery, and understanding spoken commands. They do all of this without a cloud connection, without a GPU, without even a fan. They do it in real time, on a coin-cell battery, in a device that might be buried in a wall, bolted to a turbine, or stitched into a piece of clothing.
This article is your comprehensive guide to that remarkable engineering achievement. We will travel from the theoretical foundations of why you would ever want AI on a microcontroller, through every major technique used to shrink and optimize neural networks, all the way to concrete code and deployment strategies. We will look at the tools, the trade-offs, the tricks, and the occasional traps that await the unwary engineer. Some of the mathematics is genuinely beautiful. Some of the engineering is genuinely brutal. All of it is worth understanding.
CHAPTER ONE: WHY WOULD YOU EVEN WANT DEEP LEARNING ON AN EMBEDDED SYSTEM?
Before we dive into the engineering, we need to answer the most fundamental question: why bother? The cloud exists. Powerful GPU servers exist. Why would anyone want to run a neural network on a device with 256 kilobytes of RAM and no floating-point unit?
The answer is not one reason but a constellation of compelling forces that, in 2026, have made on-device AI not just desirable but often mandatory.
LATENCY IS THE ENEMY OF REAL-TIME SYSTEMS
Consider an industrial robot arm on a factory floor. It uses a camera and a neural network to detect whether a component is correctly positioned before welding. The decision must be made in under 10 milliseconds. Sending an image to a cloud server and waiting for the round-trip — even on a fast 5G connection, you are looking at 20 to 100 milliseconds of network latency alone, before the server has even looked at your image — is simply not fast enough. The physics of the situation demand local inference. The same logic applies to automotive safety systems, medical devices that monitor vital signs, and drones that must avoid obstacles while flying at speed. For these applications, the cloud is not a solution. It is a liability.
CONNECTIVITY IS NOT GUARANTEED
A soil moisture sensor in a remote agricultural field, a vibration monitor inside a deep-sea oil pipeline, a wildlife tracking collar on a wolf in the Arctic — none of these devices can rely on a permanent internet connection. They must make intelligent decisions locally, store results, and upload them opportunistically when connectivity is available. On-device AI is not just convenient for such deployments; it is the only architecture that actually works.
PRIVACY IS A FUNDAMENTAL REQUIREMENT
When a hearing aid uses a neural network to separate speech from background noise, the audio stream contains deeply personal information. When a smart camera in a hospital monitors patient movement for fall detection, it captures sensitive medical data. Sending this data to a cloud server creates privacy risks, regulatory complications (think GDPR in Europe, HIPAA in the United States), and a fundamental breach of user trust. Running the inference locally means the sensitive data never leaves the device. In many domains this is not just a technical preference — it is a legal requirement.
COST AT SCALE IS BRUTAL
Imagine deploying 10 million IoT sensors across a smart city. Each sensor makes 10 inference calls per minute. That is 100 million inference calls per minute, or roughly 52 billion per year. Cloud inference at even the cheapest rates quickly becomes economically catastrophic. A microcontroller that costs two dollars and runs inference locally, powered by a small battery or energy harvesting, is economically transformative at that scale. The arithmetic is not subtle.
ENERGY EFFICIENCY IS PHYSICS
A microcontroller running a compressed neural network might consume 1 to 10 milliwatts during inference. A cloud round-trip — accounting for the radio transmission, the network infrastructure, and the server-side computation — consumes orders of magnitude more energy per inference. For battery-powered devices, this difference is the line between a product that lasts two years on a battery and one that lasts two weeks. You cannot negotiate with thermodynamics.
THE NUMBERS TELL THE STORY
The TinyML market reached $1.53 billion in 2025 and is projected to hit $2.49 billion by the end of 2026, growing at nearly 25% per year. Over 30 billion IoT devices are expected to be deployed globally by 2026, with a significant fraction running some form of on-device machine learning. This is not a niche curiosity pursued by a handful of academics. This is a major technological wave, and it is already breaking.
CHAPTER TWO: THE BRUTAL REALITY OF MICROCONTROLLER CONSTRAINTS
To appreciate the engineering challenge, you need to understand exactly what you are working with when you target a microcontroller. The contrast with the hardware that deep learning was designed for is almost comical — in the way that comparing a canoe to an aircraft carrier is almost comical.
A modern GPU used for training large language models might have 80 gigabytes of high-bandwidth memory, thousands of CUDA cores, and consume 400 watts of power. A popular embedded target like the STM32F4 series microcontroller has 192 kilobytes of SRAM, 1 megabyte of Flash storage, runs at 168 MHz, and consumes around 100 milliwatts under load. The gap is not just large; it is measured in orders of magnitude across every relevant dimension.
More capable embedded targets exist, of course. The STM32H7 series offers up to 1 megabyte of SRAM. The STM32N6, released in 2024, integrates a dedicated Neural-ART accelerator — a hardware NPU — and can handle significantly more complex models. The ESP32-S3 from Espressif includes vector instructions that accelerate neural network operations. The Arduino Nano 33 BLE Sense, based on the Nordic nRF52840, has 256 KB of RAM and 1 MB of Flash. These are the battlegrounds of TinyML, and understanding their constraints is not optional — it shapes every design decision you will make.
SRAM is the most critical bottleneck. During inference, the neural network must store its intermediate activations — the outputs of each layer as data flows through the network. These activations can be surprisingly large, and on a device with 256 KB of total working memory, you must be extraordinarily careful about how much each layer demands at any given moment. Run out of SRAM and your model simply does not fit. There is no swap space, no virtual memory, no second chance.
Flash memory holds the model weights. A model with 1 million parameters stored as 32-bit floats requires 4 megabytes of storage — already four times the Flash capacity of many common microcontrollers. Compression is not an optimization you apply when you feel like it. It is existential.
Compute throughput is limited by the absence of floating-point hardware on many microcontrollers, relatively low clock speeds, and the lack of SIMD instructions that make matrix multiplication fast on larger processors. Integer arithmetic is typically much faster than floating-point on these devices, which is one of the reasons quantization matters so much.
Power consumption must be managed carefully. Many embedded AI applications run on batteries, sometimes for years. The inference engine must be designed to minimize active time and exploit sleep modes aggressively. Every milliwatt counts.
With these constraints firmly in mind, let us now explore the toolkit that engineers use to make deep learning work within them.
CHAPTER THREE: QUANTIZATION — THE SINGLE MOST POWERFUL TOOL IN YOUR ARSENAL
If you had to choose just one technique to make a neural network run on a microcontroller, quantization would be that technique. It is the process of reducing the numerical precision of the numbers that represent a model's weights and activations, and it delivers dramatic benefits in memory footprint, compute speed, and energy efficiency — often all at once, often with surprisingly little accuracy loss.
To understand why quantization works, consider what a trained neural network actually is at its core. It is a collection of numbers — billions of them in large models, millions even in small ones — that represent learned relationships between inputs and outputs. During training, these numbers are stored as 32-bit floating-point values (FP32), which can represent an enormous range of values with high precision. That precision is necessary during training because the gradient updates that adjust the weights are very small, and losing precision during those updates would prevent the model from converging.
But here is the key insight: once training is complete, the model does not need that level of precision for inference. The weights have settled into values that encode the learned knowledge, and those values can be approximated with much lower precision without significantly changing the model's behavior. The network is, in a sense, more robust than it looks. This is the fundamental intuition behind quantization, and it is why the technique works as well as it does.
THE MATHEMATICS OF QUANTIZATION
The basic idea is to map a range of floating-point values to a smaller set of integer values. The simplest form of this mapping is linear (also called uniform) quantization. Given a floating-point tensor with minimum value (x_{\min}) and maximum value (x_{\max}), we want to represent it using (b)-bit integers (so (2^b) distinct values). The scale factor (s) and zero-point (z) are computed as:
$$s = \frac{x_{\max} - x_{\min}}{2^b - 1}, \qquad z = \operatorname{round}!\left(\frac{-x_{\min}}{s}\right)$$
The quantization of a floating-point value (x) to an integer (q) is:
$$q = \operatorname{clamp}!\left(\operatorname{round}!\left(\frac{x}{s}\right) + z,; 0,; 2^b - 1\right)$$
The dequantization — recovering an approximate floating-point value from the integer — is simply:
$$\hat{x} = s,(q - z)$$
For INT8 quantization ((b = 8)), we have 256 distinct values to represent the entire range of a tensor. For INT4 ((b = 4)), we have only 16. The fewer bits we use, the more aggressive the compression and the more potential accuracy loss we accept. The art lies in finding the sweet spot.
Installation
pip install numpy==1.26.4 tensorflow==2.16.1 torch==2.3.1 torchvision==0.18.1
The following script demonstrates symmetric INT8 quantization, a common simplification where the zero-point is forced to zero. This makes the arithmetic during inference simpler and faster, which matters a great deal on hardware without a hardware multiplier.
# quantize_int8.py
# Demonstrates symmetric per-tensor INT8 quantization and dequantization.
# Run: python quantize_int8.py
import numpy as np
def quantize_symmetric_int8(weights: np.ndarray) -> tuple:
"""
Performs symmetric per-tensor INT8 quantization on a weight array.
In symmetric quantization the zero-point is fixed at 0, which
simplifies multiply-accumulate operations during inference.
The scale factor maps the maximum absolute value of the tensor
to the maximum representable INT8 value (127).
Args:
weights: A float32 numpy array of weights to quantize.
Returns:
A tuple of (quantized_weights, scale_factor) where
quantized_weights is an int8 array and scale_factor is a
float32 scalar used for dequantization.
"""
max_abs_val = np.max(np.abs(weights))
# Map max_abs_val to 127 (INT8 max). Epsilon guards against
# division by zero for degenerate all-zero weight tensors.
scale = max_abs_val / 127.0 + 1e-8
# Divide by scale, round to nearest integer, clamp to [-128, 127].
quantized = np.clip(np.round(weights / scale), -128, 127)
quantized = quantized.astype(np.int8)
return quantized, np.float32(scale)
def dequantize_symmetric_int8(
quantized: np.ndarray,
scale: float,
) -> np.ndarray:
"""
Recovers an approximate float32 tensor from INT8 quantized values.
Args:
quantized: An int8 numpy array of quantized weights.
scale: The scale factor returned by quantize_symmetric_int8.
Returns:
A float32 numpy array approximating the original weights.
"""
return quantized.astype(np.float32) * scale
if __name__ == "__main__":
np.random.seed(42)
original_weights = np.random.randn(64, 3, 3, 3).astype(np.float32)
q_weights, scale_factor = quantize_symmetric_int8(original_weights)
recovered_weights = dequantize_symmetric_int8(q_weights, scale_factor)
mse = np.mean((original_weights - recovered_weights) ** 2)
compression_ratio = original_weights.nbytes / q_weights.nbytes
print(f"Original size: {original_weights.nbytes / 1024:.2f} KB")
print(f"Quantized size: {q_weights.nbytes / 1024:.2f} KB")
print(f"Compression ratio: {compression_ratio:.1f}x")
print(f"Mean Squared Error: {mse:.6f}")
When you run this script, you will see that the quantized representation is exactly 4 times smaller than the original — 32 bits down to 8 bits per value — and the mean squared error is typically on the order of 1e-4 or less for well-behaved weight distributions. That is the magic of quantization: a 4× memory reduction with negligible accuracy impact, at least for INT8. It is one of those rare engineering wins where you genuinely get most of what you want for very little of what you feared you would lose.
POST-TRAINING QUANTIZATION (PTQ)
Post-training quantization is the simplest approach: take a model that has already been fully trained in FP32, and convert it to a lower-precision format after the fact, without any additional training. This is attractive because it requires no changes to the training pipeline and can be applied to any pre-trained model you happen to have lying around.
The main challenge with PTQ is handling activations. Weight values are fixed after training and their range can be computed directly, but activation values depend on the input data and can vary significantly. To quantize activations statically — which is necessary for maximum inference speed — you need to run a small calibration dataset through the model and observe the range of activations at each layer. This calibration step is what allows PTQ to set appropriate scale factors for activations, and the quality of your calibration data has a surprisingly large impact on the final result.
TensorFlow Lite provides an excellent PTQ pipeline. The following script shows how to convert a Keras model to a fully INT8-quantized TFLite model, including both weights and activations.
# ptq_tflite.py
# Converts a trained Keras model to a fully INT8-quantized TFLite model
# suitable for deployment via TensorFlow Lite Micro.
# Run: python ptq_tflite.py
import numpy as np
import tensorflow as tf
def create_representative_dataset(num_samples: int = 100):
"""
Generator that yields calibration samples for PTQ activation ranging.
Replace the random data here with real samples from your training or
validation set. The quality of this data directly affects quantization
accuracy — random noise will produce poor scale factors.
Yields:
A list containing one float32 input tensor per iteration.
"""
for _ in range(num_samples):
# Shape (1, 28, 28, 1) matches a typical MNIST-style input.
# Adjust to match your model's expected input shape.
sample = np.random.rand(1, 28, 28, 1).astype(np.float32)
yield [sample]
def convert_model_to_int8_tflite(
keras_model: tf.keras.Model,
output_path: str,
) -> None:
"""
Converts a trained Keras model to a fully INT8-quantized TFLite model.
Applies PTQ with both weight and activation quantization. The resulting
model is optimized for integer-only inference on microcontrollers via
TensorFlow Lite Micro (TFLM).
Args:
keras_model: A trained tf.keras.Model to convert.
output_path: File path where the .tflite model will be saved.
"""
converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# The representative dataset enables static activation quantization.
# Without it, only weights are quantized — activations remain float.
converter.representative_dataset = create_representative_dataset
# Force integer-only ops. Required for Cortex-M0/M3 class devices
# that have no floating-point unit.
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS_INT8
]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_model = converter.convert()
with open(output_path, "wb") as f:
f.write(tflite_model)
print(f"INT8 TFLite model saved to '{output_path}'")
print(f"Model size: {len(tflite_model) / 1024:.2f} KB")
if __name__ == "__main__":
# Load your trained Keras model and convert it.
# Replace "./saved_model" with the path to your actual model.
keras_model = tf.keras.models.load_model("./saved_model")
convert_model_to_int8_tflite(keras_model, "./model_int8.tflite")
# Convert the .tflite file to a C array for embedding in firmware:
# xxd -i model_int8.tflite > model_data.h
# Then include model_data.h in your TFLM firmware project.
The resulting .tflite file can be converted to a C array and compiled directly into your microcontroller firmware, where TensorFlow Lite Micro handles the inference. TFLM's core runtime fits in as little as 16 KB of Flash on an ARM Cortex-M3, making it a remarkably lean inference engine for what it does.
QUANTIZATION-AWARE TRAINING (QAT)
While PTQ is convenient, it has a fundamental limitation: the model was trained without any awareness of the quantization that will be applied later. For aggressive quantization — INT4 or below — or for models that are particularly sensitive to numerical precision, PTQ can cause significant accuracy degradation. The model simply was not prepared for the rounding errors it will encounter.
Quantization-aware training solves this by simulating the effects of quantization during the training process itself. Fake quantization nodes are inserted into the computation graph. During the forward pass, these nodes quantize and then immediately dequantize the values, introducing the same rounding errors that will occur during real inference. The model then learns to be robust to these errors during backpropagation — it adapts its weights to work well under quantization pressure, rather than being surprised by it after the fact.
The key insight of QAT is that even though the forward pass uses quantized values, the backward pass still uses full-precision floating-point arithmetic. This is possible through the straight-through estimator (STE), which simply passes gradients through the quantization operation as if it were an identity function. This approximation sounds alarming but works surprisingly well in practice.
The following script demonstrates QAT setup using PyTorch's built-in quantization toolkit. In PyTorch 2.x, the preferred namespace is torch.ao.quantization.
# qat_pytorch.py
# Sets up a small CNN for Quantization-Aware Training (QAT) in PyTorch.
# Run: python qat_pytorch.py
import torch
import torch.nn as nn
import torch.ao.quantization as quant # Preferred namespace in PyTorch 2.x
class TinyConvNet(nn.Module):
"""
A small convolutional neural network designed for TinyML deployment.
Kept intentionally compact to fit within microcontroller memory
constraints. Uses QuantStub and DeQuantStub to mark the entry and
exit points of the quantized computation graph, as required by
PyTorch's QAT framework.
"""
def __init__(self, num_classes: int = 10):
super(TinyConvNet, self).__init__()
# QuantStub converts incoming float tensors to quantized form.
self.quant = quant.QuantStub()
# First convolutional block: 1 input channel, 8 output channels.
self.conv1 = nn.Conv2d(
in_channels=1, out_channels=8, kernel_size=3, padding=1
)
self.relu1 = nn.ReLU()
self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
# Second convolutional block: 8 input channels, 16 output channels.
self.conv2 = nn.Conv2d(
in_channels=8, out_channels=16, kernel_size=3, padding=1
)
self.relu2 = nn.ReLU()
self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
# Fully connected head.
# Input: 16 channels × 7 × 7 spatial (for 28×28 input after 2 pools).
self.fc = nn.Linear(16 * 7 * 7, num_classes)
# DeQuantStub converts quantized tensors back to float at the exit.
self.dequant = quant.DeQuantStub()
def forward(self, x: torch.Tensor) -> torch.Tensor:
x = self.quant(x)
x = self.pool1(self.relu1(self.conv1(x)))
x = self.pool2(self.relu2(self.conv2(x)))
x = x.view(x.size(0), -1)
x = self.fc(x)
x = self.dequant(x)
return x
def prepare_model_for_qat(model: TinyConvNet) -> TinyConvNet:
"""
Prepares a TinyConvNet for quantization-aware training.
Fuses Conv + ReLU pairs into single operations (reducing quantization
error by eliminating intermediate quantization between tightly coupled
layers), then inserts fake-quantization nodes throughout the graph.
Args:
model: The TinyConvNet to prepare.
Returns:
The model with fake-quantization nodes inserted, ready for QAT
fine-tuning. Typically run for 10–20% of original training duration.
"""
model.train()
# Fuse Conv + ReLU pairs. After fusion, relu1 and relu2 become
# identity modules in the graph; forward() still calls them safely.
model = quant.fuse_modules(model, [
["conv1", "relu1"],
["conv2", "relu2"],
])
# 'qnnpack' targets ARM/mobile hardware. Use 'fbgemm' for x86 servers.
model.qconfig = quant.get_default_qat_qconfig("qnnpack")
# Insert fake-quantization nodes. The model is now ready for QAT.
model = quant.prepare_qat(model)
return model
def finalize_qat_model(model: TinyConvNet) -> nn.Module:
"""
Converts a QAT-prepared model to a real quantized model for inference.
Call this after QAT fine-tuning is complete. Replaces fake-quantization
nodes with actual integer operations, producing a model that performs
real INT8 arithmetic during inference.
Args:
model: The QAT-fine-tuned TinyConvNet.
Returns:
A quantized model ready for INT8 inference.
"""
model.eval()
return quant.convert(model)
if __name__ == "__main__":
model = TinyConvNet(num_classes=10)
print("Before QAT preparation:")
print(model)
model = prepare_model_for_qat(model)
print("\nAfter QAT preparation (fake-quantization nodes inserted):")
print(model)
# After fine-tuning on real data with your optimizer and DataLoader:
# for epoch in range(num_qat_epochs): # 10–20% of original epochs
# for inputs, labels in train_loader:
# outputs = model(inputs)
# loss = criterion(outputs, labels) # criterion = nn.CrossEntropyLoss()
# optimizer.zero_grad()
# loss.backward()
# optimizer.step()
#
# quantized_model = finalize_qat_model(model)
# torch.save(quantized_model.state_dict(), "tiny_conv_int8.pt")
print("\nModel is ready for QAT fine-tuning.")
After fine-tuning, call finalize_qat_model to replace the fake-quantization nodes with real integer operations. The result is a model that performs actual INT8 arithmetic during inference — not a simulation of it.
ADVANCED QUANTIZATION: GPTQ AND AWQ FOR LARGE LANGUAGE MODELS
For large language models, the quantization challenge is considerably more severe. The models are enormous, and retraining them with QAT is prohibitively expensive — you cannot simply fine-tune a 7-billion-parameter model on a consumer GPU for a few hours. Two post-training quantization methods have emerged as the state of the art for LLMs in 2025 and 2026: GPTQ and AWQ.
GPTQ uses second-order information — specifically, the Hessian of the loss with respect to the weights — to minimize quantization error layer by layer. It quantizes weights one at a time, and after quantizing each weight, it adjusts the remaining unquantized weights in the same row to compensate for the error introduced. This greedy compensation strategy is remarkably effective at preserving model quality even at INT4.
AWQ takes a different and rather elegant approach. It observes that not all weights are equally important: weights that correspond to large activation values have a disproportionate impact on the model's output. AWQ identifies these salient weight channels and protects them by applying a per-channel scaling factor that effectively increases their precision relative to the less important channels. The overall model can then be quantized to INT4 with minimal accuracy loss, because the weights that matter most are the ones that are handled most carefully.
Benchmarks from 2025 show that 4-bit AWQ quantization of Llama-3-8B achieves only −0.4 percentage points on HumanEval compared to FP16, while 4-bit GPTQ shows approximately −0.7 percentage points on the same benchmark. Both methods deliver roughly a 4× reduction in weight storage versus FP16.
Installation — AWQ environment
# Python 3.10+, CUDA 11.8+ required for GPU-accelerated quantization.
pip install autoawq==0.2.6 transformers==4.41.2 torch==2.3.1 accelerate==0.30.1
# awq_quantize.py
# Applies AWQ INT4 quantization to a HuggingFace language model.
# Run: python awq_quantize.py
#
# Note: AWQ quantization requires a CUDA GPU with at least 8 GB VRAM
# for a 3B-parameter model. The quantized model can then be deployed on CPU.
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
def quantize_llm_with_awq(
model_name: str,
output_dir: str,
quant_bits: int = 4,
) -> None:
"""
Applies AWQ (Activation-aware Weight Quantization) to an LLM.
AWQ identifies the most important weight channels by analyzing
activation magnitudes, then applies per-channel scaling to protect
those channels during aggressive INT4 quantization. The result is
a model approximately 4× smaller than its FP16 counterpart with
minimal accuracy degradation.
Args:
model_name: HuggingFace model identifier or local path.
output_dir: Directory where the quantized model will be saved.
quant_bits: Target bit-width for quantization (typically 4).
"""
tokenizer = AutoTokenizer.from_pretrained(
model_name,
trust_remote_code=True,
)
# Load in full precision. AWQ needs the full-precision weights to
# compute activation-aware scaling factors before quantizing.
model = AutoAWQForCausalLM.from_pretrained(
model_name,
trust_remote_code=True,
low_cpu_mem_usage=True,
)
# group_size=128: weights are quantized in groups of 128, with a
# separate scale per group. Smaller groups preserve more accuracy
# at a slight overhead cost.
quant_config = {
"zero_point": True, # Asymmetric quantization (better accuracy).
"q_group_size": 128,
"w_bit": quant_bits,
"version": "GEMM", # GEMM kernel for inference speed.
}
# Analyzes activation statistics and computes optimal per-channel scales.
# Typically takes 5–15 minutes for a 1–3B parameter model on a GPU.
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(output_dir)
tokenizer.save_pretrained(output_dir)
print(f"AWQ-quantized model saved to '{output_dir}'")
print(f"Expected size reduction: ~{32 // quant_bits}× vs FP32")
if __name__ == "__main__":
quantize_llm_with_awq(
model_name="microsoft/Phi-4-mini-instruct",
output_dir="./phi4_mini_awq_int4",
quant_bits=4,
)
# Load the quantized model for inference:
# from awq import AutoAWQForCausalLM
# model = AutoAWQForCausalLM.from_quantized(
# "./phi4_mini_awq_int4", fuse_layers=True
# )
For a 1-billion-parameter model, AWQ INT4 quantization reduces weight storage from approximately 4 GB (FP32) to approximately 500 MB. For a 3-billion-parameter model, the reduction is from 12 GB to about 1.5 GB. This brings models that were previously GPU-only into the realm of high-end embedded processors like the Raspberry Pi 5 or NVIDIA Jetson Orin Nano. For truly tiny microcontrollers — Cortex-M class — even these compressed models are still too large, and this is where the other techniques in our toolkit become essential.
CHAPTER FOUR: PRUNING — REMOVING THE DEAD WEIGHT
Neural networks, it turns out, are remarkably redundant. When a large network is trained, many of its connections and neurons contribute very little to the final output. They are, in a sense, passengers rather than drivers — along for the ride, consuming resources, contributing nothing particularly useful. Pruning is the process of identifying and removing these redundant elements, making the network smaller and faster without significantly harming its accuracy.
The concept was inspired by neuroscience. The human brain undergoes synaptic pruning during development, eliminating weak or unused neural connections and strengthening the important ones. The result is a more efficient brain. The same principle applies to artificial neural networks, and it works for much the same reason: not all connections are created equal, and the ones that matter least can be removed without the network noticing much.
UNSTRUCTURED PRUNING
The simplest form of pruning is unstructured (or magnitude-based) pruning. Look at all the weights in the network, rank them by their absolute magnitude, and set the smallest ones to zero. The intuition is that small-magnitude weights have little influence on the network's output, so removing them causes minimal damage. A network with many zero weights is called sparse, and if 90% of the weights are zero, we say the network has 90% sparsity.
The problem with unstructured pruning is that the resulting sparsity pattern is irregular. The non-zero weights are scattered randomly throughout the weight matrices. To actually skip the zero multiplications during inference, you need specialized hardware or software that can handle irregular sparse computations efficiently. Most microcontrollers do not have such hardware. On a standard Cortex-M processor, iterating over a sparse matrix with random non-zero positions can actually be slower than computing the full dense matrix, because the irregular memory access patterns destroy cache efficiency. Unstructured pruning gives you a sparse model that is theoretically smaller but practically no faster — and sometimes slower.
STRUCTURED PRUNING
Structured pruning solves this by removing entire structural units of the network rather than individual weights. Instead of zeroing out random individual weights, it removes entire neurons, entire convolutional filters, entire attention heads, or even entire layers. The result is a smaller, denser network that runs efficiently on any hardware without requiring specialized sparse computation support. The network is genuinely smaller, not just nominally sparse.
The following script demonstrates filter-level structured pruning for a convolutional neural network. We use the L1 norm of each filter — the sum of absolute values of its weights — as a proxy for its importance. Filters with low L1 norms tend to produce undistinctive feature maps and are good candidates for removal.
# structured_pruning.py
# Filter-level structured pruning for convolutional neural networks.
# Run: python structured_pruning.py
import torch
import torch.nn as nn
def compute_filter_importance(conv_layer: nn.Conv2d) -> torch.Tensor:
"""
Computes the L1-norm importance score for each filter in a Conv2d layer.
Filters with higher L1 norms tend to produce more distinctive feature
maps and are considered more important. Filters with near-zero L1 norms
contribute little to the output and are good pruning candidates.
Args:
conv_layer: A PyTorch Conv2d layer to analyze.
Returns:
A 1-D tensor of importance scores, one per output filter.
"""
# Weight shape: [out_channels, in_channels, kH, kW].
# Sum absolute values across all dims except out_channels.
return conv_layer.weight.data.abs().sum(dim=[1, 2, 3])
def prune_conv_layer_structured(
conv_layer: nn.Conv2d,
next_conv_layer: nn.Conv2d,
prune_ratio: float,
) -> tuple:
"""
Removes the least important filters from a convolutional layer.
Because each output filter of conv_layer corresponds to an input
channel of next_conv_layer, the corresponding input channels of
next_conv_layer are also removed to maintain architectural consistency.
Args:
conv_layer: The convolutional layer to prune.
next_conv_layer: The subsequent layer that must be updated.
prune_ratio: Fraction of filters to remove (e.g., 0.5 = 50%).
Returns:
A tuple (pruned_conv, pruned_next_conv) of new nn.Conv2d layers.
"""
importance = compute_filter_importance(conv_layer)
num_filters = len(importance)
num_to_prune = int(num_filters * prune_ratio)
# argsort gives ascending order; keep the highest-importance filters.
sorted_indices = torch.argsort(importance)
keep_indices = sorted(sorted_indices[num_to_prune:].tolist())
print(f" Keeping {len(keep_indices)} of {num_filters} filters "
f"({prune_ratio * 100:.0f}% pruned)")
new_out_channels = len(keep_indices)
# Build the pruned conv_layer with fewer output channels.
pruned_conv = nn.Conv2d(
in_channels=conv_layer.in_channels,
out_channels=new_out_channels,
kernel_size=conv_layer.kernel_size,
stride=conv_layer.stride,
padding=conv_layer.padding,
bias=(conv_layer.bias is not None),
)
pruned_conv.weight.data = conv_layer.weight.data[keep_indices, :, :, :]
if conv_layer.bias is not None:
pruned_conv.bias.data = conv_layer.bias.data[keep_indices]
# Build the updated next layer accepting only the kept input channels.
pruned_next = nn.Conv2d(
in_channels=new_out_channels,
out_channels=next_conv_layer.out_channels,
kernel_size=next_conv_layer.kernel_size,
stride=next_conv_layer.stride,
padding=next_conv_layer.padding,
bias=(next_conv_layer.bias is not None),
)
pruned_next.weight.data = (
next_conv_layer.weight.data[:, keep_indices, :, :]
)
if next_conv_layer.bias is not None:
# Bias of next_conv_layer is per output channel, not affected by
# input channel pruning, so we copy it unchanged.
pruned_next.bias.data = next_conv_layer.bias.data.clone()
return pruned_conv, pruned_next
if __name__ == "__main__":
torch.manual_seed(0)
conv1 = nn.Conv2d(1, 16, kernel_size=3, padding=1, bias=True)
conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1, bias=True)
print("Original architecture:")
print(f" conv1: {conv1}")
print(f" conv2: {conv2}")
pruned_conv1, pruned_conv2 = prune_conv_layer_structured(
conv_layer=conv1,
next_conv_layer=conv2,
prune_ratio=0.5,
)
print("\nPruned architecture:")
print(f" conv1: {pruned_conv1}")
print(f" conv2: {pruned_conv2}")
print("\nFine-tune the pruned model to recover accuracy before deployment.")
After structured pruning, it is essential to fine-tune the pruned model for a few epochs to allow the remaining weights to adapt to the absence of the pruned filters. This recovery training typically restores most of the accuracy lost during pruning. The standard workflow is therefore: train to convergence, prune, fine-tune, prune again, fine-tune again — an iterative process that gradually reduces the model size while maintaining accuracy. This is called iterative magnitude pruning, and it is significantly more effective than pruning everything at once and hoping for the best.
THE LOTTERY TICKET HYPOTHESIS
In 2019, Frankle and Carlin published a paper called "The Lottery Ticket Hypothesis," which proposed that within every large neural network there exists a small subnetwork — the winning lottery ticket — that, when trained in isolation from the start with the same initial weights, can match the accuracy of the full network. This hypothesis has profound implications: large networks are not inherently necessary for high accuracy. They are simply a convenient way to find these small, efficient subnetworks through the training process. The large network is not the destination; it is the search procedure.
While finding the exact winning ticket is computationally expensive, the hypothesis has inspired a generation of more practical pruning algorithms that aim to identify and preserve the most important subnetwork within a trained model. The field is still active, and the practical implications continue to unfold.
CHAPTER FIVE: KNOWLEDGE DISTILLATION — TEACHING A SMALL STUDENT FROM A LARGE TEACHER
Quantization and pruning work by modifying an existing model to make it smaller. Knowledge distillation takes a fundamentally different approach: it trains a small model from scratch, but instead of using only the hard ground-truth labels as training targets, it also uses the outputs of a large, pre-trained model as a richer source of supervision. The large model teaches the small one, and the small one learns more than it could ever learn from the labels alone.
The intuition is beautiful. When a large, well-trained neural network — the teacher — processes an input, its output is not just a single predicted class. It is a probability distribution over all classes. For an image of a cat, the teacher might output 85% probability for "cat," 10% for "lynx," 3% for "tiger," and tiny probabilities for everything else. These soft labels contain far more information than the hard label "cat." They reveal that the teacher considers a cat to be somewhat similar to a lynx and a tiger — which is genuinely useful structural knowledge about the problem domain. The small student model is trained to match these soft distributions, not just the hard labels, and this richer training signal allows it to learn more efficiently.
Geoffrey Hinton introduced this idea formally in 2015, and it has since become one of the most powerful tools in the model compression toolkit. In 2026, knowledge distillation is routinely used to create the small language models that run on edge devices, where a large cloud-based LLM acts as the teacher and a tiny model suitable for microcontroller deployment is the student.
THE TEMPERATURE PARAMETER
A key ingredient in knowledge distillation is the temperature parameter (T). The standard softmax converts a vector of logits (\mathbf{z}) into probabilities:
$$p_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)}$$
With temperature (T), the softmax becomes:
$$p_i(T) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$
When (T > 1), the distribution becomes softer — the differences between class probabilities are reduced, and the small but non-zero probabilities for incorrect classes become more visible. This is exactly what we want: we want the student to learn from the teacher's nuanced uncertainty, not just its confident predictions. A teacher that is 85% sure about a cat and 10% sure about a lynx is telling you something important about the structure of the problem. A hard label of "cat" is not.
# knowledge_distillation.py
# Complete knowledge distillation training loop in PyTorch.
# Run: python knowledge_distillation.py
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
def distillation_loss(
student_logits: torch.Tensor,
teacher_logits: torch.Tensor,
true_labels: torch.Tensor,
temperature: float = 4.0,
alpha: float = 0.7,
) -> torch.Tensor:
"""
Computes the combined knowledge distillation loss.
Combines two components:
1. Soft loss: KL divergence between student and teacher soft
probability distributions at temperature T. Transfers the
teacher's "dark knowledge" to the student.
2. Hard loss: Standard cross-entropy against ground-truth labels.
Ensures the student still learns the correct answers directly.
Args:
student_logits: Raw output logits from the student model.
teacher_logits: Raw output logits from the teacher model.
true_labels: Ground-truth class indices.
temperature: Softmax temperature T. Higher values produce softer
distributions. Typical range: 3–6.
alpha: Weight for the soft loss. Hard loss weight is
(1 - alpha). Typical range: 0.5–0.9.
Returns:
A scalar tensor representing the combined distillation loss.
"""
# F.kl_div expects log-probabilities as input, probabilities as target.
student_soft = F.log_softmax(student_logits / temperature, dim=1)
teacher_soft = F.softmax(teacher_logits / temperature, dim=1)
# Multiply by T² to compensate for gradient scaling caused by the
# temperature division in the forward pass (Hinton et al., 2015).
soft_loss = F.kl_div(
student_soft,
teacher_soft,
reduction="batchmean",
) * (temperature ** 2)
hard_loss = F.cross_entropy(student_logits, true_labels)
return alpha * soft_loss + (1.0 - alpha) * hard_loss
def train_with_distillation(
teacher_model: nn.Module,
student_model: nn.Module,
train_loader: DataLoader,
optimizer: torch.optim.Optimizer,
device: torch.device,
temperature: float = 4.0,
alpha: float = 0.7,
num_epochs: int = 20,
) -> None:
"""
Trains a student model using knowledge distillation from a teacher.
The teacher is kept in eval mode with gradients disabled — we only
use it to generate soft targets. The student is trained normally
with gradient updates.
Args:
teacher_model: The large pre-trained teacher network (frozen).
student_model: The small student network to be trained.
train_loader: DataLoader providing (inputs, labels) batches.
optimizer: Optimizer for the student model's parameters.
device: The torch.device to run computations on.
temperature: Distillation temperature (see distillation_loss).
alpha: Distillation loss weight (see distillation_loss).
num_epochs: Number of training epochs.
"""
teacher_model.eval()
teacher_model.to(device)
student_model.train()
student_model.to(device)
for epoch in range(num_epochs):
total_loss = 0.0
num_batches = 0
for inputs, labels in train_loader:
inputs = inputs.to(device)
labels = labels.to(device)
with torch.no_grad():
teacher_logits = teacher_model(inputs)
student_logits = student_model(inputs)
loss = distillation_loss(
student_logits=student_logits,
teacher_logits=teacher_logits,
true_labels=labels,
temperature=temperature,
alpha=alpha,
)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
num_batches += 1
avg_loss = total_loss / num_batches
print(f"Epoch [{epoch + 1:3d}/{num_epochs}] "
f"Avg Distillation Loss: {avg_loss:.4f}")
The power of knowledge distillation becomes clear when you look at the results. A student model with 10 times fewer parameters than the teacher can often achieve 95 to 98% of the teacher's accuracy when trained with distillation, compared to perhaps 85 to 90% when trained on hard labels alone. For TinyML applications, this difference can be the line between a model that is good enough to deploy and one that is not. In 2026, knowledge distillation is also used at the LLM level, where large cloud-based models like GPT-4 generate high-quality training data and soft labels used to train much smaller models that run on edge devices.
CHAPTER SIX: NEURAL ARCHITECTURE SEARCH — LETTING THE MACHINE DESIGN THE MACHINE
So far, we have discussed techniques that take an existing model and make it smaller. Neural Architecture Search (NAS) takes a fundamentally different philosophical approach: instead of compressing a human-designed architecture, it automatically searches for the best architecture for a given set of constraints from the very beginning. Rather than shrinking something large, it finds something small that is already good.
The idea is to define a search space of possible architectures — different numbers of layers, different filter sizes, different activation functions, different connection patterns — and then use an optimization algorithm to find the architecture within that space that achieves the best accuracy while satisfying the hardware constraints. For TinyML, NAS is particularly powerful because the hardware constraints are so severe that human intuition about good architectures often fails. An automated search can discover non-obvious architectural choices that happen to work extremely well within the tight constraints of a specific microcontroller, choices that no human would have thought to try.
MIT's MCUNet framework, developed by the Han Lab, is the landmark example of NAS for microcontrollers. MCUNet consists of two co-designed components: TinyNAS (the architecture search algorithm) and TinyEngine (the inference engine). The key innovation is that TinyNAS does not just search for a good architecture in the abstract — it searches for an architecture specifically optimized for the memory layout and execution model of TinyEngine. This co-design approach allows MCUNet to achieve over 70% top-1 accuracy on ImageNet on a commercial microcontroller with only 256 KB of SRAM, a result that was considered essentially impossible before MCUNet appeared.
TinyEngine achieves its memory efficiency through in-place depth-wise convolution and careful memory scheduling that considers the entire network topology rather than optimizing each layer independently. By analyzing the data flow through the entire network, TinyEngine can reuse memory buffers across layers in ways that a layer-by-layer optimizer cannot, reducing peak SRAM usage by 2.7 to 4.8 times compared to TensorFlow Lite Micro.
Even without running a full NAS, several architecture families have been specifically designed for efficiency and are widely used as starting points for embedded AI. MobileNetV3 uses depthwise separable convolutions, which factorize a standard convolution into a depthwise convolution followed by a pointwise convolution, reducing multiply-accumulate operations by roughly 8 to 9 times compared to a standard convolution with the same number of parameters. EfficientNet uses compound scaling to uniformly scale network width, depth, and resolution, with the smallest variants (EfficientNet-B0 and EfficientNet-Lite0) designed for mobile and embedded deployment. SqueezeNet achieves AlexNet-level accuracy on ImageNet with 50 times fewer parameters by using Fire modules that squeeze and expand the channel count strategically. For language tasks, Phi-4-mini (3.8B parameters) from Microsoft and the Qwen series (0.8B to 9B parameters) from Alibaba represent the state of the art in small language models as of 2026, using techniques like grouped-query attention to reduce KV cache size and optimized tokenizers to reduce sequence lengths.
CHAPTER SEVEN: OPERATOR FUSION AND MEMORY OPTIMIZATION
Beyond the model-level optimizations we have discussed, there is a class of inference-level optimizations that are crucial for microcontroller deployment. These do not change the model's parameters or architecture at all — they change how the computation is executed, and the difference can be dramatic.
OPERATOR FUSION
In a naive implementation of neural network inference, each operation — convolution, batch normalization, activation function — is executed as a separate step, with intermediate results written to memory between steps. This is wasteful because the memory bandwidth required to read and write these intermediate results is often the bottleneck on memory-constrained devices. You are spending most of your time moving data around rather than computing with it.
Operator fusion combines multiple consecutive operations into a single fused kernel that keeps intermediate results in registers or cache, never writing them to main memory. The most common fusion is Conv + BatchNorm + ReLU, which is ubiquitous in convolutional neural networks. The mathematical justification is elegant: since both the convolution and batch normalization are linear operations once the training statistics are fixed, they can be folded into a single linear operation:
$$y = \underbrace{\frac{\gamma}{\sqrt{\sigma^2 + \varepsilon}}}{\text{folded weight scale}} \cdot x{\text{conv}} + \underbrace{\beta - \frac{\gamma,\mu}{\sqrt{\sigma^2 + \varepsilon}}}_{\text{folded bias}}$$
This is equivalent to a convolution with scaled weights and a bias term — a single operation instead of two. The following script demonstrates how to fold batch normalization into a preceding convolutional layer, a standard step in preparing models for embedded deployment.
# fold_batchnorm.py
# Folds a BatchNorm2d layer into a preceding Conv2d layer for inference.
# Run: python fold_batchnorm.py
import torch
import torch.nn as nn
def fold_batchnorm_into_conv(
conv: nn.Conv2d,
bn: nn.BatchNorm2d,
) -> nn.Conv2d:
"""
Folds a BatchNorm2d layer into a preceding Conv2d layer.
After training, batch normalization parameters (running_mean,
running_var, weight/gamma, bias/beta) are constants. This function
absorbs them into the convolutional layer's weights and bias,
eliminating the batch normalization operation at inference time.
The model must be in eval() mode before calling this function so
that running_mean and running_var reflect training statistics, not
the current batch statistics.
Args:
conv: The Conv2d layer to fold into. Must be on the same device as bn.
bn: The BatchNorm2d layer to absorb.
Returns:
A new Conv2d layer with batch normalization folded in.
The returned layer always has a bias term.
"""
device = conv.weight.device
gamma = bn.weight.data
beta = bn.bias.data
mean = bn.running_mean
var = bn.running_var
eps = bn.eps
std = torch.sqrt(var + eps)
scale = gamma / std # Per-channel scale factor: γ / sqrt(σ² + ε)
# Scale each output filter by its corresponding scale factor.
# conv.weight shape: [out_channels, in_channels, kH, kW].
folded_weight = conv.weight.data * scale.view(-1, 1, 1, 1)
# Compute the folded bias, absorbing both the conv bias and BN shift.
# torch.zeros is created on the same device to avoid CUDA/CPU mismatches.
conv_bias = (
conv.bias.data
if conv.bias is not None
else torch.zeros(conv.out_channels, device=device)
)
folded_bias = scale * (conv_bias - mean) + beta
fused_conv = nn.Conv2d(
in_channels=conv.in_channels,
out_channels=conv.out_channels,
kernel_size=conv.kernel_size,
stride=conv.stride,
padding=conv.padding,
dilation=conv.dilation,
groups=conv.groups,
bias=True, # Always has a bias after folding BN.
)
fused_conv.weight.data = folded_weight
fused_conv.bias.data = folded_bias
return fused_conv
if __name__ == "__main__":
torch.manual_seed(0)
conv = nn.Conv2d(3, 16, kernel_size=3, padding=1, bias=False)
bn = nn.BatchNorm2d(16)
conv.eval()
bn.eval()
x = torch.randn(1, 3, 8, 8)
with torch.no_grad():
ref_output = bn(conv(x))
fused_conv = fold_batchnorm_into_conv(conv, bn)
fused_conv.eval()
with torch.no_grad():
fused_output = fused_conv(x)
max_diff = (ref_output - fused_output).abs().max().item()
print(f"Max absolute difference (should be ~0): {max_diff:.2e}")
print("BN folding successful." if max_diff < 1e-5 else "WARNING: mismatch!")
CMSIS-NN: THE GOLD STANDARD FOR ARM CORTEX-M INFERENCE
For ARM Cortex-M microcontrollers specifically, ARM's CMSIS-NN library provides hand-optimized implementations of the most common neural network operations. CMSIS-NN exploits the SIMD instructions available on Cortex-M4 and higher — the DSP extension — which allow multiple integer operations to be performed in a single clock cycle. The Cortex-M4's SIMD instructions can perform four 8-bit multiply-accumulate operations simultaneously, effectively quadrupling the throughput of INT8 matrix multiplications compared to a naive scalar implementation. The Cortex-M55, with its M-Profile Vector Extension (MVE), can process even more operations per cycle.
TensorFlow Lite Micro automatically uses CMSIS-NN kernels when targeting ARM Cortex-M devices, which is one of the reasons it is the preferred inference framework for this class of hardware. You get the hand-optimized performance without having to write a single line of assembly.
Installation — TensorFlow Lite Micro firmware project
# Clone the TensorFlow Lite Micro repository.
git clone --depth 1 https://github.com/tensorflow/tflite-micro.git
cd tflite-micro
# Generate a standalone TFLM project with CMSIS-NN kernels for Cortex-M4.
python tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py \
--makefile_options="TARGET=cortex_m_generic \
OPTIMIZED_KERNEL_DIR=cmsis_nn \
TARGET_ARCH=cortex-m4+fp" \
/tmp/tflm_project
# Convert your quantized model to a C array for embedding in firmware.
xxd -i model_int8.tflite > /tmp/tflm_project/model_data.h
# Install the ARM GCC cross-compiler (Ubuntu/Debian).
sudo apt install gcc-arm-none-eabi
# Build the firmware project.
cd /tmp/tflm_project
make -f Makefile TARGET=cortex_m_generic
# Flash to an STM32F4 board using OpenOCD and ST-Link.
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg \
-c "program firmware.elf verify reset exit"
The following C++ file shows what the deployment side looks like — the firmware that runs on the microcontroller to perform inference using a TFLM model.
/*
* tflite_micro_inference.cc
*
* Runs inference using TensorFlow Lite Micro on an ARM Cortex-M
* microcontroller. The model is embedded as a C array (model_data.h)
* generated by: xxd -i model_int8.tflite > model_data.h
*
* Build: included in the TFLM project Makefile (see installation above).
* Flash: openocd -f interface/stlink.cfg -f target/stm32f4x.cfg \
* -c "program firmware.elf verify reset exit"
*/
#include <cstring>
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/micro/system_setup.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "model_data.h"
/*
* Tensor arena — statically allocated RAM used by TFLM for all intermediate
* tensors (activations) during inference. Must accommodate peak memory usage.
*
* Too small: AllocateTensors() returns kTfLiteError at startup.
* Too large: wastes precious SRAM.
*
* Use TFLM's RecordingMicroInterpreter or Edge Impulse's profiler to
* determine the exact value needed for your specific model.
*/
static const int TENSOR_ARENA_SIZE = 32 * 1024; /* 32 KB for this model */
alignas(16) static uint8_t tensor_arena[TENSOR_ARENA_SIZE];
/* Global TFLM objects — persist across inference calls. */
static tflite::MicroInterpreter* interpreter = nullptr;
static TfLiteTensor* input_tensor = nullptr;
static TfLiteTensor* output_tensor = nullptr;
/*
* initialize_tflm_model()
*
* Parses the embedded model and allocates all tensors in the arena.
* Call once during system initialization — not before each inference,
* since tensor allocation is relatively expensive.
*
* Returns: true on success, false on failure.
*/
bool initialize_tflm_model(void)
{
const tflite::Model* model = tflite::GetModel(g_model_data);
if (model->version() != TFLITE_SCHEMA_VERSION) {
/* Schema version mismatch — regenerate model_data.h with the
version of xxd that matches your TFLM library version. */
return false;
}
/*
* Register only the operations used by this model.
* Registering all ops wastes Flash. Check required ops with Netron
* (https://netron.app) or tflite_micro_analyze.py.
* The template parameter (5) must equal the number of Add*() calls.
*/
static tflite::MicroMutableOpResolver<5> resolver;
resolver.AddConv2D();
resolver.AddDepthwiseConv2D();
resolver.AddReshape();
resolver.AddSoftmax();
resolver.AddMaxPool2D();
static tflite::MicroInterpreter static_interpreter(
model, resolver, tensor_arena, TENSOR_ARENA_SIZE
);
interpreter = &static_interpreter;
if (interpreter->AllocateTensors() != kTfLiteOk) {
/* Arena too small or unsupported op — check resolver and arena size. */
return false;
}
input_tensor = interpreter->input(0);
output_tensor = interpreter->output(0);
return true;
}
/*
* run_inference()
*
* Copies INT8 input data into the model's input tensor and executes
* one forward pass. On Cortex-M4 with CMSIS-NN, INT8 convolutions
* use SIMD instructions for 4× throughput vs. scalar implementation.
*
* Parameters:
* input_data — Pointer to INT8 quantized input (e.g., image pixels).
* input_size — Number of bytes in input_data.
* output_scores — Caller-allocated buffer for INT8 class scores.
* num_classes — Number of output classes (size of output_scores).
*
* Returns: true on success, false on inference failure.
*/
bool run_inference(
const int8_t* input_data,
size_t input_size,
int8_t* output_scores,
size_t num_classes)
{
std::memcpy(input_tensor->data.int8, input_data, input_size);
if (interpreter->Invoke() != kTfLiteOk) {
return false;
}
std::memcpy(output_scores, output_tensor->data.int8, num_classes);
return true;
}
This C++ file is the final destination of all the Python-level optimizations we have discussed. The model, compressed through quantization and pruning, ends up as a C array compiled into the microcontroller's Flash memory. The TFLM interpreter, running from a tiny 16 to 32 KB footprint, reads that array and executes the inference using CMSIS-NN's optimized kernels. The entire inference might take 10 to 50 milliseconds on a Cortex-M4 running at 168 MHz, consuming perhaps 50 milliwatts of power. That is a remarkable amount of intelligence for a remarkable amount of energy.
CHAPTER EIGHT: DEPLOYING LANGUAGE MODELS ON THE EDGE — THE FRONTIER OF 2026
Everything we have discussed so far applies primarily to traditional deep learning models: convolutional neural networks for image classification, small recurrent networks for time-series analysis, and similar architectures. But what about large language models? Can you really run something like a language model on a microcontroller?
The honest answer in 2026 is: it depends on what you mean by "language model" and what you mean by "microcontroller."
THE SPECTRUM OF EDGE LANGUAGE MODELS
At the most capable end of the edge spectrum, devices like the Raspberry Pi 5 (with 8 GB of RAM) can run 4 to 7 billion parameter models quantized to INT4 using llama.cpp. These models can perform genuine language understanding, text generation, and even simple reasoning tasks. They are not as capable as GPT-4 or Gemini Ultra, but they are remarkably useful for a wide range of applications. Moving down the capability ladder, devices like the NVIDIA Jetson Orin Nano (with 8 GB of LPDDR5 RAM and a dedicated GPU) can run 1 to 3 billion parameter models at interactive speeds. The STM32N6 microcontroller, with its integrated Neural-ART NPU, can run models with tens of millions of parameters for tasks like keyword spotting, intent classification, and simple natural language understanding. At the very bottom of the ladder — the classic Cortex-M4 with 256 KB of RAM — you are looking at models with hundreds of thousands of parameters at most. These are not language models in the GPT sense, but they can perform useful NLP tasks like intent classification, named entity recognition, and sentiment analysis on short text inputs.
LLAMA.CPP: THE WORKHORSE OF EDGE LLM INFERENCE
For the Raspberry Pi and similar single-board computers, llama.cpp has become the de facto standard for running quantized language models. Written in pure C/C++ with no external dependencies, it supports a wide range of quantization formats (from Q8_0 down to Q2_K) and can run models like Llama 3.2, Phi-4-mini, Qwen 2.5, and Gemma 3n on ARM CPUs using NEON SIMD instructions. The GGUF file format used by llama.cpp is particularly well-designed for memory-mapped inference: the model weights are stored in a format that can be directly memory-mapped from disk, allowing inference on devices with less RAM than the model's total size by swapping layers in and out as needed.
Installation — llama.cpp on Raspberry Pi 5 (ARM64)
# Install build dependencies.
sudo apt update && sudo apt install -y git cmake build-essential
# Clone and build llama.cpp with NEON optimizations (enabled by default on ARM64).
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -DLLAMA_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)
# Download a quantized model (Phi-4-mini at Q4_K_M, approximately 2.3 GB).
pip install huggingface_hub
python -c "
from huggingface_hub import hf_hub_download
hf_hub_download(
repo_id='bartowski/Phi-4-mini-instruct-GGUF',
filename='Phi-4-mini-instruct-Q4_K_M.gguf',
local_dir='./models'
)
"
# Run inference directly from the command line to verify the model works.
./build/bin/llama-cli \
--model ./models/Phi-4-mini-instruct-Q4_K_M.gguf \
--prompt "Sensor reading: temperature=87.3C. Is this within normal range for a steam turbine?" \
--n-predict 64 \
--threads 4 \
--ctx-size 512
# Install Python bindings for programmatic access.
pip install llama-cpp-python==0.2.77
# edge_llm_inference.py
# Runs a quantized language model locally using llama-cpp-python.
# Run: python edge_llm_inference.py
from llama_cpp import Llama
class EdgeLanguageModel:
"""
A wrapper around llama.cpp for efficient edge LLM inference.
Manages the lifecycle of a quantized language model and provides
a clean interface for text generation on resource-constrained devices.
Configured for minimal memory usage while maintaining acceptable
generation quality.
"""
def __init__(
self,
model_path: str,
context_length: int = 512,
num_threads: int = 4,
num_gpu_layers: int = 0,
):
"""
Initializes the edge language model.
Args:
model_path: Path to the GGUF quantized model file.
context_length: Maximum context window in tokens. Smaller values
use less RAM (KV cache). Keep at 256–512 for
devices with limited RAM.
num_threads: Number of CPU threads for inference. Match to
the number of physical cores available.
num_gpu_layers: Number of layers to offload to GPU.
Set to 0 for CPU-only inference (Raspberry Pi).
"""
print(f"Loading model: {model_path}")
print(f"Context length: {context_length} tokens | Threads: {num_threads}")
# n_ctx controls the KV cache size, often the dominant memory
# consumer for LLMs. Reducing from the default 4096 to 512
# can save hundreds of megabytes of RAM.
self._model = Llama(
model_path=model_path,
n_ctx=context_length,
n_threads=num_threads,
n_gpu_layers=num_gpu_layers,
verbose=False,
)
print("Model loaded successfully.")
def generate(
self,
prompt: str,
max_new_tokens: int = 128,
temperature: float = 0.7,
top_p: float = 0.9,
) -> str:
"""
Generates text from a prompt using the quantized language model.
Args:
prompt: The input text prompt for the model.
max_new_tokens: Maximum number of tokens to generate. Keep small
on edge devices to limit inference time.
temperature: Sampling temperature. Lower values (0.1–0.5)
produce more deterministic outputs. Higher values
(0.7–1.0) increase creativity.
top_p: Nucleus sampling threshold.
Returns:
The generated text string (excluding the input prompt).
"""
response = self._model(
prompt,
max_tokens=max_new_tokens,
temperature=temperature,
top_p=top_p,
echo=False,
)
return response["choices"][0]["text"].strip()
def demonstrate_edge_llm() -> None:
"""
Demonstrates edge LLM inference for an industrial IoT use case.
Simulates a smart sensor that uses a compressed language model to
interpret natural language queries and generate structured responses,
all running locally without cloud connectivity.
"""
model = EdgeLanguageModel(
model_path="./models/Phi-4-mini-instruct-Q4_K_M.gguf",
context_length=512,
num_threads=4,
)
command = (
"Sensor reading: temperature=87.3C, pressure=2.1bar. "
"Is this within normal operating range for a steam turbine? "
"Answer in one sentence."
)
print(f"\nPrompt: {command}")
response = model.generate(
prompt=command,
max_new_tokens=64,
temperature=0.3,
)
print(f"Response: {response}")
if __name__ == "__main__":
demonstrate_edge_llm()
A Phi-4-mini model at Q4_K_M quantization occupies approximately 2.3 GB of RAM, fitting comfortably on a Raspberry Pi 5 with 4 or 8 GB of RAM. At Q2_K quantization, the same model drops to about 1.2 GB, enabling deployment on devices with only 2 GB of RAM, though with some quality degradation. The key parameters to tune are the context length (which controls KV cache memory usage) and the quantization level — and the trade-offs between them are surprisingly smooth.
EXECUTORCH: PYTORCH'S ANSWER TO EDGE DEPLOYMENT
Meta's ExecuTorch framework, which reached production maturity in 2025, takes a different approach to edge LLM deployment. Rather than relying on a runtime interpreter like llama.cpp, ExecuTorch uses ahead-of-time (AOT) compilation to convert PyTorch models into a highly optimized binary format (.pte files) that can be executed by a tiny C++ runtime with a base footprint of only 50 KB. ExecuTorch supports ARM Cortex-M and Ethos-U NPU backends, making it one of the few frameworks capable of running transformer-based models on true microcontroller hardware.
Installation — ExecuTorch
# Python 3.10+ required.
pip install executorch==0.4.0 torch==2.3.1 torchvision==0.18.1
# For ARM Cortex-M / Ethos-U backend support:
pip install "executorch[ethos-u]==0.4.0"
# Export a PyTorch model to .pte format.
python -c "
import torch
from executorch.exir import to_edge
from torch.export import export
model = torch.nn.Sequential(
torch.nn.Linear(128, 64),
torch.nn.ReLU(),
torch.nn.Linear(64, 10),
)
model.eval()
example_inputs = (torch.randn(1, 128),)
exported = export(model, example_inputs)
edge_program = to_edge(exported)
executorch_program = edge_program.to_executorch()
with open('model.pte', 'wb') as f:
f.write(executorch_program.buffer)
print('Exported model.pte — ready for embedded deployment.')
"
# See https://pytorch.org/executorch/ for device-specific deployment guides.
CHAPTER NINE: THE COMPLETE OPTIMIZATION PIPELINE — PUTTING IT ALL TOGETHER
In practice, achieving good accuracy on a microcontroller requires combining multiple optimization techniques in a carefully orchestrated pipeline. No single technique is sufficient on its own. The art of TinyML engineering lies in knowing which techniques to apply, in what order, and how aggressively — and in having the humility to go back and try again when the first attempt does not fit.
A typical pipeline for a classification model targeting a Cortex-M4 with 256 KB SRAM and 1 MB Flash begins with the choice of base architecture. Rather than trying to compress a large ResNet-50, you start with a MobileNetV3-Small or a MCUNet-searched architecture — something already designed for efficiency, which makes every subsequent compression step much easier. If you have access to a larger, more accurate model, you use it as a knowledge distillation teacher during training, squeezing every bit of accuracy out of your small model before you even begin compressing it.
Once the model is trained to convergence, you apply iterative structured pruning: prune 20 to 30% of the filters in each convolutional layer, fine-tune for a few epochs to recover accuracy, then prune again. You repeat this cycle until the model fits within the Flash budget with some headroom to spare. Then you apply quantization-aware training — inserting fake-quantization nodes and fine-tuning for 10 to 20% of the original training duration — which allows the model to adapt to INT8 arithmetic and typically recovers any accuracy lost during pruning. Finally, you run the TFLite PTQ converter with a calibration dataset to produce the deployable INT8 model, and you profile it on the actual target hardware to verify that both the Flash and SRAM budgets are satisfied.
The following script ties several of these steps together into a coherent workflow, tracking the key metrics at each stage so you can see exactly where you are in the accuracy-size trade-off.
# optimization_pipeline.py
# Runs a complete TFLite optimization pipeline and tracks metrics at each stage.
# Run: python optimization_pipeline.py
import numpy as np
import tensorflow as tf
from dataclasses import dataclass
@dataclass
class ModelMetrics:
"""
Stores key metrics for a model at a specific optimization stage.
Tracking these at each stage lets you make informed decisions about
which techniques to apply and how aggressively, based on the
accuracy-size trade-off at each step.
"""
stage_name: str # Human-readable name for this pipeline stage.
accuracy: float # Top-1 accuracy on the validation set (0–1).
model_size_kb: float # Model size in kilobytes.
def evaluate_tflite_model(
tflite_model_bytes: bytes,
test_data: np.ndarray,
test_labels: np.ndarray,
) -> float:
"""
Evaluates a TFLite model's top-1 accuracy on a test dataset.
Handles both float32 and INT8 quantized models automatically by
inspecting the input tensor's dtype and applying the model's
quantization parameters when needed.
Args:
tflite_model_bytes: The raw bytes of the .tflite model.
test_data: Test input data as a float32 numpy array.
test_labels: Ground-truth class indices.
Returns:
Top-1 accuracy as a float between 0 and 1.
"""
interpreter = tf.lite.Interpreter(model_content=tflite_model_bytes)
interpreter.allocate_tensors()
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
is_quantized = (input_details[0]["dtype"] == np.int8)
correct = 0
for i in range(len(test_data)):
sample = test_data[i : i + 1] # Add batch dimension.
if is_quantized:
scale = input_details[0]["quantization"][0]
zero_point = input_details[0]["quantization"][1]
sample = (sample / scale + zero_point).astype(np.int8)
interpreter.set_tensor(input_details[0]["index"], sample)
interpreter.invoke()
output = interpreter.get_tensor(output_details[0]["index"])
if np.argmax(output[0]) == test_labels[i]:
correct += 1
return correct / len(test_data)
def run_optimization_pipeline(
keras_model: tf.keras.Model,
test_data: np.ndarray,
test_labels: np.ndarray,
calibration_data: np.ndarray,
target_flash_kb: float = 512.0,
) -> list:
"""
Runs a complete optimization pipeline and tracks metrics at each stage.
Applies FP32 baseline, FP16 weight quantization, and full INT8 PTQ,
reporting the accuracy-size trade-off at each step.
Args:
keras_model: The trained Keras model to optimize.
test_data: Test dataset for accuracy evaluation (float32).
test_labels: Ground-truth labels for the test dataset.
calibration_data: Small dataset for INT8 activation calibration.
100–200 representative samples recommended.
target_flash_kb: Flash memory budget in KB.
Returns:
A list of ModelMetrics objects, one per optimization stage.
"""
history = []
# ------------------------------------------------------------------ #
# Stage 1: Baseline FP32 #
# ------------------------------------------------------------------ #
converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
fp32_model = converter.convert()
history.append(ModelMetrics(
stage_name="Baseline FP32",
accuracy=evaluate_tflite_model(fp32_model, test_data, test_labels),
model_size_kb=len(fp32_model) / 1024,
))
# ------------------------------------------------------------------ #
# Stage 2: FP16 weight-only quantization #
# ------------------------------------------------------------------ #
converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]
fp16_model = converter.convert()
history.append(ModelMetrics(
stage_name="FP16 Weight Quantization",
accuracy=evaluate_tflite_model(fp16_model, test_data, test_labels),
model_size_kb=len(fp16_model) / 1024,
))
# ------------------------------------------------------------------ #
# Stage 3: Full INT8 PTQ (weights + activations) #
# ------------------------------------------------------------------ #
def calibration_generator():
for i in range(min(100, len(calibration_data))):
yield [calibration_data[i : i + 1]]
converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = calibration_generator
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS_INT8
]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
int8_model = converter.convert()
history.append(ModelMetrics(
stage_name="Full INT8 PTQ",
accuracy=evaluate_tflite_model(int8_model, test_data, test_labels),
model_size_kb=len(int8_model) / 1024,
))
# ------------------------------------------------------------------ #
# Print summary table #
# ------------------------------------------------------------------ #
baseline_size = history[0].model_size_kb
baseline_acc = history[0].accuracy
print("\n" + "=" * 72)
print("OPTIMIZATION PIPELINE RESULTS")
print("=" * 72)
print(f"{'Stage':<30} {'Accuracy':>10} {'Size (KB)':>12} "
f"{'Reduction':>10} {'Budget':>8}")
print("-" * 72)
for m in history:
reduction = baseline_size / m.model_size_kb
acc_delta = (m.accuracy - baseline_acc) * 100
fits = "OK " if m.model_size_kb <= target_flash_kb else "OVER"
print(f"{m.stage_name:<30} "
f"{m.accuracy * 100:>9.2f}% "
f"{m.model_size_kb:>10.1f} KB "
f"{reduction:>8.1f}x "
f"[{fits}] Δacc={acc_delta:+.2f}%")
print("=" * 72)
return history
if __name__ == "__main__":
# Load your trained Keras model. Replace "./saved_model" with your path.
keras_model = tf.keras.models.load_model("./saved_model")
# Replace these random arrays with your real validation and
# calibration data. The shapes must match your model's input.
num_test_samples = 200
input_shape = (28, 28, 1)
num_classes = 10
test_data = np.random.rand(
num_test_samples, *input_shape
).astype(np.float32)
test_labels = np.random.randint(0, num_classes, num_test_samples)
calibration_data = np.random.rand(100, *input_shape).astype(np.float32)
run_optimization_pipeline(
keras_model=keras_model,
test_data=test_data,
test_labels=test_labels,
calibration_data=calibration_data,
target_flash_kb=512.0,
)
Running this pipeline on a typical MobileNetV3-Small model trained on a 10-class image classification task yields results similar to the following (representative numbers based on published benchmarks):
========================================================================
OPTIMIZATION PIPELINE RESULTS
========================================================================
Stage Accuracy Size (KB) Reduction Budget
------------------------------------------------------------------------
Baseline FP32 92.40% 4820.0 KB 1.0x [OVER]
FP16 Weight Quantization 92.30% 2410.0 KB 2.0x [OVER]
Full INT8 PTQ 91.80% 1205.0 KB 4.0x [OK ]
========================================================================
The INT8 model achieves a 4× size reduction with only 0.6 percentage points of accuracy loss. For most embedded applications, that is an excellent trade-off — and it is the kind of result that makes the effort of building this pipeline entirely worthwhile.
CHAPTER TEN: BENCHMARKS, REAL-WORLD RESULTS, AND WHAT TO EXPECT
Theory is valuable, but engineers need numbers. Let us look at what is actually achievable on real hardware in 2026, based on published benchmarks and real-world deployments.
On Cortex-M4 class devices — the STM32F4 running at 168 MHz with 192 KB of SRAM and 1 MB of Flash — a well-optimized INT8 MobileNetV1 at 0.25× width multiplier can achieve approximately 70% top-1 accuracy on a 10-class subset of ImageNet, with an inference time of around 200 to 400 milliseconds and a model size of about 200 KB. This is adequate for gesture recognition, simple object detection, and keyword spotting. MIT's MCUNet, using TinyNAS-searched architectures and TinyEngine, achieves over 70% top-1 accuracy on the full 1000-class ImageNet dataset on a Cortex-M7 running at 216 MHz with 512 KB SRAM — a landmark result that demonstrated microcontrollers could handle genuinely challenging vision tasks. For audio tasks, keyword spotting models can achieve over 95% accuracy with models that fit in under 50 KB of Flash and run in under 10 milliseconds on a Cortex-M4. This is the most mature and widely deployed form of TinyML in 2026.
On Cortex-M55 devices paired with an Ethos-U55 NPU — such as the NXP FRDM i.MX 93 — the NPU can accelerate INT8 convolutions by 32 times compared to the CPU alone. This brings person detection to under 5 milliseconds and enables real-time audio classification at under 10 milliseconds. The STM32N6 with its Neural-ART accelerator achieves similar results, enabling models with tens of millions of parameters to run at interactive speeds on a microcontroller.
On the Raspberry Pi 5 with an ARM Cortex-A76 and 4 to 8 GB of LPDDR4X, a 4-bit quantized Phi-4-mini (3.8B parameters, approximately 2.3 GB in Q4_K_M format) runs at approximately 8 to 12 tokens per second using llama.cpp with 4 threads. This is fast enough for interactive applications like voice assistants, smart home controllers, and industrial diagnostic tools. A 1B parameter model at Q4_K_M runs at 25 to 40 tokens per second, which feels genuinely responsive.
On the NVIDIA Jetson Orin Nano with 8 GB of RAM and a 1024-core Ampere GPU, a 7B parameter model quantized to INT4 runs at 20 to 30 tokens per second, enabling sophisticated language understanding and generation for robotics, autonomous vehicles, and industrial automation. This device represents the high end of the embedded AI spectrum in 2026.
CHAPTER ELEVEN: PRACTICAL TIPS, COMMON PITFALLS, AND LESSONS FROM THE FIELD
After years of deploying AI models on embedded systems, certain patterns of success and failure emerge repeatedly. Some of these lessons are obvious in retrospect and painful in practice. Others are subtle enough that even experienced engineers get caught by them. The following hard-won observations can save you weeks of debugging and frustration.
ALWAYS PROFILE BEFORE OPTIMIZING
The most common mistake in TinyML engineering is applying aggressive optimization techniques without first understanding where the bottlenecks actually are. Before you start pruning and quantizing, profile your model on the target hardware — or at least on a simulator — to understand which layers consume the most memory and time. You may find that 80% of the inference time is spent in a single depthwise convolution layer, in which case optimizing everything else is largely wasted effort. Measure first. Optimize second. It sounds obvious. It is routinely ignored.
CALIBRATION DATA QUALITY MATTERS ENORMOUSLY FOR PTQ
For post-training quantization, the calibration dataset must be representative of the real input distribution. Using random noise or a dataset from a different domain will produce poor scale factors for the activations, leading to significant accuracy degradation. The model will be quantized as if it expects inputs that look nothing like what it will actually receive. Use at least 100 to 200 samples from your actual deployment distribution for calibration, and if accuracy is still disappointing after PTQ, the calibration data is the first thing to suspect.
WATCH OUT FOR ACTIVATION OUTLIERS
One of the most common causes of large accuracy drops during quantization is the presence of outlier activation values. A single activation value that is 100 times larger than the typical values in a tensor forces the quantization scale to be large, which means the typical values are represented with very coarse granularity — essentially, most of your precision is wasted on a range that almost never gets used. Techniques like SmoothQuant (which transfers the scale of outlier activations into the weights) and per-channel quantization (which uses a separate scale for each output channel) can mitigate this problem significantly.
THE TENSOR ARENA SIZE IS NOT OBVIOUS
When deploying with TensorFlow Lite Micro, the tensor arena size is not simply the sum of all activation sizes. TFLM reuses memory buffers across layers when possible, since not all activations need to be live simultaneously. The actual peak memory usage can be significantly lower than the naive sum. Use TFLM's built-in RecordingMicroInterpreter or the profiling tools in Edge Impulse to determine the correct arena size for your model. Guessing too high wastes SRAM. Guessing too low causes a silent failure at startup that can be baffling to diagnose.
BATCH NORMALIZATION MUST BE IN EVAL MODE BEFORE EXPORT
Batch normalization layers use running statistics — mean and variance — computed during training. If you export a model without first calling model.eval() in PyTorch (or setting training=False in TensorFlow), the batch normalization layers will use batch statistics instead of running statistics during the export trace, producing wildly incorrect outputs at inference time. The model will appear to work during development and fail mysteriously on the device. Always verify that batch normalization is properly handled before export. This mistake has caused more late-night debugging sessions than almost anything else in embedded AI.
TEST WITH REAL HARDWARE, NOT JUST SIMULATORS
Simulators and emulators are useful for initial development, but they do not perfectly replicate the behavior of real hardware. Numerical precision differences, memory alignment requirements, and hardware-specific bugs can cause a model that works perfectly in simulation to fail on the actual device. Always validate your final model on the target hardware before declaring success. The hardware has a way of finding problems that the simulator missed.
CONSIDER THE FULL SYSTEM, NOT JUST THE MODEL
The neural network is only one component of an embedded AI system. The sensor interface, the preprocessing pipeline, the postprocessing logic, and the communication stack all consume memory and compute resources. A model that fits in 200 KB of Flash might leave no room for the rest of the firmware. Budget your resources holistically, not just for the model. The model is the star of the show, but the show cannot run without the rest of the crew.
CHAPTER TWELVE: THE TOOLS OF THE TRADE — A 2026 ECOSYSTEM OVERVIEW
The tooling ecosystem for embedded AI has matured dramatically in recent years. In 2026, engineers have access to a rich set of frameworks, libraries, and platforms that make the optimization and deployment pipeline significantly more accessible than it was just three years ago. The days of hand-rolling inference kernels in assembly are not entirely gone, but they are increasingly optional.
TensorFlow Lite Micro remains the most widely deployed inference framework for Cortex-M class microcontrollers. Its core runtime fits in 16 KB of Flash, it automatically uses CMSIS-NN kernels on ARM targets, and it has excellent tooling support through the TFLite converter and Edge Impulse. Its main limitation is that it supports a fixed set of operations, and models that use unusual operations may not be deployable without custom kernel development — a non-trivial undertaking.
ExecuTorch from Meta and the PyTorch team is the most exciting new entrant in the embedded AI space. Its 50 KB base footprint, native PyTorch export workflow, and support for ARM Ethos-U NPUs make it a strong choice for new projects targeting the latest hardware. Its support for LLM deployment — including Llama and Phi models — on embedded targets is a unique capability that TFLM does not yet match, and it is worth watching closely.
MicroTVM from Apache TVM provides a compiler-based approach to embedded inference. Rather than using a fixed library of hand-optimized kernels, TVM compiles the model into optimized code for the specific target hardware. This can produce better performance than library-based approaches for unusual architectures or hardware targets, but it requires considerably more expertise to use effectively. It is a powerful tool for teams that have the time to invest in it.
Edge Impulse has become the dominant end-to-end platform for TinyML development. It provides a web-based interface for data collection, model training, optimization (including quantization and pruning), and deployment to a wide range of target devices. For engineers who are not deep learning experts, Edge Impulse dramatically lowers the barrier to entry for embedded AI — sometimes to the point where a working prototype can be built in an afternoon.
Neuton.AI, acquired by Nordic Semiconductor in 2025, specializes in automatically generating ultra-compact neural networks for Nordic's nRF series microcontrollers. Its AutoML approach can produce models with as few as a few hundred parameters that achieve good accuracy on simple classification tasks. It occupies a useful niche for applications where even a MobileNet is too large.
llama.cpp remains the gold standard for running quantized language models on CPU-based edge devices. Its GGUF format and extensive quantization options — Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K — provide a smooth trade-off curve between model quality and resource consumption. The project is actively maintained, and new quantization formats continue to appear as the community finds better ways to compress models without losing quality.
CHAPTER THIRTEEN: THE FUTURE — WHERE ARE WE HEADED?
The pace of progress in embedded AI is breathtaking, and not in the way that phrase is usually deployed as a cliché. In 2020, running a neural network on a microcontroller was a research curiosity that required a PhD and a lot of patience. In 2023, it was an emerging industry trend. In 2026, it is a mainstream engineering discipline with a $2.5 billion market and billions of deployed devices. The trajectory is steep and it shows no sign of flattening.
Silicon vendors are integrating increasingly capable NPUs directly into microcontrollers. The STM32N6's Neural-ART accelerator, the Nordic nRF9280's integrated AI capabilities, and the Infineon PSoC Edge series all represent a trend toward microcontrollers designed from the ground up for AI workloads. As these NPUs become more capable and more power-efficient, the boundary between "microcontroller AI" and "edge AI" will continue to blur, and the distinction will eventually become meaningless.
The research community has made remarkable progress in training small models that punch far above their weight. Techniques like mixture-of-experts (where only a fraction of the model's parameters are active for any given input), speculative decoding (where a tiny draft model generates candidate tokens that are verified by the main model), and state-space models (which offer linear rather than quadratic scaling with sequence length) are all enabling more capable AI in smaller packages. The trend is consistent: every year, the same accuracy is achievable in a smaller model than the year before.
The Han Lab's work on on-device training under 256 KB of memory is a harbinger of things to come. The ability for embedded devices to learn from new data in the field — adapting to changing environments, personalizing to individual users, detecting novel anomalies — will unlock entirely new classes of applications. Federated learning, where models are trained collaboratively across many devices without sharing raw data, is already being deployed in production systems and will become increasingly common.
Perhaps the most exciting development on the horizon is the convergence of large language model capabilities with embedded hardware. As models like Phi-4-mini and Qwen 2.5 continue to shrink while maintaining impressive capabilities, and as hardware like the Ethos-U85 NPU provides more compute per milliwatt, the gap between cloud AI and embedded AI will narrow. By 2028, it is entirely plausible that a microcontroller-class device will be able to run a model capable of genuine natural language understanding, contextual reasoning, and multi-step task planning — all locally, privately, and on a coin-cell battery. That is not science fiction. It is engineering in progress.
CONCLUSION: THE ART AND SCIENCE OF MAKING BIG AI SMALL
We have traveled a long road in this article, from the fundamental motivations for embedded AI through the mathematical details of quantization, the surgical precision of structured pruning, the pedagogical elegance of knowledge distillation, the automated ingenuity of neural architecture search, and the practical craft of deployment on real hardware.
The central lesson is that making deep learning work on microcontrollers is not about finding a single magic technique. It is about combining multiple complementary approaches — each addressing a different aspect of the resource-efficiency problem — into a coherent optimization strategy tailored to the specific constraints of your target hardware and the specific accuracy requirements of your application. There is no shortcut, but there is a clear path.
Quantization gives you 4 to 8 times memory reduction with minimal accuracy loss. Structured pruning gives you 2 to 5 times additional compression with fine-tuning recovery. Knowledge distillation gives you a better starting point for your small model. Neural architecture search gives you an architecture natively suited to your hardware constraints. Operator fusion and CMSIS-NN give you the best possible inference speed on ARM hardware. Together, these techniques can take a model that requires a GPU to run and fit it into a device that costs two dollars and runs on a battery.
The field is moving fast. The tools are maturing. The hardware is improving. The applications — from industrial predictive maintenance to medical monitoring to agricultural intelligence to smart cities — are genuinely important and genuinely exciting. And the engineering, when it all comes together and a tiny chip does something that would have seemed impossible a few years ago, is genuinely satisfying in a way that is hard to describe to people who have not experienced it.
The era of tiny giants is here. The only question is what you will build with them.
REFERENCES AND FURTHER READING
Han, S. et al., "MCUNet: Tiny Deep Learning on IoT Devices," NeurIPS 2020. Lin, J. et al., "MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning," NeurIPS 2021. Lin, J. et al., "On-Device Training Under 256 KB Memory," NeurIPS 2022. Hinton, G. et al., "Distilling the Knowledge in a Neural Network," arXiv 2015. Frankle, J. and Carlin, M., "The Lottery Ticket Hypothesis," ICLR 2019. Frantar, E. et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers," ICLR 2023. Lin, J. et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration," MLSys 2024. Xiao, G. et al., "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models," ICML 2023. ARM CMSIS-NN Documentation: https://arm-software.github.io/CMSIS_5/ TensorFlow Lite Micro: https://www.tensorflow.org/lite/microcontrollersExecuTorch Documentation: https://pytorch.org/executorch/ llama.cpp Repository: https://github.com/ggerganov/llama.cpp Edge Impulse Platform: https://www.edgeimpulse.com AutoAWQ Library: https://github.com/casper-hansen/AutoAWQ
