Accuracy Isn’t Everything: What Deploying AI at the Edge Taught Me
For a long time, I thought one number was enough to evaluate a model. Then I had to run one on real hardware, in real time, and realized that accuracy alone doesn't tell the whole story.
When I first started working with machine learning, the process seemed straightforward:
- Train the model
- Evaluate the model
- Improve the accuracy
- Deploy it
A 99% model looks better than a 97% model on a benchmark. But if that 99% model takes too long to respond, consumes too many resources, or cannot meet the application’s timing requirements, what exactly did that extra 2% buy us?
Deploying AI at the edge changed that perspective.
My approach gradually shifted from evaluating the model to evaluating the deployment.
The factors I consider now go well beyond accuracy: latency, frame rate, memory, power, cost and reliability all get a vote — and on constrained hardware, any one of them can overrule the others.
Before choosing a model, I now start with three questions:
What problem are we solving? Where will it run? What does the application require?
When 99% Accuracy Isn’t Enough
Consider a vision system that needs to process a camera feed in real time. One model may provide slightly better accuracy but take close to a second for inference. Another may give slightly lower accuracy while responding much faster.
On a benchmark, the first model may look better. In an actual application, the second one may be more practical.
That changed how I evaluate models. Instead of chasing the highest possible accuracy, I first look at what level of accuracy the application needs and what latency it can tolerate.
The goal is not maximum accuracy at any cost. It is enough accuracy within the application’s constraints.
Smaller Doesn’t Always Mean Faster
When resources are limited, choosing a lightweight model seems obvious. But model size alone doesn’t determine how quickly it will run.
Parameter count ≠ computational cost ≠ latency.
FLOPs and MACs give an idea of computational workload, but actual latency also depends on memory access, data movement, operator efficiency, runtime support, and the hardware itself.
Input resolution adds another trade-off. Reducing an image from 518×518 to 224×224 can reduce computation significantly, but it can also remove details needed to detect small objects or subtle defects.
So, input resolution isn’t just an optimization setting. It is a design decision.
The Optimization Trap
Quantization and pruning can reduce model size and computational requirements, but they don’t automatically make inference faster.
An optimization that works well in one environment may behave differently on another. A quantized model can become smaller without becoming faster if the runtime doesn’t efficiently support its operators. Some operations may even fall back to slower execution paths.
That reinforced an important lesson for me: optimization has to be evaluated across the entire stack.
Model → Runtime → Libraries → Operators → Hardware
I ran into this directly on an FPGA deployment. The newer YOLO model I had planned to use wasn’t supported by the board’s inference toolchain, so I stepped back to an earlier architecture that it did support.
On paper, that looked like a downgrade. In practice, the question was never which model was newest. It was which model the whole stack could actually execute. That model now runs on the same board in both CPU mode and FPGA mode, which makes it easy to see exactly what the hardware acceleration buys.
Measure the Whole Pipeline
Inference is only one stage of an edge AI application.
If inference takes 100 milliseconds and preprocessing, postprocessing, and other stages add another 80 milliseconds, the application doesn’t run at 100 milliseconds.
What matters is end-to-end latency, not just the inference time reported by the model.
The Lesson I Carry Forward
Working with edge AI changed how I define a “good” model.
More accurate doesn’t always mean better.
Smaller doesn’t always mean faster.
Quantized doesn’t always mean efficient.
My approach now is straightforward: define the application needs, understand the constraints, choose the model and input size accordingly, validate the deployment stack, and measure the complete pipeline on the actual device.
In the end, I learned that the best model isn’t the one that wins the benchmark. It’s the one that survives the real world.
Beyond Pixels: What If You Could Sense Someone Inside a Zone?
How millimetre-wave radar can tell whether someone is inside a defined area — without a camera, without light, and without creating a conventional image.
Data Engineering in AI: 80% Nobody Talks About
The AI revolution isn’t really about AI—it’s about data engineering wearing a shiny new suit.
Why Snowflake and Databricks Bet Big on Structured Data
In a world where GenAI, LLMs, and unstructured data dominate headlines, two major acquisitions recently made by industry giants signal a more grounded truth.