A few years ago, on-device machine learning in a mobile app usually meant a small image classifier or a barcode scanner. Today both Apple and Google ship on-device language models as part of the operating system on supported hardware, and open runtimes can run compact models on mid-range phones. That opens up features such as summarization, smart replies, offline transcription and document extraction that do not need to send data to a server.
It also creates new decisions. Which tasks belong on the device and which still need the cloud? What happens on older phones? How do you test a feature whose output is not deterministic? This guide gives product owners and CTOs a practical view, based on how we approach AI features in mobile projects. Platform capabilities are changing quickly, so treat specific APIs as a starting point and check current documentation before committing.
Why move inference onto the device
Privacy
Data such as messages, photos, health notes or documents can be processed without leaving the phone, which reduces what you collect and simplifies privacy disclosures.
Latency
No network round trip means near-instant responses for small tasks such as autocomplete, classification or camera effects.
Offline use
Field workers, travelers and users with patchy coverage keep the feature when the network drops.
Running cost
Inference on the user's hardware does not add per-request cloud API charges, though it adds engineering and testing effort.
The trade-off is capability. On-device models are much smaller than frontier cloud models. They handle focused tasks well and struggle with broad reasoning, long documents and up-to-date world knowledge.
The frameworks and models available now
There are two broad routes: use a model the operating system provides, or bundle your own model with a runtime. The table summarizes the main options at the time of writing.
| Option | Platform | What it is | Best for |
|---|---|---|---|
| Core ML | Apple | Apple's framework for running converted ML models on CPU, GPU and Neural Engine | Custom vision, audio and text models you train or convert |
| Foundation Models framework | Apple | Access to Apple's on-device language model on Apple Intelligence-capable devices | Text generation, summarization, extraction and structured output without bundling weights |
| ML Kit | Android and iOS | Google's SDK of ready-made on-device APIs (text recognition, barcode scanning, translation and more), plus GenAI APIs on supported Android devices | Common tasks with minimal ML expertise |
| Gemini Nano via AICore | Android | Google's on-device model, managed by the system service on supported devices | Summarization, rewriting, smart reply and similar text tasks |
| LiteRT (formerly TensorFlow Lite) | Android and iOS | Google's lightweight runtime for running custom models | Cross-platform custom models with hardware acceleration |
| ONNX Runtime Mobile | Android and iOS | Microsoft's runtime for models exported to the ONNX format | Teams training in PyTorch or other frameworks who want one export path |
For cross-platform apps, these are usually reached through native modules. Our Flutter developers and React Native engineers typically wrap the native APIs behind a small plugin so the shared code calls one interface on both platforms.
Use cases that work well on the device
- Text: summarizing a note or email thread, rewriting for tone, suggesting replies, extracting fields (dates, amounts, names) into structured data.
- Vision: document scanning and OCR, receipt capture, barcode reading, object detection for inventory or inspection apps, on-device photo classification.
- Audio: offline speech-to-text for dictation or field notes, keyword spotting, sound classification.
- Personalization: ranking content or suggestions based on behavior that stays on the device.
- Safety and moderation: flagging sensitive content before upload.
A logistics app that reads labels and extracts delivery details offline is a good example: the task is narrow, connectivity is unreliable and latency matters to the driver.
Device constraints you have to design around
- Hardware support: OS-provided language models run only on certain recent devices and may need to be downloaded or enabled. Plan for users who do not have them.
- Memory: larger models compete with your app and others for RAM; the OS may terminate background apps under pressure.
- App size: bundled models can add a significant download. Consider on-demand download after install.
- Battery and heat: sustained inference drains battery and can trigger thermal throttling, slowing everything down.
- Context limits: small models accept limited input, so long documents need chunking or a cloud path.
- Variability: performance differs widely across chipsets, especially in the Android ecosystem.
Designing a hybrid on-device and cloud architecture
In most apps we design for, on-device and cloud models work together. A thin routing layer decides where each request goes based on capability, input size, connectivity and the sensitivity of the data.
- 1
Define an AI service interface
Feature code calls one interface, such as summarize or extract, not a specific model, so you can swap implementations later.
- 2
Check capability at runtime
Detect whether the OS model or your bundled model is available on this device before offering the feature.
- 3
Route by rule
For example: sensitive or short inputs stay on the device; long inputs or complex reasoning go to the cloud when the user has consented.
- 4
Degrade gracefully
If neither path is available, fall back to a non-AI experience rather than an error screen.
- 5
Log outcomes, not content
Capture latency, failure rate and user feedback without uploading the private data the on-device path was meant to protect.
How to evaluate an on-device AI feature
Model outputs are probabilistic, so traditional pass or fail tests are not enough. Build a small evaluation set of realistic inputs with expected outputs or grading criteria, and run it on every model or prompt change.
| Dimension | How to test it |
|---|---|
| Quality | Score outputs on a fixed evaluation set, with human review for a sample |
| Latency | Measure time to first result and total time on low, mid and high-end target devices |
| Resource use | Profile memory peaks, battery drain and thermal state during repeated use |
| Robustness | Test empty, very long, multilingual and adversarial inputs |
| Fallback behavior | Verify the experience on unsupported devices and offline |
Start with one narrow feature, measure it with real users and expand from there. If you want help scoping an AI feature for an existing app, our iOS engineering team and Android engineers can run a short technical spike to test feasibility on your target devices.
Frequently asked questions
Can I run a large language model entirely on a phone?
You can run small language models on capable devices, either through OS-provided models or by bundling a compact open model. They handle focused tasks such as summarization and extraction well, but they are far less capable than large cloud models for open-ended reasoning.
Does on-device AI make my app compliant with privacy laws?
It can reduce the personal data you collect and transmit, which helps, but it does not by itself make an app compliant. You still need accurate disclosures, sensible data handling and advice from your compliance team or counsel.
What happens on older devices that lack AI hardware?
OS-provided models will not be available on them. Detect capability at runtime and fall back to a cloud model, a simpler bundled model or a non-AI experience.
Will adding a model make my app much larger?
Bundled models can add a significant amount to download size, depending on the model. Using OS-provided models or downloading the model after install keeps the initial app size down.
This article is general guidance based on common delivery practice. It is not legal, tax or financial advice; platform rules and regulations change, so confirm current requirements with the relevant provider or adviser.