Running AI Models in Flutter: Local LLM Integration Guide

A client asked me last year to add a "smart reply" feature to their messaging app. The brief was straightforward — call OpenAI's API, generate three suggested responses, show them as chips below the input field. I built it in two days. Worked beautifully in testing.

Then we shipped. Users in rural areas of Pakistan complained about 4-second response times. Users on hotel WiFi got flat-out timeouts. And our monthly API bill after just 5,000 active users? $200. For suggested replies. I spent another two days adding retry logic, response caching, and graceful degradation for offline scenarios. That's when it hit me — what if the model just ran on the phone itself?

That question sent me down a rabbit hole that lasted months. I tried TensorFlow Lite, wrestled with llama.cpp through Dart FFI, spent weekends profiling memory usage on budget Android phones. After all of that, I've now shipped three production Flutter apps with on-device AI features. No cloud dependency. Sub-second responses. Zero API costs.

I'm going to walk you through everything I learned — the stuff that actually works, the approaches I abandoned, and the real performance numbers nobody shares in tutorials.

Why I Stopped Using Cloud APIs for Everything

I want to be upfront — cloud APIs like OpenAI, Gemini, and Claude are phenomenal for complex reasoning tasks. If your app needs GPT-4-level intelligence, you need the cloud. No mobile chip on earth is running 175 billion parameters. That's just physics.

But most AI features in mobile apps don't need GPT-4. When I broke down the AI functionality across my projects, roughly 70% of them worked perfectly fine with small, local models:

• Smart replies — A 3B parameter model handles this just fine
• Text classification — TFLite with a 20MB model, instant inference
• Sentiment analysis — Even smaller models work here
• Image labeling — MobileNet is only 17MB

The other pain point was latency. Cloud API calls averaged 800ms-2s depending on network conditions. My Flutter performance guide talks about keeping UI interactions under 100ms. Waiting two seconds for a "suggested reply" feature felt broken to users — they'd already typed their response before the suggestions appeared.

On-device inference brought that down to 150-300ms depending on the model. Not instant, but fast enough that suggestions appeared while users were still thinking about what to type.

Setting Up TensorFlow Lite in Flutter

For classification tasks, object detection, and any ML workload that doesn't involve text generation, tflite_flutter is the tool I reach for first. It wraps the native TFLite runtime for both iOS and Android, and the API is surprisingly clean for what's essentially a bridge to C++.

Adding tflite_flutter to Your Project

dependencies:
 flutter:
 sdk: flutter
 tflite_flutter: ^0.11.0
 tflite_flutter_helper: ^0.4.0 # optional, but useful for image preprocessing

You'll also need the native TFLite libraries. On Android, tflite_flutter pulls them automatically through Gradle. On iOS, you need to add the TensorFlowLiteC pod — the package README walks you through it, but here's the short version:

# ios/Podfile — add this inside your target block
pod 'TensorFlowLiteC', '~> 2.14.0'

Loading and Running a Model

Here's how I load and run inference on a text classification model. This example classifies user messages into categories (question, complaint, feedback) for routing in a support app:

import 'package:tflite_flutter/tflite_flutter.dart';

class TextClassifier {
 late Interpreter _interpreter;
 bool _isReady = false;

 Future<void> loadModel() async {
 try {
 _interpreter = await Interpreter.fromAsset(
 'models/text_classifier.tflite',
 options: InterpreterOptions()..threads = 4,
 );
 _isReady = true;
 } catch (e) {
 // Model failed to load — fall back to cloud API
 _isReady = false;
 }
 }

 List<double> classify(List<int> tokenizedInput) {
 if (!_isReady) throw Exception('Model not loaded');

 var input = [tokenizedInput];
 var output = List.filled(1 * 3, 0.0).reshape([1, 3]);

 _interpreter.run(input, output);
 return output[0]; // [questionProb, complaintProb, feedbackProb]
 }

 void dispose() => _interpreter.close();
}

The InterpreterOptions()..threads = 4 part matters more than you'd think. On a Pixel 7, going from 1 thread to 4 threads cut inference time from 85ms to 23ms for my text classifier. But going above 4 threads actually made it slower — the overhead of thread coordination wasn't worth it for a small model. I wrote about similar performance optimization patterns in a separate guide.

Running LLMs On-Device with llama.cpp

TFLite is great for classification, but when I needed actual text generation — smart replies, chat features, text summarization — I needed a different approach entirely. TFLite doesn't support autoregressive transformer models out of the box. That's where llama.cpp comes in.

llama.cpp is a C/C++ library that runs quantized LLMs with minimal dependencies. No Python, no PyTorch, no CUDA required. Just pure C++ that compiles everywhere — including ARM processors in phones. It's become the backbone of nearly every "run AI locally" project, and for good reason.

What GGUF Models Are (and Why They Matter)

When I first tried running a language model on a phone, I grabbed a 14GB Llama 2 model and wondered why it crashed immediately. Obviously — 14GB doesn't fit in 6GB of RAM. The solution is quantized models in GGUF format.

GGUF (GPT-Generated Unified Format) is the file format llama.cpp uses for quantized models. "Quantized" means the model weights are stored in lower precision — instead of 16-bit floats, you use 4-bit or 8-bit integers. A 7B-parameter model that's 14GB in full precision shrinks to roughly 4GB at INT4 quantization. That's the difference between "crashes on launch" and "runs on a midrange phone."

You can grab pre-quantized GGUF models from HuggingFace. I typically use models from TheBloke's collection — they're well-tested and come in multiple quantization levels.

Bridging llama.cpp to Dart with FFI

There's no official Flutter package for llama.cpp (as of March 2026). You need to bridge it yourself using Dart FFI. This took me a solid weekend to get right the first time, but the pattern is reusable across projects.

First, compile llama.cpp as a shared library for each platform. Here's the Android NDK approach:

# Clone and build llama.cpp for Android
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build-android && cd build-android

cmake .. \
 -DCMAKE_TOOLCHAIN_FILE=$NDK_HOME/build/cmake/android.toolchain.cmake \
 -DANDROID_ABI=arm64-v8a \
 -DANDROID_PLATFORM=android-24 \
 -DBUILD_SHARED_LIBS=ON

make -j$(nproc)

Then create your Dart FFI bindings:

import 'dart:ffi';
import 'dart:io';

typedef LlamaInitNative = Pointer<Void> Function(Pointer<Utf8> modelPath);
typedef LlamaInit = Pointer<Void> Function(Pointer<Utf8> modelPath);

typedef LlamaGenerateNative = Pointer<Utf8> Function(
 Pointer<Void> context,
 Pointer<Utf8> prompt,
 Int32 maxTokens,
);
typedef LlamaGenerate = Pointer<Utf8> Function(
 Pointer<Void> context,
 Pointer<Utf8> prompt,
 int maxTokens,
);

class LlamaBridge {
 late DynamicLibrary _lib;
 late LlamaInit _init;
 late LlamaGenerate _generate;
 Pointer<Void>? _context;

 LlamaBridge() {
 _lib = Platform.isAndroid
 ? DynamicLibrary.open('libllama.so')
 : DynamicLibrary.open('llama.framework/llama');

 _init = _lib.lookupFunction<LlamaInitNative, LlamaInit>('llama_init');
 _generate = _lib.lookupFunction<LlamaGenerateNative, LlamaGenerate>(
 'llama_generate',
 );
 }

 bool loadModel(String modelPath) {
 final pathPtr = modelPath.toNativeUtf8();
 _context = _init(pathPtr);
 calloc.free(pathPtr);
 return _context != null && _context != nullptr;
 }

 String generate(String prompt, {int maxTokens = 128}) {
 if (_context == null) throw Exception('Model not loaded');
 final promptPtr = prompt.toNativeUtf8();
 final result = _generate(_context!, promptPtr, maxTokens);
 calloc.free(promptPtr);
 return result.toDartString();
 }
}

Platform Channels as an Alternative

If wrestling with FFI bindings sounds painful (it is, honestly), platform channels offer a friendlier path. You write the native inference code in Swift/Kotlin, then call it from Dart through a message-passing channel.

// Dart side — calling native inference
class NativeLlmChannel {
 static const _channel = MethodChannel('com.flutterstudio/llm');

 static Future<String> generate(String prompt) async {
 final result = await _channel.invokeMethod<String>('generate', {
 'prompt': prompt,
 'maxTokens': 128,
 'temperature': 0.7,
 });
 return result ?? '';
 }

 static Future<bool> loadModel(String modelPath) async {
 return await _channel.invokeMethod<bool>('loadModel', {
 'path': modelPath,
 }) ?? false;
 }
}

The downside? You're writing the same logic twice — once in Swift for iOS, once in Kotlin for Android. For my banking app, we went with FFI because we wanted a single codebase. For a simpler project, platform channels are perfectly fine.

You could also look at flutter_rust_bridge if your team knows Rust. It generates type-safe Dart bindings from Rust code automatically, and Rust's memory safety guarantees eliminate the pointer bugs that plague raw FFI.

Offloading Inference to Dart Isolates

Here's a mistake I made on my first on-device AI app: I ran inference on the main isolate. The model loaded fine, generated text correctly, but the UI froze for 2-3 seconds during each inference call. Buttons stopped responding, scroll stalled, the whole app felt dead.

The fix is Dart isolates. Isolates are Dart's version of threads — they run in separate memory spaces and don't block the UI. For heavy compute like ML inference, they're non-negotiable.

import 'dart:isolate';

class IsolatedInference {
 static Future<String> generate(String modelPath, String prompt) async {
 return await Isolate.run(() {
 // This runs on a separate isolate — UI stays responsive
 final bridge = LlamaBridge();
 bridge.loadModel(modelPath);
 final result = bridge.generate(prompt, maxTokens: 128);
 return result;
 });
 }
}

// Usage in a widget
Future<void> onSendMessage(String userMessage) async {
 setState(() => _isGenerating = true);

 final reply = await IsolatedInference.generate(
 _modelPath,
 'Suggest a brief reply to: $userMessage',
 );

 setState(() {
 _suggestions = _parseSuggestions(reply);
 _isGenerating = false;
 });
}

For simpler cases, Flutter's compute() function works too — it's a wrapper around Isolate.run(). But for anything where you need to keep the model warm between requests, set up a persistent isolate. The state management patterns from my state management comparison guide apply here — you need a clean way to communicate between isolates.

Model Quantization — Making 7B Parameters Fit on a Phone

Quantization is the single most important concept for on-device AI. Without it, running language models on mobile hardware isn't realistic. A 7B parameter model in FP16 (half precision) takes up ~14GB. Phones don't have that kind of RAM to spare.

INT4 vs INT8 Tradeoffs

There are two quantization levels I've used in production:

• INT8 (Q8_0) — Each weight stored as an 8-bit integer. A 7B model shrinks to ~7GB. Quality is very close to the original. Good for devices with 8GB+ RAM.
• INT4 (Q4_K_M) — Each weight stored as a 4-bit integer with mixed precision. A 7B model shrinks to ~4GB. Some quality loss on complex reasoning, but perfectly fine for short-form text generation. This is what I use for mobile.

The naming convention in GGUF files looks like model-7b-q4_k_m.gguf. The q4_k_m means 4-bit quantization with K-quant method, medium quality. There's also q4_k_s (small/faster) and q5_k_m (5-bit, better quality but larger).

Real Numbers from My Testing

I tested a Llama 3.2 3B model (Q4_K_M quantization) across several devices last month. Here's what I measured — these are real numbers, not marketing claims:

• Pixel 8 Pro (12GB RAM, Tensor G3) — 11.2 tokens/sec, 280MB memory usage, model loads in 1.8s
• Samsung Galaxy A54 (6GB RAM, Exynos 1380) — 6.4 tokens/sec, 275MB memory, model loads in 3.1s
• iPhone 15 Pro (8GB RAM, A17 Pro) — 14.8 tokens/sec, 260MB memory, model loads in 1.2s
• Redmi Note 12 (4GB RAM, Snapdragon 4 Gen 1) — Failed to load. Not enough free RAM after the OS and app take their share.

A 3B parameter model at Q4_K_M generates decent smart replies and short summaries. It won't write essays or handle multi-step reasoning, but for the feature I built — suggesting three short replies to a message — it's more than adequate. The quality difference between a 3B local model and GPT-3.5 for short replies? Honest answer: users couldn't tell the difference in our A/B test.

Hardware Acceleration on iOS and Android

Running models on the CPU works, but both major mobile platforms offer hardware acceleration that can double or triple your inference speed.

On iOS, CoreML and Metal give you access to the Neural Engine and GPU. Apple's A-series and M-series chips have dedicated ML accelerators that demolish raw CPU performance. llama.cpp supports Metal out of the box — you just enable it at compile time with -DGGML_METAL=ON. That's why the iPhone 15 Pro numbers above are so good.

On Android, NNAPI (Neural Networks API) provides access to dedicated AI accelerators like Qualcomm's Hexagon DSP or Samsung's NPU. TFLite supports NNAPI through a delegate:

final interpreter = await Interpreter.fromAsset(
 'models/text_classifier.tflite',
 options: InterpreterOptions()
 ..addDelegate(NnApiDelegate()) // Use hardware accelerator
 ..threads = 2,
);

A word of caution though — NNAPI support varies wildly across Android devices. Some manufacturers implement it well, others have buggy or partial implementations. I always wrap NNAPI usage in a try/catch and fall back to CPU if it fails. The security implications of running models locally are generally positive (data stays on device), but you need to handle edge cases gracefully.

Cloud API vs On-Device — When to Pick Which

After shipping both cloud-based and on-device AI features, here's the decision framework I use now:

Go on-device when:

• The task is simple — classification, sentiment, short text generation
• Offline support matters (think travel apps, field work tools)
• Response time needs to be under 500ms
• User data is sensitive (medical, financial, personal messages)
• You want to avoid per-request API costs at scale

Go cloud when:

• You need complex reasoning (multi-step, long-context)
• The model needs to be larger than 4-5GB
• You want the latest model capabilities without app updates
• Your user base is primarily on budget devices

Building an Offline Chat Feature (Real Project Walkthrough)

Let me walk through the actual feature I built — the one that started all of this. A messaging app where users can get AI-suggested replies even when they're offline. I'll skip the full UI code (that's a separate post in itself) and focus on the AI plumbing.

Model Loading and Warm-Up

The model loads when the user opens the chat screen. I do a warm-up inference with a dummy prompt to pre-fill internal caches — the first real inference is then much faster:

class ChatAiService {
 late final SendPort _workerPort;
 bool _isReady = false;

 Future<void> initialize(String modelPath) async {
 final receivePort = ReceivePort();

 await Isolate.spawn(
 _workerEntryPoint,
 _WorkerConfig(sendPort: receivePort.sendPort, modelPath: modelPath),
 );

 // Get the worker's send port for future communication
 _workerPort = await receivePort.first as SendPort;

 // Warm-up inference — primes internal caches
 await _sendCommand('generate', 'Hello');
 _isReady = true;
 }

 Future<List<String>> suggestReplies(String lastMessage) async {
 if (!_isReady) return [];

 final prompt = '''Suggest 3 short, natural replies to this message.
Each reply should be under 15 words. Return only the replies, one per line.

Message: $lastMessage''';

 final result = await _sendCommand('generate', prompt);
 return result.split('\n').where((line) => line.trim().isNotEmpty).toList();
 }
}

Conversation Manager

The conversation manager tracks message history and formats prompts. I keep a sliding window of the last 10 messages for context — more than that and the prompt gets too long for a 3B model to handle well:

class ConversationManager {
 final List<ChatMessage> _history = [];
 final int _maxContext = 10;

 void addMessage(ChatMessage message) {
 _history.add(message);
 if (_history.length > _maxContext) {
 _history.removeAt(0);
 }
 }

 String buildPrompt() {
 final buffer = StringBuffer();
 for (final msg in _history) {
 final role = msg.isUser ? 'User' : 'Assistant';
 buffer.writeln('$role: ${msg.text}');
 }
 return buffer.toString();
 }
}

Wiring It Into the UI

The UI shows suggestion chips above the keyboard. When the user taps one, it sends that reply. I use a ValueNotifier to keep things simple — no need for Riverpod or Bloc for a feature this contained (though if you're curious about the tradeoffs, check my Bloc vs Riverpod comparison):

class SmartReplyWidget extends StatelessWidget {
 final ValueNotifier<List<String>> suggestions;
 final Function(String) onTap;

 const SmartReplyWidget({
 required this.suggestions,
 required this.onTap,
 super.key,
 });

 @override
 Widget build(BuildContext context) {
 return ValueListenableBuilder<List<String>>(
 valueListenable: suggestions,
 builder: (context, replies, _) {
 if (replies.isEmpty) return const SizedBox.shrink();
 return Wrap(
 spacing: 8,
 children: replies.map((reply) => ActionChip(
 label: Text(reply),
 onPressed: () => onTap(reply),
 )).toList(),
 );
 },
 );
 }
}

The whole flow: user receives a message → ConversationManager updates history → ChatAiService.suggestReplies() runs inference on the worker isolate → UI shows three suggestion chips → user taps one → message sends. Total time from message received to suggestions displayed: 250-400ms on a Pixel 8.

I added push notifications that pre-warm the AI model when a new message arrives, so by the time the user opens the chat, the model is already loaded and ready. That shaved another 1.5 seconds off the perceived latency.

What I'd Do Differently Next Time

After three production apps with on-device AI, here's what I've learned:

Start with the smallest model that works. I wasted two weeks trying to get a 7B model running on Android before realizing a 3B model produced nearly identical results for my use case. Always benchmark the smallest option first.

Test on actual budget devices. Every tutorial shows benchmarks on flagship phones. In Pakistan, most of my users have phones with 4-6GB RAM. Buy a few budget phones for testing — it'll save you from a rude awakening at launch. My testing strategy guide covers device matrix planning in more detail.

Ship the model separately from the app. Baking a 300MB model into your APK means a 300MB download. Instead, download the model on first launch and store it in the app's documents directory. Users on slow connections will thank you.

Always have a fallback. Not every device can run the model. Not every model load succeeds. Design your AI features as progressive enhancements — the app should work without them. Check my clean architecture guide for patterns that make fallback logic clean and testable.

On-device AI in Flutter isn't just a cool tech demo anymore — it's a practical approach that solves real problems around latency, cost, and offline functionality. The tooling isn't as polished as cloud APIs yet (no nice SDK, more manual setup), but the results are worth the effort. If you're building a Flutter app with AI features, at least evaluate whether some of those features can run locally. You might be surprised how much you can do without ever hitting the network.

Curious about other ways to push your Flutter apps further? My guides on essential Flutter packages and Supabase vs Firebase cover more of the tools I use daily for production apps.