Flutter – On-Device Machine Learning using ML Kit
Machine Learning is being used widely in projects today. It requires knowledge of Machine Learning to create models, for application developers who want to use Machine Learning in the project but don’t know how to create a model. In Flutter, certain developments have been done by Google to help developers create the application they want. Although there is a package google_ml_kit however it is only available for android yet. In this article, we will use ML Kit.
ML Kit:
ML Kit is created by Google for developers to build mobile applications that involve machine learning easily. It can be used for text recognition, face and pose detection, barcode scanning, etc. We are going to create an application that detects items in the Image and labels them.
Firstly, let us see the application we are going to create in this tutorial –
Features:
- Capture the Image
- Preprocess Image
- Identify items with label, index, and confidence.
Install the dependency:
Include the google_ml_kit package in pubspec.yaml file of the app, in order to use ML Kit features by Google.
Dart
google_ml_kit: ^0.3.0 |
To use the camera feature we first need to create a screen that will capture the image. To include the camera feature we need to use the camera library of Flutter.
Dart
camera: ^0.8.1 |
Configure both libraries by running pub get.
Create camera_screen.dart:
Import the camera library in camera_screen.dart, and initialize the CameraController. We are also finding the number of cameras available and selecting the first camera from the option to take pictures.
Dart
late final CameraController _controller; void _initializeCamera() async { final CameraController cameraController = CameraController( cameras[0], ResolutionPreset.high, ); _controller = cameraController; _controller.initialize().then((_) { if (!mounted) { return ; } setState(() {}); }); } |
Call the method inside initState(), to start the camera when the app starts:
Dart
@override void initState() { _initializeCamera(); super.initState(); } |
Before moving to the next screen we need to dispose of the _controller to avoid any memory leak:
Dart
@override void dispose() { _controller.dispose(); super.dispose(); } |
Now, to take pictures, we are creating another function _takePicture(), when the button will be clicked to capture an image this function will be invoked. It will check whether the CameraController is initialized or not. It will return the ImagePath where the image will be stored in the device after capturing. If any error occurs, it will throw an exception. After capturing the image, it will redirect us to another screen, detail_screen.dart. So, create another file detail_screen.dart to show the results.
Dart
Future<String?> _takePicture() async { if (!_controller.value.isInitialized) { print( "Controller is not initialized" ); return null; } String? imagePath; if (_controller.value.isTakingPicture) { print( "Processing is progress ..." ); return null; } try { // Turning off the camera flash _controller.setFlashMode(FlashMode.off); // Returns the image in cross-platform file abstraction final XFile file = await _controller.takePicture(); // Retrieving the path imagePath = file.path; } on CameraException catch (e) { print( "Camera Exception: $e" ); return null; } return imagePath; } |
Full Code for camera_screen.dart
Dart
import 'package:camera/camera.dart' ; import 'package:flutter/material.dart' ; import 'package:flutter_mlkit_vision/main.dart' ; import 'detail_screen.dart' ; class CameraScreen extends StatefulWidget { @override _CameraScreenState createState() => _CameraScreenState(); } class _CameraScreenState extends State<CameraScreen> { late final CameraController _controller; // Initializes camera controller to preview on screen void _initializeCamera() async { final CameraController cameraController = CameraController( cameras[0], ResolutionPreset.high, ); _controller = cameraController; _controller.initialize().then((_) { if (!mounted) { return ; } setState(() {}); }); } // Takes picture with the selected device camera, and // returns the image path Future<String?> _takePicture() async { if (!_controller.value.isInitialized) { print( "Controller is not initialized" ); return null; } String? imagePath; if (_controller.value.isTakingPicture) { print( "Processing is progress ..." ); return null; } try { // Turning off the camera flash _controller.setFlashMode(FlashMode.off); // Returns the image in cross-platform file abstraction final XFile file = await _controller.takePicture(); // Retrieving the path imagePath = file.path; } on CameraException catch (e) { print( "Camera Exception: $e" ); return null; } return imagePath; } @override void initState() { _initializeCamera(); super.initState(); } @override void dispose() { // dispose the camera controller when navigated // to a different page _controller.dispose(); super.dispose(); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text( 'w3wiki' ), ), body: _controller.value.isInitialized ? Stack( children: <Widget>[ CameraPreview(_controller), Padding( padding: const EdgeInsets.all(20.0), child: Container( alignment: Alignment.bottomCenter, child: ElevatedButton.icon( icon: Icon(Icons.camera), label: Text( "Click" ), onPressed: () async { // If the returned path is not null navigate // to the DetailScreen await _takePicture().then((String? path) { if (path != null) { Navigator.push( context, MaterialPageRoute( builder: (context) => DetailScreen( imagePath: path, ), ), ); } else { print( 'Image path not found!' ); } }); }, ), ), ) ], ) : Container( color: Colors.black, child: Center( child: CircularProgressIndicator(), ), ), ); } } |
Work on detail_screen.dart:
First, we need to process the image, get imageSize to be displayed on the screen.
Dart
Future< void > _getImageSize(File imageFile) async { final Completer<Size> completer = Completer<Size>(); final Image image = Image.file(imageFile); image.image.resolve( const ImageConfiguration()).addListener( ImageStreamListener((ImageInfo info, bool _) { completer.complete(Size( info.image.width.toDouble(), info.image.height.toDouble(), )); }), ); final Size imageSize = await completer.future; setState(() { _imageSize = imageSize; }); } |
Initialize imageLabeler from GoogleMLKit in the class.
Dart
final imageLabeler = GoogleMlKit.vision.imageLabeler(); |
Now, we will create another function _recognizeImage(), first, we will get image size then get the image file. We will store all the image labels in list labels of type List<ImageLabel>. Then, through for loop, we will retrieve label, index, and confidence for each Image label and store them in their respective lists – imagesData, indexData, and confidenceData.
Dart
void _recognizeImage() async { _getImageSize(File(_imagePath)); final inputImage = InputImage.fromFilePath(_imagePath); final List<ImageLabel> labels = await imageLabeler.processImage(inputImage); for (ImageLabel label in labels) { // retrieve label,index, and confidence from each label final String item = label.label; final int index = label.index; final double confidence = label.confidence; imagesData.add(item); indexData.add(index.toString()); confidenceData.add(confidence.toString()); } } |
Now, it’s time to display results on the screen.
Full Code for detail_screen.dart:
Dart
import 'dart:async' ; import 'dart:io' ; import 'package:flutter/material.dart' ; import 'package:google_ml_kit/google_ml_kit.dart' ; class DetailScreen extends StatefulWidget { final String imagePath; const DetailScreen({required this .imagePath}); @override _DetailScreenState createState() => _DetailScreenState(); } class _DetailScreenState extends State<DetailScreen> { late final String _imagePath; final imageLabeler = GoogleMlKit.vision.imageLabeler(); Size? _imageSize; List<String> imagesData = []; List<String> indexData = []; List<String> confidenceData = []; // Fetching the image size from the image file Future< void > _getImageSize(File imageFile) async { final Completer<Size> completer = Completer<Size>(); final Image image = Image.file(imageFile); image.image.resolve( const ImageConfiguration()).addListener( ImageStreamListener((ImageInfo info, bool _) { completer.complete(Size( info.image.width.toDouble(), info.image.height.toDouble(), )); }), ); final Size imageSize = await completer.future; setState(() { _imageSize = imageSize; }); } void _recognizeImage() async { _getImageSize(File(_imagePath)); final inputImage = InputImage.fromFilePath(_imagePath); final List<ImageLabel> labels = await imageLabeler.processImage(inputImage); for (ImageLabel label in labels) { final String item = label.label; final int index = label.index; final double confidence = label.confidence; imagesData.add(item); indexData.add(index.toString()); confidenceData.add(confidence.toString()); } } @override void initState() { _imagePath = widget.imagePath; // Initializing the Image Labeler final imageLabeler = GoogleMlKit.vision.imageLabeler(); _recognizeImage(); super.initState(); } @override void dispose() { // Disposing the imageLabeler when not used anymore imageLabeler.close(); super.dispose(); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text( "Image Details" ), ), body: _imageSize != null ? Stack( children: [ Container( width: double .maxFinite, color: Colors.black, child: AspectRatio( aspectRatio: _imageSize!.aspectRatio, child: Image.file( File(_imagePath), ), ), ), Align( alignment: Alignment.bottomCenter, child: Card( elevation: 8, color: Colors.white, child: Padding( padding: const EdgeInsets.all(16.0), child: Column( mainAxisSize: MainAxisSize.min, crossAxisAlignment: CrossAxisAlignment.start, children: <Widget>[ Padding( padding: const EdgeInsets.only(bottom: 8.0), child: Text( "IdentifiedItems Index Confidence" , style: TextStyle( fontSize: 20, fontWeight: FontWeight.bold, ), ), ), Container( height: 120, child: SingleChildScrollView( child: imagesData != null ? ListView.builder( shrinkWrap: true , physics: BouncingScrollPhysics(), itemCount: imagesData.length, itemBuilder: (context, index) { return Row( mainAxisAlignment: MainAxisAlignment.spaceBetween, children: [ Text(imagesData[index]), Text(indexData[index]), Text(confidenceData[index]) ], ); }) : Container(), ), ), ], ), ), ), ), ], ) : Container( color: Colors.blue, child: Center( child: CircularProgressIndicator(), ), ), ); } } |
Now, call CameraScreen in main.dart.
Dart
import 'package:camera/camera.dart' ; import 'package:flutter/material.dart' ; import 'camera_screen.dart' ; // Global variable for storing the list of cameras available List<CameraDescription> cameras = []; Future< void > main() async { // Fetch the available cameras before initializing the app. try { WidgetsFlutterBinding.ensureInitialized(); cameras = await availableCameras(); } on CameraException catch (e) { debugPrint( 'CameraError: ${e.description}' ); } runApp(MyApp()); } class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( debugShowCheckedModeBanner: false , title: 'Flutter MLKit Vision' , theme: ThemeData( primarySwatch: Colors.green, ), home: CameraScreen(), ); } } |
Output: