Android – Detect Open or Closed Eyes Using ML Kit and CameraX
Google’s ML kit is one of the best-trained models for face detection and its characteristics. Integrating with your own CameraX library can be quite a challenging task. so we are going to build an Android app that will detect whether a person’s eyes are open or closed in real time. This process going to be long so without delay let’s deep dive into the project. A sample video is given below to get an idea about what we are going to do in this article.
Note: Before starting the project please read about how the ML Kit detects faces and how cameraX works.
Project Setup
- Start a project with an empty activity and name your project whatever you want we are naming it EyeDetection.
- Language using Kotlin
- Minimum SDK set to Android 7.0(Nougat)
Step by Step Implementation
Adding Dependencies and Permissions
Open project-level build.gradle file, make sure to include Google’s Maven repository in both your buildscript and all projects sections. Add the dependencies for the ML Kit Android libraries to the module’s app-level gradle file, which is usually app/build.gradle.
dependencies {
// This dependency will dynamically download the model in Google Play Services
implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.1.0'
// camera dependencies
def camerax_version = "1.2.2"
implementation "androidx.camera:camera-core:${camerax_version}"
implementation "androidx.camera:camera-camera2:${camerax_version}"
implementation "androidx.camera:camera-lifecycle:${camerax_version}"
implementation "androidx.camera:camera-view:${camerax_version}"
}
Add this code in your manifest file
<uses-feature android:name="android.hardware.camera.any" />
<uses-permission android:name="android.permission.CAMERA" />
Add the following declaration to your app’s AndroidManifest.xml file. This will automatically download the model to the device if your app is installed from the Play Store.
<application ...>
...
<meta-data
android:name="com.google.mlkit.vision.DEPENDENCIES"
android:value="face" >
<!-- To use multiple models: android:value="face,model2,model3" -->
</application>
We are going to use viewbinding so don’t forget to enable it
buildFeatures {
viewBinding true
}
Configure the Layout
Open the activity_main layout file at res/layout/activity_main.xml, and replace it with the following code.
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<FrameLayout
android:id="@+id/frameLayout"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toTopOf="parent">
<androidx.camera.view.PreviewView
android:id="@+id/previewView_finder"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:scaleType="fillCenter">
</androidx.camera.view.PreviewView>
<com.example.eyedetection.GraphicOverlay
android:id="@+id/graphicOverlay_finder"
android:layout_width="match_parent"
android:layout_height="match_parent" />
</FrameLayout>
<TextView
android:id="@+id/tvWarningText"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_marginBottom="150dp"
android:background="@color/white"
android:focusableInTouchMode="false"
android:gravity="center"
android:padding="20dp"
android:text="No Face detected"
android:textColor="@android:color/holo_red_dark"
android:textSize="18sp"
android:visibility="visible"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>
This code includes the PreviewView for preview of cameraX that will let the user to preview the photo they will be taking. A textview for the indication of face detection and about eyes, whether they are open or closed and GraphicOverlay to draw the box on detected faces.
Note: This GrapichOverlay view is a custom view, that you have to create a first.
Making Classes
Create a GraphicOverlay class and make It open. We have to define some methods and logics to draw over the screen. Below is the code.
import android.content.Context
import android.content.res.Configuration
import android.graphics.Canvas
import android.graphics.Rect
import android.graphics.RectF
import android.util.AttributeSet
import android.view.View
import kotlin.math.ceil
open class GraphicOverlay(context: Context?, attrs: AttributeSet?) :
View(context, attrs) {
private val lock = Any()
private val faceBoxes: MutableList<FaceBox> = ArrayList()
var mScale: Float? = null
var mOffsetX: Float? = null
var mOffsetY: Float? = null
abstract class FaceBox(private val overlay: GraphicOverlay) {
abstract fun draw(canvas: Canvas?)
fun calculateRect(height: Float, width: Float, boundingBoxT: Rect): RectF {
// for land scape
fun isLandScapeMode(): Boolean {
return overlay.context.resources.configuration.orientation == Configuration.ORIENTATION_LANDSCAPE
}
fun whenLandScapeModeWidth(): Float {
return when(isLandScapeMode()) {
true -> width
false -> height
}
}
fun whenLandScapeModeHeight(): Float {
return when(isLandScapeMode()) {
true -> height
false -> width
}
}
val scaleX = overlay.width.toFloat() / whenLandScapeModeWidth()
val scaleY = overlay.height.toFloat() / whenLandScapeModeHeight()
val scale = scaleX.coerceAtLeast(scaleY)
overlay.mScale = scale
// Calculate offset (we need to center the overlay on the target)
val offsetX = (overlay.width.toFloat() - ceil(whenLandScapeModeWidth() * scale)) / 2.0f
val offsetY = (overlay.height.toFloat() - ceil(whenLandScapeModeHeight() * scale)) / 2.0f
overlay.mOffsetX = offsetX
overlay.mOffsetY = offsetY
val mappedBox = RectF().apply {
left = boundingBoxT.right * scale + offsetX
top = boundingBoxT.top * scale + offsetY
right = boundingBoxT.left * scale + offsetX
bottom = boundingBoxT.bottom * scale + offsetY
}
return mappedBox
}
}
fun clear() {
synchronized(lock) { faceBoxes.clear() }
postInvalidate()
}
fun add(faceBox: FaceBox) {
synchronized(lock) { faceBoxes.add(faceBox) }
}
override fun onDraw(canvas: Canvas) {
super.onDraw(canvas)
synchronized(lock) {
for (graphic in faceBoxes) {
graphic.draw(canvas)
}
}
}
}
To handle the camera we will create a cameraManager class to start the camera. Here we have not implemented the Picture taking ability as we are only doing real time detection so only preview will enough for us. CameraManager class takes five parameters context, previewView for the Preview, lifecycleOwner for the life cycle of camera. Listener to get the Status(which is an another class to track the status and change the textview). And graphicOverlay. For the analyzer we are using custom analyzer for face detection and draw boxes. Here is the code of cameraManager class.
import android.content.Context
import android.util.Log
import androidx.camera.core.Camera
import androidx.camera.core.CameraSelector
import androidx.camera.core.ImageAnalysis
import androidx.camera.core.ImageCapture
import androidx.camera.core.Preview
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.camera.view.PreviewView
import androidx.core.content.ContextCompat
import androidx.lifecycle.LifecycleOwner
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors
class CameraManager(
private val context: Context,
private val previewView: PreviewView,
private val lifecycleOwner: LifecycleOwner,
private val graphicOverlay: GraphicOverlay,
private val listener: ((Status) -> Unit),
) {
private var preview: Preview? = null
private var imageCapture: ImageCapture? = null
private var camera: Camera? = null
private lateinit var cameraExecutor: ExecutorService
private var cameraSelectorOption = CameraSelector.LENS_FACING_BACK
private var cameraProvider: ProcessCameraProvider? = null
private var imageAnalyzer: ImageAnalysis? = null
init {
createNewExecutor()
}
private fun createNewExecutor() {
cameraExecutor = Executors.newSingleThreadExecutor()
}
fun startCamera() {
val cameraProviderFuture = ProcessCameraProvider.getInstance(context)
cameraProviderFuture.addListener(
{
cameraProvider = cameraProviderFuture.get()
preview = Preview.Builder().build()
imageCapture = ImageCapture.Builder().build()
imageAnalyzer = ImageAnalysis.Builder()
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST).build().also {
it.setAnalyzer(cameraExecutor, selectAnalyzer())
}
val cameraSelector =
CameraSelector.Builder().requireLensFacing(cameraSelectorOption).build()
setCameraConfig(cameraProvider, cameraSelector)
}, ContextCompat.getMainExecutor(context)
)
}
// Custom analyzer
private fun selectAnalyzer(): ImageAnalysis.Analyzer {
return FaceDetection(graphicOverlay, listener)
}
private fun setCameraConfig(
cameraProvider: ProcessCameraProvider?, cameraSelector: CameraSelector
) {
try {
cameraProvider?.unbindAll()
camera = cameraProvider?.bindToLifecycle(
lifecycleOwner, cameraSelector, preview, imageCapture, imageAnalyzer
)
preview?.setSurfaceProvider(
previewView.surfaceProvider
)
} catch (e: Exception) {
Log.e("Error", "Use case binding failed", e)
}
}
}
This is the Status class to get the indication call back directly in our activity. that will be helpful to get the status and change the textview according to analyzer.
enum class Status { NO_FACE, MULTIPLE_FACES, LEFT_EYE_CLOSED,
RIGHT_EYE_CLOSED, BOTH_EYES_CLOSED,VALID_FACE
}
This is the face detection class that will be used by our custom analyzer. Which is inheriting the Analyzer class. And overriding its methods. The code doesn’t draw the box if multiple faces detected but will notifiy the main activity using the listener. And if it is a a single face , we will pass its properties to FaceBox class to draw the box and for eye detection.
import android.graphics.Rect
import android.util.Log
import androidx.camera.core.ImageProxy
import com.google.android.gms.tasks.Task
import com.google.mlkit.vision.common.InputImage
import com.google.mlkit.vision.face.Face
import com.google.mlkit.vision.face.FaceDetection
import com.google.mlkit.vision.face.FaceDetectorOptions
import java.io.IOException
class FaceDetection(
private val graphicOverlayView: GraphicOverlay,
private val listener: (Status) -> Unit
) : Analyzer<List<Face>>() {
private val realTimeOpts =
FaceDetectorOptions.Builder().setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
.setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
.setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL).build()
private val detector = FaceDetection.getClient(realTimeOpts)
override val graphicOverlay: GraphicOverlay
get() = graphicOverlayView
override fun detectInImage(image: InputImage): Task<List<Face>> {
return detector.process(image)
}
override fun stop() {
try {
detector.close()
} catch (e: IOException) {
Log.e("Error", "Exception thrown while trying to close Face Detector: $e")
}
}
override fun onSuccess(
results: List<Face>, graphicOverlay: GraphicOverlay, rect: Rect, imageProxy: ImageProxy
) {
graphicOverlay.clear()
// If multiple faces then don't draw
if (results.isNotEmpty()) {
if (results.size > 1) {
listener(Status.MULTIPLE_FACES)
} else {
for (face in results) {
val faceGraphic =
FaceBox(graphicOverlay, face, rect, listener)
graphicOverlay.add(faceGraphic)
}
}
graphicOverlay.postInvalidate()
} else {
listener(Status.NO_FACE)
Log.e("Error", "Face Detector failed.")
}
}
override fun onFailure(e: Exception) {
Log.e("Error", "Face Detector failed. $e")
listener(Status.NO_FACE)
}
}
The faceBox class inherting the Graphic overlay class. This class will calculate the probability of eyes of this face and also put the green color box to the detected face. We have set the probability to 0.6. if any eye probability is less than or equal to 0.6 we will consider that eye as closed eye and set status accordingly. Here is code of FaceBox class.
import android.graphics.Canvas
import android.graphics.Color
import android.graphics.Paint
import android.graphics.Rect
import com.google.mlkit.vision.face.Face
class FaceBox(
overlay: GraphicOverlay,
private val face: Face,
private val imageRect: Rect,
private val listener: (Status) -> Unit
) : GraphicOverlay.FaceBox(overlay) {
private val facePositionPaint: Paint
private val idPaint: Paint
private val boxPaint: Paint
init {
val selectedColor = Color.WHITE
facePositionPaint = Paint()
facePositionPaint.color = selectedColor
idPaint = Paint()
idPaint.color = selectedColor
boxPaint = Paint()
boxPaint.color = selectedColor
boxPaint.style = Paint.Style.STROKE
boxPaint.strokeWidth = 5.0f
}
private val greenBoxPaint = Paint().apply {
color = Color.GREEN
style = Paint.Style.STROKE
strokeWidth = 5.0f
}
override fun draw(canvas: Canvas?) {
val rect = calculateRect(
imageRect.height().toFloat(), imageRect.width().toFloat(), face.boundingBox
)
val leftEyeProbability = leftEyeProbability()
val rightEyeProbability = rightEyeProbability()
when {
// both eyes are closed
leftEyeProbability <= 0.6 && rightEyeProbability() <= 0.6 -> {
listener(Status.BOTH_EYES_CLOSED)
}
// left eye is closed
leftEyeProbability <= 0.6 -> {
listener(Status.LEFT_EYE_CLOSED)
}
// right is closed
rightEyeProbability <= 0.6 -> {
listener(Status.RIGHT_EYE_CLOSED)
}
// valid face, set face box color green
else -> {
listener(Status.VALID_FACE)
canvas?.drawRect(rect, greenBoxPaint)
}
}
}
private fun leftEyeProbability(): Float {
var probability = 0.0F
if (face.leftEyeOpenProbability != null) {
val leftEyeOpenProb = face.leftEyeOpenProbability
probability = leftEyeOpenProb!!
}
return probability
}
private fun rightEyeProbability(): Float {
var probability = 0.0F
if (face.rightEyeOpenProbability != null) {
val rightEyeOpenProb = face.rightEyeOpenProbability
probability = rightEyeOpenProb!!
}
return probability
}
}
We are almost ready. Before starting the app we have to add the camera permission and handle it accordingly. And after handling the permission in our activity out code will look like this.
import android.Manifest
import android.content.ActivityNotFoundException
import android.content.Intent
import android.content.pm.PackageManager
import android.net.Uri
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import android.provider.Settings
import android.util.Log
import androidx.activity.result.contract.ActivityResultContracts
import androidx.appcompat.app.AlertDialog
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import com.example.eyedetection.databinding.ActivityMainBinding
class MainActivity : AppCompatActivity() {
private lateinit var binding: ActivityMainBinding
private lateinit var cameraManager: CameraManager
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
binding = ActivityMainBinding.inflate(layoutInflater)
setContentView(binding.root)
createCameraManager()
// to handle the permission
cameraPermission()
}
private fun openCamera() {
// this will start the camera if permission is enabled
cameraManager.startCamera()
}
private fun cameraPermission() {
val cameraPermission = Manifest.permission.CAMERA
if (ContextCompat.checkSelfPermission(
this, cameraPermission
) == PackageManager.PERMISSION_GRANTED
) {
openCamera()
} else if (ActivityCompat.shouldShowRequestPermissionRationale(
this, cameraPermission
)
) {
val title = "Permission Required"
val message = "App needs Camera Permission to detect faces"
val builder: AlertDialog.Builder = AlertDialog.Builder(this)
builder.setTitle(title).setMessage(message).setCancelable(false)
.setPositiveButton("OK") { dialog, _ ->
requestCameraPermissionLauncher.launch(cameraPermission)
dialog.dismiss()
}.setNegativeButton("Cancel") { dialog, _ ->
dialog.dismiss()
}
builder.create().show()
} else {
requestCameraPermissionLauncher.launch(
cameraPermission
)
}
}
private val requestCameraPermissionLauncher = registerForActivityResult(
ActivityResultContracts.RequestPermission()
) { isGranted: Boolean ->
if (isGranted) {
openCamera()
} else if (!ActivityCompat.shouldShowRequestPermissionRationale(
this, Manifest.permission.CAMERA
)
) {
val title = "Permission required"
val message =
"Please allow camera permission to detect faces"
val builder: AlertDialog.Builder = AlertDialog.Builder(this)
builder.setTitle(title).setMessage(message).setCancelable(false)
.setPositiveButton("Change Settings") { _, _ ->
try {
val intent = Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS)
val uri = Uri.fromParts("package", this.packageName, null)
intent.data = uri
startActivity(intent)
} catch (e: ActivityNotFoundException) {
e.printStackTrace()
}
}.setNegativeButton("Cancel") { dialog, _ ->
dialog.dismiss()
}
builder.create().show()
} else {
cameraPermission()
}
}
private fun createCameraManager() {
cameraManager = CameraManager(
this,
binding.previewViewFinder,
this,
binding.graphicOverlayFinder,
::checkStatus
)
}
private fun checkStatus(status: Status) {
Log.e("status","$status")
when (status) {
Status.MULTIPLE_FACES -> {
binding.tvWarningText.text = "Multiple Faces detected"
}
Status.NO_FACE -> {
binding.tvWarningText.text = "No Face detected"
}
Status.LEFT_EYE_CLOSED -> {
binding.tvWarningText.text = "Left eye is closed"
}
Status.RIGHT_EYE_CLOSED -> {
binding.tvWarningText.text ="Right eye is closed"
}
Status.BOTH_EYES_CLOSED->{
binding.tvWarningText.text = "Both Eyes are closed"
}
Status.VALID_FACE ->{
binding.tvWarningText.text ="Correct Face"
}
}
}
}
checkStatus function will listen to the listener and set the text of textview as soon as any face or any changes in face is detected. And now we are finally ready. Build the app and run in your device. Our final result will be look like this.