Extract a person from an image in Google ML Kit with Selfie Segmentation in Kotlin

Question

I would like to remove a background from an image with a person in order to use this image in some other part of my android app. I have applied Google ML Kit - Selfie Segmentation and receive segmentation Mask as a result. What I want to do now is - save the image without a background to a device so I can then use it in other parts of my app.

Original image: Image after Applied Selfie Segmentation:

My problem is I need to somehow apply values from segmentationMask (256 * 256) to the Bitmap where I will remove background and save new Bitmap (with values from segmentation mask on only those pixels that are currently blue) with pixels from segmentation mask. Would anyone be so king and point me in the direction? Should I try to achieve this with Matrix?? Currently I am drawind this image on jetpack compose Canvas with drawPoints comand in DrawScope.

Answer 1

There is an enableRawSizeMask API on the SelfieSegmenterOptions.Builder. If you don't set it, the returned mask should have the same dimension as the input image.

Then a naive approach would be: Iterating through the result mask to determine whether a certain pixel should be kept, if not, call setPixel method on the original Bitmap to set the background pixel to transparent (or whatever color you want).

This is for sure not the most performant way, but maybe a good start.

Answer 2

Maybe you can try it,this is my way

private void initUI(){
    ImageView imageView=findViewById(R.id.iv);
    Paint paint=new Paint();
    paint.setAntiAlias(true);
    SelfieSegmenterOptions options=new SelfieSegmenterOptions.Builder()
                    .setDetectorMode(SelfieSegmenterOptions.STREAM_MODE)
                    //.enableRawSizeMask()
                    .build();
    Segmenter segmenter = Segmentation.getClient(options);
    Bitmap bitmap= BitmapFactory.decodeResource(getResources(), R.drawable.mv1);
    InputImage image = InputImage.fromBitmap(bitmap, 0);
    segmenter.process(image).addOnSuccessListener(segmentationMask -> {
        ByteBuffer mask = segmentationMask.getBuffer();
        int maskWidth = segmentationMask.getWidth();
        int maskHeight = segmentationMask.getHeight();
        Bitmap background=Bitmap.createBitmap(maskColorsFromByteBuffer(mask,maskWidth,maskHeight),maskWidth,maskHeight, Bitmap.Config.ARGB_8888);
        //创建一个bitmap
        Bitmap foreground=Bitmap.createBitmap(bitmap.getWidth(),bitmap.getHeight(), Bitmap.Config.ARGB_8888); //裁剪后的图像
        Canvas canvas=new Canvas(foreground);
        //绘制扣出来的背景
        canvas.drawBitmap(background,0,0,paint);
        //设置混合模式
        paint.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.SRC_OUT));
        //绘制原图
        canvas.drawBitmap(bitmap,0,0,paint);
        imageView.setImageBitmap(foreground);
    }).addOnFailureListener(e -> {
        // Task failed with an exception
        // ...
    });
}
private int[] maskColorsFromByteBuffer(ByteBuffer byteBuffer,int maskWidth,int maskHeight) {
    @ColorInt int[] colors = new int[maskWidth * maskHeight];
    for (int i = 0; i < maskWidth * maskHeight; i++) {
        float backgroundLikelihood = 1 - byteBuffer.getFloat();
        if (backgroundLikelihood > 0.9) { //128
            colors[i] = Color.argb(255, 255, 0, 255);
            //colors[i] = Color.argb(0, 255, 0, 255);
        } else if (backgroundLikelihood > 0.2) {
            // Linear interpolation to make sure when backgroundLikelihood is 0.2, the alpha is 0 and
            // when backgroundLikelihood is 0.9, the alpha is 128.
            // +0.5 to round the float value to the nearest int.
            int alpha = (int) (182.9 * backgroundLikelihood - 36.6 + 0.5);
            colors[i] = Color.argb(255, 255, 0, 255);
        }
    }
    return colors;
}