[英]MLKit Object Detection is not detecting objects
MLKit by Google (without Firebase) is new, so I'm having trouble. Google 的 MLKit(没有 Firebase)是新的,所以我遇到了麻烦。 I'm trying to follow this example here: https://developers.google.com/ml-kit/vision/object-detection/custom-models/android我试图在这里遵循这个例子: https://developers.google.com/ml-kit/vision/object-detection/custom-models/android
The app opens fine, & the camera works (As in, I can see things).该应用程序可以正常打开,并且相机可以正常工作(例如,我可以看到东西)。 But the actual detection doesn't seem to work.但实际检测似乎不起作用。
Am I missing part of the code to actually detect the object?我是否缺少实际检测 object 的部分代码? Or is it an issue with the implementation of CameraX or ImageInput?还是 CameraX 或 ImageInput 的实现有问题?
package com.example.mlkitobjecttest;
import androidx.annotation.NonNull;
import androidx.appcompat.app.AppCompatActivity;
import androidx.camera.core.Camera;
import androidx.camera.core.CameraSelector;
import androidx.camera.core.CameraX;
import androidx.camera.core.ImageAnalysis;
import androidx.camera.core.ImageProxy;
import androidx.camera.core.Preview;
import androidx.camera.core.impl.PreviewConfig;
import androidx.camera.lifecycle.ProcessCameraProvider;
import androidx.camera.view.PreviewView;
import androidx.core.app.ActivityCompat;
import androidx.core.content.ContextCompat;
import androidx.lifecycle.LifecycleOwner;
import android.content.pm.PackageManager;
import android.graphics.Rect;
import android.media.Image;
import android.os.Bundle;
import android.text.Layout;
import android.util.Rational;
import android.util.Size;
import android.view.View;
import android.widget.TextView;
import android.widget.Toast;
import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;
import com.google.common.util.concurrent.ListenableFuture;
import com.google.mlkit.common.model.LocalModel;
import com.google.mlkit.vision.common.InputImage;
import com.google.mlkit.vision.objects.DetectedObject;
import com.google.mlkit.vision.objects.ObjectDetection;
import com.google.mlkit.vision.objects.ObjectDetector;
import com.google.mlkit.vision.objects.custom.CustomObjectDetectorOptions;
import org.w3c.dom.Text;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class MainActivity extends AppCompatActivity {
private class YourAnalyzer implements ImageAnalysis.Analyzer {
@Override
@androidx.camera.core.ExperimentalGetImage
public void analyze(ImageProxy imageProxy) {
Image mediaImage = imageProxy.getImage();
if (mediaImage != null) {
InputImage image =
InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees());
// Pass image to an ML Kit Vision API
// ...
LocalModel localModel =
new LocalModel.Builder()
.setAssetFilePath("mobilenet_v1_1.0_128_quantized_1_default_1.tflite")
// or .setAbsoluteFilePath(absolute file path to tflite model)
.build();
CustomObjectDetectorOptions customObjectDetectorOptions =
new CustomObjectDetectorOptions.Builder(localModel)
.setDetectorMode(CustomObjectDetectorOptions.SINGLE_IMAGE_MODE)
.enableMultipleObjects()
.enableClassification()
.setClassificationConfidenceThreshold(0.5f)
.setMaxPerObjectLabelCount(3)
.build();
ObjectDetector objectDetector =
ObjectDetection.getClient(customObjectDetectorOptions);
objectDetector
.process(image)
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
//Toast.makeText(getApplicationContext(), "Fail. Sad!", Toast.LENGTH_SHORT).show();
//textView.setText("Fail. Sad!");
imageProxy.close();
}
})
.addOnSuccessListener(new OnSuccessListener<List<DetectedObject>>() {
@Override
public void onSuccess(List<DetectedObject> results) {
for (DetectedObject detectedObject : results) {
Rect box = detectedObject.getBoundingBox();
for (DetectedObject.Label label : detectedObject.getLabels()) {
String text = label.getText();
int index = label.getIndex();
float confidence = label.getConfidence();
textView.setText(text);
}}
imageProxy.close();
}
});
}
//ImageAnalysis.Builder.fromConfig(new ImageAnalysisConfig).setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST);
}
}
PreviewView prevView;
private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;
private ExecutorService executor = Executors.newSingleThreadExecutor();
TextView textView;
private int REQUEST_CODE_PERMISSIONS = 101;
private String[] REQUIRED_PERMISSIONS = new String[]{"android.permission.CAMERA"};
/* @NonNull
@Override
public CameraXConfig getCameraXConfig() {
return CameraXConfig.Builder.fromConfig(Camera2Config.defaultConfig())
.setCameraExecutor(ContextCompat.getMainExecutor(this))
.build();
}
*/
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
prevView = findViewById(R.id.viewFinder);
textView = findViewById(R.id.scan_button);
if(allPermissionsGranted()){
startCamera();
}else{
ActivityCompat.requestPermissions(this, REQUIRED_PERMISSIONS, REQUEST_CODE_PERMISSIONS);
}
}
private void startCamera() {
cameraProviderFuture = ProcessCameraProvider.getInstance(this);
cameraProviderFuture.addListener(new Runnable() {
@Override
public void run() {
try {
ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
bindPreview(cameraProvider);
} catch (ExecutionException | InterruptedException e) {
// No errors need to be handled for this Future.
// This should never be reached.
}
}
}, ContextCompat.getMainExecutor(this));
}
void bindPreview(@NonNull ProcessCameraProvider cameraProvider) {
Preview preview = new Preview.Builder()
.build();
CameraSelector cameraSelector = new CameraSelector.Builder()
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
.build();
preview.setSurfaceProvider(prevView.createSurfaceProvider());
ImageAnalysis imageAnalysis =
new ImageAnalysis.Builder()
.setTargetResolution(new Size(1280, 720))
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build();
imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this), new YourAnalyzer());
Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview, imageAnalysis);
}
private boolean allPermissionsGranted() {
for(String permission: REQUIRED_PERMISSIONS){
if(ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED){
return false;
}
}
return true;
}
@Override
public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
if(requestCode == REQUEST_CODE_PERMISSIONS){
if(allPermissionsGranted()){
startCamera();
} else{
Toast.makeText(this, "Permissions not granted by the user.", Toast.LENGTH_SHORT).show();
this.finish();
}
}
}
}
Nothing is detected because you defined the wrong path to tflite
model file.未检测到任何内容,因为您定义了tflite
model 文件的错误路径。 You emulator or physical device cannot resolve given path as it doesn't exists on mobile device: C:\\Users\\dude\\Documents\\mlkitobjecttest\\app\\src\\main\\assets\\mobilenet_v1_1.0_128_quantized_1_default_1.tflite
您的模拟器或物理设备无法解析给定路径,因为它在移动设备上不存在: C:\\Users\\dude\\Documents\\mlkitobjecttest\\app\\src\\main\\assets\\mobilenet_v1_1.0_128_quantized_1_default_1.tflite
Copy your model mobilenet_v1_1.0_128_quantized_1_default_1.tflite
into assets
directory under you app's project src/main
directory.将 model mobilenet_v1_1.0_128_quantized_1_default_1.tflite
复制到应用程序项目src/main
目录下的assets
目录中。
If you do not have that directory just create a new one named assets
.如果您没有该目录,只需创建一个名为assets
的新目录。
At the end it should look like this:最后它应该是这样的:
After that fix LocalModel
initialization code:之后修复LocalModel
初始化代码:
LocalModel localModel =
new LocalModel.Builder()
.setAssetFilePath("mobilenet_v1_1.0_128_quantized_1_default_1.tflite")
// or .setAbsoluteFilePath(absolute file path to tflite model)
.build();
ImageAnalysis
instance was not bound to CameraProvider
: ImageAnalysis
实例未绑定到CameraProvider
:
...
ImageAnalysis imageAnalysis = ...
Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview); // imageAnalysis is not used
To fix it just pass as the last argument imageAnalysis
variable into bindToLifecycle
method:要修复它,只需将imageAnalysis
变量作为最后一个参数传递给bindToLifecycle
方法:
Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview, imageAnalysis);
MLKit cannot process an image because it was closed while it was processing or right before processing started. MLKit 无法处理图像,因为它在处理过程中或处理开始之前已关闭。 I'm talking about imageProxy.close()
line of code declared inside of public void analyze(ImageProxy imageProxy)
.我说的是在public void analyze(ImageProxy imageProxy)
内部声明的imageProxy.close()
代码行。
Java documentation of close()
method: Java close()
方法的文档:
/**
* Free up this frame for reuse.
* <p>
* After calling this method, calling any methods on this {@code Image} will
* result in an {@link IllegalStateException}, and attempting to read from
* or write to {@link ByteBuffer ByteBuffers} returned by an earlier
* {@link Plane#getBuffer} call will have undefined behavior. If the image
* was obtained from {@link ImageWriter} via
* {@link ImageWriter#dequeueInputImage()}, after calling this method, any
* image data filled by the application will be lost and the image will be
* returned to {@link ImageWriter} for reuse. Images given to
* {@link ImageWriter#queueInputImage queueInputImage()} are automatically
* closed.
* </p>
*/
To fix that move imageProxy.close()
into failure and success listeners:要修复将imageProxy.close()
移动到失败和成功侦听器中的问题:
objectDetector
.process(image)
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
Toast.makeText(getApplicationContext(), "Fail. Sad!", Toast.LENGTH_LONG).show();
...
imageProxy.close();
}
})
.addOnSuccessListener(new OnSuccessListener<List<DetectedObject>>() {
@Override
public void onSuccess(List<DetectedObject> results) {
Toast.makeText(getBaseContext(), "Success...", Toast.LENGTH_LONG).show();
...
imageProxy.close();
}
});
The fixed solution was tested with image classification model from Tensorflow and test was successful.使用来自 Tensorflow 的图像分类 model对固定解决方案进行测试,测试成功。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.