Translate detected rectangle from portrait CIImage to landscape CIImage

Question

I am modifying code found here . In the code we are capturing video from the phone camera using AVCaptureSession and using CIDetector to detect a rectangle in the image feed. The feed has an image which is 640x842 (iphone5 in portrait). We then do an overlay on the image so the user can see the detected rectangle (actually it's a trapezoid most of the time).

When the user presses a button on the UI, we capture an image from the video and re-run the rectangle detection on this larger image (3264x2448) which as you can see is landscape. We then do a perspective transform on the detected rectangle and crop on the image.

This is working pretty well but the issue I have is say 1 out of 5 captures the detected rectangle on the larger image is different to the one detected (and presented to the user) from the smaller image. Even though I only capture when I detect the phone is (relatively) still, the final image then does not represent the rectangle the user expected.

To resolve this my idea is to use the coordinates of the originally captured rectangle and translate them to a rectangle on the captured still image. This is where I'm struggling.

I tried this with the detected rectangle:

CGFloat radians = -90 * (M_PI/180);
CGAffineTransform rotation = CGAffineTransformMakeRotation(radians);

CGRect rect = CGRectMake(detectedRect.bounds.origin.x, detectedRect.bounds.origin.y, detectedRect.bounds.size.width, detectedRect.bounds.size.height);

CGRect rotatedRect = CGRectApplyAffineTransform(rect, rotation);

So given a detected rect:

TopLeft: 88.213425, 632.31329 TopRight: 545.59302, 632.15546 BottomRight: 575.57819, 369.22321 BottomLeft: 49.973862, 369.40466

I now have this rotated rect:

origin = (x = 369.223206, y = -575.578186) size = (width = 263.090088, height = 525.604309)

How do I translate the rotated rectangle coordinates in the smaller portrait image to coordinates to the 3264x2448 image?

Edit Duh.. reading my own approach realised that creating a rectangle out of a trapezoid will not solve my problem!

Supporting code to detect the rectangle etc...

// In this method we detect a rect from the video feed and overlay

-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);

    CIImage *image = [CIImage imageWithCVPixelBuffer:pixelBuffer];

   // image is 640x852 on iphone5

    NSArray *rects = [[CIDetector detectorOfType:CIDetectorTypeRectangle context:nil options:@{CIDetectorAccuracy : CIDetectorAccuracyHigh}] featuresInImage:image];

    CIRectangleFeature *detectedRect = rects[0]; 

    // draw overlay on image code....
}

This is a summarized version of how the still image is obtained:

// code block to handle output from AVCaptureStillImageOutput
[self.stillImageOutput captureStillImageAsynchronouslyFromConnection:videoConnection completionHandler: ^(CMSampleBufferRef imageSampleBuffer, NSError *error)
     {
         NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageSampleBuffer];
         CIImage *enhancedImage = [[CIImage alloc] initWithData:imageData options:@{kCIImageColorSpace:[NSNull null]}];
             imageData = nil;

         CIRectangleFeature *rectangleFeature = [self getDetectedRect:[[self highAccuracyRectangleDetector] featuresInImage:enhancedImage]];

         if (rectangleFeature) {
             enhancedImage = [self   correctPerspectiveForImage:enhancedImage withTopLeft:rectangleFeature.topLeft andTopRight:rectangleFeature.topRight andBottomRight:rectangleFeature.bottomRight andBottomLeft:rectangleFeature.bottomLeft];
         }
    }

Thank you.

Answer 1

I had the same issue while doing this kind of stuff.I have resolved it by doing below code in swift. Take a look if it can help you.

 if let videoConnection = stillImageOutput.connection(withMediaType: AVMediaTypeVideo) {
            stillImageOutput.captureStillImageAsynchronously(from: videoConnection) {
                (imageDataSampleBuffer, error) -> Void in
                let imageData = AVCaptureStillImageOutput.jpegStillImageNSDataRepresentation(imageDataSampleBuffer)
                var img = UIImage(data: imageData!)!

                let outputRect = self.previewLayer?.metadataOutputRectOfInterest(for: (self.previewLayer?.bounds)!)
                let takenCGImage = img.cgImage
                let width = (takenCGImage?.width)!
                let height = (takenCGImage?.height)!
                let cropRect = CGRect(x: (outputRect?.origin.x)! * CGFloat(width), y: (outputRect?.origin.y)! * CGFloat(height), width: (outputRect?.size.width)! * CGFloat(width), height: (outputRect?.size.height)! * CGFloat(height))
                let cropCGImage = takenCGImage!.cropping(to: cropRect)
                img = UIImage(cgImage: cropCGImage!, scale: 1, orientation: img.imageOrientation)

                let cropViewController = TOCropViewController(image: self.cropToBounds(image: img))
                cropViewController.delegate = self
                self.navigationController?.pushViewController(cropViewController, animated: true)
            }
        }