Straigthening hand images by drawing a line through the middle finger in OpenCV

Question

I want to straighten the hand in the image based on the middle finger. I have about 10000 of these hand X-ray images to preprocess. The preprocess I did so far:

1. Apply Gaussian Blur and Threshold (Binary + Otsu) on the image.
2. Apply dilation to get a single object (in this case a hand).
3. Used cv2.findContours() to draw outline along the edges around the hand.
4. Apply cv2.convexHull() and then cv2.convexityDefects() to mark the points that are farthest away from the convex hull. To filter out the ones of interest between the fingers, I only considered those that are more than a certain distance from the convex hull.

The code below describes the above-mentioned:

import cv2
import numpy as np
from matplotlib import pyplot as plt

img_path = "sample_image.png"

# Getting the threshold of the image:
image = cv2.imread("sample_image.png")
original = image.copy() 
blank = np.zeros(image.shape[:2], dtype = np.uint8)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (127,127), 0)
thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

# Merge into a single contour
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
dilate = cv2.dilate(thresh, kernel, iterations = 2)

# Drawing the contours along the edges of the hand in X-ray:
contours, hierarchy = cv2.findContours(dilate, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
contours = max(contours, key = lambda x: cv2.contourArea(x))

cv2.drawContours(image, [contours], -1, (255,255,0), 2)

# To mark the points that are farthest away from the convex hull, one for each "convexity defect"
hull = cv2.convexHull(contours, returnPoints = False)
defects = cv2.convexityDefects(contours, hull)
for i in range(defects.shape[0]):
    _, _, farthest_point_index, distance = defects[i, 0]
    farthest_point = contours[farthest_point_index][0]
    if distance > 50_000:
        circle = cv2.circle(image, farthest_point, 20, [0,0,255], -1)

• Input image = Sample_image.png

• After running Contour on the image = Contour on the hand

• Final output of the code after convexity hull = Convexity Hull

The next step would be:

1. To find out which of four points are the most central, since those are next to the middle finger.

2. Draw a line along the identified middle finger and rotate the image straight.

How would one approach and code this? I am really struggling...

• What I want:

1. Draw Line on the middle finger
2. Rotate along the axis to straighten it

Answer 1

After struggling for a while with trying to reproduce your results (there is an error with the distance threshold in your code example), I came up with an alternate solution that should do the trick. Instead of attempting to find the line that goes through the two central points, I suggest fitting an ellipse to the contour of the hand, and then rotating the image by the ellipse_angle .

STEP1: Fitting the contours with an ellipse:

(x,y),(MA,ma),ellipse_angle = cv2.fitEllipse(contours)
  
cv2.ellipse(image, (int(x),int(y)), (np.int(MA),np.int(ma)), ellipse_angle, np.int(ellipse_angle), np.int(ellipse_angle+360), (255,0,255), 1)
x1=np.int((int(x) + np.int(MA)*np.sin(ellipse_angle * np.pi / 180.0)))
y1=np.int((int(y) - np.int(MA)*np.cos(ellipse_angle * np.pi / 180.0)))     
x2=np.int((int(x) - np.int(MA)*np.sin(ellipse_angle * np.pi / 180.0)))
y2=np.int((int(y) + np.int(MA)*np.cos(ellipse_angle * np.pi / 180.0))) 
cv2.line(image, (x1, y1), (x2, y2),(255,0,255),4) 

This snippet outputs:

Note that you can barely see the ellipse here, so I plotted the ellipse's longest axis using cv2.line(image, (x1, y1), (x2, y2),(255,0,255),4) .

STEP2: Rotating the image:

Following this pyimagesearch tutorial :

(h, w) = image.shape[:2]
(cX, cY) = (w // 2, h // 2)

M = cv2.getRotationMatrix2D((cX, cY), ellipse_angle, 1.0)
rotated = cv2.warpAffine(image, M, (w, h))
cv2.imshow("Rotated by 45 Degrees", rotated)

This returns:

I would be curious to know how this script performs on the remainder of your data.