用 opencv 找到手绘线的终点

Question

I'm trying to find the two end points of a hand drawn line I have written this snippet which finds the contours, but the end points is not correct:

img = cv2.imread("my_img.jpeg")

img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Binary Threshold:
_, thr_img = cv2.threshold(img_gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)

cv2.imshow(winname="after threshold", mat=thr_img)
cv2.waitKey(0)

contours, _ = cv2.findContours(image=thr_img, mode=cv2.RETR_TREE, method=cv2.CHAIN_APPROX_SIMPLE)

for idx, cnt in enumerate(contours):
    print("Contour #", idx)
    cv2.drawContours(image=img, contours=[cnt], contourIdx=0, color=(255, 0, 0), thickness=3)
    cv2.circle(img, tuple(cnt[0][0]), 5, (255, 255, 0), 5) # Result in wrong result
    cv2.circle(img, tuple(cnt[-1][0]), 5, (0, 0, 255), 5)  # Result in wrong result
    cv2.imshow(winname="contour" + str(idx), mat=img)
    cv2.waitKey(0)

Original image:

I have also tried cornerHarris but it gave me some extra points,

Can someone please suggest an accurate and better approach?

Answer 1

This solution uses a Python implementation of this approach . The idea is to convolve the image with a special kernel that identifies starting/ending points in a line. These are the steps:

1. Resize your image a little bit, because its too big.
2. Convert the image to grayscale
3. Get the skeleton
4. Convolve the skeleton with the endpoint kernel
5. Get the coordinates of the endpoints

Now, that would be first iteration of the proposed algorithm. However, depending on the input image, there could be duplicated endpoints - individual points that are too close to each other and could be joined. So, let's incorporate some additional processing to get rid of such duplicated points.

6. Identify possible duplicated points
7. Join the duplicated points
8. Compute final points

These last steps are too general, let me further elaborate on the idea behind the duplicates elimination when we get to that step. Let's see the code for the first part:

# imports:
import cv2
import numpy as np

# image path
path = "D://opencvImages//"
fileName = "hJVBX.jpg"

# Reading an image in default mode:
inputImage = cv2.imread(path + fileName)

# Resize image:
scalePercent = 50  # percent of original size
width = int(inputImage.shape[1] * scalePercent / 100)
height = int(inputImage.shape[0] * scalePercent / 100)

# New dimensions:
dim = (width, height)

# resize image
resizedImage = cv2.resize(inputImage, dim, interpolation=cv2.INTER_AREA)

# Color conversion
grayscaleImage = cv2.cvtColor(resizedImage, cv2.COLOR_BGR2GRAY)
grayscaleImage = 255 - grayscaleImage

So far I've resized the image (to a 0.5 of the original scale) and converted it to grayscale (in reality an inverted binary image). Now, the first step to detect endpoints is to normalize the lines width to 1 pixel . This is achieved by computing the skeleton , which can be implemented using OpenCV's Extended Image Processing module :

# Compute the skeleton:
skeleton = cv2.ximgproc.thinning(grayscaleImage, None, 1)

This is the skeleton:

Now, let's run the endpoint detection portion:

# Threshold the image so that white pixels get a value of 0 and
# black pixels a value of 10:
_, binaryImage = cv2.threshold(skeleton, 128, 10, cv2.THRESH_BINARY)

# Set the end-points kernel:
h = np.array([[1, 1, 1],
              [1, 10, 1],
              [1, 1, 1]])

# Convolve the image with the kernel:
imgFiltered = cv2.filter2D(binaryImage, -1, h)

# Extract only the end-points pixels, those with
# an intensity value of 110:
endPointsMask = np.where(imgFiltered == 110, 255, 0)

# The above operation converted the image to 32-bit float,
# convert back to 8-bit uint
endPointsMask = endPointsMask.astype(np.uint8)

Check out the original link for information on this method, but the general gist is that the kernel is such that convolution with ending points in a line will produce values of 110 , as a result of neighborhood summation. There are float operations involved, so gotta be careful with data types and conversions. The result of the procedure can be observed here:

Those are the endpoints, however, note that some points could be joined, IF they are too close. Now comes the duplicate elimination steps. Let's first define the criteria to check if a point is a duplicate or not. If the points are too close, we will join them. Let's propose a morphology-based approach to points closeness. I'll dilate the endpoints mask with a rectangular kernel of size 3 and 3 iterations. If two or more points are too close, their dilation will produce a big, unique, blob:

# RGB copy of this:
rgbMask = endPointsMask.copy()
rgbMask = cv2.cvtColor(rgbMask, cv2.COLOR_GRAY2BGR)

# Create a copy of the mask for points processing:
groupsMask = endPointsMask.copy()

# Set kernel (structuring element) size:
kernelSize = 3
# Set operation iterations:
opIterations = 3
# Get the structuring element:
maxKernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernelSize, kernelSize))
# Perform dilate:
groupsMask = cv2.morphologyEx(groupsMask, cv2.MORPH_DILATE, maxKernel, None, None, opIterations, cv2.BORDER_REFLECT101)

This is the result of the dilation. I refer to this image as the groupsMask :

Note how some of the points now share adjacency. I will use this mask as a guide to produce the final centroids. The algorithm is like this: Loop through the endPointsMask , for every point, produce a label. Using a dictionary , store the label and all centroids sharing that label - use the groupsMask for propagating the label between different points via flood-filling . Inside the dictionary we will store the centroid cluster label, the accumulation of the centroids sum and the count of how many centroids where accumulated, so we can produce a final average. Like this:

# Set the centroids Dictionary:
centroidsDictionary = {}

# Get centroids on the end points mask:
totalComponents, output, stats, centroids = cv2.connectedComponentsWithStats(endPointsMask, connectivity=8)

# Count the blob labels with this:
labelCounter = 1

# Loop through the centroids, skipping the background (0):
for c in range(1, len(centroids), 1):

    # Get the current centroids:
    cx = int(centroids[c][0])
    cy = int(centroids[c][1])

    # Get the pixel value on the groups mask:
    pixelValue = groupsMask[cy, cx]

    # If new value (255) there's no entry in the dictionary
    # Process a new key and value:
    if pixelValue == 255:

        # New key and values-> Centroid and Point Count:
        centroidsDictionary[labelCounter] = (cx, cy, 1)

        # Flood fill at centroid:
        cv2.floodFill(groupsMask, mask=None, seedPoint=(cx, cy), newVal=labelCounter)
        labelCounter += 1

    # Else, the label already exists and we must accumulate the
    # centroid and its count:
    else:

        # Get Value:
        (accumCx, accumCy, blobCount) = centroidsDictionary[pixelValue]

        # Accumulate value:
        accumCx = accumCx + cx
        accumCy = accumCy + cy
        blobCount += 1

        # Update dictionary entry:
        centroidsDictionary[pixelValue] = (accumCx, accumCy, blobCount)

Here are some animations of the procedure, first, the centroids being processed one by one. We are trying to join those points that seem to close to each other:

The groups mask being flood-filled with new labels. The points that share a label are added together to produce a final average point. It is a little hard to see, because my label starts at 1 , but you can barely see the labels being populated:

Now, what remains is to produce the final points. Loop through the dictionary and check out the centroids and their count. If the count is greater than 1 , the centroid represents an accumulation and must be diveded by its count to produce the final point:

# Loop trough the dictionary and get the final centroid values:
for k in centroidsDictionary:
    # Get the value of the current key:
    (cx, cy, count) = centroidsDictionary[k]
    # Process combined points:
    if count != 1:
        cx = int(cx/count)
        cy = int(cy/count)
    # Draw circle at the centroid
    cv2.circle(resizedImage, (cx, cy), 5, (0, 0, 255), -1)

cv2.imshow("Final Centroids", resizedImage)
cv2.waitKey(0)

This is the final image, showing the end/starting points of the lines:

Now, the endpoints detection method, or rather, the convolution step, is producing an apparent extra point on the curve, that could be because of a segment on the lines is too separated from its neighborhood - splitting the curve in two parts. Maybe applying a little bit of morphology before the convolution could get rid of the problem.

Answer 2

I would like to suggest a simpler, more efficient method and more importantly, it does not produce false endpoints:

The idea is quite simple, after thinning, count neighbouring pixels (8-connectivity) if neighbours count equals 1 --> the point is an end point

The code is self-explanatory:

def get_end_pnts(pnts, img):
    extremes = []    
    for p in pnts:
        x = p[0]
        y = p[1]
        n = 0        
        n += img[y - 1,x]
        n += img[y - 1,x - 1]
        n += img[y - 1,x + 1]
        n += img[y,x - 1]    
        n += img[y,x + 1]    
        n += img[y + 1,x]    
        n += img[y + 1,x - 1]
        n += img[y + 1,x + 1]
        n /= 255        
        if n == 1:
            extremes.append(p)
    return extremes

main:

img = cv2.imread(p, cv2.IMREAD_GRAYSCALE)
img = cv2.threshold(img, 128, 255, cv2.THRESH_OTSU + cv2.THRESH_BINARY_INV)
img = cv2.ximgproc.thinning(img)
pnts = cv2.findNonZero(img)
pnts = np.squeeze(pnts)
ext = get_end_pnts(pnts, img)
for p in ext:
    cv2.circle(img, (p[0], p[1]), 5, 128)

Output:

Edit: you might be interested in visiting my answer to this similar question . It has some extra functionality, it detects end-points and connector-points as well.

Answer 3

I would like to propose another solution, involving the Hit-or-Miss Transform . This transform helps identify certain patterns in a binary image. The popular erosion and dilation operations form the basis of this transform. See this page for more details, formula, illustration and code

One of the operations I performed on the binary image is skeletonization, which brings down the width of any shape down to just one pixel . Using hit-or-miss transform finding endpoints is pretty easy, ie if you know how they can be found and what kernels need to be used to spot them.

Variations:

These end points can be in eight different positions. The end pixel is the one at the center of each kernel below:

1. One variation that can occur is when the end pixel is between 2 other pixels like the following:

1 --> foreground pixel (white pixel)

0 --> background pixel (black pixel)

|1 1 1|
|0 1 0|
|0 0 0|

|0 0 1|
|0 1 1|
|0 0 1|

|0 0 0|
|0 1 0|
|1 1 1|

|1 0 0|
|1 1 0|
|1 0 0|

2. Another variation is when the end pixel is at the end of a diagonal; like the following:

|0 0 0|
|0 1 0|
|1 0 0|

|1 0 0|
|0 1 0|
|0 0 0|

|0 0 1|
|0 1 0|
|0 0 0|

|0 0 0|
|0 1 0|
|0 0 1|

Code:

Read the image, inversely binarize it and obtain its skeleton:

img = cv2.imread(r'C:\Users\524316\Desktop\Stack\Code\Find_endpoints\lines.jpg')
img2 = img.copy()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
th = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]    
skeleton = cv2.ximgproc.thinning(th, None, 1)  

According to the documentation in OpenCV:

• 1 -> foreground pixel
• -1 -> background pixel
• 0 -> foreground/background pixel

We build 4 different kernels of variation 1 from k1 to k4 below:

k1 = np.array(([ 0, 0,  0], 
               [-1, 1, -1], 
               [-1,-1, -1]), dtype="int")

k2 = np.array(([0, -1, -1], 
               [0,  1, -1], 
               [0, -1, -1]), dtype="int")

k3 = np.array(([-1, -1, 0],  
               [-1,  1, 0], 
               [-1, -1, 0]), dtype="int")

k4 = np.array(([-1, -1, -1], 
               [-1,  1, -1], 
               [ 0,  0,  0]), dtype="int")

Perform hit-or-miss transform using all the above kernels and sum the images:

o1 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k1)
o2 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k2)
o3 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k3)
o4 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k4)
out1 = o1 + o2 + o3 + o4

We build the next 4 kernels of variation 2 from k5 to k8 below:

k5 = np.array(([-1, -1, -1], 
               [-1,  1, -1], 
               [ 0, -1, -1]), dtype="int")

k6 = np.array(([-1, -1, -1], 
               [-1,  1, -1], 
               [-1, -1,  0]), dtype="int")

k7 = np.array(([-1, -1,  0], 
               [-1,  1, -1], 
               [-1, -1, -1]), dtype="int")

k8 = np.array(([ 0, -1, -1], 
               [-1,  1, -1], 
               [-1, -1, -1]), dtype="int")               

o5 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k5)
o6 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k6)
o7 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k7)
o8 = cv2.morphologyEx(skeleton, cv2.MORPH_HITMISS, k8)
out2 = o5 + o6 + o7 + o8

Add both the results and draw the end points:

out = cv2.add(out1, out2)
pts = np.argwhere(out == 255)
for pt in pts:
    img2 = cv2.circle(img2, (pt[1], pt[0]), 15, (0,255,0), -1)