为什么kivy read_pixel不返回预期的颜色？

Question

I am trying to create markers on an image that would allow a user to select colors, mark features etc. Ultimately I would like to have the corresponding image pixel for further use via opencv.

I'm having a lot of trouble getting the expected color under the touch and it sometimes returns colors like magenta which are not even in the example image.

I am pretty sure the problem is with how I am converting the touch position to the values I am handing to the read_pixel function.

I have tried many different solutions without success so I think there is something I am missing here.

main.py

from kivy.app import App
from kivy.properties import ListProperty, ObjectProperty
from kivy.uix.image import AsyncImage
from kivy.uix.relativelayout import RelativeLayout
from kivy.uix.widget import Widget
from kivy.uix.screenmanager import ScreenManager, Screen


class Marker(Widget):
    selected_color = ListProperty([0,1,0])

    def __init__(self, **kwargs):
        super(Marker, self).__init__(**kwargs)
        self.selected_pos = None

    def on_touch_down(self, touch):
        if self.collide_point(*touch.pos):
            print("Touched at Marker: {0}".format(touch.spos))

    def on_touch_move(self, touch):
        self.set_position_from_touch(touch.spos)

    def set_position_from_touch(self, spos):
        # print("touch: {0}".format(touch))
        self.image = self.parent.parent.image
        x = spos[0] * self.image.width
        y = spos[1] * self.image.height

        # setting position of the widget relative to touch
        self.pos = (x-self.width/2, y-self.height*(2/3))
        # self.pos = (x, y)

        print("widget position : {0}".format(self.pos))
        # converting widget position to pixel(row, column of
        selected_pixel = self.image.to_row_col(self.pos)
        print("selected Pixel: {0}".format(selected_pixel))

        try:
            self.selected_color = self.image._coreimage.read_pixel(
                selected_pixel[0],
                selected_pixel[1])
                # this skips conversion and just uses pos
                # self.pos[0],
                # self.pos[1])
        except IndexError:
            print("position out of range")


class MarkerManager(RelativeLayout):
    def __init__(self, **kwargs):
        super(MarkerManager, self).__init__(**kwargs)
        self.marker_mode = None
        self.features = []

    def on_touch_down(self, touch):
        if self.collide_point(*touch.pos):
            child_touched = False
            print("Touched: {0}".format(touch))
            if self.children:
                for child in self.children[:]:
                    if child.collide_point(touch.pos[0], touch.pos[1]):
                        child_touched = True
                        child.dispatch('on_touch_down', touch)
            if not child_touched:
                print("Touched only Image at: {0}".format(touch.spos))
                marker = Marker()
                self.features.append(marker)
                self.add_widget(marker)
                marker.set_position_from_touch(touch.spos)

class SelectedImage(AsyncImage):
    def __init__(self, **kwargs):
        super(SelectedImage, self).__init__(**kwargs)
        self.allow_stretch=True
        self.keep_ratio=False

    def to_row_col(self, pos):
        pixels_x = self._coreimage.width
        pixels_y = self._coreimage.height
        pixel_x = (pos[0] / self.width) * pixels_x
        # pixel_y = (pos[1] / self.height) * self.pixels_y
        pixel_y = (1 - (pos[1] / self.height)) * pixels_y
        # should correspond to row column of image
        return [int(pixel_x), int(pixel_y)]

class ImageScreen(Screen):
    image = ObjectProperty()
    manager = ObjectProperty()
    def __init__(self, **kwargs):
        super(ImageScreen, self).__init__(**kwargs)

class PointsSelectorApp(App):
    def build(self):
        return ImageScreen()

if __name__ == "__main__":
    PointsSelectorApp().run()

pointsselector.kv

<ImageScreen>:
    image: image_id
    manager: manager_id

    SelectedImage:
        id: image_id
        source: "rainbow_checkerboard.jpg"
        keep_data: True

    MarkerManager:
        id: manager_id


<Marker>:
    size_hint: None, None
    size: "40dp", "40dp"
    canvas:
        Color:
            rgb: self.selected_color
        Ellipse:
            pos: self.pos[0]+self.width/4, self.pos[1]+self.height/3
#            pos: self.pos[0], self.pos[1]
            size: self.width*.6, self.height*.6

here is my image i have been using to test with "rainbow_checkerboard.jpg"

Answer 1

I believe this is a bug in the Kivy itself. Particularly I think that line 901 at the kivy/core/image/__init__.py

    index = y * data.width * size + x * size
    raw = bytearray(data.data[index:index + size])
    color = [c / 255.0 for c in raw]

is wrong. It should be

    index = y * data.rowlength + x * size

instead.

Important thing here is that for performance reasons bitmap images in memory are alligned at 4 bytes addresses. Thus there is an explicit data.rowlength field. Usually that line works fine because images are typically well alligned so that data.rowlength = data.width * size . But your particular image is different: it uses 3-bytes RGB format (no alpha) and its width is odd 561 ie data.width * size = 1683 and so data.rowlength should be rounded up to 1684 , which it actually is but that code doesn't take it into account. It means that often you read colors from two consecutive pixels and with RGB components randomly rotated.

Additionally since you use JPEG format for your image, borders between "cells" are not really strict. If you zoom in really hard, you can see some compression artifacts such as these from around bottom-left corner.

Those artifacts mutliplied by the aforementioned random color components rotation bug give you very strange (seemingly non-existent) colors.

Possible workarounds

1. Change your image so that it is pre-aligned in the memory and the bug doesn't affect you (for example use mutliple of 4 as width)
2. Submit a bug or even a pull request to the Kivy and wait for it to be fixed
3. Patch kivy/core/image/__init__.py file locally (I tried it and it seems to work fine).