如何在Sikuli获取图像的坐标？

Question

I have this GUI on our project (please bear with my illustration, I'm not allowed to take a screenshot at work)

----------------------------------------------
    (1)     Header                            |
----------------------------------------------|
             (2)      Files             | (3) | Drop-down button (Options)
----------------------------------------|-----|          
                                        |     |
----------------------------------------|-----|
                                        |     |
----------------------------------------|-----|
                                        |     |
-----------------------------------------------

I need to test one file in each file type our product can support. The ultimate goal is to use that (3) drop down button. Here is the summary of my code when I'm in this part

find_filetype = find.imagedict(filetypeicon.png) #(2) filetype

freg = find_filetype.right()

if freg.exists(drop-down.png):                   #(3) drop-down icon
    freg.click(drop-down.png)

Now after clicking (3), a drop-down menu with 4 action items will appear. So for example if I click (3a): it will appear below from (3b - 3d) showing at least 2-3 drop-down items.

----------------------------------------------
            Header                            |           
----------------------------------------------|
                                        | (3a)|
----------------------------------------------|
                                |  (a1)   |   |   #a1 = action 1
--------------------------------|---------|---|   
                                |  (a2)   |   |   #a2 = action 2
--------------------------------|---------|---|
                                |__(a3)___|   |
-----------------------------------------------

If I click (3c): The drop-down menu will choose to appear above (fad) Showing at least 2 drop-down items.

----------------------------------------------
            Header                            |           
----------------------------------------------|
                                |  (a1)   |   |
--------------------------------|---------|---|         
                                |  (a2)   |   |
--------------------------------|_________|---|
                                        | (3c)|           
----------------------------------------|-----|
                                        |     |
----------------------------------------------

My workaround for this are :

1. screenshot all action items and

2. if/elifs + dragDrop()

which makes my scripts slow (I perform hundred of iterations everyday)

Is there a way I can find (a1) (because its always present after clicking (3)) , save its coordinates and use it as a param for scrolldown_codes like this

   #after clicking drop-down
   #find and get (action1) coordinates
   def scroll_down(coordinates):              #pass it here
       while not exist(action(n).png):        #n = action choice 
          end_Y = coordinates
          start_Y = coodinates - few_Y
          dragdrop(start_Y, end_Y)
          wait(1)
       click(action(n).png)

Please help me! Feel free to suggest anything helpful!

Note: Im only allowed to click and drag while on that GUI because its a touchscreen product we're developing.

Answer 1

Actual scrolling, as you would do with the scrollbar, is probably not the best solution as it requires many additional actions that all prone to failure due to various reasons. What I usually suggest in such cases is using keyboard to control the selection. There are few options really and that depends on how your application behaves. So I will list few tricks that might be used in your scenario and you decide what suits you best.

First of all, keep in mind that you can speed up the search time by limiting the area you are looking in. There are many ways to do that. For example:

1. Split the screen if you know that the patters you are looking for are expected to appear only in certain area.

2. Relative search / Region extension . You can find the relevant (3) item and then expand the region to its right to capture the area where the (4) selector (3 dots) is located. This will speed up the search significantly as it will only look in a small area rather than the whole screen and it will ensure you are clicking the correct (4) item. More info and examples about region expansion here . More info on optimizing search time here .

Now to the actual item selection in a menu. So, as I mentioned, I reckon you better off using your keyboard to select the item. That means selecting/clicking on the first item in the list and then pressing the down arrow button x times until you reach the menu item you are after. There are few ways to hit the down arrow button correct number of times:

1. Hard code it. If you know that the specific item is fourth in list just click 4 times.

2. Press arrow down button until you reach the desired item. That might be a bit slower and might require taking additional screenshots but might prove useful if you do not know what order the items are in or if that is prone to change.