Automator - AppleScript recording and changing into values

Question

I'm recording an Automator Script to help me scrape an URL that I use to get information from a week at a time. I would like to specific value to be a vaiable.

When I have recorded the script I would like to have the specific values (ie 2018-01-02) be able to change into a variable that can be changed within the script (ie a day at a time to 2018-01-03). I can not change the date to a variable in Automator. If I copy the script to a text document I can not get it back into Automator.

Is there a way to get this done?

The script as text looks like this:

-- Click the “2018-01-02” button.
delay 2.631525
set timeoutSeconds to 2.000000
set uiScript to "click UI Element \"2018-01-03\” of group 7 of UI Element 1 of scroll area 1 of group 23 of UI Element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window \"Historical Data Feed\" of application process \"Safari\""
my doWithTimeout( uiScript, timeoutSeconds )

in Automator it looks like:

• Click the "2018-01-02" button

Answer 1

You can set variables in Automator, by using the "run applescript" command. Search for the command by typing "applescript" into the search box in automator. Then you can set the variable within the script. Here's an example of how it could be done:

set dateVar to "2018-01-02"
-- Click the “2018-01-02” button.
delay 2.631525
set timeoutSeconds to 2.0
set uiScript to "click UI Element \"" & dateVar & "\" of group 7 of UI Element 1 of scroll area 1 of group 23 of UI Element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window \"Historical Data Feed\" of application process \"Safari\""
my doWithTimeout(uiScript, timeoutSeconds)

You could also use other methods of setting the date. This article has a lot of information about using dates with Applescript https://macscripter.net/viewtopic.php?id=24737

The UI element numbers can also vary, so you would need to account for this to get your script to work with variables.

Edit: As you're getting assistance to use another method, I can see how my answer wouldn't be relevant to your situation now. Although Applescript will get the job done, it's quite a difficult way to accomplish the end goal of downloading the files. I've found that using Selenium, Firefox and Python to be a far easier way to automate the browser to download files. It works best when you understand how to use xpath, and occasionally execute some javascript, but if it's something you're doing more of in the future, you might want to consider this method. You can also use tag, id or class selectors instead of xpath.

Answer 2

@Matts gave a fine answer about using variables in Automator . But, having discussed the nature of your objective in more depth, it's clear that knowing how to reference variables in Automator is not at all the issue, and won't help you.

In fact, don't even bother with Automator . It's not going to do what you want it to. The whole endeavour of trying to automate physical interaction with Safari means it's going to involve a fair bit of AppleScripting, and it's the worst kind— GUI (graphical user interface) scripting .

Come out of Automator and use Script Editor . Make sure Script Editor has relevant accessibility permissions in order for GUI scripting to work. You can do this by going to System Preferences > Security & Privacy > Privacy > Accessibility , unlocking the padlock in the bottom left of the window, and then finding Script Editor in the list of apps allowed to control your computer. Make sure it's checked.

Briefly, GUI scripting is pretty much doing just that, in a nutshell: controlling your computer, in a way that allows it to issue mouse clicks and press buttons or menu items as if the user were there physically doing it themselves.

But it's a hell of a nightmare figuring out how to reference objects in the GUI hierarchy of on-screen elements that can be pressed or clicked.

I think your snippet of Automator -manufactured code was misleading you:

    click UI Element "2017-01-03” of group 7 of UI Element 1 of scroll area 1 of ¬
        group 23 of UI Element 1 of scroll area 1 of group 1 of group 1 of tab group 1 ¬
        of splitter group 1 of window "..." of application process "Safari"

I think you saw UI Element "2017-01-03" and deduced that one must be able to access date-specific data by substituting the appropriate date string in for the one that's there currently. Hence, why you asked about using variables in Automator . But it wouldn't have worked. That date string just reports the name of the element (which happens to be something called a static text element) that contained a piece of text with a date in it, but not one that was editable. Therefore, referencing UI Element "2017-01-02" would just throw an error, because no such element exists.

So here's what I've come up with so far. It's by no means a full package: I haven't written the script that will achieve your objective straight out of the box, but I'm definitely providing you with all the necessary components for you to be able to piece together a finished product. Believe me, I did the most grueling part of it, which a painstaking search to obtain all the specific object references for every button you will want to be able to click within the GUI hierarchy.

First, an important note: I didn't use the URL you posted in the comments to do my GUI inspection. The webpage was far too busy and convoluted, which would have made the job much harder. But luckily, they had a link to a page that contains only the widget with which we'll be interacting. But, because the GUI makeup of the page is different (the widget is identical to the one your original link, it's just the rest of the page that's different), none of the code I've written will work with your original link. So you'll have to use the URL that I've provided in the script, which is https://www.dukascopy.com/plugins/fxMarketWatch/detach.php?id=ct23 .

---

Referencing the GUI objects

Declare some property variables, which you can change according to what month and year you want to retrieve the datasets from. You can also specify your instrument, by either the short abbreviation code, or its full description:

    property |URL| : "https://www.dukascopy.com/plugins/fxMarketWatch/detach.php?id=ct23"
    property Instrument : {|short|:"AAPL.US/USD", |description|:"APPLE INC"}
    property |month| : "December"
    property |year| : "2017"

This next part is where System Events and the Safari process are employed, which is the root of the GUI scripting commands. I also lumped in with it a reference to all the elements that were common to all ones we'll be interacting with. Otherwise, you'd have to write them out over and over, whereas it's easier to just reference them once as a group of UI elements that form the basis of a tell... block:

    tell application "System Events" to ¬
        tell application process "Safari" to ¬
            tell (window named |URL|) to ¬
                tell UI element 1 of scroll area 1 of group 1 of ¬
                    UI element 1 of scroll area 1 of group 1 of ¬
                    group 1 of tab group 1 of splitter group 1
                                  .
                                  .
                    (* The rest of our code goes
                       inside this tell block  *)
                                  .
                                  .
                end tell

Note also that the commands are being sent to the window that's named the same as the URL of the webpage. So this script assumes that the webpage has already been loaded up in Safari—it doesn't load it for you (although that wouldn't be difficult to implement).

Also make sure you log in after you load the page. If you don't, clicking the download button brings up the login box, which will interrupt the whole script and throw an error. But, once you're logged in and click the box to "stay logged in", from then on it will proceed to the download options by itself.

The next part is the meat of the endeavour. This is where I define variables, each of which reference a GUI object that you will want to be able to issue clicks and presses to from the script.

These three form the main components for each row of available instruments that you can select:

    set Instruments to a reference to groups of group 1 of group 1 of group 3
        --> All Instruments (410 of them)
    set Instrument_Abbreviations to a reference to static texts of group 1 of group 2 of Instruments
        --> The short text code for each instrument
    set Instrument_Descriptions to a reference to static texts of group 2 of Instruments
        --> The long text for each instrument

This one is the pair of elements for selecting the timezone, local or GMT:

    set LOCAL_GMT to a reference to checkboxes of group 1 of group 7 of group 4

This next block of elements took some work to navigate through. These are the ones responsible for selecting the date. Pressing the date button allows a calendar pop-up to appear on-screen, in which you choose the year, month and day. This will also form the core of what will become your program loop, as you use the variables to cycle through the days of a month to retrieve the data files for all of, say, December.

The thing about menus and objects such as this calendar that "pop up" is that they don't physically exist until the moment they are created and appear on our screens. Therefore, there's potential in GUI scripting for errors to get thrown when the script tries to reference an object that will be there at some future point, but is not currently there and doesn't exist yet. Using a reference to is one way to cater for this problem, and I've done this for most of these variable declarations.

    set Date_button to a reference to button 1 of group 3 of group 4
        --> The date button
    set Calendar_year_button to a reference to button 2 of UI element 2 of row 1 of table 1 of group 5
        --> The button that pops up the year menu (as shown)
    set Calendar_year_list_items to a reference to static texts of groups of group 1 of group 1 of UI element 1 of row 1 of table 1 of group 5
        --> The year menu
    set Calendar_month_button to a reference to button 2 of UI element 1 of row 1 of table 1 of group 5
        --> The button that pops up the month menu
    set Calendar_month_list_items to a reference to static texts of groups of group 1 of group 1 of UI element 1 of row 1 of table 1 of group 5
        --> The month menu
    set Calendar_days to a reference to static texts of UI elements 2 thru -1 of rows 3 thru -2 of table 1 of group 5
        --> All the day text objects visible at any one time
        --> 6 rows x 7 days = 42 text objects

The next three variables are for clicking the "Tick" button, which brings up a pop-up menu. Last in this group are the pair of BID/ASK buttons.

    set Tick_button to a reference to static text 1 of group 1 of group 1 of list 2 of group 1 of group 4
        --> The tick button
    set Tick_menu_items to a reference to menu items of list 1
        --> The pop-up menu items as shown below
    set BID_ASK to a reference to checkboxes of group 1 of group 2 of group 4
        --> The "BID" and "ASK" buttons

Next is the download button:

    set Download_button to a reference to button "Download" of group 8 of group 4

Now, when the download button is clicked, the GUI goes under a reconstruction in this bottom section. Firstly, a progress bar is shown in place of all the buttons that temporarily vanish:

I ignored these elements as I didn't see any need to have access to the cancel button in such a short space of time.

Then another reconstruction and the "Save As CVS" and "Reset" buttons come into existence. We need both:

    set Reset_button to a reference to button "Reset" of group 2 of group 4
    set Save_button to a reference to button "Save as .csv" of group 2 of group 2 of group 4

And now we have all the GUI objects we need access to, stored by reference in a set of variables that we (by which I mean you) can make use of to carry out the actual process of automation.

---

Performing Some Actions

Some of the variables contain references to elements that can perform actions straight out of the box, eg click Date_button . However, others will throw an error or not perform quite as you expect them to. This is because some of the variables contain a reference to a single object, such as Date_button , whilst others contains references to several different objects or groups of objects, in a nested list, such as Calendar_year_list_items .

Before utilising these nested lists, we have to flatten them into a single list. The following function will do this, and can be added at the very end of the script (after the end tell :

    on flattened(L) -- recurse nested lists and return a one-dimensional list
        if class of L is not list then
            return [L]
        else if length of L is 0 then
            return L
        else
            return flattened(the first item of L) & ¬
                flattened(the rest of L)
        end if
    end flattened

Now we have everything we need to start getting these objects to perform some actions. For this, I think it's best to write another function. It can just before the on flattened function declaration:

    on Download_CSV_Data for _instrument from _date
        local _instrument, _date
        set [_day, _month, _year] to [day, month, year] of date _date

        global Instruments, Instrument_Abbreviations, Instrument_Descriptions, LOCAL_GMT, ¬
            Date_button, Calendar_year_button, Calendar_year_list_items, Calendar_month_button, ¬
            Calendar_month_list_items, Calendar_days, Tick_button, Tick_menu_items, ¬
            BID_ASK, Download_button, Reset_button, Save_button

        tell application "System Events"

            click 1st item of my flattened(Instrument_Abbreviations whose name is _instrument)
            -- click 1st item of my flattened(Instrument_Descriptions whose name is Instrument's |description|)

            click Tick_button
            click (Tick_menu_items whose name is "Second")
            click (BID_ASK whose name is "ASK")

            click (LOCAL_GMT whose name is "GMT")

            click Date_button
            click Calendar_year_button
            click 1st item of my flattened(Calendar_year_list_items whose name is _year)
            click Calendar_month_button
            click 1st item of my flattened(Calendar_month_list_items whose name is (_month as text))
            click 1st item of my flattened(Calendar_days whose name is _day)

            click Download_button

            repeat while not (exists Reset_button)
                delay 1
            end repeat

            click Save_button
            click Reset_button

        end tell
    end Download_CSV_Data

In order to download a data file pertaining to a specific date, all you have to do is call that function. You would place that function call between the first end tell statement (after the variable declarations), ad before the first function declaration, so your script will look something like this:

    property |URL| : "https://www.dukascopy.com/plugins/fxMarketWatch/detach.php?id=ct23"
    property Instrument : {|short|:"AAPL.US/USD", |description|:"APPLE INC"}
    property |month| : "December"
    property |year| : "2017"


    -- GUI object variable declarations
    tell application "System Events" to ¬
        tell application process "Safari" to ¬
            tell (window named |URL|) to ¬
                tell UI element 1 of scroll area 1 of group 1 of ¬
                    UI element 1 of scroll area 1 of group 1 of ¬
                    group 1 of tab group 1 of splitter group 1
                    .
                    .
                    .
                end tell


    (*** Insert function calls here ***)
    Download_CSV_Data for Instrument's |short| from "17/12/2017"


    on Download_CSV_Data for _instrument from _date
        .
        .
        .
    end Download_CSV_Data

    on flattened(L) -- recurse nested lists and return a one-dimensional list
        .
        .
        .
    end flattened

If you're in America, you might have you switch the month and day around in the short date. So where I've written Download_CSV_Data for Instrument's |short| from "17/12/2017" Download_CSV_Data for Instrument's |short| from "17/12/2017" , you might have to write Download_CSV_Data for Instrument's |short| from "12/17/2017" Download_CSV_Data for Instrument's |short| from "12/17/2017" (use whatever short-date format your computer system uses).