Can we login on a website with pure Python urllib, modify some form values, and submit? (without Selenium browser automation)

Question

Is it possible to make a script that automatically:

1. Connects to the PayPal HTTPS website
2. Logs in with username + password
3. Goes to a specific URL
4. Modifies a few values in the dropdown lists
5. Clicks on a button

All of this from Python with standard methods like urllib.request ?

import urllib.request; s = urllib.request.urlopen('https://example.com').read().decode()

Or because of the login involved, do we have to do it by "automating" or "faking" a real browser, using Selenium methods / or other methods à la PhantomJS ? (I'd rather not have to do that, if possible)

---

Context : I've looked into various places and asked their support by email + phone, but it seems that PayPal doesn't offer access to an API to trigger the generation and download of a CSV report of all transactions between eg 01/01/2020 and 31/01/2020 ... except if you have > 20k sales per month and in this case you have access to their Reporting API + reports on a FTP server .

If this is not the case this Reporting API is not available, and you have to manually log in, go to the page https://business.paypal.com/merchantdata/reportHome?reportType=DLOG (see screenshot above), click in various lists, click Create report , wait a few minutes/hours, re log-in, go back on this page, and finally click on Download . This is a boring task to do each month (and their website is rather slow in general) that I would like to automate.

Answer 1

The correct link to SFTP reports is: https://developer.paypal.com/docs/reports/sftp-reports/

And those are all stricly daily report files, not monthly or any other time period. You'd have to read and parse in each daily file to cover the time period you desire.

---

It is theoretically possible to emulate the PayPal website UI with something like urllib or mechanize.Browser and automate requesting an activity log file and downloading it. But that's a fair amount of work for something you're only going to be doing once a month, and may break when PayPal changes their site. The login portion is particularly tricky to automate, since PayPal can require additional verification beyond just a password.

If you are going to automate some portion of this, I would recommend only doing so inside the browser (Tampermonkey, Selenium, etc; note that Tampermonkey blacklists paypal.tld by default, for your own protection against importing a malicious script)