How do I retrieve a variable's content using Python Requests module?

Question

I started a little personal project just for fun. I hope posting what I'm doing here doesn't break any of the local rules. If so, let me know and I'll take the question down. No need to flag me for anything

I'm trying to pull the background image URL of my chromium homepage. Just for reference, the URL is https://www.mystart.com/new-tab/newtab/ When going to this page, nice background images are loaded. I'm trying to grab those images for personal, not commercial, use.

What I've traced down is that the page listed above calls out to another similar page: https://www.mystart.com/new-tab/newtab/newtab/ Currently, on line #1622 through #1636, two significant lines read:

var fastload    = JSON.parse(localStorage.getItem('FASTLOAD_WALLPAPER_557b2c52a6fde1413ac3a48a'))
...
var url = fastload.info.cache_url || fastload.info.data_uri || fastload.info.image;

The value returned in the url is the URL to the background image. If I drop into the Chromium console and use: console.log(url) , I see the exact data I'm trying to scrape. I'm wondering how I do that through python, since the actual textValue of url is not seen.

I have looked all over to try to find the localStorage object definition with no luck. I'm pulling the page with result = requests.get("https://www.mystart.com/new-tab/newtab/newtab/"); and then looking through result.text . I've also tried using BeautifulSoup to parse through things, not that this is really any different, but still not getting the results I'm looking for.

Being that I'm a hobbyist coder, I feel like I'm missing something simple. I've searched for answers, but I must be using the wrong keywords. I'm finding a lot of answers for parsing the urls that can be read, but not from the contents of a variable.

Answer 1

if you look at the requests being made, there is JSON response with info for 350 images. image_id is used in the url, eg https://gallery.mystartcdn.com/mystart/images/<image_id>.jpeg so for id=154_david-wilson-moab :

https://gallery.mystartcdn.com/mystart/images/154_david-wilson-moab.jpeg

Parse the JSON and get url for all images.

Note: this is not an answer of your question, but it looks like XY problem - this solves the underlying problem of retrieving image urls.