How to split Javascript code (bs4) using Python

Question

So I have been having some issues when trying to scrape Javascript values from a bs4 code.

Basically the javascript looks like

<script type="text/javascript">
var FancyboxI18nClose = 'Close';
var FancyboxI18nNext = 'Next';
var FancyboxI18nPrev = 'Previous';
var PS_CATALOG_MODE = false;
var ajaxsearch = true;
var attribute_anchor_separator = '-';
var blocksearch_type = 'top';
var combinationsFromController = {"163972":{"attributes_values":{"15":"40"},"attributes":[75],"price":0,"specific_price":false,"ecotax":0,"weight":0.6,"quantity":1,"reference":"IDP20059--IDPA163972","unit_impact":0,"minimal_quantity":"1","date_formatted":"","available_date":"","id_image":-1,"list":"'75'"}};
var comparator_max_item = 0;
</script>

and what I am trying to do here is to scrape the value var combinationsFromController = however what I tried to do is:

bs4 = soup(requests.text, 'html.parser')

for nosto_sku_tag in bs4.find_all('script', {'type': 'text/javascript'}):
    if 'combinationsFromController' in nosto_sku_tag.text.strip():
        print(nosto_sku_tag)
        for att, values in json.loads(
                re.findall('var combinationsFromController = (\{.*}?);', nosto_sku_tag.text.strip())[0][:-1]).values():
            print(values)

Which gives me an error of Expecting ',' delimiter: line 1 column 4112 (char 4111)

I did realized that whenever I try to do

for nosto_sku_tag in bs4.find_all('script', {'type': 'text/javascript'}):
    if 'combinationsFromController' in nosto_sku_tag.text.strip():
        print(nosto_sku_tag)
        print("---------")

The outprint gives me:

var FancyboxI18nClose = 'Close';
var FancyboxI18nNext = 'Next';
var FancyboxI18nPrev = 'Previous';
var PS_CATALOG_MODE = false;
var ajaxsearch = true;
var attribute_anchor_separator = '-';
var blocksearch_type = 'top';
var combinationsFromController = {"163972":{"attributes_values":{"15":"40"},"attributes":[75],"price":0,"specific_price":false,"ecotax":0,"weight":0.6,"quantity":1,"reference":"IDP20059--IDPA163972","unit_impact":0,"minimal_quantity":"1","date_formatted":"","available_date":"","id_image":-1,"list":"'75'"}};
var comparator_max_item = 0;
----------------------------

Which seems to mean that the javascript code is as one code which I believe maybe needs to split, However I tried to use regex for it but it didn't help me.

So my question is how am I able to scrape ONLY the value var combinationsFromController = ?

Answer 1

Use the following regex pattern to isolate the entire javascript object which is assigned to that variable.

combinationsFromController = (.*?);

Try it here .

Eg

import requests, re, json

r = requests.get(url)
p = re.compile(r'combinationsFromController = (.*?);', re.DOTALL)
data = json.loads(p.findall(r.text)[0])

---