Extract data from string using Scrapy

Question

<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">
{"InstrumentID":85,"BuyPrice":24677.0,"SellPrice":24671.0,"HighPrice":24671.0,"LowPrice":24212.0,"ChangePercent":2.1,"ChangePercentText":"2.10%","UsersBuyPercentage":56.0,"UsersSellPercentage":44.0,"IsValid":true}
</string>

I need to extract BuyPrice, SellPrice with Scrapy, but I don't know how. Could someone help?

Answer 1

Looks like you have json inside xml, so extracting data will be a 2-part task:

1. Extract the json string
2. Load the information you need using the json module

An example of how this might be done (using scrapy shell here):

>>> import json
>>> sel = scrapy.Selector(text='''<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">
... {"InstrumentID":85,"BuyPrice":24677.0,"SellPrice":24671.0,"HighPrice":24671.0,"LowPrice":24212.0,"ChangePercent":2.1,"ChangePercentText":"2.10%","UsersBuyPercentage":56.0,"UsersSellPercentage":44.0,"IsValid":true}
... </string>''')
>>> sel.remove_namespaces()
>>> json.loads(sel.xpath('//string/text()').get())
{'InstrumentID': 85, 'BuyPrice': 24677.0, 'SellPrice': 24671.0, 'HighPrice': 24671.0, 'LowPrice': 24212.0, 'ChangePercent': 2.1, 'ChangePercentText': '2.10%', 'UsersBuyPercentage': 56.0, 'UsersSellPercentage': 44.0, 'IsValid': True}