解析键，值对的请求响应

Question

I'm storing the response from a POST request to Instagram's API into a text file. What's contained in this response is HTML, which includes an access token I'd like to dig out. The reason it's HTML is because this POST response is really meant to be dealt with by the end user, wherein they click a button and are then provided with the access code. However I need to do this on the backend, hence needing to deal with the HTML response.

In any event, here's my code so far (real client ID is obscured for this post obviously):

OAuthURL = "https://api.instagram.com/oauth/authorize/?client_id=cb0096f08a3848e65f&redirect_uri=https://www.smashboarddashboard.com/whathappened&response_type=code"
OAuth_AccessRequest = requests.post(OAuthURL).text 
#print OAuth_AccessRequest

with open('response.txt', 'w') as OAuthResponse:
        OAuthResponse.write(OAuth_AccessRequest.encode("UTF-8"))

OAuthReady = open('response.txt', 'r')
OAuthView = OAuthReady.read()
print OAuthView 

What I'm left with after this is HTML stored in a text file. Among the HTML however are dictionaries, which I need to access the value, pairs for — some of it, for example, looks like this:

</div> <!-- .root -->

    <script src=//instagramstatic-a.akamaihd.net/bluebar/422f3d9/scripts/polyfills/es5-shim.min.js></script>
<script src=//instagramstatic-a.akamaihd.net/bluebar/422f3d9/scripts/polyfills/es5-sham.min.js></script>
<script type="text/javascript">window._sharedData = {"static_root":"\/\/instagramstatic-a.akamaihd.net\/bluebar\/422f3d9","entry_data":{},"hostname":"instagram.com","platform":{"is_touch":false,"app_platform":"web"},"qe":{"su":false},"display_properties_server_guess":{"viewport_width":360,"pixel_ratio":1.5},"country_code":"US","language_code":"en","gatekeepers":{"tr":false},"config":{"dismiss_app_install_banner_until":null,"viewer":null,"csrf_token":"2aedabf96ad1fe86fab0"},"environment_switcher_visible_server_guess":true};</script>

    </body>
</html>

It's the string of numbers that is the value for the key "csfr_token" that I need to grab. What's the best approach for digging this out of the HTML that's stored in the txt file?

Answer 1

If the csrf_token string is the only such string in the whole page, it'll be trivial to extract it with a regular expression:

import re

token_pattern = re.compile(r'"csrf_token":\s*"([^"]+)"')

token = token_pattern.search(requests.post(OAuthURL).content).group(1)

Note that I used the content attribute of the response, there is no point in decoding the whole response to Unicode when all you need is a few ASCII characters.

Demo:

>>> import requests, re
>>> token_pattern = re.compile(r'"csrf_token":\s*"([^"]+)"')
>>> OAuthURL = "https://api.instagram.com/oauth/authorize/?client_id=cb0096f08a3848e65f&redirect_uri=https://www.smashboarddashboard.com/whathappened&response_type=code"
>>> token_pattern.search(requests.post(OAuthURL).content).group(1)
'3fd6022ac344c3eaea46e87e258ef9c6'

You may want to look at the headers and cookies of the response as well; a CSRF token is usually also set as a cookie (or at the very least as a value in the session).

For this specific request for example, the token is also stored as a cookie, matching the value in the JavaScript block:

>>> r = requests.post(OAuthURL)
>>> r.cookies
<RequestsCookieJar[Cookie(version=0, name='csrftoken', value='b2b621c198642e26a19fc9bf1b38d246', port=None, port_specified=False, domain='instagram.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=1467828030, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]>
>>> r.cookies['csrftoken']
'b2b621c198642e26a19fc9bf1b38d246'
>>> 'b2b621c198642e26a19fc9bf1b38d246' in r.content
True
>>> token_pattern.search(r.content).group(1)
'b2b621c198642e26a19fc9bf1b38d246'