Encountering an error while calling get function using torpy in python

Question

The following code works perfectly, thanks to @AKX

from pprint import pprint

sess = requests.Session()

# Do initial GET request, grab CSRF token
resp = sess.get("https://chartink.com/")
resp.raise_for_status()
csrf_token_m = re.search(r'<meta name="csrf-token" content="(.+?)" />', resp.text)

csrf_token = csrf_token_m.group(1)

# Do data query
resp = sess.post("https://chartink.com/screener/process",
                 data={"scan_clause":"( {cash} ( latest count( 90, 1 where latest ha-low > latest ichimoku cloud top( 9 , 26 , 52 ) ) = 90 ) )"},
                 headers={"Referer": "https://chartink.com/","x-csrf-token": csrf_token,"x-requested-with": "XMLHttpRequest"})
resp.raise_for_status()
data = resp.json()
pprint(data)

But when I try an implement the same using torpy, I get an error message : "AttributeError: '_GeneratorContextManager' object has no attribute 'get'", on this statement resp = sess.get("https://chartink.com/")

req = TorRequests()
sess = req.get_session()

# Do initial GET request, grab CSRF token
resp = sess.get("https://chartink.com/")
resp.raise_for_status()

csrf_token_m = re.search(r'<meta name="csrf-token" content="(.+?)" />', resp.text)
csrf_token = csrf_token_m.group(1)

# Do data query
resp = sess.post("https://chartink.com/screener/process", data={"scan_clause": "( {cash} ( latest count( 90, 1 where latest ha-low > latest ichimoku cloud top( 9 , 26 , 52 ) ) = 90 ) )"},
                                    headers={"Referer": "https://chartink.com/", "x-csrf-token": csrf_token,
                                             "x-requested-with": "XMLHttpRequest"})
resp.raise_for_status()
pprint(resp.json())

Any ideas what's wrong here? Thanks

@AKX As per your suggestion, I changed the code and used the with statement and it worked. But I am not able to understand how it works with "with" and can the code be written without the "with" statement?

Here is the modified code:

with TorRequests() as req:
    with req.get_session() as sess:
        # Do initial GET request, grab CSRF token

        resp = sess.get("https://chartink.com/")
        resp.raise_for_status()

        csrf_token_m = re.search(r'<meta name="csrf-token" content="(.+?)" />', resp.text)
        csrf_token = csrf_token_m.group(1)

        # Do data query
        resp = sess.post("https://chartink.com/screener/process", data={"scan_clause": "( {cash} ( latest count( 90, 1 where latest ha-low > latest ichimoku cloud top( 9 , 26 , 52 ) ) = 90 ) )"},
                                            headers={"Referer": "https://chartink.com/", "x-csrf-token": csrf_token,
                                                     "x-requested-with": "XMLHttpRequest"})
        resp.raise_for_status()
        pprint(resp.json())

Answer 1

You need to use with ... with torpy's TorRequests , since the __enter__() function (which is what with uses) is used to open the Tor connection for the session .

The library does provide a functionally equivalent helper context manager tor_requests_session() , so instead of

with TorRequests() as req:
    with req.get_session() as sess:

you can do

from torpy.requests import tor_requests_session

with tor_requests_session() as sess:

You could write the code without using the with block (a hint as to how is one of the links above), but you really should not. Besides, there's no good reason to - just formulate your program to use functions á la

def get_chartink_data(sess):
    # ...

def main():
    with tor_requests_session() as sess:
        data = get_chartink_data(sess)
    # ...

and you don't really need to care about that with .