Python script running curl commands fail with an 500 Internal error

Question

I am running the below code in Google Colab and get The server encountered an internal error or misconfiguration and was unable to complete your request . If I am running the command without passing in the variable $data like below, it runs perfectly fine. Only when I'm looping through the file and passing variables it seems to be failing

import pandas as pd
import requests
import csv
import json

reader = csv.reader(open('/content/drive/MyDrive/file5.csv'))
for row in reader:
    data = {"snps": row[0], "pop": "YRI", "r2_threshold": "0.9", "maf_threshold": "0.01"}
    data = json.dumps(data)
    data = "'{}'".format(data)
    !curl -k -H "Content-Type: application/json" -X POST -d "$data" 'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'

This works:

!curl -k -H "Content-Type: application/json" -X POST -d '{"snps": "rs3\nrs4", "pop":"YRI", "r2_threshold": "0.1", "maf_threshold": "0.01"}' 'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'

Answer 1

The ! escape tells your notebook (Google Collab, Jupyter, or what have you; basically whatever is running ipython as a kernel or similar) to leave Python and run a shell command. Python itself has no support for this; the closest approximation would be something like

import subprocess
...
for row in reader:
    data = {"snps": row[0], "pop": "YRI", "r2_threshold": "0.9", "maf_threshold": "0.01"}
    data = json.dumps(data)
    # This was wrong on so many levels
    # data = "'{}'".format(data)
    subprocess.run(['curl', '-k',
        '-H', "Content-Type: application/json",
        '-X', 'POST', '-d', data,
        'https://ldlink.nci.nih.gov/LDlinkRest/snpclip?token=e3e559472899'],
        text=True, check=True)

though avoiding subprocess and running Python urllib or requests code to perform the POST would be more efficient and elegant, and give you more control over what gets sent and how it gets handled.

How to properly quote strings when translating between shell commands and Python requires you to understand the shell's quoting behavior. I'll just briefly note that I left double quotes where theiy were not incorrect in your original comman, but otherwise preferred single quotes, and of course, data now refers to a proper Python variable with that name, not a shell variable with the same name.

To reiterate: ipython (which is what your notebook is an interface to) knows how to run both Python code and shell scipt code via ! ; but once you ask it to run Python code, ipython hands it over to Python proper, and you are no longer in ipython .