Parsing inline python command via linux using pdsh

Question

So, I am trying to issue this command from a python script that collects cpu information across a predetermined number of nodes in a cluster. Here I use a fanout of 2 and only run it on nodes b127 through b129 for testing purposes.

    pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
    num_cpu = multiprocessing.cpu_count()
    stat_fd = open('/proc/stat')
    stat_fd.close()"'

I printed the command and this is what it shows on the terminal. Thus, telling me that the quotes and commands are properly formatted. I get this string by executing the following code:

    python_command = "'python -c "\
                 + "\"import multiprocessing\n"\
                 + "num_cpu = multiprocessing.cpu_count()\n"\
                 + "stat_fd = open(\'/proc/stat\')\n"\
                 + "stat_fd.close()\"'"

    command = "pdsh -f2 -w b[127-129] " + python_command
    print command

Unfortunately, the line with open(\\'/proc/stat\\') seems to be the problem as that is the only line that causes the parser to fail due to the nested single quotes. I've tried numerous combinations of quoting and escaping in order to make it work to no avail. I've omitted some code between the opening and closing of the file to minimize the posted code.

I searched around and found this link , but found it was too simple of an example because I could replicate those commands. And yes, I know I can use bash commands to get what I want done and I may end up doing so, but this one has me beating my head on the wall. I also have scripts that gather data using top and ps so I don't need an explanation using them. I greatly appreciate the help in advance.

Answer 1

Try this:

python_command = """pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open(\\"/proc/stat\\")
stat_fd.close()"'"""

In Python, you can use triple quotes ( """...""" or '''...''' ) for strings containing new lines and single/double quotes.

The last level of quotes (on the open() line) will need to be escaped so that they don't conflict with outer quotes. You also need to escape the backslashes so they aren't immediately consumed by Python when interpreting the string: \\\\" .