这甚至可能吗？ 将命令/对象从一个 python shell 发送到另一个？

Question

I have a question I wasn't really able to solve after doing a little digging, and this is also not my area of expertise so I really don't even know what I'm looking for.

I'm wondering if it's possible to "link" together two python shells?

This is the actual use case...

I am working with a program that has it's own dedicated python shell built into the GUI. When you run commands in the internal python shell, the GUI updates in real-time reflecting the commands you ran.

The problem is, the scripting environment is terrible. It's basically a textpad next to a shell and is just constant copy and pasting, no serious development could ever really be achieved.

What I want to do is open my IDE (VSCode/Spyder) so I can have a proper environment, but be able to run commands in my IDE that somehow get sent to softwares internal python shell.

Is it possible to somehow detect the open shell in the software and connect/link or make a pipe between the two python instances? So I can pass commands / python objects between the two and basically have the same state of variables in each?

The closest I've come to seeing something like what I want is using the multiprocessing module. Or perhaps socket or pexpect ?

Passing data between separately running Python scripts

How to share variables across scripts in python?

Even if it's just one way communication that might work, just want to be able to use this software in a proper development env.

Honestly I don't really have a clue what I'm doing and hoping for some help here..

Answer 1

TL.DR;

It is a complex request. I don't think can be achieved with a trick. Or you own the process in which is running blender (meaning that you import it's api), or you attach to the process (using gdb, but I don't know if then you can use the IDE you wanted) or you use an IDE with pydevd included. And even so, I don't know how much you can achieve.

Synchronize two python process is not trivial. The answer shows a little of that.

---

PyDev.Debugger

You want to find a way to synchronize two python objects that lives in different python instances. I think that the only real solution to your problem is to setup a pydevd server and connect to it. It is simpler if you use one of the supported IDE, like PyDEV or PyCharm since they have everything in place to do so:

• PyDEV configuration page: https://www.pydev.org/manual_adv_remote_debugger.html
• PyCharm configuration page: https://www.jetbrains.com/help/pycharm/remote-debugging-with-product.html

The work carried out by pydev is not trivial, the repository is quite a project. It is your best bet, but I cannot guarantee you it will work.

---

Process Communication

The usual communication solution will not work because they serialize and deserialize (pickle and unpickle) data behind the scenes. Let's have an example, implementing a server in the blender process that receives arbitrary code as a string and execute it, sending back the last result of the code. The result will be received by the client as a Python object, thus you can use your IDE interface to inspect it, or you can even run some code on it. There are limitations:

• not everything can be received by the client (eg class definition must exist in the client)
• only pickable objects can transit on the connection
• the object in the client and in the server are different: modifications made on the client will not be applied on the server without additional (quite complex) logic

This is the server that should run on your Blender instance

from multiprocessing.connection import Listener
from threading import Thread
import pdb
import traceback

import ast
import copy


# This is your configuration, chose a strong password and only open
# on localhost, because you are opening an arbitrary code execution
# server. It is not much but at least we cover something.
port = 6000
address = ('127.0.0.1', port)
authkey = b'blender'
# If you want to run it from another machine, you must set the address
# to '0.0.0.0' (on Linux, on Windows is not accepted and you have to
# specify the interface that will accept the connection) 


# Credits: https://stackoverflow.com/a/52361938/2319299
# Awesome piece of code, it is a carbon copy from there

def convertExpr2Expression(Expr):
    r"""
    Convert a "subexpression" of a piece of code in an actual
    Expression in order to be handled by eval without a syntax error

    :param Expr: input expression
    :return: an ast.Expression object correctly initialized
    """
    Expr.lineno = 0
    Expr.col_offset = 0
    result = ast.Expression(Expr.value, lineno=0, col_offset=0)
    return result


def exec_with_return(code):
    r"""
    We need an evaluation with return value. The only two function 
    that are available are `eval` and `exec`, where the first evaluates
    an expression, returning the result and the latter evaluates arbitrary code
    but does not return.

    Those two functions intercept the commands coming from the client and checks
    if the last line is an expression. All the code is executed with an `exec`,
    if the last one is an expression (e.g. "a = 10"), then it will return the 
    result of the expression, if it is not an expression (e.g. "import os")
    then it will only `exec` it.

    It is bindend with the global context, thus it saves the variables there.

    :param code: string of code
    :return: object if the last line is an expression, None otherwise
    """
    code_ast = ast.parse(code)
    init_ast = copy.deepcopy(code_ast)
    init_ast.body = code_ast.body[:-1]
    last_ast = copy.deepcopy(code_ast)
    last_ast.body = code_ast.body[-1:]
    exec(compile(init_ast, "<ast>", "exec"), globals())
    if type(last_ast.body[0]) == ast.Expr:
        return eval(compile(convertExpr2Expression(last_ast.body[0]), "<ast>", "eval"), globals())
    else:
        exec(compile(last_ast, "<ast>", "exec"), globals())

# End of carbon copy code


class ArbitraryExecutionServer(Thread):
    r"""
    We create a server execute arbitrary piece of code (the most dangerous
    approach ever, but needed in this case) and it is capable of sending
    python object. There is an important thing to keep in mind. It cannot send
    **not pickable** objects, that probably **include blender objects**!

    This is a dirty server to be used as an example, the only way to close 
    it is by sending the "quit" string on the connection. You can envision
    your stopping approach as you wish

    It is a Thread object, remeber to initialize it and then call the
    start method on it.

    :param address: the tuple with address interface and port
    :param authkey: the connection "password"
    """

    QUIT = "quit" ## This is the string that closes the server

    def __init__(self, address, authkey):
        self.address = address
        self.authkey = authkey
        super().__init__()

    def run(self):
        last_input = ""
        with Listener(self.address, authkey=self.authkey) as server:
            with server.accept() as connection:
                while last_input != self.__class__.QUIT:
                    try:
                        last_input = connection.recv()
                        if last_input != self.__class__.QUIT:
                            result = exec_with_return(last_input) # Evaluating remote input                       
                            connection.send(result)
                    except:
                        # In case of an error we return a formatted string of the exception
                        # as a little plus to understand what's happening
                        connection.send(traceback.format_exc())


if __name__ == "__main__":
    server = ArbitraryExecutionServer(address, authkey)
    server.start() # You have to start the server thread
    pdb.set_trace() # I'm using a set_trace to get a repl in the server.
                    # You can start to interact with the server via the client
    server.join() # Remember to join the thread at the end, by sending quit

While this is the client in your VSCode

import time
from multiprocessing.connection import Client


# This is your configuration, should be coherent with 
# the one on the server to allow the connection
port = 6000
address = ('127.0.0.1', port)
authkey = b'blender'


class ArbitraryExecutionClient:
    QUIT = "quit"

    def __init__(self, address, authkey):
        self.address = address
        self.authkey = authkey
        self.connection = Client(address, authkey=authkey)

    def close(self):
        self.connection.send(self.__class__.QUIT)
        time.sleep(0.5)  # Gives some time before cutting connection
        self.connection.close()

    def send(self, code):
        r"""
        Run an arbitrary piece of code on the server. If the
        last line is an expression a python object will be returned.
        Otherwise nothing is returned
        """
        code = str(code)
        self.connection.send(code)
        result = self.connection.recv()
        return result

    def repl(self):
        r"""
        Run code in a repl loop fashion until user enter "quit". Closing
        the repl will not close the connection. It must be manually 
        closed.
        """
        last_input = ""
        last_result = None
        while last_input != self.__class__.QUIT:
            last_input = input("REMOTE >>> ")
            if last_input != self.__class__.QUIT:
                last_result = self.send(last_input)
                print(last_result)
        return last_result


if __name__ == "__main__":
    client = ArbitraryExecutionClient(address, authkey)
    import pdb; pdb.set_trace()
    client.close()

At the bottom of the script there is also how to launch them while having pdb as "repl". With this configuration you can run arbitrary code from the client on the server (and in fact it is an extremely dangerous scenario , but for your very specific situation is valid, or better "the main requirement").

Let's dive into the limitation I anticipated.

You can define a class Foo on the server:

[client] >>> client = ArbitraryExecutionClient(address, authkey)
[client] >>> client.send("class Foo: pass")

[server] >>> Foo
[server] <class '__main__.Foo'>

and you can define an object named "foo" onthe server, but you will immediately receive an error because the class Foo does not exist in the local instance (this time using the repl):

[client] >>> client.repl()
[client] REMOTE >>> foo = Foo()
[client] None
[client] REMOTE >>> foo
[client] *** AttributeError: Can't get attribute 'Foo' on <module '__main__' from 'client.py'>

this error appears because there is no declaration of the Foo class in the local instance, thus there is no way to correctly unpickle the received object (this problem will appear with all the Blender objects. Take notice, if the object is in some way importable, it may still work, we will see later on this situation).

The only way to not receive the error is to previously declare the class also on the client, but they will not be the same object, as you can see by looking at their ids:

[client] >>> class Foo: pass
[client] >>> client.send("foo")
[client] <__main__.Foo object at 0x0000021E2F2F3488>

[server] >>> foo
[server] <__main__.Foo object at 0x00000203AE425308>

Their id are different because they live in a different memory space: they are completely different instances, and you have to manually synchronize every operation on them!

If the class definition is in some way importable and the object are pickable, you can avoid to multiply the class definition, as far as I can see it will be automatically imported:

[client] >>> client.repl()
[client] REMOTE >>> import numpy as np
[client] None
[client] REMOTE >>> ary = np.array([1, 2, 3])
[client] None
[client] REMOTE >>> ary
[client] [1 2 3]
[client] REMOTE >>> quit
[client] array([1, 2, 3])
[client] >>> ary = client.send("ary")
[client] >>> ary
[client] array([1, 2, 3])
[client] >>> type(ary)
[client] <class 'numpy.ndarray'>

We never imported on the client numpy but we have correctly received the object. But what happen if we modify the local instance to the remote instance?

[client] >>> ary[0] = 10
[client] >>> ary
[client] array([10,  2,  3])
[client] >>> client.send("ary")
[client] array([1, 2, 3])

[server] >>> ary
[server] array([1, 2, 3])

we have no synchronization of the modifications inside the object.

What happens if an object is not pickable? We can test with the server variable, an object that is a Thread and contains a connection, which are both non pickable (meaning that you cannot give them an invertible representation as a list of bytes):

[server] >>> import pickle
[server] >>> pickle.dumps(server)
[server] *** TypeError: can't pickle _thread.lock objects

and also we can see the error on the client, trying to receive it:

[client] >>> client.send("server")
[client] ... traceback for "TypeError: can't pickle _thread.lock objects" exception ...

I don't think there is a "simple" solution for this problem, but I think there is some library (like pydevd ) that implements a full protocol for overcoming this problem.

I hope now my comments are more clear.

Answer 2

Here is all the pieces put together!

• Multithreading so that both the file processing system and the Python interactive shell can work at the same time.
• Variables are updated between the interactive shell and the file. In other words, the file and the interactive shell shares variables, functions, classes, etc.
• Instant update between shell and file.

import threading
import platform
import textwrap
import traceback
import hashlib
import runpy
import code
import time
import sys
import os


def clear_console():
    """ Clear your console depending on OS. """

    if platform.system() == "Windows":
        os.system("cls")
    elif platform.system() in ("Darwin", "Linux"):
        os.system("clear")


def get_file_md5(file_name):
    """ Grabs the md5 hash of the file. """

    with open(file_name, "rb") as f:
        return hashlib.md5(f.read()).hexdigest()


def track_file(file_name, one_way=False):
    """ Process external file. """

    # Grabs current md5 of file.
    md5 = get_file_md5(file_name)

    # Flag for the first run.
    first_run = True

    # If the event is set, thread gracefully quits by exiting loop.
    while not event_close_thread.is_set():

        time.sleep(0.1)

        # Gets updated (if any) md5 hash of file.
        md5_current = get_file_md5(file_name)
        if md5 != md5_current or first_run:
            md5 = md5_current

            # Executes the content of the file.
            try:
                # Gather the threads global scope to update the main thread's scope.
                thread_scope = runpy.run_path(file_name, init_globals=globals())

                if not one_way:
                    # Updates main thread's scope with other thread..
                    globals().update(thread_scope)

                # Prints updated only after first run.
                if not first_run:
                    print(f'\n{"="*20} File {file_name} updated! {"="*20}\n>>> ', end="")
                else:
                    first_run = False

            except:
                print(
                    f'\n{"="*20} File {file_name} threw error! {"="*20}\n {traceback.format_exc()}\n>>> ',
                    end="",
                )


def track(file_name):
    """ Initializes tracking thread (must be started with .start()). """

    print(f'{"="*20} File {file_name} being tracked! {"="*20}')
    return threading.Thread(target=track_file, args=(file_name,)).start()


if __name__ == "__main__":
    clear_console()

    # Creates a thread event for garbage collection, and file lock.
    event_close_thread = threading.Event()

    banner = textwrap.dedent(
        f"""\
        {"="*20} Entering Inception Shell {"="*20}\n
        This shell allows the sharing of the global scope between
        Python files and the Python interactive shell. To use:

        \t >>> track("script.py", one_way=False)

        On update of the file 'script.py' this shell will execute the
        file (passing the shells global variables to it), and then, if
        one_way is False, update its own global variables to that of the
        file's execution.
        """
    )

    # Begins interactive shell.
    code.interact(banner=banner, readfunc=None, local=globals(), exitmsg="")

    # Gracefully exits the thread.
    event_close_thread.set()

    # Exits shell.
    print(f'\n{"="*20} Exiting Inception Shell {"="*20}')
    exit()

One liner:

exec("""\nimport threading\nimport platform\nimport textwrap\nimport traceback\nimport hashlib\nimport runpy\nimport code\nimport time\nimport sys\nimport os\n\n\ndef clear_console():\n    \"\"\" Clear your console depending on OS. \"\"\"\n\n    if platform.system() == "Windows":\n        os.system("cls")\n    elif platform.system() in ("Darwin", "Linux"):\n        os.system("clear")\n\n\ndef get_file_md5(file_name):\n    \"\"\" Grabs the md5 hash of the file. \"\"\"\n\n    with open(file_name, "rb") as f:\n        return hashlib.md5(f.read()).hexdigest()\n\n\ndef track_file(file_name, one_way=False):\n    \"\"\" Process external file. \"\"\"\n\n    # Grabs current md5 of file.\n    md5 = get_file_md5(file_name)\n\n    # Flag for the first run.\n    first_run = True\n\n    # If the event is set, thread gracefully quits by exiting loop.\n    while not event_close_thread.is_set():\n\n        time.sleep(0.1)\n\n        # Gets updated (if any) md5 hash of file.\n        md5_current = get_file_md5(file_name)\n        if md5 != md5_current or first_run:\n            md5 = md5_current\n\n            # Executes the content of the file.\n            try:\n                # Gather the threads global scope to update the main thread's scope.\n                thread_scope = runpy.run_path(file_name, init_globals=globals())\n\n                if not one_way:\n                    # Updates main thread's scope with other thread..\n                    globals().update(thread_scope)\n\n                # Prints updated only after first run.\n                if not first_run:\n                    print(f'\\n{"="*20} File {file_name} updated! {"="*20}\\n>>> ', end="")\n                else:\n                    first_run = False\n\n            except:\n                print(\n                    f'\\n{"="*20} File {file_name} threw error! {"="*20}\\n {traceback.format_exc()}\\n>>> ',\n                    end="",\n                )\n\n\ndef track(file_name):\n    \"\"\" Initializes tracking thread (must be started with .start()). \"\"\"\n\n    print(f'{"="*20} File {file_name} being tracked! {"="*20}')\n    return threading.Thread(target=track_file, args=(file_name,)).start()\n\n\nif __name__ == "__main__":\n    clear_console()\n\n    # Creates a thread event for garbage collection, and file lock.\n    event_close_thread = threading.Event()\n\n    banner = textwrap.dedent(\n        f\"\"\"\\\n        {"="*20} Entering Inception Shell {"="*20}\\n\n        This shell allows the sharing of the global scope between\n        Python files and the Python interactive shell. To use:\n\n        \\t >>> track("script.py", one_way=False)\n\n        On update of the file 'script.py' this shell will execute the\n        file (passing the shells global variables to it), and then, if\n        one_way is False, update its own global variables to that of the\n        file's execution.\n        \"\"\"\n    )\n\n    # Begins interactive shell.\n    code.interact(banner=banner, readfunc=None, local=globals(), exitmsg="")\n\n    # Gracefully exits the thread.\n    event_close_thread.set()\n\n    # Exits shell.\n    print(f'\\n{"="*20} Exiting Inception Shell {"="*20}')\n    exit()\n""")

---

Try the following for your Blender shell:

import threading
import traceback
import hashlib
import runpy
import time


def get_file_md5(file_name):
    """ Grabs the md5 hash of the file. """

    with open(file_name, "rb") as f:
        return hashlib.md5(f.read()).hexdigest()


def track_file(file_name, one_way=False):
    """ Process external file. """

    # Grabs current md5 of file.
    md5 = get_file_md5(file_name)

    # Flag for the first run.
    first_run = True

    # If the event is set, thread gracefully quits by exiting loop.
    while not event_close_thread.is_set():

        time.sleep(0.1)

        # Gets updated (if any) md5 hash of file.
        md5_current = get_file_md5(file_name)
        if md5 != md5_current or first_run:
            md5 = md5_current

            # Executes the content of the file.
            try:
                # Gather the threads global scope to update the main thread's scope.
                thread_scope = runpy.run_path(file_name, init_globals=globals())

                if not one_way:
                    # Updates main thread's scope with other thread..
                    globals().update(thread_scope)

                # Prints updated only after first run.
                if not first_run:
                    print(
                        f'\n{"="*20} File {file_name} updated! {"="*20}\n>>> ', end=""
                    )
                else:
                    first_run = False

            except:
                print(
                    f'\n{"="*20} File {file_name} threw error! {"="*20}\n {traceback.format_exc()}\n>>> ',
                    end="",
                )


def track(file_name):
    """ Initializes tracking thread (must be started with .start()). """

    print(f'{"="*20} File {file_name} being tracked! {"="*20}')
    return threading.Thread(target=track_file, args=(file_name,)).start()


if __name__ == "__main__":
    # Creates a thread event for garbage collection, and file lock.
    event_close_thread = threading.Event()

    # Gracefully exits the thread.
    event_close_thread.set()

One liner:

exec("""\nimport threading\nimport traceback\nimport hashlib\nimport runpy\nimport time\n\n\ndef get_file_md5(file_name):\n    \"\"\" Grabs the md5 hash of the file. \"\"\"\n\n    with open(file_name, "rb") as f:\n        return hashlib.md5(f.read()).hexdigest()\n\n\ndef track_file(file_name, one_way=False):\n    \"\"\" Process external file. \"\"\"\n\n    # Grabs current md5 of file.\n    md5 = get_file_md5(file_name)\n\n    # Flag for the first run.\n    first_run = True\n\n    # If the event is set, thread gracefully quits by exiting loop.\n    while not event_close_thread.is_set():\n\n        time.sleep(0.1)\n\n        # Gets updated (if any) md5 hash of file.\n        md5_current = get_file_md5(file_name)\n        if md5 != md5_current or first_run:\n            md5 = md5_current\n\n            # Executes the content of the file.\n            try:\n                # Gather the threads global scope to update the main thread's scope.\n                thread_scope = runpy.run_path(file_name, init_globals=globals())\n\n                if not one_way:\n                    # Updates main thread's scope with other thread..\n                    globals().update(thread_scope)\n\n                # Prints updated only after first run.\n                if not first_run:\n                    print(\n                        f'\\n{"="*20} File {file_name} updated! {"="*20}\\n>>> ', end=""\n                    )\n                else:\n                    first_run = False\n\n            except:\n                print(\n                    f'\\n{"="*20} File {file_name} threw error! {"="*20}\\n {traceback.format_exc()}\\n>>> ',\n                    end="",\n                )\n\n\ndef track(file_name):\n    \"\"\" Initializes tracking thread (must be started with .start()). \"\"\"\n\n    print(f'{"="*20} File {file_name} being tracked! {"="*20}')\n    return threading.Thread(target=track_file, args=(file_name,)).start()\n\n\nif __name__ == "__main__":\n    # Creates a thread event for garbage collection, and file lock.\n    event_close_thread = threading.Event()\n\n    # Gracefully exits the thread.\n    event_close_thread.set()\n""")