IPC between VBA and Python

Question

I am facing the following problem: In our company we use a software whose GUI was programmed with MS Access/VBA. Now a part of the business logic should be moved to Python, but the MS Access part should remain. The following scenario is now implemented and works: The user types in a string in Access, this string is read out in VBA and by means of a command line the Python script is called and the string is given to the script as a command line argument. Python in turn connects to a vendor's database, uses the passed string as a parameter, and stores the result in our MS SQL database. The vendor offers a Python API for its database, hence the necessary intermediate step via Python. This happens several times a day and starting the script or interpreter takes around 3 seconds each time. This takes too long. The following is not desired: convert Python scripts to a web server or reprogram the GUI using Python.

Sub CallPython()


Dim PythonExe As String, PythonScript As String, PythonArgs As String, PythonOutput As String
Dim PythonCommand As String
Dim objShell As Object

PythonExe = """C:\Program Files\Python37\python.exe"""
PythonScript = """[path_to_our_script]\insert_article.py"""
PythonArgs = "-id 123456"

Set objShell = VBA.CreateObject("Wscript.Shell")

PythonCommand = PythonExe & " " & PythonScript & " " & PythonArgs 
'MsgBox PythonCommand
objShell.Run PythonCommand


End Sub

I have seen the following page regarding IPC techniques, but I don't have much experience in this area, so I can't say much about the complexity. Does anyone have experience with the above scenario and can share their knowledge on smarter solutions?

Answer 1

This subject is really very broad and complex.

I've done this myself using bidirectional direct communication between R and Access via named pipes, which are handled very similar to files on the Python (or R) side. However, the Access side needs many API declarations to set up the pipes, and for my case, peek at progress so we can asynchronously report progress without locking up the application.

The basics on named pipes can be found here:

https://docs.microsoft.com/en-us/windows/win32/ipc/multithreaded-pipe-server

and here:

https://docs.microsoft.com/en-us/windows/win32/ipc/named-pipe-client

Named pipe servers can be single-threaded if there's only one client (your Python application) so you can ignore most of the multithreading stuff.

The declarations I needed in VBA are:

Private Type SECURITY_ATTRIBUTES
    nLength As Long
    lpSecurityDescriptor As LongPtr
    bInheritHandle As Long
End Type
 
Private Type PROCESS_INFORMATION
    hProcess As LongPtr
    hThread As LongPtr
    dwProcessId As Long
    dwThreadId As Long
End Type
 
Private Type STARTUPINFO
    cb As Long
    lpReserved As LongPtr
    lpDesktop As LongPtr
    lpTitle As LongPtr
    dwX As Long
    dwY As Long
    dwXSize As Long
    dwYSize As Long
    dwXCountChars As Long
    dwYCountChars As Long
    dwFillAttribute As Long
    dwFlags As Long
    wShowWindow As Integer
    cbReserved2 As Integer
    lpReserved2 As LongPtr
    hStdInput As LongPtr
    hStdOutput As LongPtr
    hStdError As LongPtr
End Type
 
Private Const STARTF_USESHOWWINDOW  As Long = &H1
Private Const STARTF_USESTDHANDLES  As Long = &H100
Private Const SW_HIDE               As Long = 0&
Private Const ERROR_SUCCESS As Long = 0
Private Const STILL_ACTIVE As Long = 259
Private Const PIPE_TYPE_BYTE As Long = 0
Private Const PIPE_ACCESS_INBOUND = 1
Private Const PIPE_ACCESS_OUTBOUND = 2
Private Const PIPE_ACCESS_DUPLEX As Long = 3
Private Const PIPE_WAIT As Long = 0
Private Const PIPE_NOWAIT As Long = 1
Private Const PIPE_ACCEPT_REMOTE_CLIENTS As Long = 0
Private Const ERROR_PIPE_CONNECTED = 535
Private Const ERROR_PIPE_LISTENING = 536

Private Declare PtrSafe Function CreatePipe Lib "kernel32" (ByRef hReadPipe As LongPtr, ByRef hWritePipe As LongPtr, ByVal lpPipeAttributes As LongPtr, ByVal nSize As Long) As Long
Private Declare PtrSafe Function CreateProcess Lib "kernel32" Alias "CreateProcessW" (ByVal lpApplicationName As LongPtr, ByVal lpCommandLine As LongPtr, ByVal lpProcessAttributes As LongPtr, ByVal lpThreadAttributes As LongPtr, ByVal bInheritHandles As Long, ByVal dwCreationFlags As Long, ByVal lpEnvironment As LongPtr, ByVal lpCurrentDirectory As LongPtr, lpStartupInfo As STARTUPINFO, lpProcessInformation As PROCESS_INFORMATION) As Long
Private Declare PtrSafe Function ReadFile Lib "kernel32" (ByVal hFile As LongPtr, lpBuffer As Any, ByVal nNumberOfBytesToRead As Long, lpNumberOfBytesRead As Long, ByVal lpOverlapped As LongPtr) As Long
Private Declare PtrSafe Function WriteFile Lib "kernel32" (ByVal hFile As LongPtr, lpBuffer As Any, ByVal nNumberOfBytesToWrite As Long, ByRef nNumberOfBytesWritten As Long, ByVal lpOverlapped As LongPtr) As Long
Private Declare PtrSafe Function FlushFileBuffers Lib "kernel32" (ByVal hFile As LongPtr) As Long
Private Declare PtrSafe Function CloseHandle Lib "kernel32" (ByVal hObject As LongPtr) As Long

Private Declare PtrSafe Function GetExitCodeProcess Lib "kernel32" (ByVal hProcess As LongPtr, ByRef lpExitCode As Long) As Long
Private Declare PtrSafe Function TerminateProcess Lib "kernel32" (ByVal hProcess As LongPtr, ByVal uExitCode As Long) As Long

Private Declare PtrSafe Function CreateNamedPipeW Lib "kernel32" (ByVal lpName As LongPtr, ByVal dwOpenMode As Long, ByVal dwPipeMode As Long, ByVal nMaxInstances As Long, ByVal nOutBufferSize As Long, ByVal nInBufferSize As Long, ByVal nDefaultTimeOut As Long, lpSecurityAttributes As Any) As LongPtr
Private Declare PtrSafe Function ConnectNamedPipe Lib "kernel32" (ByVal hNamedPipe As LongPtr, lpOverlapped As Any) As Long
Private Declare PtrSafe Function DisconnectNamedPipe Lib "kernel32" (ByVal hNamedPipe As LongPtr) As Long
Private Declare PtrSafe Function PeekNamedPipe Lib "kernel32" (ByVal hNamedPipe As LongPtr, lpBuffer As Any, ByVal nBufferSize As Long, ByRef lpBytesRead As Long, ByRef lpTotalBytesAvail As Long, ByRef lpBytesLeftThisMessage As Long) As Long

And the essential stuff on the VBA part is:

1. Create a named pipe via CreateNamedPipeW with PIPE_ACCESS_OUTBOUND (or two, if you want input and output, one in and one out)
2. Spawn a listener process (your Python process) via CreateProcess so you can get its ID
3. When sending a command, check if the process is alive via GetExitCode , connect to the pipe via ConnectNamedPipe , write to the pipe using WriteFile , then FlushFileBuffers , release the file handle via CloseHandle , then disconnect from the pipe using DisconnectNamedPipe

From the Python process, in a loop, open up the pipe via open , read and process the message, then open up again via open . open should halt until the next message is sent.

And if you want to use return messages for progress, make sure to use PeekNamedPipe to not halt your Access application if Python is slow or has encountered an error.

You'd probably want to wrap all this in a predeclared self-healing VBA class to keep your Python program active while VBA is active and not have to wait for Python to start up/read your program/etc., all nontrivial stuff.

In the end, it's much simpler to just use local http since you can use pre-existing tools for sending and receiving http requests. But direct IPC between VBA and Python/R/any programming languages that can read named pipes (= read files) can be done.

This, unfortunately, is as far as I can take you. Really, reconsider "not webserver". If the processes are on the same machine the webserver can be firewalled, going for a webserver has some more overhead over named pipes but it's a lot easier. I kinda wish I went that route.