FileNotFoundError: [WinError 2] Using Microsoft BCP Utility with Python Subprocess

Question

I want to load a text file (TSV, tab-separated values) to SQL Server using Microsoft's Bulk Copy Program (BCP) utility. I want to call BCP in a python script via python's subprocess module.

Here's the command I'm entering:

subprocess.check_output(['bcp', 'TEST_load_hh20220818', 'in', 'C:\\Users\\xxxx\\some-folder\\_household.tsv', '-S', 'SQL-SVR', '-d', 'MTP2024', '-T', '-q', '-c', '-t', '\\t', '-F', '2'])

But when I run it, I get the error:

*** FileNotFoundError: [WinError 2] The system cannot find the file specified

So far I tried the following:

• Confirmed that the file path is correct by running os.path.exists('C:\\Users\\xxxx\\some-folder\\_household.tsv') and also successfully loading the file into a Pandas dataframe--again, just to confirm that the file path is correct.

• Successfully ran the script on three other computers , making me suspect it is a machine issue and not a syntax/script error. All machines that I tested on are running Windows 10.

• Tried adding shell=True to the subprocess.check_output() parameters, as suggested on this thread . This did not fix the issue.

Any idea what's going on?

Answer 1

The path is likely not valid. You mentioned that you verified the path, but hear me out.

When you execute your python command to verify the path, the command is running on the CPU of the same machine you executed the command on. This is likely a separate/distributed machine/CPU than where your SQL Server is running. Is SQL Server running on a different computer than where you run your python command?

If so, when your python script executes the BCP command, the BCP command is sent TO the computer that SQL Server is running on (the SQL Server named with the -S option).

Now, SQL Server has taken that command and is looking at its LOCAL C: drive for the path you've given it. But it's likely that this path is not valid ON the SQL Server. That path only exists on your local computer (workstation?).

To do something like this (build a BCP command that is sent to a remote or distributed SQL Server) you need to use full UNC paths that the SQL Server can see (or wrestle with drive mappings, but that's ugly). Also, the account that SQL Server is running as will need to have access to whatever UNC path you place the file in.