script works in Jupyter notebook, fails from command line or Atom IDE

Question

I have python installed via Anaconda, and want to serialize a rather large dataframe to a disk file. This runs well on a Jupyter notebook, but fails when run from the Atom IDE or the command prompt (ie python script_name.py)

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, index=['a', 'b', 'c'])
df.to_hdf('data.h5', key='df', mode='w')
pd.read_hdf('data.h5')

Here is the error:

 line 3, in <module>
    df.to_hdf('data.h5', key='df', mode='w')
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\generic.py", line 2530, in to_hdf
    pytables.to_hdf(path_or_buf, key, self, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 276, in to_hdf
    path_or_buf, mode=mode, complevel=complevel, complib=complib
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\pytables.py", line 484, in __init__
    tables = import_optional_dependency("tables")
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\compat\_optional.py", line 93, in import_optional_dependency
    raise ImportError(message.format(name=name, extra=extra)) from None
ImportError: Missing optional dependency 'tables'.  Use pip or conda to install tables.

I added the hdf5.dll path to the list of environment variables but no luck. Both pip and conda tell me that I have a version of hdf5 and tables so I do not understand how this code can run in Jupyter notebook but not outside the notebook. BTW, I have a similar issue with matplotlib and a few other packages which run without errors from a Jupyter notebook but throw up package not found, dll missing errors when run from the command line. Can anyone please help? Thank you!

My path looks like:

c:\users\UserID\appdata\roaming\python\python37\site-packages;
C:\ProgramData\Anaconda3\pkgs\m2w64-openssl-1.0.2.g-2\Library\mingw-w64\bin;
C:\ProgramData\Anaconda3\pkgs\pytables-3.6.1-py37h1da0976_0\Lib\site-packages\tables\;
C:\ProgramData\Anaconda3\Lib\site-packages\tables;
C:\ProgramData\Anaconda3;C:\ProgramData\Anaconda3\Scripts;
C:\ProgramData\Anaconda3\Library\usr\bin;
C:\Users\UserID\AppData\Local\atom;
C:\ProgramData\Anaconda3\Library\mingw-w64\bin;
C:\Program Files (x86)\ActiveState Komodo Edit 11\;
C:\ProgramData\Anaconda3\envs\pudl;
C:\Users\UserID\AppData\Roaming\Python\Python37\Scripts;
C:\WINDOWS\system32;
C:\WINDOWS;
C:\WINDOWS\System32\Wbem;
......
C:\Users\UserID\AppData\Local\atom\bin

OS: Win 10 Python version: 3.7

Answer 1

You probably need to activate the environment where your Jupyter kernel resides, as that will be pulling in the necessary dependencies on your behalf.

Assuming you've already created an environment using conda create -n my_conda_env you'll also have to activate it using conda activate -n my_conda_env then your cmd prompt will be prefixed with the name of the environment.

If you're not running in an environment - I'd really recommend it - as it can avoid some dependency issues.