在 Pyspark 中使用时，具有静态文件依赖项的 python 包无法读取静态文件

Question

I am trying to resolve an issue with python packages PySpark. I developed a python package which has the following structure.

sample_package/
  |-config/
       |-sample.ini
  |-main.py
  |-__init__.py

Inside my main.py , I have a code snippet that reads the config file from the config/ directory as follows

import ConfigParser, os
def sample_func():
    config = ConfigParser.ConfigParser()
    configfile = os.path.join(os.path.dirname(__file__), 'config', 'sample.ini')
    config.read(configfile)
    return config.sections()

I created a zip file of the above package as sample_package.zip and included the zip as a pyspark dependency

addPyFile(path/to/zip/file)

In my pyspark job when i import the sample_package the import works fine and i'm able to call the sample_func inside main, but however my python package is unable to read the sample.ini file. When executed inside a plain python program, it works fine but not inside a pyspark job. Is there any path manipulation being done in a pyspark environment when accessing static files? How can I get my python package to properly read the config file?

Answer 1

I figured out the answer by myself. It is more of a python packaging issue rather than pyspark environment issue. Looks like I had to use pkgutil module to reference my static resources which modifies my function as below

import ConfigParser, os, pkgutil, StringIO
def sample_func():
    config = ConfigParser.ConfigParser()
    configfile = pkgutil.get_data('sample_package', 'config/sample.ini')
    cf_buf = StringIO.StringIO(configfile)
    config.readfp(cf_buf)
    return config.sections()

Answer 2

More simple version:

from configparser import ConfigParser
import pkgutil

def sample_func():
    config = ConfigParser()
    # os.path.join is not needed.
    config_data = pkgutil.get_data(__name__, 'config/sample.ini').decode()
    config.read_string(config_data)
    return config.sections()