在Python中使用PySpark：WindowsError：[错误5]访问被拒绝

Question

I'm trying to run PySpark unit tests using PyBuilder under Windows 10

pyb_ clean install

All the necessary libraries are included into PYTHONPATH and they are available.

But when running I get the following error:

..\..\src\unittest\python\client_tests.py:11: in <module>
    from ut_setting import spark
..\..\src\unittest\python\ut_setting.py:75: in <module>
    spark = SparkSession.builder.master("local[4]").getOrCreate()
..\python\pyspark\sql\session.py:169: in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
..\python\pyspark\context.py:307: in getOrCreate
    SparkContext(conf=conf or SparkConf())
..\python\pyspark\context.py:115: in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
..\python\pyspark\context.py:256: in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
..\python\pyspark\java_gateway.py:82: in launch_gateway
    proc = Popen(['spark-submit.cmd', "-f"])
D:\Python27\Lib\subprocess.py:672: in __init__
    errread, errwrite)
D:\Python27\Lib\subprocess.py:882: in _execute_child
    startupinfo)
E   WindowsError: [Error 5] Access is denied

I've also tried to use different approaches: running using admin permissions, moving to another drive and etc.

If I do the same on Linux, it works fine without any tuning.

Do you have any thoughts?

Thanks!

Answer 1

Try this:

1. Right click the Start button and click "Command Line (Admin)" (or "PowerShell (Admin)")
2. Run the installation command.
3. ...
4. PROFIT!

Answer 2

Thank you guys!

I've managed to solve the issue.

The main reason was that I've tried to set admin permissions to the single processes, while I need to set the permissions to the whole pyspark directory.

Right click on the directory -> Properties -> Security tab I've set "Full control" for "Everyone" and enabled inheritance.