使用 pyspark windows 写入本地 hadoop

Question

我有使用 pipenv 的虚拟环境，我有

pyspark==3.1.2

当我做

print(f"Hadoop version = {SparkContext._jvm.org.apache.hadoop.util.VersionInfo.getVersion()}")

我明白了

3.2.0

我的 hadoop 设置：-

1. Downloaded hadoop-3.2.0.tar.gz from https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/
2. Unpacked it using : tar -xvzf hadoop-3.2.0.tar.gz to C:\hadoop (this command unpacked the files but complained of "exited with error because of previous errors", but I could not see an actual error
3. Downloaded the windows native IO libraries from https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.0/bin
4. Added the files from point 3 to C:\hadoop\bin , 7 in total, making the total files in the directory to be 18.
5. Copied the hadoop.dll to C:\WINDOWS\System32.
6. Created an environmental variable HADOOP_HOME with a VALUE of C:\hadoop
7. Added the bin folder to the PATH variable i.e. C:\hadoop\bin

但是，即使做了所有这些（在我在网上查看的所有地方都指定了），每当我这样做

 self.spark = SparkSession.builder.master("local[1]").getOrCreate() 
 self.test_config = {
            "output_format": "parquet",
            "output_path": os.path.join(self.test_dir, "output"),
        }
 df.write.format(self.conf["output_format"]).mode("overwrite").save(
            self.conf["output_path"]
        )

为线

df.write.format(self.conf["output_format"]).mode("overwrite").save(
            self.conf["output_path"]
        )

我不断收到错误

py4j.protocol.Py4JJavaError: An error occurred while calling o31.save.
E                   : org.apache.spark.SparkException: Job aborted.
E                       at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231)
E                       at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188)
E                       at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)      
E                       at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
E                       at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)
E                       at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
E                       at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
E                       at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
E                       at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
E                       at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
E                       at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
E                       at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)
E                       at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
E                       at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
E                       at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
E                       at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
E                       at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
E                       at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
E                       at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
E                       at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)
E                       at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)
E                       at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
E                       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
E                       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
E                       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
E                       at java.lang.reflect.Method.invoke(Unknown Source)
E                       at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
E                       at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E                       at py4j.Gateway.invoke(Gateway.java:282)
E                       at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
E                       at py4j.commands.CallCommand.execute(CallCommand.java:79)
E                       at py4j.GatewayConnection.run(GatewayConnection.java:238)
E                       at java.lang.Thread.run(Unknown Source)
E                   Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z   
E                       at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
E                       at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:645)
E                       at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1230)
E                       at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1435)
E                       at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:493)
E                       at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
E                       at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
E                       at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678)
E                       at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
E                       at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
E                       at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:332)  
E                       at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:402)
E                       at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:375)
E                       at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48)
E                       at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182)      
E                       at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:220)
E                       ... 32 more

我什至重新启动了我的电脑（以防环境变量没有启动），尝试了 hadoop 版本 3.2.1 和 3.2.2 但没有任何效果。

我怎样才能让它工作？