通过 databricks-connect 将带有 mlflow 的 Spark 模型记录到 databricks 注册表时出错

Question

我正在尝试使用 databricks-connect 在 mlflow 上记录经过训练的 spark 模型。 我希望将此模型记录在 Databricks 注册表中。 目前，我的代码如下所示：

mlflow.set_tracking_uri("databricks")
mlflow.set_experiment("/Users/xxxxx/experiment_name")

with mlflow.start_run(run_name="my_run") as _:
    mlflow.spark.log_model(my_spark_model, "my_model")

当它运行 log_model 行时，执行中断并出现以下堆栈跟踪：

22/07/21 11:05:03 WARN ProtoSerializer: 无法反序列化远程异常 java.io.InvalidClassException: 无法在 java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) 在 java.io.ObjectInputStream.readClassDesc( Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source ) 在 org.apache.spark.sql 的 scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) 的 org.apache.spark.sql.util.ProtoSerializer.$anonfun$deserializeObject$1(ProtoSerializer.scala:6618)。 util.ProtoSerializer.deserializeObject(ProtoSerializer.scala:6603) at org.apache.spark.sql.util.ProtoSerializer.deserializeException(ProtoSerializer.scala:6634) at com.databricks.service.SparkServiceRemoteFuncRunner.executeRPC(SparkServiceRemoteFuncRunner.scala:188)在 com.databricks.service.SparkServiceRemoteF uncRunner.$anonfun$execute0$1(SparkServiceRemoteFuncRunner.scala:121) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at com.databricks.service.SparkServiceRemoteFuncRunner.withRetry(SparkServiceRemoteFuncRunner.scala:135) at com.databricks .service.SparkServiceRemoteFuncRunner.execute0(SparkServiceRemoteFuncRunner.scala:113) 在 com.databricks.service.SparkServiceRemoteFuncRunner.$anonfun$execute$1(SparkServiceRemoteFuncRunner.scala:86) 在 com.databricks.spark.util.Log4jUsageLogger.recordOperation(UsageLogger.scala) :247) 在 com.databricks.spark.util.UsageLogging.recordOperation(UsageLogger.scala:429) 在 com.databricks.spark.util.UsageLogging.recordOperation$(UsageLogger.scala:408) 在 com.databricks.service.SparkServiceRPCClientStub .recordOperation(SparkServiceRPCClientStub.scala:58) at com.databricks.service.SparkServiceRemoteFuncRunner.execute(SparkServiceRemoteFuncRunner.scala:78) at com.databricks.service.SparkServiceRemoteFuncRunner.execute$(SparkServiceRemoteFuncRu nner.scala:67) 在 com.databricks.service.SparkServiceRPCClientStub.execute(SparkServiceRPCClientStub.scala:58) 在 com.databricks.service.SparkServiceRPCClientStub.fileSystemOperation(SparkServiceRPCClientStub.scala:297) 在 com.databricks.service.FSClient.send (FSClient.scala:51) 在 com.databricks.service.FSClient.getFileStatus(FSClient.scala:181) 在 org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) 在 org.apache.spark。 ml.util.FileSystemOverwrite.handleOverwrite(ReadWrite.scala:675) at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:167) at org.apache.spark.ml.PipelineModel$PipelineModelWriter.super$在 org.apache.spark.ml.MLEvents.withSaveInstanceEvent(events.scala: 175) 在 org.apache.spark.ml.util.Instrumentation.withSaveInstanceEvent(Instrumentation.scala:43) 在 org.apache.spark 的 org.apache.spark.ml.MLEvents.withSaveInstanceEvent$(events.scala:170)。毫升 .PipelineModel$PipelineModelWriter.$anonfun$save$3(Pipeline.scala:344) at org.apache.spark.ml.PipelineModel$PipelineModelWriter.$anonfun$save$3$adapted(Pipeline.scala:344) at org.apache.spark .ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:284) at scala.util.Try$.apply(Try.scala:213) at org.apache.spark.ml.util.Instrumentation$。在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 在 sun.reflect.NativeMethodAccessorImpl.invoke( Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine .invoke(ReflectionEngine.java:380) at py4j.Gateway.invoke(Gateway.java:295) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execu te(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:251) at java.lang.Thread.run(Unknown Source) 引起：java.lang.ClassNotFoundException: com.databricks.backend.daemon。 data.common.InvalidMountException at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.Class.forName0( Native Method) at java.lang.Class.forName(Unknown Source) at org.apache.spark.util.Utils$.classForName(Utils.scala:242) at org.apache.spark.sql.util.SparkServiceObjectInputStream.readResolveClassDescriptor( SparkServiceObjectInputStream.scala:60) at org.apache.spark.sql.util.SparkServiceObjectInputStream.readClassDescriptor(SparkServiceObjectInputStream.scala:55) ... 51 更多 **22/07/21 11:05:03 错误检测：com.databricks .service.SparkServiceRemoteException：com.databricks.backend.daemon.data.common.InvalidMountException：使用路径 /databricks/mlflow-trackin 时出错 g/000000000000000/0a0a0a0a0a0a0a0a0a0a/artifacts\experiment_name/sparkml 用于解析路径 '/000000000000000/0a0a0a0a0a0a0a0a0a0a/artifacts\experiment_name/sparkml' 在 mount at '/databricks/mlflow-tracking.>

<...>>

原因：java.io.IOException: No FileSystem for scheme: unsupported-access-mechanism-for-path--use-mlflow-client at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache .getInternal(FileSystem.java:2703) 在 org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) 在 org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) 在com.databricks.backend.daemon.data.client.DatabricksFileSystemV2Factory.createFileSystem(DatabricksFileSystemV2Factory.scala:124) 在 com.databricks.backend.daemon.data.filesystem.MountEntryResolver.$anonfun$resolve$1(MountEntryResolver.scala:67) 在com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:395) 在 com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:484) 在 com.databricks.logging.Usag eLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:504) at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:266) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)在 com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:261) 在 com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:258) 在 com.databricks.common.util.locks.LoggedLock$.withAttributionContext (LoggedLock.scala:73) 在 com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:305) 在 com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:297) 在 com.databricks.common.util .locks.LoggedLock$.withAttributionTags(LoggedLock.scala:73) at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:479) at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:404) at com.databricks.common.util.locks.LoggedLock$.recordO perationWithResultTags(LoggedLock.scala:73) 在 com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:395) 在 com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:367) 在 com.databricks.common。 util.locks.LoggedLock$.recordOperation(LoggedLock.scala:73) 在 com.databricks.common.util.locks.LoggedLock$.withLock(LoggedLock.scala:120) 在 com.databricks.common.util.locks.PerKeyLock。 withLock(PerKeyLock.scala:36) at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.resolve(MountEntryResolver.scala:64)>

<...> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _>

answer = 'xro1535' gateway_client = <py4j.java_gateway.GatewayClient 对象位于 0x0000023716B5BE20> target_id = 'o1532', name = 'copyToLocalFile'>

 def get_return_value(answer, gateway_client, target_id=None, name=None): """Converts an answer received from the Java gateway into a Python object. For example, string representation of integers are converted to Python integer, string representation of objects are converted to JavaObject instances, etc. :param answer: the string returned by the Java gateway :param gateway_client: the gateway client used to communicate with the Java Gateway. Only necessary if the answer is a reference (eg, object, list, map) :param target_id: the name of the object from which the answer comes from (eg, *object1* in `object1.hello()`). Optional. :param name: the name of the member from which the answer comes from (eg, *hello* in `object1.hello()`). Optional. """ if is_error(answer)[0]: if len(answer) > 1: type = answer[1] value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) if answer[1] == REFERENCE_TYPE: raise Py4JJavaError( "An error occurred while calling {0}{1}{2}.\n". format(target_id, ".", name), value)

E py4j.protocol.Py4JJavaError: 调用 o1532.copyToLocalFile 时出错。 E : java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\itscarlayall\AppData\Local\Temp\tmpalmxdo16\model\sparkml\metadata_SUCCESS E at org.apache.hadoop.util.Shell $ShellCommandExecutor.execute(Shell.java:773) E at org.apache.hadoop.util.Shell.execCommand(Shell.java:869) E at org.apache.hadoop.util.Shell.execCommand(Shell.java:852 ) E at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733) E at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225) E at org.apache.hadoop。 fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209) E at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307) E at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java :296) E 在 org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)

Answer 1

解决了！ 我需要安装winutils。 显然，即使 databricks-connect 将执行发送到远程 databricks，mlflow 仍然需要一些本地操作来保存 spark 模型🤷‍♀️