[英]Azure databricks - not able to read .csv files using spark jobs from datalake storage gen2 service
I have a databricks clister running fine.我有一个运行良好的 databricks 模块。 And using the following code I can mount my "datalake storage gen2" account as well.并使用以下代码我也可以挂载我的“datalake storage gen2”帐户。 I am mounting everything on /mnt/data1我正在 /mnt/data1 上安装所有内容
val configs = Map("fs.azure.account.auth.type" -> "OAuth",
"fs.azure.account.oauth.provider.type" -> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id" -> appID,
"fs.azure.account.oauth2.client.secret" -> password,
"fs.azure.account.oauth2.client.endpoint" -> ("https://login.microsoftonline.com/" + tenantID + "/oauth2/token"),
"fs.azure.createRemoteFileSystemDuringInitialization"-> "true")
dbutils.fs.mount(
source = "abfss://" + fileSystemName + "@" + storageAccountName + ".dfs.core.windows.net/",
mountPoint = "/mnt/data1",
extraConfigs = configs)
Until this point everything is fine and working.到目前为止,一切都很好并且可以正常工作。 But when I'm trying to access one file from the mount location with the following command但是当我尝试使用以下命令从挂载位置访问一个文件时
val df = spark.read.csv("/mnt/data1/creodemocontainer/movies.csv")
I'm getting following error我收到以下错误
java.io.FileNotFoundException: dbfs:/mnt/data1/creodemocontainer2/movies.csv
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.$anonfun$getFileStatus$2(DatabricksFileSystemV2.scala:775)
Though I can connect and load those files in PowerBI without any issue.尽管我可以毫无问题地在 PowerBI 中连接和加载这些文件。 I'm not getting any clue from last 2 days So any help will be really appreciated.我没有从过去 2 天得到任何线索所以任何帮助将不胜感激。
thanks in advance.提前致谢。
Sharing the answer as per the comment by the original poster:根据原始海报的评论分享答案:
I'm not supposed to add container name while reading.我不应该在阅读时添加容器名称。
val df = spark.read.csv("/mnt/data1/creodemocontainer/movies.csv")
Removed container name, since it has already called in the mount point.删除了容器名称,因为它已经在挂载点中调用了。 Everything is working fine now现在一切正常
val df = spark.read.csv("/mnt/data1/movies.csv")
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.