Hi i want to upload Parquet files to ADLS gen 2 blob. I am using below line of code to create blob and upload parquet file in it.
blob = BlobClient.from_connection_string(conn_str="Connection String", container_name="parquet", blob_name=outdir)
df.to_parquet('logs.parquet',compression='GZIP') #df is dataframe
with open("./logs.parquet", "rb") as data:
blob.upload_blob(data)
os.remove("logs.parquet")
i do not face any error and the files are also written in the blob. But, i don't think i am doing it right as ADX/kusto query can not understand the file and no data is visible there.
Below are the steps i performed in Azure Data Explorer to fetch records from uploaded parquet file in ADLS gen 2.
Created External Table:
.create external table LogDataParquet(AppId_s:string,UserId_g:string,Email_s:string,RoleName_s:string,Operation_s:string,EntityId_s:string,EntityType_s:string,EntityName_s:string,TargetTitle_s:string,TimeGenerated:datetime)
kind=blob
dataformat=parquet
(
h@'https://streamoutalds2.blob.core.windows.net/stream-api-raw-testing;secret'
)
with
(
folder = "ExternalTables"
)
External Table Column Mapping:
.create external table LogDataParquet parquet mapping "LogDataMapparquet" '[{ "column" : "AppId_s", "path" : "$.AppId_s"},{ "column" : "UserId_g", "path" : "$"},{ "column" : "Email_s", "path" : "$.Email_s"},{ "column" : "RoleName_s", "path" : "$.RoleName_s"},{ "column" : "Operation_s", "path" : "$.Operation_s"},{ "column" : "EntityId_s", "path" : "$.EntityId_s"}]'
External Tables Gives no records
external_table('LogDataParquet')
No records
external_table('LogDataParquet') | count
1 record - count 0
I have used similar scenario using stream analytics, where i receive incoming streams and save it in parquet format to ADLS. External table in ADX fetches records well in that case. I feel i am making mistake in the way parquet files are written in blob - (with open("./logs.parquet", "rb") as data: )
According to logs, external table was defined as follows:
.create external table LogDataParquet(AppId_s:string,UserId_g:string,Email_s:string,RoleName_s:string,Operation_s:string,EntityId_s:string,EntityType_s:string,EntityName_s:string,TargetTitle_s:string,TimeGenerated:datetime)
kind=blob
partition by
AppId_s,
bin(TimeGenerated,1d)
dataformat=parquet
(
'******'
)
with
(
folder = "ExternalTables"
)
The PARTITION BY
clause tells ADX that the expected folder layout is:
<AppId_s>/<TimeGenerated, formatted as 'yyyy/MM/dd'>
For example:
https://streamoutalds2.blob.core.windows.net/stream-api-raw-testing;secret/SuperApp/2020/01/31
You can find more info on how ADX locates files on external storage during a query, in this section: https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/external-tables-azurestorage-azuredatalake#artifact-filtering-logic
To fix the external table definition in accordance to folders layout, please use .alter
command:
.alter external table LogDataParquet(AppId_s:string,UserId_g:string,Email_s:string,RoleName_s:string,Operation_s:string,EntityId_s:string,EntityType_s:string,EntityName_s:string,TargetTitle_s:string,TimeGenerated:datetime)
kind=blob
dataformat=parquet
(
h@'https://streamoutalds2.blob.core.windows.net/stream-api-raw-testing;secret'
)
with
(
folder = "ExternalTables"
)
BTW, if a mapping is naive (eg mapped column names match data source column names), then it's not needed for Parquet format.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.