Transform.parquet 文件 varchar 列到 Azure Synapse 中的日期时间

Question

I am sending JSON telemetry data from our IoT Hub to Azure Data Lake Gen2 in the form of.parquet files. From the data lake I've then created a view in my Azure Synapse Serverless SQL pool that I can connect to and query data for reports.

CREATE VIEW DeviceTelemetryView
AS SELECT * FROM
    OPENROWSET(
        BULK 'https://test123.dfs.core.windows.net/devicetelemetry/*/*/*/*/*/',
        FORMAT = 'PARQUET'
    ) AS [result]

This is what my view data looks like:

Most of these reports are based on date time ranges. Therefore I want to be able to write SQL queries that use my date time stamp.

The Current Issue

When I look at the current data type for the dateTimeStamp column, it defaults to varchar(8000) even though I believe my JSON is in the correct datetime format: "2021-11-29T21:45:00" . How can I transform this specific field to a datetime field in my view to run queries on it?

Answer 1

When I look at the current data type for the dateTimeStamp column, it defaults to varchar(8000)

I think you would have to look at the data type for that column in the parquet file, it's likely to be a string in your case. Sql interpret as a varchar(8000).

even though I believe my JSON is in the correct datetime format: "2021-11-29T21:45:00".

Even if the timestamp format is correct, I think you'd have to tip system so it knows to cast that string to a datetime

How can I transform this specific field to a datetime field in my view to run queries on it?

I'm not an expert in sql but I think you can convert a string to a timestamp using cast and convert

CREATE VIEW DeviceTelemetryView
AS SELECT corporationid, deviceid, version, Convert(dateTimestamp, 126), data FROM
    OPENROWSET(
        BULK 'https://test123.dfs.core.windows.net/devicetelemetry/*/*/*/*/*/',
        FORMAT = 'PARQUET'
    ) AS [result]