Meta data in Azure data lake

Question

I have written a Azure finction in C# that recursivly goes through the data lake and generates a file with metadata (filename,path,size mofied date etc) of all files and folders in the datalake.

This takes quite a while since we have a lot of files and foders. So I was just wondering if there was a meta data store that we could pull this data from directly? I thinking of something like sys tables in SQL Server.

Thanks in advance!

Answer 1

There are some features around file information that will soon be released that give you some of the file system meta data properties. But you would still need to enumerate your folder hierarchies yourself.

For example:

@data = 
  EXTRACT 
    vehicle_id int
  , entry_id long
  , event_date DateTime
  , latitude float
  , longitude float
  , speed int
  , direction string
  , trip_id int?
  , uri = FILE.URI()
  , modified_date = FILE.MODIFIED()
  , created_date = FILE.CREATED()
  , file_sz = FILE.LENGTH()
FROM "/Samples/Data/AmbulanceData/vehicle{*}"
USING Extractors.Csv();

OUTPUT @data
TO "/output/releasenotes/winter2018/fileprops.csv"
USING Outputters.Csv(outputHeader : true);

I suggest that you file a request for a file system meta-data catalog view (eg, usql.files and usql.filesystem ) at http://aka.ms/adlfeedback to augment our metadata catalog views.