在Azure Data Lake中计数行

Question

I have some files in Azure Data Lake and I need to count how many lines they have to make sure they are complete. What would be the best way to do it?

I am using Python:

from azure.datalake.store import core, lib
adl_creds = lib.auth(tenant_id='fake_value', client_secret='fake_another value', client_id='fake key', resource='https://my_web.azure.net/')
adl = core.AzureDLFileSystem(adl_creds, store_name='fake account')

file_path_in_azure = "my/path/to/file.txt"
if adl.exists(file_path_in_azure) is True:
    # 5 megs 5242880 500megs 524288000 100megs 104857600 1meg 1048576
    counter = 0
    with adl.open(file_path_in_azure, mode="rb", blocksize=5242880) as f:
        # i try to use list comprehension but the memory increase since make a list of 1 [1,1,1,1,1,1,1] and then sums all
        # counter1 = sum(1 for line in f)
        for line in f:
            counter = counter + 1

print(counter)

This works, but it takes hours for files that are 1 or 2 gigabytes. Shouldn't this be faster? Might there be a better way?

Answer 1

Do you need to count lines? Maybe it is enough to get size of the file? You have AzureDLFileSystem.stat to get the file size, If you know how long is an average line size you could calculate the expected line count.

Answer 2

You could try:

for file in adl.walk('path/to/folder'):
    counter += len(adl.cat(file).decode().split('\n'))

I'm not sure if this is actually faster, but it uses the unix built ins to get file output which might be quicker than explicit I/O

EDIT: The one pitfall of this method is in the case that file sizes exceed the RAM of the device you run this on, as cat will throw the contents into memory explicitly

Answer 3

The only faster way i found, was to actually download the file locally to where the script is running with

 adl.put(remote_file, locally)

and then count line by line with out putting all file into the memory, download 500mgs takes around 30secs and reading 1mill lines around 4 secs =)