读取 SAS 文件以获取元信息

Question

Very new to data science technologies. Currently working on reading a SAS File (.sas7dbat).

Able to read the file using :

SAS7BDAT('/dbfs/mnt/myMntScrum1/sasFile.sas7bdat') as f:
    for row in f:
      print(row)

Row prints all the data.

When we view SAS files in SAS viewer we can see metadata Eg Label Information & variable (column names) used on actual data

How can I read this metadata in Spark (Databricks) using Python ?

Answer 1

Did you try pyreadstat ?

It can directly read metadata.

    import pyreadstat

    df, meta = pyreadstat.read_sas7bdat('/path/to/a/file.sas7bdat')

Answer 2

Most data analysis in Python is done using the pandas library which has a method called 'read_sas' which preserves the meta-data unless you are being ordered to use spark I strongly recommend pandas. Here is a set of instructions for SAS users: https://blog.dominodatalab.com/pandas-for-sas-users-part-1/

Answer 3

You can use a Spark external package called spark-sas7bdat for reading sas_file_name.sas7bdat

Here is how to install it into Spark Application https://spark-packages.org/package/saurfang/spark-sas7bdat and some examples on its github page https://github.com/saurfang/spark-sas7bdat

Then just using Spark read method

spark.read.format("com.github.saurfang.sas.spark")
          .load("path to the sas_file_name.sas7bdat", inferLong=True)

Answer 4

If you are interested on metadata only, you can use pyreadstat passing metadataonly parameter as True , it will not read any data, but just the metadata, so size of the file is going to have no impact on the amount of time required to read the metadata.

import pyreadstat

df, meta = pyreadstat.read_sas7bdat('/dbfs/mnt/myMntScrum1/sasFile.sas7bdat', metadataonly=True)

Note that df will be an empty dataframe when metadataonly=True is passed, you may want to omit this if you want both the data as well as the metadata

You can access variable labels using meta.column_names_to_labels , it will give a dictionary where variable name is the key and variable label is the value.

Other useful metadata are: metadata.number_columns , metadata.number_rows , metadata.file_encoding , metadata.file_label , etc.

You can find the complete list of available metadata at pyreadstat documentation