How to write pytest by taking dataframe as an input to fetch all the required values in palantir foundry

Question

I'm able to write pytest functions by manually giving column names and values to create a data frame and passing it to the production code to check all the transformed fields values in palantir foundry code repository.

Instead of manually passing column names and their respective values I want to store all the required data in the dataset and import that dataset into pytest function to fetch all the required values and passing over to the production code to check all the transformed field values.

Is there anyways to accept the dataset as input to the test function in planatir code repository.

Answer 1

You can't read from a dataset at the time of running the tests. However, perhaps you can create the test dataset, save it as a CSV, and then in the test read the CSV using the test's spark session. The code for that would be like spark.read.csv("path") .

This will add file I/O time and slow your test down, so my recommendation is to just create the dataframe using test data that is already in memory. The code for that would be spark.createDataFrame(data) .