如何在Azure Databricks Spark中从DataFrame获取特定的行和列

Question

I have a DataFrame in Azure Databricks which looks like

Col a| Col b
------------
Marc | Taylor
John | McC
Bill | Gates

I would like to extract a specfic column and row. I know how to extract a specific column and assign it to a variable

result = ds.select(Col a)

But how to get row number 2 for example in this line of code?

Answer 1

You can use monotonically_increasing_id() function to generate a new column with serial number and then use filter function to filter in the row

from pyspark.sql.functions import *
ds.withColumn('sn', monotonically_increasing_id())\
    .filter(col('sn') == 1)\
    .drop('sn')\
    .show(truncate=False)

which would give you

+-----+-----+
|Col a|Col b|
+-----+-----+
|John |McC  |
+-----+-----+

Note: monotonically_increasing_id will generate increasing order numbers but not guaranteed to generate serial numbers starting from 0.

To sum up, filter() and select()/where() functions are two ways to select rows from a dataframe

I hope the answer is helpful

Answer 2

I can get the value with python using this:

df_sample = yourDataFrame.select(collect_list("Col b").alias("a")) value = df_sample.select(col("a").getItem(1).alias("x")) display(value)

hope it helps.