Pyspark：如何添加带有行号的列？

Question

I have a pyspark dataframe. I would like to add a column with that contains the row number.

This is what I am doing

stop_df = stop_df.withColumn("stop_id", monotonically_increasing_id())

If I check the maximum value of stop_id , I get

stop_df.agg(max("stop_id")).show()
+--------------+
|  max(stop_id)|
+--------------+
|32478542692458|
+--------------+

while the number of rows is

stop_df.count()
Out[4]: 8134605

Answer 1

From spark monotonically_increasing_id docs:

A column that generates monotonically increasing 64-bit intege rs .

The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record number within each partition in the lower 33 bits. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.

Use window row_number function to get the row number.

df=spark.createDataFrame([("a",),("b",)],["id"])
from pyspark.sql.window import Window
from pyspark.sql.functions import *
#add partition by and order by clause if ordering required with in window.
w=Window.orderBy(lit(1))

df.withColumn("stop_id",row_number().over(w)).show()
#+---+-------+
#| id|stop_id|
#+---+-------+
#|  a|      1|
#|  b|      2|
#+---+-------+

df.withColumn("stop_id",row_number().over(w)).agg(max("stop_id")).show()
#+------------+
#|max(stop_id)|
#+------------+
#|           2|
#+------------+

df.count()
#2