在另一个 dask 数据帧上使用 apply 函数将行附加到一个 dask 数据帧

Question

I want to run the following operation using dask.

df1 = pd.DataFrame()

def foo(row):
    global df1
    df1.append(row)

def main():
    global df1
    df2.apply(foo , axis = 1)

When I run the following operation without Dask , it runs perfectly fine, but when I convert both of my dataframes to dask, then I do not get any data in my df2 dataframe after computing.

df1 = pd.DataFrame()
df1 = from_pandas(df1, npartitions=10)

def foo(row):
    global df1
    df1.append(row)

def main():
    global df1
    df2 = from_pandas(df2 , npartitions = 10)
    df2.apply(foo , axis = 1, meta = df2)

I am not sure, what I am doing wrong, or if there is a better way for this in dask, my main objective is to run the entire code using dask, since the data is quite large to work on.

Answer 1

The result of .apply method should be assigned to a new dask DataFrame:

df2 = df2.apply(foo , axis = 1, meta = df2)

However, it's likely that this is not efficient when working with data at scale. What is more efficient will depend on the specific problem being solved.