在 Pyspark 中从 memory 问题中获取 Java 堆

Question

I have tried to read Multiple CSV files with a size of around 100MB using the pandas package and try to convert the file into Spark.sql.data frame and append it to a list. And this list of the spark dataset is converted to a single dataframe.

In Spark, I have used the master as local and deployment mode as the client and my system spec is 16 RAM and 8 cores.

The java heap issue occurred while converting the pandas data frame to the pyspark data frame.

As I check through the web, the most common solution is to increase the driver execution memory, which I have increased up to 6 GB but still get the same error.

I have been stuck in this error for a couple of days. Can anyone provide a solution/thought to this?

My code:

def _load_and_normalize(self, glob_paths, renames=None, columns=[]):
    renames = renames or {}
    files = sorted(glob.glob(glob_paths))
    dfs = []
    for filename in files:
        logger.info(f'adding {basename(filename)}')
        df_list = self.read_csv(filename)
        for df_chunk in df_list:
            if len(columns) > 0:
                df_chunk = df_chunk[columns]
            df = self.spark_session.createDataFrame(df_chunk.astype('str'))
            df = self.add_procuredate(df, filename)
            dfs.append(df)
        logger.info(f'Added the procure date {basename(filename)} and append to list')
    df = reduce(DataFrame.unionByName, dfs).repartition(5000)
    logger.info(f' Combine the list of dataframes into single dataframe of size {df.count()}')
    for rename_columns in renames:
        df = df.withColumnRenamed(rename_columns, renames[rename_columns])
    df = self.uppercase_and_trim_all_columns(df)
    df = df.dropDuplicates()
    all_cols_except_procure = [col for col in df.schema.names if col != 'procure_date']
    df = df.dropDuplicates(all_cols_except_procure)
    df = self.get_normalized_address(df)
    df = self.get_normalized_address(df, col_name='orig_normalized_address',
                                        full_address_col='orig_address', city_col='orig_city', state_col='orig_state',
                                        zip_col='orig_zip')
    return df

Error log:

2022-12-29 17:55:06,163 - prepare_ncoa_sp - INFO - [542587] 0.43GB Starting
        Setting default log level to "WARN".
        To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
        22/12/29 17:55:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        2022-12-29 17:55:11,712 - prepare_ncoa_sp - INFO - [542587] 0.55GB adding 20210203_agent_addresses_1_validated.csv
        2022-12-29 17:56:45,989 - prepare_ncoa_sp - INFO - [542587] 0.58GB adding 20210203_agent_addresses_2_validated.csv
        2022-12-29 17:57:51,402 - prepare_ncoa_sp - INFO - [542587] 0.60GB adding 20210203_agent_addresses_3_validated.csv
        2022-12-29 17:59:11,865 - prepare_ncoa_sp - INFO - [542587] 0.60GB adding 20210203_agent_addresses_4_validated.csv
        2022-12-29 18:00:33,017 - prepare_ncoa_sp - INFO - [542587] 0.61GB adding 20210203_agent_addresses_5_validated.csv
        2022-12-29 18:01:58,095 - prepare_ncoa_sp - INFO - [542587] 0.62GB adding 20210203_agent_addresses_6_validated.csv
        2022-12-29 18:03:08,589 - prepare_ncoa_sp - INFO - [542587] 0.63GB adding 20210203_agent_addresses_7_validated.csv
        2022-12-29 18:04:17,571 - prepare_ncoa_sp - INFO - [542587] 0.63GB adding 20210203_agent_addresses_8_validated.csv
        2022-12-29 18:05:32,933 - prepare_ncoa_sp - INFO - [542587] 0.67GB adding 20210306_agent_addresses_1_validated.csv
        2022-12-29 18:06:43,819 - prepare_ncoa_sp - INFO - [542587] 0.68GB adding 20210306_agent_addresses_2_validated.csv
        2022-12-29 18:08:04,511 - prepare_ncoa_sp - INFO - [542587] 0.68GB adding 20210306_agent_addresses_3_validated.csv
        Traceback (most recent call last):
        File "manage.py", line 21, in <module>
        main()
        File "manage.py", line 17, in main
        execute_from_command_line(sys.argv)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
        utility.execute()
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
        self.execute(*args, **cmd_options)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
        output = self.handle(*args, **options)
        File "/home/ubuntu/backend/first_class/core/management/commands/prepare_ncoa_sp.py", line 24, in handle
        step.start()
        File "/home/ubuntu/backend/first_class/core/management/commands/prepare_ncoa_sp.py", line 34, in start
        self.prepare_agent_address_updates()
        File "/home/ubuntu/backend/first_class/core/management/commands/prepare_ncoa_sp.py", line 122, in prepare_agent_address_updates
        prepare_agent_data = self._load_and_normalize(file_glob, {
        File "/home/ubuntu/backend/first_class/core/management/commands/prepare_ncoa_sp.py", line 93, in _load_and_normalize
        df = self.spark_session.createDataFrame(df.astype('str'))
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/sql/session.py", line 891, in createDataFrame
        return super(SparkSession, self).createDataFrame(  # type: ignore[call-overload]
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/sql/pandas/conversion.py", line 437, in createDataFrame
        return self._create_dataframe(converted_data, schema, samplingRatio, verifySchema)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/sql/session.py", line 936, in _create_dataframe
        rdd, struct = self._createFromLocal(map(prepare, data), schema)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/sql/session.py", line 648, in _createFromLocal
        return self._sc.parallelize(internal_data), struct
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/context.py", line 674, in parallelize
        jrdd = self._serialize_to_jvm(c, serializer, reader_func, createRDDServer)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/context.py", line 720, in _serialize_to_jvm
        return reader_func(tempFile.name)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/context.py", line 668, in reader_func
        return self._jvm.PythonRDD.readRDDFromFile(self._jsc, temp_filename, numSlices)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/py4j/java_gateway.py", line 1321, in __call__
        return_value = get_return_value(
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/pyspark/sql/utils.py", line 190, in deco
        return f(*a, **kw)
        File "/home/ubuntu/.local/share/virtualenvs/first_class-iV0cREgX/lib/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value
        raise Py4JJavaError(
        py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.readRDDFromFile.
        : java.lang.OutOfMemoryError: Java heap space
        at org.apache.spark.api.java.JavaRDD$.readRDDFromInputStream(JavaRDD.scala:252)
        at org.apache.spark.api.java.JavaRDD$.readRDDFromFile(JavaRDD.scala:239)
        at org.apache.spark.api.python.PythonRDD$.readRDDFromFile(PythonRDD.scala:274)
        at org.apache.spark.api.python.PythonRDD.readRDDFromFile(PythonRDD.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.base/java.lang.Thread.run(Thread.java:829)

Answer 1

I'm going to start by saying: we don't have enough info to properly judge this code. There are a bunch of functions you haven't shared with us ( self.read_csv(filename) , self.add_procuredate(df, filename) , self.uppercase_and_trim_all_columns(df) , ...) and we don't know the data you're working with. Stackoverflow is not really a code-reviewing website but it seems like you're struggling so I'll give you some pointers.

This bit of code:

files = sorted(glob.glob(glob_paths))
dfs = []
for filename in files:
    logger.info(f'adding {basename(filename)}')
    df_list = self.read_csv(filename)
    for df_chunk in df_list:
        if len(columns) > 0:
            df_chunk = df_chunk[columns]
        df = self.spark_session.createDataFrame(df_chunk.astype('str'))
        df = self.add_procuredate(df, filename)
        dfs.append(df)
    logger.info(f'Added the procure date {basename(filename)} and append to list')
df = reduce(DataFrame.unionByName, dfs).repartition(5000)

is filled with things I believe can be better. Let's have a look at this and understand why it can be improved so much.

1. You're doing df_list = self.read_csv(filename) . That seems not to be the spark.read.csv(filename) function. Why are you using something different? You're creating a list of dataframes, that scares me a little bit. (could be a legit reason, but this smells bad to me). Why not simply reading all your data into a single dataframe by doing spark.read.csv(sorted(glob.glob(glob_paths))) ? That would make a single dataframe on which you can do whatever operations you want.

2. Then you're doing the following (code block underneath). So for each file, you seem to create a list of Dataframes, df_list , and then looping over that list, creating a bunch of Dataframes (using createDataFrame ) per file??? That sounds like a good way to fill up your Java heap space: creating unnecessary objects everywhere you go. Also, within this loop it seems like the only thing you're doing is selecting some columns ( df_chunk = df_chunk[columns] ) and adding something to the df (with add_procuredate ). It sounds like you could perfectly do this on the final Dataframe. You're also appending to a list with an unknown length ( dfs.append(df) ), which is classical performance hogger.

for df_chunk in df_list:
    if len(columns) > 0:
        df_chunk = df_chunk[columns]
    df = self.spark_session.createDataFrame(df_chunk.astype('str'))
    df = self.add_procuredate(df, filename)
    dfs.append(df)

3. Then you're creating a single DataFrame and repartitioning it: df = reduce(DataFrame.unionByName, dfs).repartition(5000) . First of all, your code does not even get here (because if you look at the Java error stack trace you're getting a failure when reading RDDs, which is the problematic first 2 points). But there are 2 things here:

   3.1. In the end, you're making a single DataFrame, unioning all of your smaller Dataframes. It really looks like there are many ways to do this better. Of course this will depend on many things: your data, your cluster, your other code that you haven't shown us, ...

   3.2. You're doing .repartition(5000) . Why? This triggers a shuffle: are you positive you need this?

Conclusion: This code does not seem to have been written in the Spark mindset. There are too many things to discuss (and that is not the Stackoverflow way) so I'm just dumping them here. My main advice is : read everything into a single dataframe (don't create multiple dataframes unnecessarily) and do all of your operations on that single dataframe.

PS You'll possibly have some questions/remarks after reading this. I suggest that we don't make a huge string of comments underneath this answer but that you chop up your question into minimal reproducible examples and ask them one question at a time (after having researched/googled around extensively of course).