![](/img/trans.png)
[英]Spark Caused by: java.lang.StackOverflowError Window Function?
[英]java.lang.StackOverFlowError when converting an RDD to a DataFrame
嘗試計算大量 RDD 文檔的 tf-idf 分數,每當我嘗試將其轉換為 dataframe 時,它總是崩潰。 我得到的最初錯誤是
org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError
緊接着這個,重復了很多次:
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
其次是
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1171)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:1069)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:1013)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
at org.apache.spark.api.python.PythonRDD$.runJob(PythonRDD.scala:153)
at org.apache.spark.api.python.PythonRDD.runJob(PythonRDD.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
我做了一些研究,似乎這個與 dataframe 相關的 DAG(有向無環圖)太大了,我應該對我的數據進行某種緩存/檢查點/持久化來解決它。 每次還是崩潰。 為了避免混淆問題,我從下面的代碼中省略了那些緩存/檢查點/持久性行:
from pyspark.sql import SQLContext
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('app').getOrCreate()
rdd = spark.sparkContext.parallelize([])
data = []
count = 0
for sentence in giant_list_of_sentences:
words = sentence.split(' ')
data.append((words, count)) #Count is the index of the document
count += 1
if (len(data) > 5000):
rdd = rdd.union(spark.sparkContext.parallelize(data))
if (len(data) > 0):
rdd = rdd.union(spark.sparkContext.parallelize(data))
df_txts = sqlContext.createDataFrame(data, ["list_of_words",'index'])
總是跑到最后一行然后失敗,除非它只運行在它正常工作的一小部分數據上。
所以解決方案實際上非常簡單 - 事實證明,將巨型 RDD 轉換為巨型 dataframe 很難,但是將幾個較小的 RDD 轉換為幾個較小的數據幀,然后加入數據幀,效果很好。
from pyspark.sql import SQLContext
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('app').getOrCreate()
rdds = [spark.sparkContext.parallelize([])]*6
data = []
count = 0
turn = 0
for sentence in giant_list_of_sentences:
words = sentence.split(' ')
data.append((words, count)) #Count is the index of the document
count += 1
if (len(data) > 5000):
rdds[turn] = rdds[turn].union(spark.sparkContext.parallelize(data))
if (len(data) > 0):
rdds[turn] = rdds[turn].union(spark.sparkContext.parallelize(data))
df_txts = rdds[0].toDF(['list_of_words', 'index'])
for i in range(1, len(rdds)):
df_txts = df_txts.union(rdds[i].toDF(['list_of_words', 'index'])
df_txts = sqlContext.createDataFrame(data, ["list_of_words",'index'])
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.