Spark错误：没有足够的空间来缓存内存中的分区rdd_8_2！可用内存为58905314字节
[英]Spark Error: Not enough space to cache partition rdd_8_2 in memory! Free memory is 58905314 bytes
问题描述
When I run a Spark job using its example code BinaryClassification.scala with my own data, it always shows the errors like "Not enough space to cache partition rdd_8_2 in memory! Free memory is 58905314 bytes.". 
当我使用其示例代码BinaryClassification.scala运行Spark作业和我自己的数据时，它总是显示错误，例如“没有足够的空间来缓存内存中的分区rdd_8_2！可用内存是58905314字节。”。
I set the memory to 4G via conf = new SparkConf().setAppName(s"BinaryClassification with $params").set("spark.executor.memory", "4g"), and it doesn't work. 我通过conf = new SparkConf（）。setAppName（s“BinaryClassification with $ params”）。set（“spark.executor.memory”，“4g”）将内存设置为4G，但它不起作用。 Does anyone get any ideas? 有没有人有任何想法？ Thank you:) 谢谢：）
I run it locally on a Macbook Pro with 16GB ram. 我在带有16GB内存的Macbook Pro上本地运行它。
bin/spark-submit --class BinaryClassification ~/dev/scalaworkspace/BinaryClassification/BinaryClassification_fat.jar ~/data/trajectory.libsvm --algorithm LR
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/11/22 17:07:24 INFO SecurityManager: Changing view acls to: wangchao,
14/11/22 17:07:24 INFO SecurityManager: Changing modify acls to: wangchao,
14/11/22 17:07:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(wangchao, ); users with modify permissions: Set(wangchao, )
14/11/22 17:07:24 INFO Slf4jLogger: Slf4jLogger started
14/11/22 17:07:24 INFO Remoting: Starting remoting
14/11/22 17:07:24 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@wangchaos-mbp-2.lan:65405]
14/11/22 17:07:24 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@wangchaos-mbp-2.lan:65405]
14/11/22 17:07:24 INFO Utils: Successfully started service 'sparkDriver' on port 65405.
14/11/22 17:07:24 INFO SparkEnv: Registering MapOutputTracker
14/11/22 17:07:24 INFO SparkEnv: Registering BlockManagerMaster
14/11/22 17:07:24 INFO DiskBlockManager: Created local directory at /var/folders/6k/nww6s1p52yg424zdcckvpwvc0000gn/T/spark-local-20141122170724-b202
14/11/22 17:07:24 INFO Utils: Successfully started service 'Connection manager for block manager' on port 65406.
14/11/22 17:07:24 INFO ConnectionManager: Bound socket to port 65406 with id = ConnectionManagerId(wangchaos-mbp-2.lan,65406)
14/11/22 17:07:24 INFO MemoryStore: MemoryStore started with capacity 265.1 MB
14/11/22 17:07:24 INFO BlockManagerMaster: Trying to register BlockManager
14/11/22 17:07:24 INFO BlockManagerMasterActor: Registering block manager wangchaos-mbp-2.lan:65406 with 265.1 MB RAM
14/11/22 17:07:24 INFO BlockManagerMaster: Registered BlockManager
14/11/22 17:07:24 INFO HttpFileServer: HTTP File server directory is /var/folders/6k/nww6s1p52yg424zdcckvpwvc0000gn/T/spark-cf2b6d19-1db6-4e66-b080-8558a281cd50
14/11/22 17:07:24 INFO HttpServer: Starting HTTP Server
14/11/22 17:07:24 INFO Utils: Successfully started service 'HTTP file server' on port 65407.
14/11/22 17:07:25 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/11/22 17:07:25 INFO SparkUI: Started SparkUI at http://wangchaos-mbp-2.lan:4040
14/11/22 17:07:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/11/22 17:07:25 INFO SparkContext: Added JAR file:/Users/wangchao/dev/scalaworkspace/BinaryClassification/BinaryClassification_fat.jar at http://192.168.199.120:65407/jars/BinaryClassification_fat.jar with timestamp 1416647245419
14/11/22 17:07:25 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@wangchaos-mbp-2.lan:65405/user/HeartbeatReceiver
14/11/22 17:07:29 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 238168618 bytes.
14/11/22 17:07:29 WARN CacheManager: Not enough space to cache partition rdd_4_0 in memory! Free memory is 238168618 bytes.
14/11/22 17:07:30 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 238168618 bytes.
14/11/22 17:07:30 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 238168618 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_6_1 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_6_3 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_4_0 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_6_0 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_7_0 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_6_2 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:32 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 166721050 bytes.
14/11/22 17:07:35 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 158787330 bytes.
14/11/22 17:07:35 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 158787330 bytes.
14/11/22 17:07:35 WARN CacheManager: Not enough space to cache partition rdd_6_1 in memory! Free memory is 158787330 bytes.
14/11/22 17:07:35 WARN CacheManager: Not enough space to cache partition rdd_8_1 in memory! Free memory is 158787330 bytes.
14/11/22 17:07:35 WARN CacheManager: Not enough space to cache partition rdd_6_2 in memory! Free memory is 158787330 bytes.
14/11/22 17:07:35 WARN CacheManager: Not enough space to cache partition rdd_8_2 in memory! Free memory is 158787330 bytes.
14/11/22 17:07:36 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 158791130 bytes.
14/11/22 17:07:36 WARN CacheManager: Not enough space to cache partition rdd_6_3 in memory! Free memory is 158791130 bytes.
14/11/22 17:07:36 WARN CacheManager: Not enough space to cache partition rdd_8_3 in memory! Free memory is 158791130 bytes.
14/11/22 17:07:36 WARN CacheManager: Not enough space to cache partition rdd_4_0 in memory! Free memory is 158791130 bytes.
14/11/22 17:07:36 WARN CacheManager: Not enough space to cache partition rdd_6_0 in memory! Free memory is 158791130 bytes.
14/11/22 17:07:36 WARN CacheManager: Not enough space to cache partition rdd_8_0 in memory! Free memory is 158791130 bytes.
Training: 385580, test: 96120.
14/11/22 17:07:41 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72899530 bytes.
14/11/22 17:07:41 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72899530 bytes.
14/11/22 17:07:41 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72899530 bytes.
14/11/22 17:07:41 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72899530 bytes.
14/11/22 17:07:41 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72899530 bytes.
14/11/22 17:07:41 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72899530 bytes.
14/11/22 17:07:44 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72905066 bytes.
14/11/22 17:07:44 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72905066 bytes.
14/11/22 17:07:44 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72905066 bytes.
14/11/22 17:07:44 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72905066 bytes.
14/11/22 17:07:45 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72905066 bytes.
14/11/22 17:07:45 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72905066 bytes.
14/11/22 17:07:47 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
14/11/22 17:07:47 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
14/11/22 17:07:48 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72900010 bytes.
14/11/22 17:07:48 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72900010 bytes.
14/11/22 17:07:48 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72900010 bytes.
14/11/22 17:07:48 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72900010 bytes.
14/11/22 17:07:48 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72900010 bytes.
14/11/22 17:07:48 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72900010 bytes.
14/11/22 17:07:51 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72892226 bytes.
14/11/22 17:07:51 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72892226 bytes.
14/11/22 17:07:51 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72892226 bytes.
14/11/22 17:07:51 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72892226 bytes.
14/11/22 17:07:52 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72892226 bytes.
14/11/22 17:07:52 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72892226 bytes.
14/11/22 17:07:55 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:07:55 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:07:55 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:07:55 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:07:55 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:07:55 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:07:58 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:07:58 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:07:58 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:07:58 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:07:59 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:07:59 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:02 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:02 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:02 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:02 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:02 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:02 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:05 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:05 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:05 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:05 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:05 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:05 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:08 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:08 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:08 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:08 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:09 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:09 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:12 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:12 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:12 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:12 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:12 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:12 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:15 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:15 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:15 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:15 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:16 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:16 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:19 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72887738 bytes.
14/11/22 17:08:19 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72887738 bytes.
14/11/22 17:08:19 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72887738 bytes.
14/11/22 17:08:19 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72887738 bytes.
14/11/22 17:08:19 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72887738 bytes.
14/11/22 17:08:19 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72887738 bytes.
14/11/22 17:08:22 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:22 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:22 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:22 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:23 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:23 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:26 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:26 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:26 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:26 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:26 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:26 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:29 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72882858 bytes.
14/11/22 17:08:29 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72882858 bytes.
14/11/22 17:08:29 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72882858 bytes.
14/11/22 17:08:29 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72882858 bytes.
14/11/22 17:08:30 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:30 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:33 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:33 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:33 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:33 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:33 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:33 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:08:36 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:36 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:36 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:36 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:37 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:37 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:40 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:40 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:40 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:40 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:40 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:40 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:43 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:43 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:43 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:43 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:44 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:44 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:47 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:47 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:47 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:47 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:47 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:47 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:51 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:51 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:51 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:51 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:51 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:51 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:54 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:54 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:54 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:54 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:54 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:54 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72893082 bytes.
14/11/22 17:08:58 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:58 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:58 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:58 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72885298 bytes.
14/11/22 17:08:58 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:08:58 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:09:01 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:01 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:01 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:01 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:02 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:02 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:05 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72887738 bytes.
14/11/22 17:09:05 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72887738 bytes.
14/11/22 17:09:05 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72887738 bytes.
14/11/22 17:09:05 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72887738 bytes.
14/11/22 17:09:05 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72887738 bytes.
14/11/22 17:09:05 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72887738 bytes.
14/11/22 17:09:08 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:09:08 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:09:08 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72903306 bytes.
14/11/22 17:09:08 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72903306 bytes.
14/11/22 17:09:09 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:09:09 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72903306 bytes.
14/11/22 17:09:12 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:12 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:12 WARN CacheManager: Not enough space to cache partition rdd_7_2 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:12 WARN CacheManager: Not enough space to cache partition rdd_7_3 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:12 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:12 WARN CacheManager: Not enough space to cache partition rdd_7_1 in memory! Free memory is 72895522 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_8_1 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_4_1 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_8_1 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_8_3 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_8_2 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_4_3 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_4_2 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_8_3 in memory! Free memory is 58905314 bytes.
14/11/22 17:09:15 WARN CacheManager: Not enough space to cache partition rdd_8_2 in memory! Free memory is 58905314 bytes.
Test areaUnderPR = 0.8666401497180547.
Test areaUnderROC = 0.7099423688872086.
1 个解决方案
解决方案1
20 已采纳 2014-11-22 16:48:17
Currently you are running with the default memory options, like indicated in the logs: 目前，您正在使用默认内存选项运行，如日志中所示：
14/11/22 17:07:24 INFO MemoryStore: MemoryStore started with capacity 265.1 MB
If you are running locally, you need to set the option --driver-memory 4G instead. 如果您在本地运行，则需要设置选项--driver-memory 4G 。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
什么时候实现了自动 Spark RDD 分区缓存驱逐？ - When was automatic Spark RDD partition cache eviction implemented?
为什么RDD没有在火花的每次迭代中都保留在内存中？ - Why the RDD is not persisted in memory for every iteration in spark?
在 spark 中均匀划分一个 rdd - uniformly partition a rdd in spark
Spark：打破分区迭代器以获得更好的内存管理？ - Spark: break partition iterator for better memory management?
从内存中触发非缓存的RDD逐出 - Spark non-cached RDD Eviction from memory
如何Spark实现交互式内存缓存？ - How to Spark implement the interactive In-Memory Cache?
如何通过Spark中的键对RDD进行分区？ - How to partition RDD by key in Spark?
Spark SQL：通过“order by”改善缓存内存占用 - Spark SQL: Cache Memory footprint improves with 'order by'
Spark配置因内存不足错误 - Spark configuration for Out of memory error
OutOfMemoryError：Spark中的Java堆空间和内存变量 - OutOfMemoryError: Java heap space and memory variables in Spark
相关标签