[英]java.lang.OutOfMemoryError: Java heap space error while running seq2sparse in mahout
我正在嘗試在mahout中使用k均值來聚類一些手工制作的日期。 我創建了6個文件，每個文件中幾乎沒有1或2個單詞的文字。 使用./mahout seqdirectory從其中創建一個序列文件。 嘗試使用./mahout seq2sparse命令將序列文件轉換為矢量時，出現java.lang.OutOfMemoryError：Java堆空間錯誤。 序列文件的大小為0.215 KB。
命令：./mahout seq2sparse -i mokha / output -o mokha / vector -ow
錯誤日志：
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/bitnami/mahout/mahout-distribution-0.5/m
ahout-examples-0.5-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/bitnami/mahout/mahout-distribution-0.5/l
ib/slf4j-jcl-1.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Apr 24, 2013 2:25:11 AM org.slf4j.impl.JCLLoggerAdapter warn
WARNING: No seq2sparse.props found on classpath, will use command-line arguments
only
Apr 24, 2013 2:25:12 AM org.slf4j.impl.JCLLoggerAdapter info
INFO: Maximum n-gram size is: 1
Apr 24, 2013 2:25:12 AM org.slf4j.impl.JCLLoggerAdapter info
INFO: Deleting mokha/vector
Apr 24, 2013 2:25:12 AM org.slf4j.impl.JCLLoggerAdapter info
INFO: Minimum LLR value: 1.0
Apr 24, 2013 2:25:12 AM org.slf4j.impl.JCLLoggerAdapter info
INFO: Number of reduce tasks: 1
Apr 24, 2013 2:25:12 AM org.apache.hadoop.metrics.jvm.JvmMetrics init
INFO: Initializing JVM Metrics with processName=JobTracker, sessionId=
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 1
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Running job: job_local_0001
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 1
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commi
ting
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapred.Task commit
INFO: Task attempt_local_0001_m_000000_0 is allowed to commit now
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapreduce.lib.output.FileOutputCommitt
er commitTask
INFO: Saved output of task 'attempt_local_0001_m_000000_0' to mokha/vector/token
ized-documents
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Apr 24, 2013 2:25:12 AM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0001_m_000000_0' done.
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 100% reduce 0%
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0001
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: Counters: 5
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: FileSystemCounters
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_READ=1471400
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_WRITTEN=1496783
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: Map-Reduce Framework
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: Map input records=6
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: Spilled Records=0
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.Counters log
INFO: Map output records=6
Apr 24, 2013 2:25:13 AM org.apache.hadoop.metrics.jvm.JvmMetrics init
INFO: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - al
ready initialized
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 1
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Running job: job_local_0002
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 1
Apr 24, 2013 2:25:13 AM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Apr 24, 2013 2:25:14 AM org.apache.hadoop.mapred.LocalJobRunner$Job run
WARNING: job_local_0002
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:
781)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.ja
va:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:1
77)
Apr 24, 2013 2:25:14 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 0% reduce 0%
Apr 24, 2013 2:25:14 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0002
Apr 24, 2013 2:25:14 AM org.apache.hadoop.mapred.Counters log
INFO: Counters: 0
Apr 24, 2013 2:25:14 AM org.apache.hadoop.metrics.jvm.JvmMetrics init
INFO: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - al
ready initialized
Apr 24, 2013 2:25:15 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 1
Apr 24, 2013 2:25:15 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Running job: job_local_0003
Apr 24, 2013 2:25:15 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 1
Apr 24, 2013 2:25:15 AM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Apr 24, 2013 2:25:15 AM org.apache.hadoop.mapred.LocalJobRunner$Job run
WARNING: job_local_0003
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:
781)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.ja
va:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:1
77)
Apr 24, 2013 2:25:16 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 0% reduce 0%
Apr 24, 2013 2:25:16 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0003
Apr 24, 2013 2:25:16 AM org.apache.hadoop.mapred.Counters log
INFO: Counters: 0
Apr 24, 2013 2:25:16 AM org.apache.hadoop.metrics.jvm.JvmMetrics init
INFO: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - al
ready initialized
Apr 24, 2013 2:25:16 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 0
Apr 24, 2013 2:25:16 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Running job: job_local_0004
Apr 24, 2013 2:25:16 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat li
stStatus
INFO: Total input paths to process : 0
Apr 24, 2013 2:25:16 AM org.apache.hadoop.mapred.LocalJobRunner$Job run
WARNING: job_local_0004
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:1
24)
Apr 24, 2013 2:25:17 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 0% reduce 0%
Apr 24, 2013 2:25:17 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0004
Apr 24, 2013 2:25:17 AM org.apache.hadoop.mapred.Counters log
INFO: Counters: 0
Apr 24, 2013 2:25:17 AM org.slf4j.impl.JCLLoggerAdapter info
INFO: Deleting mokha/vector/partial-vectors-0
Apr 24, 2013 2:25:17 AM org.apache.hadoop.metrics.jvm.JvmMetrics init
INFO: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - al
ready initialized
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputExc
eption: Input path does not exist: file:/home/bitnami/mahout/mahout-distribution
-0.5/bin/mokha/vector/tf-vectors
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
InputFormat.java:224)
at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listSta
tus(SequenceFileInputFormat.java:55)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
nputFormat.java:241)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
79)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
at org.apache.mahout.vectorizer.tfidf.TFIDFConverter.startDFCounting(TFI
DFConverter.java:350)
at org.apache.mahout.vectorizer.tfidf.TFIDFConverter.processTfIdf(TFIDFC
onverter.java:151)
at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(Spars
eVectorsFromSequenceFiles.java:262)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(Spar
seVectorsFromSequenceFiles.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(Progra
mDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:187)
bin / mahout腳本讀取環境變量'MAHOUT_HEAPSIZE'（以兆字節為單位），並從中設置'JAVA_HEAP_MAX'變量（如果存在）。 我正在使用的mahout版本（0.8）將JAVA_HEAP_MAX設置為3G。 執行
export MAHOUT_HEAPSIZE=10000m
在機蓋群集運行似乎幫助我的運行在一台計算機上保持更長的時間之前。 但是，我懷疑最好的解決方案是切換到群集上運行。
供參考，還有另一篇相關文章： Mahout用完了堆空間
我不知道您是否嘗試過此功能，只是將其發布以防萬一您錯過了它。
'Set the environment variable 'MAVEN_OPTS' to allow for more memory via 'export MAVEN_OPTS=-Xmx1024m'
請參考（在常見問題部分） 在這里
帶有Java Vector.addElement（Object o）方法的java.lang.OutOfMemoryError
[英]java.lang.OutOfMemoryError with Java Vector.addElement(Object o) method
[java.lang.String中; 無法強制轉換為java.lang.String
[英][java.lang.String; cannot be cast to java.lang.String
為什么會出現此錯誤：java.lang.Exception：XMLEncoder：丟棄語句Vector.add（）？
[英]Why did I get this error : java.lang.Exception: XMLEncoder: discarding statement Vector.add()?
java.lang.ClassCastException：無法將java.lang.String強制轉換為java.util.Vector
[英]java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Vector
我在Vector中插入元素時收到錯誤java.lang.NullPointerException
[英]I am getting error java.lang.NullPointerException when insert element in Vector
錯誤：10 [1.1、1.2、1.3、1.4、1.5、1.6] 線程“主”java.lang.ArrayIndexOutOfBoundsException 中的異常：數組索引超出范圍：0 處為 ja
[英]Error: 10 [1.1, 1.2, 1.3, 1.4, 1.5, 1.6] Exception in thread “main” java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0 at ja
java.lang.ClassCastException：[Ljava.lang.Object; 無法轉換為[Ljava.lang.Comparable;
[英]java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.lang.Comparable;
Java：無法修復未經檢查的轉換需要java.util.Collection <? extends java.lang.String>
[英]Java: Unable to fix unchecked conversion required java.util.Collection<? extends java.lang.String>
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.