[英]golang syscall, locked to thread
I am attempting to create an program to scrape xml files.我正在尝试创建一个程序来抓取 xml 文件。 I'm experimenting with go because of it's goroutines.
我正在试验 go 因为它是 goroutines。 I have several thousand files, so some type of multiprocessing is almost a necessity...
我有几千个文件,所以某种类型的多处理几乎是必要的......
I got a program to successfully run, and convert xml to csv(as a test, not quite the end result), on a test set of files, but when run with the full set of files, it gives this:我有一个程序成功运行,并将 xml 转换为 csv(作为测试,不是最终结果),在测试文件集上,但是当使用全套文件运行时,它给出了:
runtime: program exceeds 10000-thread limit
I've been looking for similar problems, and theres a couple, but i haven't found one that was similar enough to solve this.我一直在寻找类似的问题,并且有几个,但我还没有找到一个足够相似的问题来解决这个问题。
and finally heres some code im running:最后继承一些我正在运行的代码:
// main func (start threads)
for i := range filelist {
channels = append(channels, make(chan Test))
go Parse(files[i], channels[len(channels)-1])
}
// Parse func (individual threads)
func Parse(fileName string, c chan Test) {
defer close(c)
doc := etree.NewDocument()
if err := doc.ReadFromFile(fileName); err != nil {
return
}
root := doc.SelectElement("trc:TestResultsCollection")
for _, test := range root.FindElements("//trc:TestResults/tr:ResultSet/tr:TestGroup/tr:Test") {
var outcome Test
outcome.StepType = test.FindElement("./tr:Extension/ts:TSStepProperties/ts:StepType").Text()
outcome.Result = test.FindElement("./tr:Outcome").Attr[0].Value
for _, attr := range test.Attr {
if attr.Key == "name" {
outcome.Name = attr.Value
}
}
for _, attr := range test.FindElement("./tr:TestResult/tr:TestData/c:Datum").Attr {
if attr.Key == "value" {
outcome.Value = attr.Value
}
}
c <- outcome
}
}
// main (process results when threads return)
for c := 0; c < len(channels); c++ {
for i := range channels[c] {
// csv processing with i
}
}
I'm sure theres some ugly code in there.我敢肯定那里有一些丑陋的代码。 I've just picked up go recently from other languages...so i apologize in advance.
我刚刚从其他语言中挑选了 go ......所以我提前道歉。 anyhow
无论如何
any ideas?有任何想法吗?
I apologize for not including the correct error.我很抱歉没有包括正确的错误。 as the comments pointed out i was doing something dumb and creating a routine for every file.
正如评论指出的那样,我正在做一些愚蠢的事情并为每个文件创建一个例程。 Thanks to JimB for correcting me, and torek for providing a solution and this link.
感谢 JimB 纠正我,并感谢 torek 提供解决方案和此链接。 https://gobyexample.com/worker-pools
https://gobyexample.com/worker-pools
jobs := make(chan string, numJobs)
results := make(chan []Test, numJobs)
for w := 0; w < numWorkers; w++ {
go Worker(w, jobs, results)
wg.Add(1)
}
// give workers jobs
for _, i := range files {
if filepath.Ext(i) == ".xml" {
jobs <- ("Path to files" + i)
}
}
close(jobs)
wg.Wait()
//result processing <- results
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.