使用Python通過MapReduce在Hadoop中解析HTML .txt文件

Question

我對使用Hadoop平台和定義MapReduce函數是非常陌生的，並且我很難理解為什么該Mapper在我的MapReduce腳本中不起作用。 我試圖解析.txt文件中以字符串形式編寫的頁面集合，其中每個“行”都表示<page>...</page> 。 這個腳本有什么錯誤？ 感謝您的幫助！

from mrjob.job import MRJob
from mrjob.step import MRStep
from mrjob.compat import jobconf_from_env
import lxml
import mwparserfromhell
import heapq
import re

class MRParser(MRJob):
    def mapper(self, _, line):
        bigString = ''.join(re.findall(r'(<text xml:space="preserve">.*</text>)',line))
        root = etree.fromstring(bigString.decode('utf-8'))
        if not(bigString == ''):
            bigString = etree.tostring(root,method='text', encoding = "UTF-8")    
            wikicode = mwparserfromhell.parse(bigString)
            bigString = wikicode.strip_code()
            yield None, bigString

    def steps(self):
        return [
            MRStep(mapper=self.mapper)
        ]

Answer 1

您缺少減速器功能。 您需要將映射器中的行作為“鍵”（沒有值）傳遞給化簡器。 嘗試這個：

def mapper(self, _, line):
    bigString = ''.join(re.findall(r'(<text xml:space="preserve">.*</text>)',line))
    root = etree.fromstring(bigString.decode('utf-8'))
    if not(bigString == ''):
        bigString = etree.tostring(root,method='text', encoding = "UTF-8")    
        wikicode = mwparserfromhell.parse(bigString)
        bigString = wikicode.strip_code()
        yield bigString, None

def reducer(self, key, values):
    yield key, None

def steps(self):
    return [
        MRStep(mapper=self.mapper, reducer=self.reducer)
    ]