蜂巢：消耗大量小型XML文件的最佳方法

Question

I would like to get advices regarding the best approach to store my data in HDFS and to further retrieve values from it using SQL through Hive.

I receive a lot of files in XML format, basically tens of thousands a day. Each file has about 10 kB and obeys a given XSD schema. Currently I have more than 120 TB of these XML files stored in a filesystem.

I was wondering to ingest all these XML files into HDFS in order to offer SQL interface to some applications perform relational queries against the data.

What key technologies do you think I'll need to build this solution?

For efficient processing, perhaps I'd need to convert these XML files in a better format for Hadoop (ie, RCfile or ORC) and store them in HDFS. The problem is the schema of these files is expected to change over time. The nature of my data seems to benefit from partitioning (ie, by date/time or state). Also, I don't know if data compression is a good idea.

Here is a sample content I have inside a single XML file:

<invoice schema_version="1.1">
  <general id="123456798">
    <creationdate>2016-03-21 16:25:09-03:00</creationdate>
  </general>
  <buyer id="11">
    <name>The Buyer</name>
    <address>
      <street>1st St</street>
      <city>Los Angeles</city>
      <state>CA</state>
    </address>
  </buyer>
  <seller id="22">
    <name>The Seller</name>
    <address>
      <street>2nd Ave</street>
      <city>Miami</city>
      <state>FL</state>
    </address>
  </seller>
  <items>
    <product id="123">
      <name>Blue Pen</name>
      <price>1.50</price>
      <quantity>4</quantity>
      <subtotal>6.00</subtotal>
    </product>
    <product id="456">
      <name>White Board</name>
      <price>5.20</price>
      <quantity>2</quantity>
      <subtotal>10.40</subtotal>
    </product>
  </items>
  <amount>
    <products>16.40</products>
    <shipping>2.35</shipping>
    <total>18.75</shipping>
  </amount>
</invoice>

Thus, I'd like to perform SQL queries similar to these:

SELECT general.creationdate, buyer.name, amount.total
FROM invoice
WHERE general_id = '123456798';

SELECT count(*) AS qty, sum(amount.total) AS total
FROM invoice
WHERE general.creationdate >= '2016-03-01'
GROUP BY seller.address.state;

SELECT b.name, avg(b.price) AS avg_price, sum(b.quantity) AS sum_quantity
FROM invoice a
  JOIN invoice_items b ON (...)
WHERE a.buyer.address.state = 'CA'
GROUP BY b.name
ORDER BY sum_quantity DESC;

Thanks in advance!

Answer 1

You can write xslt-file to translate incoming xml's into csv format and apply it to your files eg using streaming job:

hadoop jar hadoop-streaming.jar \
    -mapper 'xsltproc file.xslt -' -file file.xslt \
    -input /path/to/your/xmls \
    -output /path/to/resulting/files

look on https://github.com/whale2/iow-hadoop-streaming if you want to use avro or parquet instead of simple text, this lib also can handle multiple outputs, so you can save each table in separate folder (and ofcourse subfolders if you want partitioning).

Next, just create external table in hive to your resulting files and make your sql queries.

if your schema will change, you can just change xslt-file.

add: to make it working you should delete newlines from input xmls or write wrapper (see http://www.science.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.1_--_Streaming_XML_Files )

upd You should write 1 xslt to produce alll record in file like this:

header\tval1,val2,val3
details\tval1,val2,val3,val4

next, add option -outputformat net.iponweb.hadoop.streaming.io.ByKeyOutputFormat to you command and you get different files for each key.

What about hadoop profit in this task - distributed processing, if you have small amount of data, you dont need hadoop