如何用具有多对多关系的两个表在Java Mapreduce上执行减少侧连接？

Question

First of all, I am not sure whether it is possible or not. If it is possible, I am still not sure whether it is the correct way of doing it.

What I have is:

• Two large csv files called A and B on HDFS
• A has the followin columns: a1, a2, a3, a4
• B has the following columns: b1, b2, b3, b4, b5

What I want is:

• To join the two files if, let's say, a1=b1

The problem I have is:

• If there is a many-to-many relationship between the two files on join keys, how can I perform this with Hadoop Mapreduce on Java?

  As you can see from the illustration below, A has 4 matching row for a1=x and B has 2 matching row for b1=x. Thus, joining the two tables on a1=b1=x produce 4*2 = 8 rows(combinations) as it is shown on the last table. With a reduce side join, I could not manage to do that because this means increasing the key and value pairs which is against the nature of MapReduce.

How can I perform such a thing?

Why it is a problem is:

Let's say the table A is:

a1  a2  a3          a4
x   1   somevalue   somevalue
x   2   somevalue   somevalue
x   3   somevalue   somevalue
x   4   somevalue   somevalue

Let's say the table B is:

b1  b2  b3          b4          b5
x   i   somevalue   somevalue   somevalue
x   j   somevalue   somevalue   somevalue

The result of joining two files on a1=b1:

a1  a2  b2
x   1   i
x   2   i
x   3   i
x   4   i
x   1   j
x   2   j
x   3   j
x   4   j

Answer 1

A full join will always produce M x N output values for each key.

Note that, with a reduce side join, the number of intermediate keys pairs as emitted by the mappers would still be N + M and it is the reducer who does the Cartesian product. So there is nothing wrong about that. Since you control the reducer, you can do further filtering and output only what you need.