从当前行向后查找第一个非空值扫描行

Question

Software

I'm using Pentaho Data Integration 5.4

Input data & explanation

Input data from a file (simplified, there are more columns):

number       name
1009       ProductA
2150       ProductB
3235       ProductC
           ProductD
           ProductE
1234       ProductF
7765       ProductG
4566       ProductH
           ProductI
9907       ProductJ

The issue is that I had an Excel file format xlsx which has the data with merged cells, and for one value of id there are 1..n rows of values.

After converting that file to csv values for next rows (other than first) are missing, despite the one column which was not merged (see example id=3 , id=6 ).

I'm generating a sequence using step Add sequence , the input is sorted the way it was originally stored in a file.

Steps to achieve the goal

Basically what I need to do is:

• Find first non-null value that has sequence_number less than current_row.sequence_number
• Concatenate the value from field name to that matching row
• Keep scanning next rows with sequence_number higher than the last scanned

As stated before, there can be 1..n rows of values for such case.

Expected output

number       name
1009       ProductA
2150       ProductB
3235       ProductC; ProductD; ProductE
1234       ProductF
7765       ProductG
4566       ProductH; ProductI
9907       ProductJ

My approach

I believe I'm able to do this in a loop, by using Analytic Query and calculating LAG(1) and then concatenating the column name for one row with null values and discarding other column values from null row - and then doing this in a loop (for like 20 times assuming this is maximum), but I do consider this a bad idea.

There are probably better ways to achieve this result using for example Java Script step with scanning the rows backward from current (based on sequence number), but I'm unaware of those functions, if they do exist.

How can I achieve this using Modified Java Script Value step, or any other efficient way without using a loop for entire content of the file until there are no empty rows?

Answer 1

To solve this, I would use Modified Java Script Value to save the last seen product and use this for all rows, and then use Group By to group the columns.

Answer 2

Introduction

Merged adjacent cells in Excel files are presented on the image below.

When opened as a plain text file, it actually creates gaps (data from merged cell is missing) for every row but first that contains the merged cell.

number  name
1000/P  um6p1
        um1p2
        um1p3
1500    um2p1
9823    um3p1
83424   um4p1
        um4p2
        um4p3
        um4p4
21390   um5p1

While @bolav answer addresses the problem, there is a simplier and probably more efficient approach to this issue in Kettle .

---

Approach

• In Microsoft Excel Input step go to Fields tab and mark Repeat option as Y for columns that store values in merged cells
• Use Sort rows on number column because Group by step needs the input to be sorted
• Group by on field number and aggregate name with Concatenate strings separated by as type and ; as value

From Pentaho User Guide :

Repeat If set to Y, will repeat this value if the field in the next row is empty.