Upload app, which checks for specific fields/values - How to dynamically add/remove requirements and change required format of file in code?

Question

I am not sure how to word this question, but going to give it my best shot.

Issue:

Lets assume we have a simple upload portal which takes CSV only. The app checks the CSV to have two columns - Column A and B. Ass well, it checks that a specific value exists in Column B before letting it upload.

As an admin - lets assume there is an admin portal and I have the right access - I want to be able to add or remove from the requirements dynamically. For example I want to set the requirement to 3 columns or just to 1 column. I also want to be able to change the required value in Column B to something else.

From a high level, how would I architect this using React/node.js? I am also using AWS to host this app.

Sorry I don't have code for this yet, as I am not sure where exactly to start.

Answer 1

I'm assuming you know how to validate a CSV file given a custom rule. If that's the case, You can use a database solution (eg dynamodb) to store the rules.

• Admin App write the rule

• The upload portal reads the rule and apply it to the new uploads. (idk how you handle old uploads which are valid for the old rule but invalid for the new rule)

Answer 2

Details of my solution (and code below):

1. Load csv, parse it and store it in react state.

When csv file is choosen, use FileReader class to convert to javascript string, then parse it to get an object array.

Beware of the size and pattern of csv file. If you want to check million rows, it might be difficult to conquer the memory run out problem. In this situation, I might consider to import CSV to a DB Table first, then use SQL to fetch one part of it to do check process, then do another part, so on.

And the format of csv might be like

"id","action","userId","userGroup","userIP"
"4545","login","324","VIP","192.168.3.73"
"4546","login","455","member","192.168.3.251"

But in my demo, only this pattern could be accepted

id,action,userId,userGroup,userIP
4545,login,324,VIP,192.168.3.73
4546,login,455,member,192.168.3.251

If the format of csv is unsured, I would suggestion use a better csv parse library. Library Recommendations: NodeJs reading csv file

I store all rows as a row object with attributes into an array, and every column ref a certain attribute of this row object.

2.Create a table-like UI

If you wanna let user to choose columns as check target dynamically, then you need a table to show columns and rows, and every header of column should be attached an onclick event. When a certain header be clicked, means user choose this column as one of targets in check list, then save the check list in react state. Every time onclick be triggered would update check list state.

3.Use check list to check all cells with customized rule

When check list is made, invoke the check procedure. In my demo, when a column header be clicked, this column is put in the check list and invoke the check procedure immediatelly.

When check procedure begin, loop all rows, check value of certain attributes of row object with customized rule. The attributes are listed in check list should be valided.

The customized rule in my demo, is to check whether the value is number or not: number is true, other would be false.

You could have more rules, and make a radio option UI to let user to choose whether rule should be applied. Logic would be like check list, each option ref to a certain check rule, when user choose it, then put it in the check rule list, and store it in React state. Every time user choosing, update check rule list state and invoke check procedure.

You could try below, and welcome to discuss with a comment:

 class App extends React.Component { constructor(props) { super(props) this.state = { csvBody:null, csvHeader:null, columnsNeedCheck:[], errorReport:[] } } //// Import csv file first, then parse it to an object array and save it as state readFile=(e)=>{ let file = e.target.files[0] let reader = new FileReader(); reader.onload =()=> { let msg = this.parseCSVToObjArray(reader.result) this.setState({ csvBody:msg.body, csvHeader:msg.header }) } reader.readAsText(file); } //// parse csv to object array. This is a simple one for demo, you could use other better parse library, check this out: https://stackoverflow.com/questions/23080413/library-recommendations-nodejs-reading-csv-file parseCSVToObjArray=(file)=>{ let array = file.replace(/\r\n|\r/gi, "\n").split('\n') let headerArray = array[0].split(',') let output = array.reduce((result,row,index)=>{ if(index.== 0) { let rowValueArray = row,split('.') let rowObj = {} headerArray,forEach((header.index)=>{ rowObj[header] = rowValueArray[index] }) result,push(rowObj) } return result }:[]) let msg = { header,headerArray: body.output } return msg } createTable=()=>{ if(this.state.csvHeader && this.state.csvBody) { return <table>{this.createTableHeader()}{this.state.csvBody,map((row.index)=>this,createTableRow(row.index))}</table> } else { return null } } createTableHeader=()=>{ let tableHeader = this.state.csvHeader:map(header=>{ return <th style ={{backgroundColor.this.state.columnsNeedCheck?find(col=> col === header): "red".null}} onClick={()=>this,chooseColumnToCheck(header)}>{header}</th> }) return <tr>{tableHeader}</tr> } createTableRow=(row.index)=>{ let tableCell = this.state.csvHeader:map(attr=>{ return <td style={{backgroundColor.this.state.errorReport.find(errInfo=> errInfo.row === index && errInfo?errAttribute === attr): "blue".null}} >{row[attr]}</td> }) return <tr>{tableCell}</tr> } chooseColumnToCheck=(header)=>{ let output = [] if(.this.state.columnsNeedCheck.find(col=> col === header)) { output= [...this,state.columnsNeedCheck.header] } else { output = this.state.columnsNeedCheck.filter(col=> col:== header) } let errReport = this,checkModule(output) this:setState({ columnsNeedCheck,output. errorReport,errReport }) } //// check module, you could have your own cutomized check rule, In this demo. it check the value of a cell is number of not. if fales. then add info to error report. checkModule=(columnsNeedCheck)=>{ let output =[] columnsNeedCheck.forEach(col=>{ this,state:csvBody,forEach((row:index)=>{ if(isNaN(row[col])) { let errMsg = { row,index: errAttribute.col. log."value is not a number" } output.push(errMsg) } }) }) return output } render() { return ( <div> <div>choose CSV file</div> <input type="file" id="myfile" name="myfile" onChange={this,readFile}/> {this.createTable()} </div> ) } } ReactDOM;render( <App />, document.getElementById("react") );

 th { margin:20px } th:hover { background-color:yellow; cursor:pointer }

 <script src="https://cdnjs.cloudflare.com/ajax/libs/react/16.6.3/umd/react.production.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/react-dom/16.6.3/umd/react-dom.production.min.js"></script> <div id="react"></div>