交易中春季的性能优化

Question

I am making one enterprise application. In which i have task to parse one csv file that contain suppose 50,000 k records. This csv file supplied by the end user at the time of registration. I made one program to parse the csv file into java object and then save all these object into database. This file contain mobile number and before save the csv file as a java object it firstly validated for mobile number. either it exist on database or not. It it is exist then it fails the validation and stop the exection. Now Suppose two different user called A and B send a request for registration.Controller listen this request by the following code. controller layer

@Transactional
    @RequestMapping(value = "/saveCsvData")
    public ModelAndView saveVMNDataFromCsv(@ModelAttribute("vmn") @Valid VMN vmn,   BindingResult result, HttpSession session, HttpServletRequest request) {

    String response = parseCsvService.parseAndPersistCsv(vmn.getVmnFile(),vmn.getNumberType(), request);
}

on the controller method i have user @Transactional annotation so that this method can complete its work completely.This controller call the helper call to read line by line from csv file and put them into java object.After getting the list of VMN Object with the help of loop i call service method which again call dao method for ecah line. helper class

public String parseAndPersistCsv(MultipartFile csvFile,String numberType, HttpServletRequest request){
            List<VMN> vmnList = new ArrayList<VMN>();
                if(save){
                    for(VMN vmn : vmnList){
                    System.out.println("Remote Host :" + request.getRemoteHost());
                    System.out.println("Remote Add :" + request.getRemoteAddr());

                    vmnService.saveVmn(vmn, numberType); 
                    }
                    response = constantService.getSuccess();
                }
}

service Layer

public String saveVmn(final VMN vmn, String numberType) {
    vmnService.saveVmn(vmn, numberType); 
}

At Dao Layer method looks like this. This method insert record into multiple tables as it can be seen in method code.

Dao Layer

public String saveVmn(final VMN vmn, String numberType) {
String result = "error";
try {
final StringBuffer sql = new StringBuffer();
sql.append(constantService.getInsertInto());
sql.append(VMNTableSingleton.getInstance().getTableName());
sql.append(" (");
sql.append(VMNTableSingleton.getInstance().getVmnNo());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getNumberType());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getOperator());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getCircle());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getBuyingPrice());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getRecurringPrice());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getCreationDate());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getUpdationDate());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getActive());
sql.append(constantService.getComma());
sql.append(VMNTableSingleton.getInstance().getStatus());
sql.append(")");
sql.append(constantService.getValues());
sql.append(" (?,?,?,?,?,?,?,?,?,?)");
logger.info("Saving Vmn..." + sql);
KeyHolder keyHolder = new GeneratedKeyHolder();
int response = jdbcTemplate.update(new PreparedStatementCreator() {
@Override
public PreparedStatement createPreparedStatement(Connection conn) throws SQLException {
PreparedStatement ps = conn.prepareStatement(sql.toString(), Statement.RETURN_GENERATED_KEYS);
ps.setObject(1, vmn.getVmnNo());
ps.setObject(2, vmn.getNumberType());
ps.setObject(3, vmn.getOperator());
ps.setObject(4, vmn.getCircle());
ps.setObject(5, vmn.getBuyingPrice());
ps.setObject(6, vmn.getBuyingPrice());
ps.setObject(7, new Date());
ps.setObject(8, new Date());
ps.setObject(9, true);
ps.setObject(10, vmn.getStatus());
return ps;
}
}, keyHolder);
logger.info("Saved Successfully");
if (response == 1) {
if(vmn.getMappedVmn() != null){
Long vmnId = keyHolder.getKey().longValue();
if(vmnId > 0){
StringBuffer mappedsql = new StringBuffer();
mappedsql.append(constantService.getInsertInto());
mappedsql.append(MapDIDVMNTableSingleton.getInstance().getTableName());
mappedsql.append(" (");
mappedsql.append(MapDIDVMNTableSingleton.getInstance().getDidId());
mappedsql.append(constantService.getComma());
mappedsql.append(MapDIDVMNTableSingleton.getInstance().getMappedId());
mappedsql.append(constantService.getComma());
mappedsql.append(MapDIDVMNTableSingleton.getInstance().getType());
mappedsql.append(constantService.getComma());
mappedsql.append(MapDIDVMNTableSingleton.getInstance().getCreationDate());
mappedsql.append(constantService.getComma());
mappedsql.append(MapDIDVMNTableSingleton.getInstance().getModifiedDate());
mappedsql.append(")");
mappedsql.append(constantService.getValues());
mappedsql.append(" (?,?,?,?,?)");
logger.info("Mapping... DID with VMN");
int mappedresponse = jdbcTemplate.update(mappedsql.toString(),
new Object[] {vmn.getMappedVmn().getVmnId(),vmnId ,vmn.getNumberType(),new Date(),new Date()});
logger.info("Mapped Successfully");
if(mappedresponse == 1){
stringBuffer updatesql = new StringBuffer();
updatesql.append(constantService.getUpdate());
updatesql.append(VMNTableSingleton.getInstance().getTableName());
updatesql.append(constantService.getSet());
updatesql.append(VMNTableSingleton.getInstance().getStatus());
updatesql.append(constantService.getEqual());
updatesql.append(constantService.getQuestionMark());
updatesql.append(constantService.getComma());
updatesql.append(VMNTableSingleton.getInstance().getAllocationDateTime());
updatesql.append(constantService.getEqual());
updatesql.append(constantService.getQuestionMark());
updatesql.append(constantService.getWhere());
updatesql.append(VMNTableSingleton.getInstance().getVmnId());
updatesql.append(constantService.getEqual());
updatesql.append(constantService.getQuestionMark());
logger.info("Updating Vmn..." + updatesql);
jdbcTemplate.update(updatesql.toString(),
new Object[] { constantService.getMapped(),new Date(), vmn.getMappedVmn().getVmnId()});
logger.info("Saved Successfully");
}
}
}
result = "success";
} else {
result = "error";
}
} catch (Exception ex) {
}
return result;
}

Now when i have sent to request toward controller for registration. then i have seen on console both thread accessing method one by by one. By this code inside for loop.

System.out.println("Remote Host :" + request.getRemoteHost()); //10.0.0.0114 System.out.println("Remote Add :" + request.getRemoteAddr());//110.0.0.115

Not this can be dirty read problem because if one thread is reading data then other might be inserting. so to resolve this i have used sync block. Like this

synchronized (this) {
                if(save){
                    for(VMN vmn : vmnList){
                    System.out.println("Remote Host :" + request.getRemoteHost());
                    System.out.println("Remote Add :" + request.getRemoteAddr());

                    vmnService.saveVmn(vmn, numberType); 
                    }
                    response = constantService.getSuccess();
                }               
            }

Now my question is, is this is the right way to do this or it can be done in some other way too.

Answer 1

this can be dirty read problem because if one thread is reading data then other might be inserting

If your concern is visibility then I think this can be taken care of by transaction isolation configured ( usually READ COMMITTED by default) where a thread can only see the committed data and the data it is trying to update.

Also you should consider using batchUpdate method of JDBCTemplate which uses jdbc batching feature. Read and check the existence of number in batches and then update in batches.

Typically such features are handled well by Spring Batch framework. Your use case fits the description of a Spring job which can utilize inbuilt csv readers, processors and chunk based processing to achieve high volume processing of data. The job can be triggered from UI using Spring MVC controllers. For more details you can take a look here .