Deeplearning4j predicting used cars prices

Question

I want to predict prices for used cars and I have historical data of sold cars. I scaled numeric values into 0-1 and made other features one hot.

public RestResponse<JSONObject> buildModelDl4j( HttpServletRequest request, HttpServletResponse response, @RequestBody Map<String, String> json ) throws IOException
{
    RestResponse<JSONObject> restResponse = ControllerBase.getRestResponse( request, response, null ) ;

    String path = "\\HOME_EXCHANGE\\uploads\\" + json.get( "filePath" ) ;

    int numLinesToSkip = 1 ;
    char delimiter = ',' ;

    RecordReader recordReader = new CSVRecordReader( numLinesToSkip, delimiter ) ;

    try
    {
        recordReader.initialize( new FileSplit( new File( path ) ) ) ;
    }
    catch( InterruptedException e )
    {
        e.printStackTrace( ) ;
    }

    DataSetIterator iter = new RecordReaderDataSetIterator( recordReader, batchSize, indexToCalc, indexToCalc, true ) ;
    json.put( "numAttr", String.valueOf( numAttr ) ) ;

    //        ds.shuffle( ) ;   //TODO should I shuffle the data ?

    MultiLayerNetwork net = buildNetwork( json ) ;

    net.init( ) ;

    net.setListeners( new ScoreIterationListener( 30 ) ) ;

    DataSet testData = null ;

    for( int i = 0; i < nEpochs; i++ )
    {
        iter.reset( ) ;

        while( iter.hasNext( ) )
        {
            DataSet ds = iter.next( ) ;
            SplitTestAndTrain testAndTrain = ds.splitTestAndTrain( splitRate / 100.0 ) ;
            DataSet trainingData = testAndTrain.getTrain( ) ;
            testData = testAndTrain.getTest( ) ;
            net.fit( trainingData ) ;
        }

        iter.reset( ) ;

        int cnt = 0 ;
        while( iter.hasNext( ) && cnt++ < 3 )
        {
            DataSet ds = iter.next( ) ;
            SplitTestAndTrain testAndTrain = ds.splitTestAndTrain( splitRate / 100.0 ) ;
            testData = testAndTrain.getTest( ) ;
            String testResults = testResults( net, testData, indexToCalc ) ;
            System.err.println( "Test results:  [" + i + "]  \n" + testResults ) ;
        }

    }

    RegressionEvaluation eval = new RegressionEvaluation( ) ;
    INDArray output = net.output( testData.getFeatures( ) ) ;
    eval.eval( testData.getLabels( ), output ) ;
    System.out.println( eval.stats( ) ) ;

    String testResults = testResults( net, testData, indexToCalc ) ;

    result.put( "testResults", testResults ) ;

    System.err.println( "Test results last: \n" + testResults ) ;

    restResponse.setData( result ) ;

    return restResponse ;
}

I build model with parameters passed from front-end, I read data from csv files then train the model. Am I doing the right thing? How should I use test and train data? There are two approached in examples, they use

DataSet ds = iter.next( ) ;
SplitTestAndTrain testAndTrain = ds.splitTestAndTrain( splitRate / 100.0 ) ;
DataSet trainingData = testAndTrain.getTrain( ) ;
testData = testAndTrain.getTest( ) ;
net.fit( trainingData ) ;

or

for( int i = 0; i < nEpochs; i++ )
{
  net.fit( iter ) ;
  iter.reset( ) ;
}

Which one is the right approach?

Answer 1

I build model with parameters passed from front-end, I read data from csv files then train the model. Am I doing the right thing? How should I use test and train data?

A much better approach would be using DataSetIteratorSplitter as below:

DataSetIteratorSplitter dataSetIteratorSplitter = new DataSetIteratorSplitter(dataSetIterator,totalNumBatches,ratio);
multiLayerNetwork.fit(dataSetIteratorSplitter.getTrainIterator(),epochCount);

totalNumBatches would be total number data sets divided by mini batch size. For example if you have 10000 datasets and suppose we assign 8 samples in a single batch, then there are 1250 batches in total.