Problems Synchronizing threads in Java

Question

ok so I did my research there is plenty of questions here on thread synchronization but non of them really hit the point. I am currently working in Opencv, I get a frame from the camera containing vehicles, remove the background and track the vehicles, but before I do this I do some pre-processing and post-processing like removing noise with blur, all this runs in a single thread and it works great but here comes an issue, I now want to read number plates, for this i need a higher resolution frame otherwise for every frame I will not detect a single plate, but as soon as i increase my frame size I get a performance hit,my threads slows down to the point that my program no longer qualifies to be a real time system.

So I thought of adding more threads to my scene each to specialize on one task here is a list of my tasks

 //recieves fame from Camera
 1. preprocess
 //recieves a Frame from preprocess and removes the background
 2. remove background
//recieves a Frame from backgroundremover and tracks the vehicles
 3. postprocess

If I run the threads one by one am thinking it will still be slow instead I thought or running the threads simultenously but the issues it they use the same objects, declaring them volatile will mean threads waiting for the thread with lock to complete for it to use the object which will mean a slow system again so my question is how can I run these threads simultaneously without having to wait for others?

I have looked at a dozen of multithreading techniques in Java but finding it really hard to come up with a way of making this work. So far I have looked at

 1. Thread synchronization using the keyword volatile
 2. Thread synchronization using the keyword synchronized
 3. Multiple thread locks using a lock object
 4. Using threadpools
 5. Using the Countdown Latch
 6. Wait and motify
 7. Using Semaphores(which seemed like a good idea).

Here is the code I want to break down into those threads

public void VideoProcessor()
{


    videProcessorThread = new Thread(new Runnable()
            {

        @Override
        public void run()
        {

            try{

                int i = 0;
                while (isPlaying() && isMainScreenONOFF()) {

                    camera.read(frame);
                    //set default and max frame speed
                    camera.set(Videoio.CAP_PROP_FPS, 25);
                    //get frame speed just incase it did not set
                    fps = camera.get(Videoio.CAP_PROP_FPS);
                    //if(frame.height() > imgHeight || frame.width() > imgWidth)
                    Imgproc.resize(frame, frame, frameSize);
                    //check if to convert or not
                    if(getblackAndWhite())
                    Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGB2GRAY);

                    imag = frame.clone();
                    if(rOI){

                    //incase user adjusted the lines we try calculate there new sizes
                    adjustLinesPositionAndSize(xAxisSlider.getValue(), yAxisSlider.getValue());
                    //then we continue and draw the lines

                    if(!roadIdentified)
                    roadTypeIdentifier(getPointA1(), getPointA2());
                    }

                    viewClass.updateCarCounter(tracker.getCountAB(), tracker.getCountBA());


                    if (i == 0) {
                        // jFrame.setSize(FRAME_WIDTH, FRAME_HEIGHT);
                        diffFrame = new Mat(outbox.size(), CvType.CV_8UC1);
                        diffFrame = outbox.clone();
                    }

                    if (i == 1) {
                        diffFrame = new Mat(frame.size(), CvType.CV_8UC1);

                        removeBackground(frame, diffFrame, mBGSub, thresHold.getValue(), learningRate.getValue());

                        frame = diffFrame.clone();
                        array = detectionContours(diffFrame, maximumBlob.getValue(), minimumBlob.getValue());
                        Vector<VehicleTrack> detections = new Vector<>();
                        Iterator<Rect> it = array.iterator();
                        while (it.hasNext()) {
                            Rect obj = it.next();           
                            int ObjectCenterX = (int) ((obj.tl().x + obj.br().x) / 2);
                            int ObjectCenterY = (int) ((obj.tl().y + obj.br().y) / 2);
                            //try counter

                            //add centroid and bounding rectangle
                            Point pt = new Point(ObjectCenterX, ObjectCenterY);
                            VehicleTrack track = new VehicleTrack(frame, pt, obj);

                            detections.add(track);
                        }
                        if (array.size() > 0) {
                            tracker.update(array, detections, imag);
                            Iterator<Rect> it3 = array.iterator();
                            while (it3.hasNext()) {
                                Rect obj = it3.next();

                                int ObjectCenterX = (int) ((obj.tl().x + obj.br().x) / 2);
                                int ObjectCenterY = (int) ((obj.tl().y + obj.br().y) / 2);
                                Point pt = null;
                                pt = new Point(ObjectCenterX, ObjectCenterY);                                   
                                Imgproc.rectangle(imag, obj.br(), obj.tl(), new Scalar(0, 255, 0), 2);
                                Imgproc.circle(imag, pt, 1, new Scalar(0, 0, 255), 2);
                                //count and eleminate counted
                                tracker.removeCounted(tracker.tracks);
                            }
                        } else if (array.size() == 0) {
                            tracker.updateKalman(imag, detections);
                        }
                    }

                    i = 1;  
                    //Convert Image and display to View
                   displayVideo();
                }
                //if error occur or video finishes
                Image image = new Image("/assets/eyeMain.png");  
                viewClass.updateMainImageView(image);

                }catch(Exception e)
                {
                    e.printStackTrace();
                    System.out.println("Video Stopped Unexpectedly");
                }
            //thread is done    

          }
        });videProcessorThread.start();

}

Answer 1

As no-one else has replied, I'll have a go.

You've already covered the main technical aspects in your questions (locking, synchronisation etc). Whichever way you look at it, there is no general solution to designing a multi-threaded system. If you have threads accessing the same objects you need to design your synchronisation and you can get threads blocking each other, slowing everything down.

The first thing to do is to do some performance profiling as there is no point making things run in parallel if they are not slowing things down.

That said, I think there are three approaches you could take in your case.

1. Have a single thread process each frame but have a pool of threads processing frames in parallel. If it takes a second to process a frame and you have 25fps you'd need at least 25 threads to keep up with the frame rate. You'd always be about a second behind real time but you should be able to keep up with the frame rate.

A typical way to implement this would be to put the incoming frames in a queue. You then have a pool of threads reading the latest frame from the queue and processing it. The downside of this design is that you can't guarantee in which order you would get the results of the processing so you might need to add some more logic around sorting the results.

The advantages are that:

• There is very little contention, just around getting the frames off the queue, and that should be minimal
• It is easy to tune and scale by adjusting the number of threads. It could even run on multiple machines, depending on how easy it is to move frames between machines
• You avoid the overhead of creating a new thread each time as each thread processing one frame after another
• It is easy to monitor as you can just look at the size of the queue

• Error handling can be implemented easily, eg use ActiveMQ to re-queue a frame if a thread crashes.

  2. Run parts of your algorithm in parallel. The way you've written it (pre-process, process, post-process), I don't see this is suitable as you can't do the post processing at the same time as the pre-processing. However, if you can express your algorithm in steps that can be run in parallel then it might work.

  3. Try and run specific parts of your code in parallel. Looking at the code you posted the iterators are the obvious choice. Is there any reason not to run the iterator loops in parallel? If you can, experiment with the Java parallel streams to see if that bring any performance gains.

Personally I'd try option 1 first as its quick and simple.