deep learning for 3d segmentation

Question

I have 20 heart images taken at the same location and there are 50 of those images. So, 20 images are one input and I have 50 inputs. Because heart is moving, all images are different. I want to make those images as input and then segmented binary mask images as output using deep learning.

I treat my inputs as 3D and found a lot of 3D CNN or FCN online. But those are the case of x,y,z 3D. My case is x, y, t.

Any suggestion?

Answer 1

This problem is similar to video object segmentation , where an object needs to be semantically segmented across several frames in a video by utilizing temporal information across frames. In your case, each set of input image can be thought of as a video with 20 frames with dataset containing 50 videos and you can use supervised, semi-supervised or unsupervised video object segmentation models (depending on the availability/cost of obtaining ground truth masks). This formulation solves the task of segmenting input image stack in (x,y,t) order with FCNs.

Below are references I found for research work related to video object segmentation:

• https://paperswithcode.com/task/video-object-segmentation
• https://paperswithcode.com/task/semi-supervised-video-object-segmentation
• https://paperswithcode.com/task/unsupervised-video-object-segmentation