使用ffmpeg或Python从视频中删除随机背景

Question

I want to remove background from a person's video using ffmpeg or Python . If I record a video at any place, detect the person in the video and then remove anything except that person. Not asking for green or single color background as that can be done through chromakey and I am not looking for that.

I've tried this ( https://tryolabs.com/blog/2018/04/17/announcing-luminoth-0-1/ ) approach but it is giving me output of rectangular box. It is informative enough as area to explore is narrow down enough but still need to remove total background. I've also tried grabcut ( https://docs.opencv.org/4.1.0/d8/d83/tutorial_py_grabcut.html ) but that need user interaction otherwise result isn't too good. I've also tried to use ffmpeg and found this example ( http://oioiiooixiii.blogspot.com/2016/09/ffmpeg-extract-foreground-moving.html ) but it needs still image so I tried to take background picture before recording video with a person but there are many things required to take difference between background image and video frame.

For opencv approach, I've tried this.

img = cv.imread('pic.png')
mask = np.zeros(img.shape[:2], np.uint8)
bgdModel = np.zeros((1, 65), np.float64)
fgdModel = np.zeros((1, 65), np.float64)
rect = (39, 355, 1977, 2638)
cv.grabCut(img, mask, rect, bgdModel, fgdModel, 5, cv.GC_INIT_WITH_RECT)
mask2 = np.where((mask==2)|(mask==0), 0, 1).astype('uint8')
img = img*mask2[:, :, np.newaxis]
plt.imshow(img), plt.colorbar(), plt.show()

But it is removing some of person's part too. Also tried ffmpeg way but not a good result.

ffmpeg -report -y -i "img.jpg" -i "vid.mov" -filter_complex "[1:v]format=yuva444p,lut=c3=128[video2withAlpha],[0:v][video2withAlpha]blend=all_mode=difference[out]" -map "[out]" "output.mp4"

All I need is just a person's image/video take under any normal background without user interaction like area selection or any other thing like that. Luminoth has trained data but that is giving box of person not exact person so that I can remove. Any help or guidance to remove background will be appreciated.

Answer 1

You could try a deep learning based approach. Here is an example repo:

https://github.com/TianzhongSong/Person-Segmentation-Keras

This will give you a mask for a person on arbitrary background. Just load the images with opencv and put them through the network.