将YUV420（NV12）上采样到YUV422的正确方法是什么？

Question

I have a YUV420 image (NV12 image, but it shouldn't matter). I am trying to upsample it to YUV422.

The problem is I am not able to find the right weightage that needs to be given to the UV samples in YUV420 to figure out the UV in YUV422 images.

x -> Y 
0 -> UV

YUV420       YUV422
x x x x      x x x x
o   o        o   o  
x x x x      x x x x
         to  o   o 
x x x x      x x x x
o   o        o   o
x x x x      x x x x
             o   o 

Right now I am just repeating the UV samples, However that is not the right way. So, the question is, Is there a standard way of doing the chroma upsampling? Can someone direct me to the theory of it?

NOTE: I want to implement this and am not interested in tools that will do it. Interested if you can direct me to the source code of these tools that do it according to some standard (assuming there is one :D)

Thanks

Answer 1

You're basically asking that if I have an array of N (where N=height/2) [vertical] samples (which happen to be U - or perhaps V), how can I convert that to an array of N*2 samples with correct interpolation? The answer is indeed interpolation. I'm going to ignore the horizontal aspect because of the scope of your question, but it should be easy to understand that also.

First of all: chroma positioning . Let's assume I had an array of N*2 Y [vertical] samples, and the array of size of U (or V) is only N. It's clear that chroma subsampling implies that for every 2 Y samples, there's only one U (or V) sample [vertically]. But it doesn't tell you where the U/V samples are located. In yuv422 [vertical], this is obvious, the vertical position of each U (or V) aligns perfectly with the Y sample's vertical position. But for subsampled yuv420? Is the center of the vertical position of the first U value aligned with the vertical position of the first Y value ["top"]? Or exactly in between the first and second Y sample ["middle"]? Or (this would be strange, but theoretically it could happen) the center of the second Y sample ["bottom"]?

Y1 U <- top    Y1                Y1
.              .  U <- center    .
Y2             Y2                Y2 U <- bottom

For context, this is the "chroma_sample_location_type" element in the VUI of the SPS in the H.264 header.

Next, what do we do with this information? Well, interpolating from yuv420 to yuv422 basically means [vertically] increasing the resolution times two. Imagine now that you have a grayscale image and you want to increase the resolution. You use a scaling algorithm, and scaling means interpolation. The fact that the target and source height are exact multiples of each other is a special case, but the fact that you have to use a scaling algorithm (ie a scaling filter) doesn't change. So, what filter do you use?

Nearest neighbour is easiest, it means you pick the value from the closest source position:

Y1 U1in <- top               Y1 U1out=U1in
.                            .
Y2                           Y2 U2out=U1in?
.                 becomes    .
Y3 U2in                      Y3 U3out=U2in
.                            .
Y4                           Y4 U4out=U2in?

Mathematically, U2out could also be U2in, since the distance is equal. Here, it also becomes obvious why chroma positioning is important, compare it with center:

Y1                              Y1 U1out=U1in
.  U1in <- center               .
Y2                              Y2 U2out=U1in
.                    becomes    .
Y3                              Y3 U3out=U2in
.  U2in                         .
Y4                              Y4 U4out=U2in

Note how the question marks disappeared. Now, there's not actually any filtering going on yet, so let's get into that.

The easiest filter is bilinear (or in 1D: linear). Here, you use two U samples and interpolate them into one, where the weight of each source pixel is decided by their relative distance to the destination pixel.

Y1 U1in <- top               Y1 U1out=U1in
.                            .
Y2                           Y2 U2out=(U1in+U2in)/2
.                 becomes    .
Y3 U2in                      Y3 U3out=U2in
.                            .
Y4                           Y4 U4out=(U2in+U3in)/2

or:

Y1                              Y1 U1out=U1in
.  U1in <- center               .
Y2                              Y2 U2out=(U1in*3+U2in)/4
.                    becomes    .
Y3                              Y3 U3out=(U1in+U2in*3)/4
.  U2in                         .
Y4                              Y4 U4out=(U2in*3+U3in)/4

As you search more filtering algorithms on eg wikipedia, you'll notice that this is a whole research area and there's more complicated algorithms available, such as bicubic (or in 1D: cubic) or lanczos. For these, IMO it goes too far to explain them here, just look up the functions on wikipedia and do as you need. Which one is right for you is a matter of taste - or better, it basically depends on how you want to balance quality and speed. Higher-tap filters (lanczos > cubic > linear > nearest-neighbour) will give better quality, but will also be computationally slower.

Lastly, you mentioned that you're interested in doing this yourself, which is why I explain all this here. But please understand that writing a bug-free, high-quality multi-tap filtering function (eg for lanczos, or even bicubic) will actually take quite some time/effort and will require significant knowledge in vector processing (SIMD, eg x86 AVX/SSE or arm Neon) to be practically useful. If your end goal is to use this in any serious setting, you probably do want to use existing software that implements these algorithms, eg swscale in ffmpeg, simply because they already implement all of this.

Answer 2

Just double every byte of U and V planes. Or you may take average between consecutive bytes.

You may as well try to step through with debugger in libswscale (from ffmpeg) to see what they do. If it's difficult for you or you don't know how to do that, you can take some picture and convert it to YUV 420 and then convert that YUV420 to YUV422 and then print a few bytes from source U frame and bytes from result U frame and see what kind of math was done. Most likely simply doubling you'll get visually acceptable results.