NDI Lib YUV Color Profile

M

matt.beghin

New member
Our application uses NDI for input and output. We try to do as much on the GPU, so we have fragment shaders for YUV => RGB conversion. Those are made for standard color profiles.
For NDI out, we actually download an RGBA texture and transmit it as is to NDI. But I can't find what color profile NDI library uses internally for RGB<=>YUV conversions.
The best I can get (I do a NDI loop: a color pattern is sent to NDI, and the NDI input stream-loopback, is displayed offsetted vertically, so I can see what the color looks like after N iterations) is by doing this transform (I tweak values manually until it looks perfect with a wide range of colors):

yuv.r = (yuv.r - 0.0625);
yuv.gb -= 0.5;
vec4 color = vec4(mat3(1.1643, 1.1643, 1.1643,
0.0, -0.2, 2.1,
1.7748, -0.4991, 0) * yuv.rgb,
1.0);

Screenshot 2022-10-04 at 12.24.44.png

But this matrix does not match any of the standard conversions described here: https:en.wikipedia.org/wiki/YCbCr#Y'CbCr_to_xvYCC

BT.709 with Limited Range (without yuv.r = 1.164383561643836 * (yuv.r - 0.0625)) seems the closer, but colors are becoming desaturated after some iterations:

Screenshot 2022-10-04 at 12.24.08.png

Here are the
// Color profile ITU-R BT.601 -
yuv.r = 1.164383561643836 * (yuv.r - 0.0625);
yuv.g -= 0.5;
yuv.b -= 0.5;
vec4 color = vec4(yuv.r + 1.6007 * yuv.b,
yuv.r - 0.391762290094914 * yuv.g - 0.812967647237771 * yuv.b,
yuv.r + 2.017232142857143 * yuv.g,
1);

// Color profile ITU-R BT.709 - https:en.wikipedia.org/wiki/YCbCr#Y'CbCr_to_xvYCC
yuv.r = 1.164383561643836 * (yuv.r - 0.0625);
yuv.g -= 0.5;
yuv.b -= 0.5;
vec4 color = vec4(mat3(1, 1, 1,
0.0, -0.1873, 1.8556,
1.5748, -0.4681, 0) * yuv.rgb,
1.0);

// Color profile ITU-R BT.2020
yuv.r = 1.164383561643836 * (yuv.r - 0.0625);
yuv.g -= 0.5;
yuv.b -= 0.5;
vec4 color = vec4(mat3(1, 1, 1,
0.0, -0.1645531, 1.8814,
1.4746, -0.571353, 0) * yuv.rgb,
1.0);

Is there any documentation on how the library converts RGB to YUV ?
I tested on Apple M1 & Apple Intel.
 
Last edited:
Very interesting research.
Sorry, I have not seen any specifics about how NDI convert RGB to YUV.
What NDIlib_recv_create_v3_t.color_format are you setting when creating your recv_create_v3?
What FourCC are you using when sending your video frame?
 
Per the Advanced SDK Documentation (19.1 Video Frames) SD resolutions use BT.601, HD (xres>720 || yres>576) use Rec.709, and UHD (xres>1920 || yres>1080) use Rec.2020. Note a color-space conversion is only performed if you are passing or receiving something other than YUV video.

Can you describe your test setup in a bit more detail? It is unclear where you are performing GPU based color-space conversion and the formats you are passing to/from the NDI library (eg: are you sending RGB and receiving YUV or vice/versa or something else?).
 
What NDIlib_recv_create_v3_t.color_format are you setting when creating your recv_create_v3?
What FourCC are you using when sending your video frame?
Click to expand...

The application sends video streams using FourCC = NDIlib_FourCC_type_BGRX (the data comes from a RGB texture in the graphics card)
I receives video using color_format = NDIlib_recv_color_format_fastest / bandwidth = NDIlib_recv_bandwidth_highest. The pixel format received in this case is NDIlib_FourCC_type_UYVY.

Per the Advanced SDK Documentation (19.1 Video Frames) SD resolutions use BT.601, HD (xres>720 || yres>576) use Rec.709, and UHD (xres>1920 || yres>1080) use Rec.2020.
Click to expand...
I tried with the above conversion matrices all of those formats and the result is not what I sent... Those transforms come from https://en.wikipedia.org/wiki/YCbCr#Y'CbCr_to_xvYCC

If I force getting the input stream (back from my output) as RGB using color_format = NDIlib_recv_color_format_BGRX_BGRA then everything looks ok. But if I request UYVY and do the YUV=>RGB conversion in a shader with the above matrices, I can't get the same colors than in the image I sent...
Is there something I am missing ?
 
Last edited:
Indeed, I just had to rescale the UV values into 0-1 range as I was doing for Y. With this shader, the loopback is exactly the same when requesting YUV from NDI lib converted with the shader than when directly requesting RGB from the NDI library.

// Color profile ITU-R BT.2020 (input is limited range, Y:16-235, UV:16-240)
// Rescale YUV input to 0-1
yuv.r = 1.164383561643836 * (yuv.r - 0.0625);
yuv.g = 1.138392857142857 * (yuv.g - 0.0625);
yuv.b = 1.138392857142857 * (yuv.b - 0.0625);

// Convert to RGB
yuv.g -= 0.5;
yuv.b -= 0.5;
vec4 color = vec4(mat3(1, 1, 1,
0.0, -0.1645531, 1.8814,
1.4746, -0.571353, 0) * yuv.rgb,
1.0);
 
If anyone is looking for it in the future:

// Color profile ITU-R BT.601 Limited Range
yuv.r = 1.164383561643836 * (yuv.r - 0.0625);
yuv.g = 1.138392857142857 * (yuv.g - 0.0625) - 0.5;
yuv.b = 1.138392857142857 * (yuv.b - 0.0625) - 0.5;
vec4 color = vec4(mat3(1.1643, 1.16430, 1.1643,
0.0, -0.39173, 2.0170,
1.5958, -0.81290, 0) * yuv.rgb,
1.0);


// Color profile ITU-R BT.709 Limited Range
yuv.r = 1.164383561643836 * (yuv.r - 0.0625);
yuv.g = 1.138392857142857 * (yuv.g - 0.0625) - 0.5;
yuv.b = 1.138392857142857 * (yuv.b - 0.0625) - 0.5;
vec4 color = vec4(mat3(1, 1, 1,
0.0, -0.1873, 1.8556,
1.5748, -0.4681, 0) * yuv.rgb,
1.0);


// Color profile ITU-R BT.2020 Limited Range
yuv.r = 1.164383561643836 * (yuv.r - 0.0625);
yuv.g = 1.138392857142857 * (yuv.g - 0.0625) - 0.5;
yuv.b = 1.138392857142857 * (yuv.b - 0.0625) - 0.5;
vec4 color = vec4(mat3(1, 1, 1,
0.0, -0.1645531, 1.8814,
1.4746, -0.571353, 0) * yuv.rgb,
1.0);
 
The RGB => YUV conversion done by NDI SDK seems to have changed since my conversion is no more working. Our app sends a RGB stream to NDI, and might get NDI in as YUV (which will be converted optimally in the graphics card). If I do a loopback I don't get the same colors with latest NDI versions.
I did a few SDK updates since this post and have no tried to see which update has changed the behavior.
Is there a documentation on NDI library RGBA => YUV conversion so I can get a correct matrix for it ?
 
Last edited:
You must log in or register to reply here.
Back
Top