How to dynamically change video encoding format during runtime - AN75779

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
AbMa_4712156
Level 1
Level 1
First like given

We are successfully streaming a video from an image sensor to our host PC through an FX3 module. An FPGA acts as a bridge between the FX3 module and the sensor.

Our FX3 firmware is based on AN75779. We have read the AN75779 chapter 2 to get an understanding of the UVC control and streaming interface. We have also referred to the following threads to get an idea of the modifications we need to make:

Use FX3 as UVC camera based on AN75779

Re: How to dynamically change resolution of image sensor - AN75779

Exchange resolution control messages between the Host and FX3 configured as UVC-class device - AN757...

We wish to be able to change the streaming video encoding format from the default 'YUY2' to 'Y16 ' dynamically during runtime by sending an input from the host PC. For that we have made the following modifications:

1.) In the cyfxuvcdsrc.c file, within CyFxUSBSSConfigDscr[] under /* Class-specific Video Streaming Input Header Descriptor */ we have changed  the number of format desciptor follows from 0x01 to 0x02.

2.) We have added an additional /* Class specific Uncompressed VS format descriptor */  for the Y16 format along with its own Class specific Uncpompressed VS frame descriptor. We have set the Format desciptor index as 0x02 for Y16.

3.) We have added a probe structure in the uvc.h file for streaming Y16 format video upon receiving the SET_CUR from the probe request, and modified the following line in the probe structure:

0x02,                       /* Use 1st Video format index */

4.) We have added the following lines in the uvc.c file under UVCHandleVideoStreamingRqts{}:

case CY_FX_USB_UVC_GET_DEF_REQ: /* There is only one setting per USB speed. */

                    if (usbSpeed == CY_U3P_SUPER_SPEED)

                    {

                        if(glProbeCtrl[2] == 1)

                                   CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl);

                        else

                                   CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl_Y16);  //Probe structure for streaming Y16 encoded video

                    }

                    else

                    {

                        CyU3PUsbSendEP0Data (CY_FX_UVC_MAX_PROBE_SETTING, (uint8_t *)glProbeCtrl20);

                    }

                    break;

5.) We understand we have to edit the descriptor size in each of the appropriate descriptors in the cyfxuvcdsrc.c file, but we are unsure how to calculate the appropriate size of all the class-specific descriptors as well as the overall size. For trial purposes we have made an upper limit estimation of the descriptor sizes.

Currently we have an OpenCV Python program to stream the video. The stream is coming out fine when using the unmodified firmware(default YUY2 format basic firmware that came directly with FX3 SDK from the cypress website) to run the UVC device. However the program is unable to detect the video stream from the modified firmware.

Is our understanding correct ? Have all the above mentioned modifications been done correctly and with the proper understanding ? If not what are the required modifications that need to be done ?

We sincerely thank you for your time and efforts !

Sincerely,

Abdul Mannan

0 Likes
1 Solution

Hello,

Please find the following snapshot which shows the sequence of requests that host use to switch the streaming format or resolution.

pastedImage_0.png

I see that you have handled PROBE: SET_CUR request properly in the firmware. But after this, the host will issue PROBE: GET_CUR request to read the updated values from the UVC device. This is not handled in your firmware. Also, if the updated probe/commit control structure sent by the device matches with that sent by the host, then the host will send COMMIT:SET_CUR request. When this is received, you should update the sensor registers to stream with the new format.

Please make the following modifications to your code:

1. Use two global variables glCurrentFormatIndex and glCurrentFrameIndex.

2. When PROBE: SET_CUR request is obtained, replace the existing code by the following lines of code:

  

      glCurrentFrameIndex = glCommitCtrl[3];

     glCurrentFormatIndex = glCommitCtrl[2];

3. Now, when PROBE:GET_CUR request is received, under the check for usb speed, check what is the value of  glCurrentFormatIndex. If the value is 1, then send the probe control structure for YUY2. And if it is 2, then send the probe control structure for Y16. (You have made this change in COMMIT: GET_CUR which is not needed).

4. After this, the host will issue COMMIT: SET_CUR. At that time, you need to configure the sensor to output the corresponding format (by checking glCurrentFormatIndex) by writing the sensor registers. At the moment, always it is calling the function SensorScaling_HD720p_30fps. Which I assume is writing the sensor registers for streaming YUY2 data and not Y16.

Please let me know if you have any queries on this.

Best Regards,

Jayakrishna

Best Regards,
Jayakrishna

View solution in original post

14 Replies