Integrating OV2311 with EZ-USB SDK CX3 project

[NiNa_4798196]

I'm trying to bring up OV2311 sensor using CYUSB3065.

I have the corresponding register settings for this sensor and set them accordingly in cyu3imagesensor.c file.

I need help in understanding and setting CX3 receiver configuration, Image sensor configuration (1600x1300 @30fps) and UVC descriptors for RAW10 output format

Please help and guide me in this regards.

[Rashi_Vatsa]

Hello,

Please let me know the following details:

- MIPI data lanes:

-  CSI clock (MHz):

- THS -Prepare (ns):

- THS Zero (ns):

- H active(pixel):

- H blanking(pixel):

- V Active  (line):

- V Blanking (line):

- Data format: - RAW10

- FPS: 30

Please note the above details should be the same as the settings of the sensor.

As the video format is RAW 10 (10bits/pixel) and if the GPIF bus width is configured for 16 bits then there are two options:

- Padding of the pixel: non-image data (6 zeros) will be padded to the image data

- Packing of the pixel: In packing, the image data will be packed to 16 bits

Please refer to this KBA Streaming RAW10 Format Input Data to 16/24-bit Output Format in CX3 MIPI CSI-2 - KBA224387  and let me know which of the above two options you want in your application

Please refer to this KBA for the steps Steps to Setup up MIPI CSI Camera Solution with CX3 – KBA225748

Regards,

Rashi

[NiNa_4798196]

Hi RashiV_61

Sorry for the delayed reply. Here are the requested configurations :

- MIPI data lanes: 2

-  CSI clock (MHz): 400

- THS -Prepare (ns): 60

- THS Zero (ns): 120

- H active(pixel): 1600

- H blanking(pixel): 560

- V Active  (line): 1200

- V Blanking (line): 50

- Data format: - RAW10

- FPS: 30

For the output format we'll be going ahead with padding extra 6-bits and then streaming the output.

[Rashi_Vatsa]

Hello,

Thank you for the details.

The CX3 MIPI receiver settings seems fine.

But in your first post you mentioned the video resolution as "Image sensor configuration (1600x1300 @30fps) and UVC descriptors for RAW10 output format" but in the CX3 Configuration tool it says 1600*1200 @30fps and RAW 10 format.

Please confirm that the settings under "MIPI CSI 2 Inputs (From Image sensor)" should be same as  the sensor settings.

Regards,

Rashi

[NiNa_4798196]

Hello,

Yes the sensor settings are set with respect to 1600*1200. I made a mistake while mentioning previously

The USB descriptors are set correctly now. The device is now enumerating correctly as a UVC camera.

However, I have not tested streaming yet. I'll test and update on the same.

Regards,

Ninad

[NiNa_4798196]

Hello,

On connecting the camera to PC I'm getting following dmesg logs :

[16958.784348] usb 3-6: USB disconnect, device number 15

[16961.755305] usb 3-6: new high-speed USB device number 16 using xhci_hcd

[16967.195397] usb 3-6: device descriptor read/64, error -110

[16967.457294] usb 3-6: New USB device found, idVendor=04b4, idProduct=00c3, bcdDevice= 0.00

[16967.457298] usb 3-6: New USB device strings: Mfr=1, Product=2, SerialNumber=0

[16967.457301] usb 3-6: Product: CX3-UVC

[16967.457303] usb 3-6: Manufacturer: Cypress

[16967.460001] uvcvideo: Found UVC 1.10 device CX3-UVC (04b4:00c3)

[16967.460637] uvcvideo: UVC non compliance - GET_DEF(PROBE) not supported. Enabling workaround.

[16967.460750] uvcvideo: Failed to query (129) UVC probe control : -32 (exp. 34).

[16967.460755] uvcvideo: Failed to initialize the device (-5).

Where I have gone wrong?

This is my MIP CSI-2 configuration settings :

/* OV2311_RAW10_2MP :  */

CyU3PMipicsiCfg_t OV2311_RAW10_2MP = 

{

    CY_U3P_CSI_DF_RAW10,  /* CyU3PMipicsiDataFormat_t dataFormat */

    2,                          /* uint8_t numDataLanes */

    2, /* uint8_t pllPrd */

    89, /* uint16_t pllFbd */

    CY_U3P_CSI_PLL_FRS_250_500M, /* CyU3PMipicsiPllClkFrs_t pllFrs */ 

    CY_U3P_CSI_PLL_CLK_DIV_4, /* CyU3PMipicsiPllClkDiv_t csiRxClkDiv */

    CY_U3P_CSI_PLL_CLK_DIV_4, /* CyU3PMipicsiPllClkDiv_t parClkDiv */

    0,                 /* uint16_t mClkCtl */

    CY_U3P_CSI_PLL_CLK_DIV_2, /* CyU3PMipicsiPllClkDiv_t mClkRefDiv */

    1600,         /* uint16_t hResolution */

    50                         /* uint16_t fifoDelay */

};

This is my USB descriptor :

    /* Class specific Uncompressed VS format descriptor */

        0x1B,                               /* Descriptor size */

        CX3_CS_INTRFC_DESCR,                /* Class-specific VS interface Type */

        0x04,                               /* Subtype : VS_FORMAT */

        0x01,                               /* Format desciptor index */

        0x01,                               /* Number of Frame Descriptors that follow this descriptor: 1 */

        /* GUID, globally unique identifier used to identify streaming-encoding format*/

        0x59, 0x55, 0x59, 0x32, /*MEDIASUBTYPE_YUY2 GUID: 32595559-0000-0010-8000-00AA00389B71 */

0x00, 0x00, 0x10, 0x00,

0x80, 0x00, 0x00, 0xAA,

0x00, 0x38, 0x9B, 0x71,

        0x10,                               /* Number of bits per pixel: 16*/

        0x01,                               /* Optimum Frame Index for this stream: 1 (2MP) */

        0x00,                               /* X dimension of the picture aspect ratio; Non-interlaced */

        0x00,                               /* Y dimension of the pictuer aspect ratio: Non-interlaced */

        0x00,                               /* Interlace Flags: Progressive scanning, no interlace */

        0x00,                               /* duplication of the video stream restriction: 0 - no restriction */

        /* Class specific Uncompressed VS Frame Descriptor 1 - 2MP */

        0x1E,                               /* Descriptor size */

        CX3_CS_INTRFC_DESCR,                /* Descriptor type*/

        0x05,                               /* Subtype:  frame interface*/

        0x01,                               /* Frame Descriptor Index: 1 */

        0x00,                               /* No Still image capture method supported */

        0x40,0x06,                         /* Width in pixel:  1600 */

        0xb0,0x04,                         /* Height in pixel: 1200 */

        0x00,0x80,0xee,0x36,             /* Min bit rate (bits/s): 1600 x 1200 x No Of Bits per Pixel x FrameRate = 921600000 */

        0x00,0x80,0xee,0x36,             /* Max bit rate (bits/s): Fixed rate so same as Min */

        0x00,0x98,0x3a,0x00,             /* Maximum video or still frame size in bytes(Deprecated): 1600 x 1200 x 2 */

        0x15,0x16,0x05,0x00,             /* Default frame interval (in 100ns units): (1/30)x10^7 */

        0x01,                               /* Frame interval type : No of discrete intervals */

        0x15,0x16,0x05,0x00,             /* Frame interval 3: Same as Default frame interval */

       /* Still image descriptor -YUV with QVGA resolution */

    0x0b, /*Size of this descriptor*/

    0x24, /*CS_INTERFACE descriptor type*/

    0x03, /*VS_STILL_IMAGE_FRAME descriptor subtype*/

    0x00, /*If method 3 is used, endpoint address is mentioned here*/

    0x01, //No of frame Resolutions Follows

    0x40,0x06, /* Width in pixel:  1600=640*/ //1

    0xb0,0x04, /* Height in pixel: 1200=4b0*/

    0x01, /*Compression patterns supported*/

    0x00, /*No compression*/

And is my UVC probe control settings :

/* UVC Probe Control Setting - 2MP*/

uint8_t const gl2MPProbeCtrl[CX3_UVC_MAX_PROBE_SETTING] = {

    0x00, 0x00,                         /* bmHint : No fixed parameters */

    0x01,                               /* Use 1st Video format index */

    0x01,                               /* Use 1st Video frame index */

    0x15,0x16,0x05,0x00,             /* Desired frame interval in 100ns = (1/30)x10^7 */

    0x00, 0x00,                         /* Key frame rate in key frame/video frame units */

    0x00, 0x00,                         /* PFrame rate in PFrame / key frame units */

    0x00, 0x00,                         /* Compression quality control */

    0x00, 0x00,                         /* Window size for average bit rate */

    0x00, 0x00,                         /* Internal video streaming i/f latency in ms */

    0x00,0x98,0x3a,0x00,             /* Max video frame size in bytes = 1920 x 1080 x 2 */

#ifdef CX3_UVC_1_0_SUPPORT

    0x00, 0x90, 0x00, 0x00              /* No. of bytes device can rx in single payload: 36KB */

#else

    /* UVC 1.1 Probe Control has additional fields from UVC 1.0 */

    0x00, 0x90, 0x00, 0x00,             /* No. of bytes device can rx in single payload: 36KB */

    0x00, 0x60, 0xE3, 0x16,             /* Device Clock */

    0x00,                               /* Framing Information - Ignored for uncompressed format*/

    0x00,                               /* Preferred payload format version */

    0x00,                               /* Minimum payload format version */

    0x00                                /* Maximum payload format version */

#endif

};

Please help in this regard as I have been stuck here for quite some time now.

[Rashi_Vatsa]

Hello,

Please let me know where in the firmware (or near which APIs) do you print these messages.

[16967.460637] uvcvideo: UVC noncompliance - GET_DEF(PROBE) not supported. Enabling workaround.

[16967.460750] uvcvideo: Failed to query (129) UVC probe control : -32 (exp. 34).

[16967.460755] uvcvideo: Failed to initialize the device (-5).

If possible please share the firmware files (without the sensor configuration) for us to check. Also,. confirm if you use the default OV5640 project of the SDK as the base or generate the firmware using the CX3  MIPI configuration tool as per this KBA Steps to Setup up MIPI CSI Camera Solution with CX3 – KBA225748 or is it a custom firmware

Also, let me know which host application are you using for video streaming.

Regards,

Rashi

[NiNa_4798196]

Hi,

I'm creating firmware using CX3 MIPI configuration tool as per KBA listed above.

I have managed to get past this error now. I added my 1600*1200 resolutions in switch cases in cycx3_uvc.c files.

Now I am able to see /dev/video2 and /dev/video3 device.

However, eCam View shows a black screen.

I tried using gstreamer to capture video as but the output got stuck as follows :

cmd >gst-launch-1.0 v4l2src device=/dev/video2 num-buffers=100 ! video/x-raw,width=1600,height=1200 ! filesink location=~/test_h264.avi

o/p>

Setting pipeline to PAUSED

Pipeline is live and does not need PREROLL

Setting pipeline to PLAYING

New clock: GstSystemClock   (Got Stuck at this point)

How do I move forward in debugging this issue?

Regards,

Ninad

[Rashi_Vatsa]

Hello,

Please share the UART debug prints when the streaming is started and this error is seen on the host side.

Please enable the macro CX3_DEBUG_ENABLED and build the firmware again before starting the streaming

Regards,

Rashi

[NiNa_4798196]

Hi,

These are my UART logs whenever I start streaming,

AppInit:GpifSMStart passed

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

Sensor Initialization

Sensor Sleep

Event Flag = 0x20

Sensor Sleep

EnterÓÕÍÁ¹5½?ÑÕ?êÂÁbº­ÕÁ?¹êÂáRjTe¹Ñ2±êÂÉj¤iUËk×KÒ*«V.¡CQ·ºerSuspendMode Status =  0x0, Wakeup reason = 0x8

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

Event Flag ýÂÉj¤iUËk×KÒ*«V.¡CQ·ºerSuspendMode Status =  0x0, Wakeup reason = 0x8

bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0x21, bRequest = 0x1, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x83, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x82, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0x21, bRequest = 0x1, wValue = 0x200, wIndex = 0x1, wLength= 0x22

Sensor Initialization : 2MP

AplnStrt:SMState = 0x2

Sensor Wakeup

Event Flag = 0x10

CB failure : Event Flag = 16

glIsApplnActive = 1

Ninad : App Stop

Sensor Sleep

AplnStop:SMState = 0x5

Ninad : App Start

AplnStrt:SMState = 0x1

Sensor Wakeup

Event Flag = 0x10

CB failure : Event Flag = 16

glIsApplnActive = 1

Ninad : App Stop

Sensor Sleep

AplnStop:SMState = 0x5

Ninad : App Start

AplnStrt:SMState = 0x1

Sensor Wakeup

Event Flag = 0x10

CB failure : Event Flag = 16

glIsApplnActive = 1

Ninad : App Stop

Sensor Sleep

AplnStop:SMState = 0x5

Ninad : App Start

AplnStrt:SMState = 0x1

Sensor Wakeup

Event Flag = 0x10

CB failure : Event Flag = 16

These keep on repeating continuously until I close the streaming app.

We probed the MIPI clock, and we're correctly getting 400MHz over there. And the sensor is streaming some data.

But on the Host side with eCam we can only see a black screen. As well, with v4l2 we're not able to get any dumps.

[Rashi_Vatsa]

Hello,

From the debug prints it seems that the CX3_DMA_RESET_EVENT event is triggered due to the failure of CyU3PDmaMultiChannelCommitBuffer API.

Can you please check the return value of this API so that we can know the error code? For checking the return value use a global variable to copy the return value f the API when it fails (doesn't return success) and print the global variable along with the print "Cb Failure"

Also, to check if the frame size and frame rate received by the CX3 from the sensor is as expected, please enable the PRINT_FRAME_INFO macro and share the debug prints.

Regards,

Rashi

[NiNa_4798196]

Hello,

This is the return value for CyU3PDmaMultiChannelCommitBuffer API.

CB failure : CyU3PDmaMultiChannelCommitBuffer = 71

Also, the 'PRINT_FRAME_INFO' is already enabled.

Debug logs :

AppInit:GpifSMStart passed

Chip ID High Read= 0x2311

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100 Òªë­Â)jIQåÁêÂÅbIÅÕÍÑêÂá

       bºY±ÕêÂÉbº%¹áêÂÁbº1¹Ñ¡õÂÅRjT?ÉMÕÍU¹5½?ÑÕ?êÂÁbº­ÕÁ?¹êÂáRjT?ÉMÕÍU¹5½?ÑÕ?êÂÁbº­ÕÁ?¹êÂáþ

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

bRType = 0xA1, bRequest = 0x87, wValue = 0x1400, wIndex = 0x100, wLength= 0xA

bRType = 0xA1, bRequest = 0x81, wValue = 0x200, wIndex = 0x0, wLength= 0x1

EnterSuspendMode Status =  0x0, Wakeup reason = 0x8

bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0x21, bRequest = 0x1, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x81, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x83, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0xA1, bRequest = 0x82, wValue = 0x100, wIndex = 0x1, wLength= 0x22

bRType = 0x21, bRequest = 0x1, wValue = 0x200, wIndex = 0x1, wLength= 0x22

Chip ID High Read= 0x2311

AplnStrt:SMState = 0x2

CB failure : CyU3PDmaMultiChannelCommitBuffer = 71

AplnStop:SMState = 0x5

AplnStrt:SMState = 0x1

CB failure : CyU3PDmaMultiChannelCommitBuffer = 71

AplnStop:SMState = 0x5

AplnStrt:SMState = 0x1

CB failure : CyU3PDmaMultiChannelCommitBuffer = 71

AplnStop:SMState = 0x5

AplnStrt:SMState = 0x1

Regards,

Ninad

[Rashi_Vatsa]

Hello Ninad,

Please refer to this KBA Invalid Sequence Error in Multi-Channel Commit Buffer - KBA218830 which mentions the reason of commit buffer Api failure with error code 71.

To follow the first point for recovering from this error,  the DMA buffer size needs to be increased.

Please let me know what is current buffer size used in the firmware.

Also, 'PRINT_FRAME_INFO' is already enabled.

>> If the firmware is generated from the CX3 MIPI configuration tool, on enabling the PRINT_FRAME_INFO the below-mentioned prints will be enabled

CyU3PDebugPrint(4,"\n\rProd = %d Cons = %d  Prtl_Sz = %d Frm_Cnt = %d Frm_Sz = %d B", TxCountflag, RxCountflag, PartialBufSize, FrameCount, ((TxCountflag*CX3_UVC_DATA_BUF_SIZE)+PartialBufSize));

Please refer to the attached file generated from the tool which prints the information about the frames received and confirm if the same implementation is done in your project.

If the same implementation is being done in your firmware this means the end of the frame is not reached which prints the above-mentioned prints ( by enabling Printflag = 1).

In this case, please let me know whether the device enumerates as a USB 3.0 device or a USB 2.0 device.

You can call CyU3PUsbGetSpeed after CyCx3UvcAppInit or CyU3PConnectState to check the device connection speed.

Regards,

Rashi

[NiNa_4798196]

Hello Rashi,

I followed this (Invalid Sequence Error in Multi-Channel Commit Buffer - KBA218830) Still I could see

CB failure : CyU3PDmaMultiChannelCommitBuffer = 71

Current DMA buffer size is 36838

The device enumerates as Full speed device

I'm connecting 3.0 cable to 3.0 port. I have checked with both orientations

[NiNa_4798196]

Hi Rashi,

I have an update. We made some corrections on the hardware.

The device is now detected as a USB2.0 device. But CyU3PUsbGetSpeed still return '1' as its return value.

I have added sensor configuration for 640*480@30fps for UsbHS descriptor and on 'guvcview' on Ubuntu I am able to see some stream. But this stream hangs after some time and then the application itself crashes.

But still, the UART logs are exactly same as above (during streaming) and this following line is still not getting printed

CyU3PDebugPrint(4,"\n\rProd = %d Cons = %d  Prtl_Sz = %d Frm_Cnt = %d Frm_Sz = %d B", TxCountflag, RxCountflag, PartialBufSize, FrameCount, ((TxCountflag*CX3_UVC_DATA_BUF_SIZE)+PartialBufSize));

The implementation is same as in the file shared by you.

Please guide me in where should I debug the issues next.

Regards,

Ninad

[Rashi_Vatsa]

Hello Ninad,

Thank you for the update.

So, the device now enumerates as High speed device at the video resolution to be streamed is 640*480 @30 FPS and the video format is RAW 10 which is padded with 6 bits non image data. Am I right?

I have added sensor configuration for 640*480@30fps for UsbHS descriptor and on 'guvcview' on Ubuntu I am able to see some stream. But this stream hangs after some time and then the application itself crashes.

>> Please let me know which among the two crashes is it the host application or the firmware.

Please try printing TxCount and RxCount variables in the for loop in CyCx3UvcAppThread_Entry and also print the usual commit buffer failure and share the traces

    CyCx3UvcAppInit();

    for (;;)

    {

        eventFlag = 0;

          CyU3PDebugPrint(4,"\n\rProd = %d Cons = %d  ", TxCount, RxCount));

....

}

Regards,

Rashi

[NiNa_4798196]

Hi Rashi,

>>So, the device now enumerates as High speed device at the video resolution to be streamed is 640*480 @30 FPS and the video format is RAW 10 which is padded with 6 bits non image data. Am I right?

- Yes, 640*480@30fps in RAW10 format with 6 bits non image data.

However, there's another update. We made more fixes on the h/w and the device now enumerates as USB3.0 Superspeed device.

So we're able to stream 1600*1300@30fps as well

Here is the UART trace when the crash occurs,

Prod = 76 Cons = 75  Prtl_Sz = 1824 Frm_Cnt = 52 Frm_Sz = 2799840 B           

0 0 0 0 0 0 0 0 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 15 Cons = 14                                                           

Prod = 24 Cons = 23                                                           

Prod = 33 Cons = 32                                                           

Prod = 43 Cons = 42                                                           

Prod = 52 Cons = 51                                                           

Prod = 62 Cons = 61                                                           

Prod = 71 Cons = 70                                                           

Prod = 76 Cons = 75  Prtl_Sz = 8416 Frm_Cnt = 53 Frm_Sz = 2806432 B           

0 0 0 0 0 0 0 0 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 12 Cons = 11                                                           

Prod = 20 Cons = 19                                                           

Prod = 30 Cons = 29                                                           

Prod = 39 Cons = 38                                                           

Prod = 48 Cons = 48                                                           

Prod = 58 Cons = 57                                                           

Prod = 67 Cons = 62                                                           

Event Flag = 0x10                                                             

CB failure : Event Flag = 16 CommitBiff = 0                                   

glIsApplnActive = 1                                                           

Ninad : App Stop                                                               

Sensor Sleep                                                                 

AplnStop:SMState = 0x2                                                         

Ninad : App Start                                                             

AplnStrt:SMState = 0x2                                                         

Sensor Wakeup                                                                 

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Event Flag = 0x10                                                             

CB failure : Event Flag = 16 CommitBiff = 0                                   

glIsApplnActive = 1                                                           

Ninad : App Stop                                                               

Sensor Sleep                                                                 

AplnStop:SMState = 0x81                                                       

Ninad : App Start                                                             

AplnStrt:SMState = 0x1                                                         

Sensor Wakeup                                                                 

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Prod = 0 Cons = 0                                                             

Event Flag = 0x10                                                             

CB failure : Event Flag = 16 CommitBiff = 0

[Rashi_Vatsa]

Hello Ninad,

However, there's another update. We made more fixes on the h/w and the device now enumerates as USB3.0 Superspeed device.

So we're able to stream 1600*1300@30fps as well

>> Glad to hear that the streaming works fine after fixing the hardware

From the UART traces as the Procuder events (prod) are 0, which means that the video data is not coming up to CX3 after the sensor initialization is done. So, please check if the sensor is streaming the data and the CX3 MIPI configuration settings match to the sensor settings. This can be checked by probing the PCLK, HYSNC, VSYNC test signals mentioned in Q10 of this KBA CX3 Hardware: Frequently Asked Questions - KBA91295

Please let me know after the hardware changes are there any problems still seen

Regards,

Rashi

[NiNa_4798196]

Hi RashiV,

I probed the mentioned test points and got these as result,

1. Pclk

2. Hsync_zoom_out

3. Hsync_zoom_in

4. Vsync

A] After this, one of the issue I am facing is, for 640*480 resolution I am able stream a proper video, meaning I can distinguish the subject being captured correctly

1. 640*480p

But for higher resolutions this is not the case. It does stream something, if I move the sensor around I see variations on the screen. But the subjects are not visible correctly and rather seems noisy. This is how it looks,

2. 1280*720

3. 1600*1300

I have tried varying CyU3PMipicsiSetPhyTimeDelay() according to CX rx config. but the results were same.

Another way to debug, I sent a solid frame (0x3f value at each pixel) from sensor and took raw dumps on the host. The dump captured the accurate pixel values, meaning there is no data corruption happening during the usb transfer as well. How do I enable higher resolution streaming successfully.

B] We have a tuning pipeline already set up for this sensor. But it works on a Imx platform where our camera driver also creates a v4l2-subdev node on which this pipeline is called. The pipeline reads and sets exposure values, analogue and digital gain parameters on the sensor through i2c sequences. How do we add such APIs in CX3 fw? Or do we already have any such APIs already present in the SDK?

Regards,

Ninad

[Rashi_Vatsa]

Hello Ninad,

Please confirm with which sensor configuration (video resolution) are the traces (PCLK, HSYNC, VSYNC) taken.

I have tried varying CyU3PMipicsiSetPhyTimeDelay() according to CX RX config. but the results were the same.

Another way to debug, I sent a solid frame (0x3f value at each pixel) from the sensor and took raw dumps on the host. The dump captured the accurate pixel values, meaning there is no data corruption happening during the USB transfer as well. How do I enable higher resolution streaming successfully?

>> Please share the UART debug prints which print the producer and consumer events so that we can know if CX3 receives the data from the sensor properly

CyU3PDebugPrint(4,"\n\rProd = %d Cons = %d  Prtl_Sz = %d Frm_Cnt = %d Frm_Sz = %d B", TxCountflag, RxCountflag, PartialBufSize, FrameCount, ((TxCountflag*CX3_UVC_DATA_BUF_SIZE)+PartialBufSize));

Another way to debug, I sent a solid frame (0x3f value at each pixel) from the sensor and took raw dumps on the host

>> Please let me know if the frame size  (UART Debug Print) received on FX3 is as expected

Please share the CX3 configuration tool snippets for both the video resolution

1) 1600*1300

2) 1280*720

The pipeline reads and sets exposure values, analog and digital gain parameters on the sensor through i2c sequences. How do we add such APIs in CX3 fw? Or do we already have any such APIs already present in the SDK?

>> From this I understand, you want to add some controls to the application like setting exposure, brightness. Is my understanding correct?

Regards,

Rashi

[NiNa_4798196]

Hi Rashi,

Please find the following details as required,

Please confirm with which sensor configuration (video resolution) are the traces (PCLK, HSYNC, VSYNC) taken.

>> The probes were noted for 480p resolution

Please share the UART debug prints which print the producer and consumer events so that we can know if CX3 receives the data from the sensor properly

>>

1600*1300 UART logs,

Prod = 76 Cons = 75  Prtl_Sz = 18080 Frm_Cnt = 712 Frm_Sz = 2816096 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18048 Frm_Cnt = 713 Frm_Sz = 2816064 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18080 Frm_Cnt = 714 Frm_Sz = 2816096 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 15104 Frm_Cnt = 715 Frm_Sz = 2813120 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18176 Frm_Cnt = 716 Frm_Sz = 2816192 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18144 Frm_Cnt = 717 Frm_Sz = 2816160 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18080 Frm_Cnt = 718 Frm_Sz = 2816096 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 15008 Frm_Cnt = 719 Frm_Sz = 2813024 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18176 Frm_Cnt = 720 Frm_Sz = 2816192 B          

TimeDiff = 1046 ms FPS = 28                                                    

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18144 Frm_Cnt = 721 Frm_Sz = 2816160 B          

0 0 0 0 0 0 0 0 0                                                              

Prod = 76 Cons = 75  Prtl_Sz = 18144 Frm_Cnt = 722 Frm_Sz = 2816160 B          

0 0 0 0 0 0 0 0 0      

1280*720 uart logs,

Prod = 42 Cons = 42  Prtl_Sz = 13472 Frm_Cnt = 1221 Frm_Sz = 1559744 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 5920 Frm_Cnt = 1222 Frm_Sz = 1552192 B          

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 28672 Frm_Cnt = 1223 Frm_Sz = 1574944 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 13504 Frm_Cnt = 1224 Frm_Sz = 1559776 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 18528 Frm_Cnt = 1225 Frm_Sz = 1564800 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 41 Cons = 41  Prtl_Sz = 34608 Frm_Cnt = 1226 Frm_Sz = 1544064 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 23616 Frm_Cnt = 1227 Frm_Sz = 1569888 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 26144 Frm_Cnt = 1228 Frm_Sz = 1572416 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 23584 Frm_Cnt = 1229 Frm_Sz = 1569856 B         

0 0 0 2 0 0 0 0 0                                                              

Prod = 42 Cons = 42  Prtl_Sz = 23616 Frm_Cnt = 1230 Frm_Sz = 1569888 B         

TimeDiff = 1962 ms FPS = 15

640*480 UART logs,

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 37 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 38 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 39 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 40 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 41 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 42 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 43 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 44 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 45 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 46 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 47 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                              

Prod = 16 Cons = 16  Prtl_Sz = 24064 Frm_Cnt = 48 Frm_Sz = 613120 B            

0 0 0 2 0 0 0 0 0                                                             

Please let me know if the frame size  (UART Debug Print) received on FX3 is as expected

>> No, the size seems incorrect considering streaming in raw10. It should be 1600*1300*16/10 if I am not wrong. Please correct me in case.

Please share the CX3 configuration tool snippets for both the video resolution

>>

1. 1600*1300

2. 1280*720

3. 640*480

From this I understand, you want to add some controls to the application like setting exposure, brightness. Is my understanding correct?

>> Yes, we want to control exposure and gain values.

Regards,

Ninad

[Rashi_Vatsa · Accepted solution]

Hello Ninad,

As  per your earlier repsonse, the RAW10 data is padded with 6 bits (non-image data), the frame size can be  calculated like this

640*480@ RAW10 = 640*480*2(bytes) as 6 bits are padded = 614400 bytes

1600*1300 @ RAW10 = 1600*1300*2 = 4160000 bytes

1280*720 @ RAW10 =  1280*720*2 = 1843200 bytes

According to the calculation, for the working case also, the frame size is not as expected.

1) For 640*480@ RAW10, from the traces I see that PCLK frequency is as expected i.e ~100MHz.

The H_Total time is ~ 11us in the trace and as per the CX3 MIPI tool settings it is expected to be 62.80us

Similarly, the V_Total is expected as 33 ms for 30 FPS

It seems that the sensor settings and the MIPI receiver tool settings are not same. Please check if the sensor is configured as per the MIPI receiver settings.

Also, confirm if the SYS_CLK is set to 403.2 MHZ as the PCLK used is 100MHz.

clockConfig.setSysClk400 = CyTrue;

2) Similarly, for the other two resolutions also the frame size is not as expected. We can first work for 640*480, RAW10 @30fps and then can switch to other two resolutions.

We can first debug this issue and then I can help you with the controls of exposure and gain

Regards,

Rashi

[NiNa_4798196]

Hi Rashi,

the RAW10 data is padded with 6 bits (non-image data), the frame size can be  calculated like this

640*480@ RAW10 = 640*480*2(bytes) as 6 bits are padded = 614400 bytes

1600*1300 @ RAW10 = 1600*1300*2 = 4160000 bytes

1280*720 @ RAW10 =  1280*720*2 = 1843200 bytes

>> Perfect. Now I have a clear understanding on this. Thanks

It seems that the sensor settings and the MIPI receiver tool settings are not same. Please check if the sensor is configured as per the MIPI receiver settings.

>> We have raised request with the sensor ovm to get these details to match the configurations correctly. I'll update here as soon as we get those.

Also, confirm if the SYS_CLK is set to 403.2 MHZ as the PCLK used is 100MHz.

>> Yes, clockConfig.setSysClk400 is set to CyTrue

then I can help you with the controls of exposure and gain

>> While we get the response for the sensor timings, can you please guide me on how to set external controls. So that we can move ahead on writing our tuning related functions on the host. Or else they may be blocked for time being.

Also we have IR Leds and a flash LED along with the sensor . I have already integrated code for their initialization. However, we would also like to control their intensity values as a part of the tuning algorithm. Can we have something similar for the same like what we'll be doing for exposure and gain.

Regards,

Ninad

[Rashi_Vatsa]

Hello Ninad,

While we get the response for the sensor timings, can you please guide me on how to set external controls. So that we can move ahead on writing our tuning related functions on the host. Or else they may be blocked for time being.

>> Please find the steps to add controls:  exposure and gain controls to UVC firmware

1) Enable the controls in descriptors, which need to be added

-   Exposure - is supported by the Camera terminal so in /* Input (Camera) Terminal Descriptor */ the bit specific to exposure needs to be enabled.

-   Similarly, Gain is supported by the Processing Unit so in /* Processing Unit Descriptor */ the bit specific to gain needs to be enabled

2) The UVC requests on the control interface CX3_UVC_CONTROL_INTERFACE needs to be handled in the firmware.

The terminal ID/UNIT ID needs to be parsed when the CX3_UVC_CONTROL_INTERFACE request is received. Based on the ID, the request needs to be parsed and then the SET/GET requests need to be handled.

For this process, you can refer to the firmware attached with AN75779  https://www.cypress.com/documentation/application-notes/an75779-how-implement-image-sensor-interface...   which supports Brightness control (Processing Unit) and Pan Tilt Zoom (Camera terminal). The AN75779 firmware is for FX3.

In a similar way, the requests need to be handled in CX3 firmware.

I have already integrated code for their initialization. However, we would also like to control their intensity values as a part of the tuning algorithm. Can we have something similar for the same like what we'll be doing for exposure and gain.

>> Please confirm if you want to control the LED through the UVC interface.

Please let me know if any queries on this

Regards,

Rashi

[NiNa_4798196]

Hi Rashi,

We got the new settings from ovt. Setting those, I find the the same issues as mentioned previously.

The settings are for 1600*1300@60fps. These are verified already on a different platform.

I have one quick question. Why is the H-Total value in MIP_CSI2 Timing not changing even on varying the H-Blanking value?

I am attaching the probes for VSync and HSync below,

Fig. VSync - Active

Fig. VSync - Blanking

Fig. HSync - Active

Fig. HSync - Blanking

These are the logs taken over UART :

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2759 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2760 Frm_Sz = 2892768 B

TimeDiff = 498 ms FPS = 60

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2761 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2762 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2763 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2764 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2765 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2766 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2767 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2768 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2769 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2770 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 78  Prtl_Sz = 18624 Frm_Cnt = 2771 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

Prod = 78 Cons = 77  Prtl_Sz = 18624 Frm_Cnt = 2772 Frm_Sz = 2892768 B

0 0 0 0 0 0 0 0 0

I believe H-Blanking as seen from above might be the problem. So I tried to increase it but could not.

Could this be causing the issue? Please help me out here.

Regards,

Ninad

[Rashi_Vatsa]

Hello Ninad,

From the settings, I see that the FPS has been increased to 60 fps.  Please let me know what is the target FPS as earlier 30 FPS was used.

Please refer to this KBA Analysis of CX3 Video Timing Parameters – KBA226779  which mentions about virtual parameter pixel clock.

As H - Total and V - total will change the pixel clock value will change accordingly

As H total time is calculated based on this pixel clock the change h total time will not be seen.

As seen in the CX3 MIPI configuration tool settings the H blanking (on bottom right) period is -337 us, which is not valid. This is causing the error shown on the top of the utility.

As the error mentions the output pixel clock or H blanking period needs to be increased to resolve the error.

I tried increasing the pixel clock to 100MHz i.e. the maximum supported but that doesn't solve the problem. So we need to increase the H blanking of the image sensor as high as possible (to do this vertical blanking can be reduced thereby maintaining the same frame rate). While doing this make sure that vertical blanking does not go below ~350 us.

Is it possible for you to change the output bus width to 24 bits with that the error will be gone (attached). As per the KBA Streaming RAW10 Format Input Data to 16/24-bit Output Format in CX3 MIPI CSI-2 - KBA224387 when output bus width is 24 bit the data will be packed as it was done earlier.

If yes, as the MIPI output bus width is configured for 24 bits i.e. same as GPIF interface bus width, the maximum bandwidth supported is 16*100MHz i.e. 2.4 Gbps and the video bandwidth is ~1.2 Gbps so it seems there won't be a problem to stream @ 60fps

Please let me know if the SYS_CLK in the firmware is configured for 403.2MHz using

clockConfig.setSysClk400  = CyTrue;

Please let me know if this works

Regards,

Rashi