Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob

EZ-USB™ CX3: Interfacing CX3 MIPI camera solution demo kit with the Sony IMX715 sensor and streaming RAW format video with custom host application - KBA238505

lock attach
Attachments are accessible only for community members.

EZ-USB™ CX3: Interfacing CX3 MIPI camera solution demo kit with the Sony IMX715 sensor and streaming RAW format video with custom host application - KBA238505

Infineon_Team
Employee
Employee
50 replies posted 25 likes received 25 replies posted

1.1  Introduction


Infineon’s EZ-USB™ CX3 bridge controller can connect devices with the Mobile Industry Processor Interface – Camera Serial Interface 2 (MIPI CSI-2) interface to any USB 3.0 host. For more details, see the EZ-USB™ CX3 MIPI CSI-2 to SuperSpeed USB bridge controller datasheet.

This article explains how the CX3 MIPI camera kit can be interfaced with the Sony IMX715 sensor over the MIPI CSI-2 interface and also how to stream RAW format video with the custom host application.

Infineon_Team_0-1698739061221.pngFigure 1  System block diagram

 

1.2 Software prerequisites

  1. Download and install the EZ-USB™ FX3 software development kit.
  2. (Optional) Download and install a serial terminal application such as the Tera term application to observe the debug prints.

1.3  Hardware requirements

1. CX3 MIPI camera kit

The Infineon’s CX3 MIPI camera kit is a USB 3.0 reference design kit based on the Infineon’s EZ-USB™ CX3 USB 3.0 camera controller. The CX3 MIPI camera kit contains the Raspberry Pi compatible 22-pin MIPI CSI camera connector. This kit is not available for sale. Kit schematic and board files are attached to this KBA for reference.

Infineon_Team_1-1698739201712.pngFigure 2 CX3 MIPI camera kit

2. SONY IMX715 image sensor board

The Infineon’s Sony IMX715 image sensor board has a Raspberry Pi compatible 22-pin MIPI CSI camera connector. This kit is not available for sale. Kit schematic and board files are attached to this KBA for the reference.

Infineon_Team_2-1698739290000.pngFigure 3  Sony IMX715 image sensor board

 

Infineon_Team_3-1698739332270.png

Figure 4  CX3 MIPI camera kit connected to Sony IMX715 Image Sensor Board

Note: Connect the Sony IMX715 image sensor board with CX3 MIPI camera kit using a Raspberry Pi 22-pin MIPI CSI camera ribbon cable, see Figure 4.

1.4 Firmware details


The firmware attached with this KBA supports the following video resolutions:

  • Video format: RAW12
  • Video resolutions for USB 3.0: 3864 x 2229 @ 18fps

Sony IMX715 supports the 3864 x 2229 video resolution (full-frame resolution of Sony IMX715), which is added to this firmware.

On programming EZ-USB™ CX3 with the firmware image file, the CX3 device will enumerate as a “CX3-UVC” camera. The firmware source is attached to this KBA.

1.5 Programming CX3 MIPI camera kit


1. On the EZ-USB™ CX3 MIPI camera kit, set the PMODE lines (SW4) to USB boot mode position, see
Figure 5.

Infineon_Team_4-1698739553348.png

Figure 5  Boot switch position for USB boot

2. Open the “Control Center” application from the Start menu.

The board must appear as Cypress USB Bootloader Device in the left pane of the USB Control Center application, see Figure 6.

Infineon_Team_5-1698739639677.pngFigure 6  Device enumeration in USB Control Center application


3. Download the Cx3UvcSonyIMX715.img (see attachments) file into FLASH/RAM of the kit. Select Program > FX3 > SPI FLASH/RAM, see
Figure 7.

Infineon_Team_6-1698739818815.pngFigure 7  Programming EZ-USB CX3™

 

4. After loading the firmware image to SPI flash, set the PMODE Lines (SW4) to the SPI flash boot mode and power cycle of the board, as shown in Figure 8.

Infineon_Team_0-1698743433688.pngFigure 8  Boot switch position for SPI flash boot
 
Infineon_Team_1-1698743519054.pngFigure 9 “CX3-UVC” enumerating in Device Manager

1.6 Streaming RAW video data format using custom host application


UVC host applications supporting RAW12 video format can be used to view the video streams by selecting the “CX3-UVC” camera. UVC host applications such as Windows Camera, VLC Media Player, and MPC-HC player cannot be used because UVC applications do not support the RAW12 video format. Therefore, a host application is created using Python to stream the video in RAW format. It uses OpenCV library for color conversion of the video frames. Windows host application for streaming RAW video format is attached with this KBA.

Host application supports the following color conversion formats:

  1. RG2BGR - Converts RGGB Pixel arrangement to BGR format.
  2. RG2RGB - Converts RGGB Pixel arrangement to RGB format.
  3. GB2BGR - Converts GBRG Pixel arragement to BGR format.
  4. GB2RGB - Converts GBRG Pixel arrangement to RGB format.
  5. None - No color conversion is done.

Because SonyIMX715 has an RGGB Pixel arrangement, you must choose RG2BGR for proper color conversion. Other formats can be used for experimentation purposes.

Do the following to use the custom host application:

1. Launch sonyIMX715_CX3_v5.0.0.exe. A console window and GUI application window open, as shown in Figure 10 and Figure 11.

Infineon_Team_0-1698747201206.png

Figure 10   Python application console

Infineon_Team_0-1698744849552.png

Figure 11  Python application GUI

2. Click on “Select Cameraand select CX3 UVC (see Figure 12).

 
Infineon_Team_1-1698744866011.pngFigure 12  Select “CX3-UVC” Camera in Host Application

3. Click on “Select Camera Resolutions” and select  2160p (see Figure 13).

Infineon_Team_8-1698744046131.png

Figure 13 Select camera resolution in host application


 4. Click on “Display Resolutions” and select display resolution as per the monitor resolution (see Figure 14) or it will overflow the monitor screen. You can also resize the window while video streaming.

Infineon_Team_7-1698744013070.png

 Figure 14  Select display resolution in host application


5. Click on “Bayer2RGB Codes” and select RG2BGR (see Figure 15).

Note: RG2BGR should be chosen for proper color conversion for SonyIMX715.

Infineon_Team_6-1698743927877.png

 Figure 15 Select Bayer2RGB code in host application


6. Click on START button (see Figure 16).

Infineon_Team_5-1698743865825.png

 Figure 16 Start video streaming in host application

 7. Once the streaming starts, you can take a snapshot or stop the streaming (see Figure 17). Additionally, you can use the Keyboard shortcut “ctrl+s” to take a snapshot.

Note: While taking a snapshot, the cursor should be at streaming window.

Infineon_Team_4-1698743779357.pngFigure 17  Snapshot and Stop button in Host Application

8. FPS and STATUS can be seen at the bottom of the application (see Figure 18)

Note: FPS can vary based on the host PC specifications.

Infineon_Team_3-1698743654267.pngFigure 18  FPS and STATUS in host applicationInfineon_Team_2-1698743600892.png
Figure 19  RAW12 format video streaming with custom Python host application
Attachments
0 Likes
550 Views
Contributors