Implementing RGB + IR camera with FX3

[RSar475]

Hello,

I'm trying to implement a camera using FX3 SDK 1.3.4 based upon the AN75779 firmware example.

I've come to a few troubles that I detailed in another topic here. However, I want to ask about a different thing.

My product connects a single image sensor to the CYUSB3014 directly and uses a modified firmware based upon the AN75779 to stream the video on PC using VLC. The final application needs to use the same image sensor in two separate modes: RGB and IR. These two modes need to work as if they are different cameras, so I changed the descriptors to have a RGB Camera and an IR Camera. I've seen that in order to separate the images I need to process the pixel data and then channel the RGB data to one endpoint and the IR data to another endpoint using different DMAs.

My question is: is it possible to use this hardware configuration to perform these tasks or do I need some kind of DSP or ISP to perform the pixel processing?

Also, if it is possible, can you tell me where is the best place, in the code, to process the data? I'm assuming the DMA callback won't be good, but it seems to be the only place to change data from the sensor before being output to the PC.

Thank you for your help!

Best regards.

[AliAsgar]

Hi,

It would be better to use an ISP/DSP or an FPGA to process the RGB-IR output. 
As the data is time critical, processing on the FX3 side may result in errors and slows down the system.

Nevertheless, could you let us know exactly what processing needs to be done on the image sensor output, to get the RGB and IR outputs separately?

Best Regards,
AliAsgar

[RSar475]

Hello,

Thank you for your answer.

Yes, I thought that time would be a limitation as well...

According to the documentation, there are two methods to separate the images, the first one is based on Bayer Image Processing and has the following steps:

1. Black Level Subtraction and Lens Shading Correction (Bayer Processing);
2. IR Pixel Interpolation - the IR image is taken from here;
3. IR Subtraction;
4. Green Pixel Interpolation;
5. Resume Bayer Processing which includes RGB demosaic, Color Correction Matrix, tone and gamma adjustment and sharpening - the RGB image is taken from here.

The second is more custom and has the following steps:

1. Black Level Subtraction and Lens Shading Correction;
2. Demosaic;
3. Denoise;
4. IR Subtraction Matrix - the IR image is taken from here;
5. White balance and Color Correction Matrix;
6. Tone Map and Gamma - the RGB image is taken from here.

I haven't tried to implement any of the methods yet, so I still don't know what each of the operations requires when it comes to processing.

[AliAsgar · Accepted solution]

Hi,

We recommend you to use an FPGA/ISP/DSP to do the processing and send the resultant output to FX3. 

Processing at the FX3 side will result in timing violations.

Best Regards,
AliAsgar

[RSar475]

Thank you very much for you reply!