FX3 12 bit image sensor issue

[JaKo_4554081]

Hello,

We have sucessfuly build an application based on AN75779. Our setup is pretty much the same as mentioned in the doc, except for the image sensor board. We are using AR0330 from OnSemi which is pin to pin compatible. To make things work, except for developing a sensor driver, we have modified GPIF to use 16 bit bus, as the sensor output is 12 bpp. With this setup we were able to see an image (greenish, but that is expected due to YUY2).

Upon this we have decided to create a custom interconnect board, to make use of all 12 bits of data. Initial setup was using only 8MSB (camera bits 11-4 were connected to bits 7-0 of FX3, remaining 8 bits were padded). The custom interconnect board utilized all 12 bits: (camera bits 11- 0 are now connected to bits 11-0 of FX3).

Upon doing so, the image is a complete garbage (not that all the PCB traces were length matched). No changes were made to FX3 firmware. We have then made a test and changed the interconnect board once again, this time the camera bits 11-0 were connected to bits 15-4 of FX3 bus. The result was pretty much the same  - junk:

Unfortunately i did not make any pictures of the initial setup, where bits 11-0 of camera were connected to bits 11-0 of FX3 bus, but the output was similar to this:

Have we missted any step ? We can not find any logic behind this behavior. Initially, when 8 bits were used and rest padded the image was sort of ok, and now, when 12 bits are used and only 4 are being padded the output is a complete junk.

Thanks for help in advance

BR

[Rashi_Vatsa]

Hello,

Please let me know which host application is used to display the video.

To debug the problem is it possible to change the bits/pixel from camera to 8 bits again and test it with the custom interconnect board.

Also, please check the UART debug prints to check if the frame size received by FX3 after connecting the interconnect board is as expected You can track the producer events for one frame to check the frame size received by FX3 (GPIF II interface).

[JaKo_4554081]

Hi again,

We have modified one of produced interconnect boards to work with 8 bit bus. This changes pretty much causes our interconnect board to work like cyusb3acc-004a. So, bits 11-4 of image sensor are now connected to bits 7-0 of FX3 bus. With this change everything is working as intended (without changing firmware). Still no luck with using 12 bit bus, it makes no sense.

[JaKo_4554081]

Hi,

Thanks for reply.

We have tried few host applications - custom with OpenCV (simple open),  e-CAMView and VirtualDub. All give us the same result.

UART logs show a buffer failure from time to time, but this happens rarely and has been happening even on old board. We have tested current firmware on original interconnect board - everything is fine.

We will try to modify our custom interconnect board to reproduce the original one for verification.

[Rashi_Vatsa]

Hello,

We have tested the current firmware on the original interconnect board - everything is fine.

>> Please let me know if the sensor in this test was configured for 12 bpp and streaming was proper

[JaKo_4554081]

In all tests we keep the 12bpp sensor output (even when we have only used 8 bit data bus). The image was greenish but this is expected (RAW12 to YUY2 conversion). This is why the issues is so weird - we did not touch the FX3 firmware at all, simply added the missing bytes.

[Rashi_Vatsa · Accepted solution]

Hello,

Streaming RAW 12 using the UVC host application could also cause the issue as the UVC host application would sample the video data 16 bits in a YUV format and RAW12 sampling would be different.

To test if the interconnect board is working fine, you can track the number of producer events or no. of full buffers and partial buffer per frame. If the frame size/buffers received per frame received by FX3 is as expected, then the problem could be due to the host application sampling.

[JaKo_4554081]

Thanks for reply.

I agreed with You, but wouldn't this issue occur when we have been using 8 bits on 16 bit bus ? Since very beginning we have been using a 16 bit bus and YUV encoding. But initially, like I have mentioned, bits 11-4 of our image sensor were connected to bits 7-0 on FX3 bus. This means that remaining bits were padded. If You are correct, then I think that we would have had issues at this point, or am I wrong ?

[Rashi_Vatsa]

Hello,

I was thinking when bits 11-4 of the image sensor were connected to bits 7-0 on the FX3 bus, the MSB side of the pixel is considered as Y component of YUV format but when 12 bits are sent by the sensor the MSB will be sampled as U or V component. 

But if the streaming (12bpp) through the original board (cyusb3acc-004a) seems fine then maybe the issue is with the interconnect board

[JaKo_4554081]

You are right, but when we have used cyusb3acc-004a the sensor was still configured as 12bpp, but only 11-4 were taken into consideration.

We really doubt that this has to do anything with interconnect board. The design was as simple as copying the original cyusb3acc-004a and connecting bits 0-0 1-1 ... 11-11 of parallel interface.

I am beginning to think that this might be a colouring issue after all. On FAQ we can find:

Where can I find resources which would help me ? (RAW12 as YUY2 to RGB)

[Rashi_Vatsa]

Hello,

The KBA actually means that to view the RAW video you would need to write a custom host application that would sample the video data as per the  RAW format you are streaming.  This is because the RAW formats are not supported by the UVC based host applications

[JaKo_4554081]

Thanks for info,

Lets for a moment assume that our interconnect board is working as intended and 12 bits parallel bus is connected to 16 bit parallel bus of FX3 in way described above (11-0 from image sensor connected to 11-0 of FX, bits 15-12 of FX3 are then equal to 0). Our image sensor output is RAW12 (bayer) AR0330 from OnSemi.

Within FX3 firmware changes has been made with GPIF designer to work with 16 bit bus, as described in AN75779:

Sensor has been configured to work with 1280x720p and we should achieve 60fps, so following descriptor change has been made:

No further changes to FX3 firmware were made (as far as I recall).

Should, at this point, we see a proper image, but greenish only ? At the moment what we receive is this:

For comparison, same firmware (no changes) but with original interconnect board (utilizing only 8 bits of image sensor):

We are struggling with this issue for days now, and can't proceed with the project further. Any kind of help will be appreciated, thanks.

BR

[Rashi_Vatsa]

Hello,

Thank you for the details.

For comparison, same firmware (no changes) but with original interconnect board (utilizing only 8 bits of image sensor):

>> Is it possible to connect 12 bits of the sensor to FX3 via the original interconnect board? This helps us to understand if the issue is from the firmware or hardware side

If the above test cannot be done, we can check the frame size received by the FX3. This can be done by counting the CY_U3P_DMA_CB_PROD_EVENT events for one frame.  Also, please share your firmware so that I can help you to add the debug prints to calculate the frame size or you can capture USB trace using Wireshark and calculate the full and partial buffers per frame

[JaKo_4554081]

Thanks for reply.

Yes it is possible to connect 12bits of the sensor to FX3 via original board, but it requires manually wireing bus - which we have done 🙂 Perhaps I will tell You what was done as a part of debugging this issue:

1. Initial test - original interconnect board. Bits 11-4 are connected to bits 7-0 of FX3. This setup gives us greenish, but ok image.

2. We have taken the original interconnect board, cut all the traces except for clock, wired it up like it was original (to verify connections and performance of kynar wrapping). So once again btis 11-4 of image sensor were connected to bits 7-0 of FX. Image was OK, there was no performance penalty when bus was wired (we have wired faster interfaces in the past as the proof of concept).

3. Wires were removed, and this time connected the proper way - bits 11-0 were connected to bits 11-0 of FX3 board. This resulted in junk image. We have desoldered wires once again, done many verifications and everything was ok. We have even once went back to wiring as described in point 2, and it was ok.

All of these experiments were done on the same firmware (16 bus enabled with steps done as per AN75779).

I am attaching the firmware code to this email. I am forced to remove sensor related functionalities, as these were developed based on documents shared upon signing NDA. I hope that this will not introduce any issues. There might be a lot happening within cyfxuvcdscr.c file (many commented out bytes). This is due to the fact that I have been experimenting with the descriptor. However firmware as it is now (with sensor file) compiles and is able to stream data.

Thank You very much for assistance. I am looking forward hearing from You.

BR

Jakub

[Rashi_Vatsa]

Hello,

Understood.

Please find the attached .c file with the added UART debug prints. Please share the complete UART prints when the issue is seen

[JaKo_4554081]

Hi, thanks for the code.

Below is the result (captured by opencv):

EDIT:

We have set the image sensor to test pattern, and received this:

Upon some dumb colouring:

Image looks ok 

[Rashi_Vatsa]

Hello,

It seems there is a problem in the debug prints. I have re-attached the uvc.c. 

Please share the complete .txt with the logs

Image looks ok 

>> Do you mean you don't see any problems now?

[JaKo_4554081]

Hi,

I have been working on this issue over the weekend. Turns out that it was an issue with image intepretation, nothing related to hardware nor firmware.

What we have done to make things work - stream RAW12 properly (this might come in handy for someone in the future):

1. Connected bits 11..0 of image sensors to bits 14_3 of FX3.

2. Changed the FX3 USB descriptor to RGB555 type - bits 15 is ignored, so we start from 14.

3. Such frame is captured by OpenCV.

4. Underneath OpenCV creates a buffer containing RGB values for us. So in case of 1080p image (1920x1080) we end up with a buffer of size 1920x1080x3 (3 channels 8 bit each). The order of colouring withing this array is as follows:
- image[0] = B, image[1] = G, image[2] = R

5. Since we are using RAW12, our image streams data in bayer order RGRGRG on rows 0,2,4 and GBGBGB on rows 1,3,5. In order to get the RAW image a simple loop must be made:

std::vector<unsigned short int> data;

unsigned short int raw12_pixel_val ;

for (size_t i = 0; i<image_size; i+=3) {

  raw12_pixel_val = 0xFFF0 & (((image.data[i + 2] >> 3) << 10 | ((image.data[i + 1] >> 3) << 5) | (image.data[i] >> 3))

  data.push_back(raw12_pixel_val)

}

Bit shift comes from the fact, that OpenCV shifts R G B values left by 3 bits automatically, so this must be reversed.

After this You end up with an image of size 1080/2 x 1920/2. What is left it to convert this uint16_t buffer into opencv buffer:

memmove(imBayer.data, data.data(), data.size() * 2);

6. imgBayer is now suitable for further OpenCV processing.

Perhaps this post will help someone in similar situation. I have seen many people struggingling with RAW12/16 streaming...

I can share OpenCV c++ code for this if anyone is intereseted.

Thank You for all the assistance !

BR

Jakub

[Rashi_Vatsa]

Hello,

Glad to hear that the problem is resolved. The actual problem was on RAW12 is interpreted by the Host application.

It would be great if you are willing to share your custom host app. It will be helpful to other community members.

[JaKo_4554081]

Hi,

I am attaching a c++ OpenCv code.

It can be most likely optimized. Some summary of what has been done can be found within comments, but I will share this here for the sake of completness:

1. Our camera is an RGB bayer RAW12 image sensor.

2. FX3 descriptor has been modify to stream RGB555 data (5 bits for each color, MSB is automatically discarded - THIS IS IMPORTANT!).

3. Bits 14_3 of FX3 were connected to bits 11_0 of our image sensor. LSB are ignored.

4. The C++ code simply reorganizes capture frame by OpenCV into a single matrix of RAW values and puts it in the OpenCV matrix for further processing. As an example I have converted our image into grayscale.

Some notes for anyone that will struggle with the same topic:

1. Make sure You know the order of Your RAW camera stream (RGGB, GRBG etc.).

2. If Your sensor has 16 bit bus (RAW16) then modify the FX3 descriptor to use  RGB565 (6 bits for G, 5 for R and B). Bit shifting within the code will have to be adjusted a little bit.

GUIDs for various UVC formats can be found here:

https://docs.microsoft.com/en-us/windows/win32/wmformat/media-type-identifiers?redirectedfrom=MSDN

3. Alaways connect MSB of Your image sensor to MSB of FX3 (except for RGB55 where bit 15 is discarded 🙂 In this case MSB is bit 14 of FX3).

I hope that his will help someone in the future.

Thanks for the assistance once again.

BR

Jakub

[xoslot]

xoslot slotxoc4