- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am facing error handling problem with FX3-based camera which leads to significant frame loss after an error occurs.
Camera configuration: 640 x 512, 16-bit, 50fps, ~262MBit/s = 32.7MB / s
FX3 configuration: USB 2.0 only, AN75779 based firmware, SDK version 1.3.4 and 1.3.5 (unofficial release)
After the host detects an error -71 it initiates a restart of the video stream, which results in a pause of about 1000 milliseconds between issuing CY_U3P_USB_SC_CLEAR_FEATURE and CY_FX_UVC_STREAM_EVENT which leads to a loss of 50 frames. Most likely, such a pause in case of an error -71 is set by the software on the host side, which, unfortunately, I cannot influence.
Some FX3 UVC device debug UART logs ( timestamp format: seconds.milliseconds )
9390.335 - UVC: Completed 5149 frames and 2 buffers, DMA(Watermark) = 2
9391.335 - UVC: Completed 5199 frames and 2 buffers, DMA(Watermark) = 3
9391.644 - Clear feature request detected..., DMA(WM) = 2
9391.645 - Application Stopped
9391.645 - UVC: Completed 0 frames and 0 buffers, DMA(Watermark) = 2
9392.645 - UVC: Completed 0 frames and 0 buffers, DMA(Watermark) = 0
9392.648 - Application Started
9392.648 - UVC: Completed 0 frames and 0 buffers, DMA(Watermark) = 0
9393.648 - UVC: Completed 49 frames and 29 buffers, DMA(Watermark) = 2
9394.648 - UVC: Completed 99 frames and 30 buffers, DMA(Watermark) = 2
9395.648 - UVC: Completed 149 frames and 30 buffers, DMA(Watermark) = 2
9396.648 - UVC: Completed 199 frames and 30 buffers, DMA(Watermark) = 2
9397.648 - UVC: Completed 249 frames and 30 buffers, DMA(Watermark) = 2
9398.648 - UVC: Completed 299 frames and 30 buffers, DMA(Watermark) = 3
The goal is to reduce this pause as much as possible. I am thinking about strategy described in KBA231382 article with monitoring difference between commited but not consumed DMA buffers to avoid this error at all, but it looks like I didn't quite get the idea.
The first thing I would like to get is a number of DMA buffers already sent by FX3 but not yet received by host as a threshold criteria to temporarely stop producing more data from GPIF state machine:
static uint8_t DMABufferWatermark = 0;
static uint8_t DMABufferWatermark_Max = 0
void CyFxUvcApplnDmaCallback (
CyU3PDmaMultiChannel *chHandle,
CyU3PDmaCbType_t type,
CyU3PDmaCBInput_t *input
)
{
CyU3PDmaBuffer_t dmaBuffer;
CyU3PReturnStatus_t status = CY_U3P_SUCCESS;
if (type == CY_U3P_DMA_CB_PROD_EVENT)
{
#ifdef FRAME_TIMER_ENABLE
/* Received data from the sensor so stop the frame timer */
CyU3PTimerStop(&UvcTimer);
/* Restart the frame timer so that we receive the next buffer before timer overflows */
CyU3PTimerModify(&UvcTimer, glFrameTimerPeriod, 0);
CyU3PTimerStart(&UvcTimer);
#endif
/* There is a possibility that CyU3PDmaMultiChannelGetBuffer will return CY_U3P_ERROR_INVALID_SEQUENCE here.
* In such a case, do nothing. We make up for this missed produce event by making repeated commit actions
* in subsequent produce event callbacks.
*/
status = CyU3PDmaMultiChannelGetBuffer (chHandle, &dmaBuffer, CYU3P_NO_WAIT);
while (status == CY_U3P_SUCCESS)
{
/* Add Headers*/
if (dmaBuffer.count == CY_FX_UVC_BUF_FULL_SIZE)
{
/* A full buffer indicates there is more data to go in this video frame. */
CyFxUVCAddHeader (dmaBuffer.buffer - CY_FX_UVC_MAX_HEADER, CY_FX_UVC_HEADER_FRAME);
}
else
{
/* A partially filled buffer indicates the end of the ongoing video frame. */
CyFxUVCAddHeader (dmaBuffer.buffer - CY_FX_UVC_MAX_HEADER, CY_FX_UVC_HEADER_EOF);
#ifdef DEBUG_PRINT_FRAME_COUNT
glFrameCount++;
glDmaDone = 0;
#endif
}
/* Commit Buffer to USB*/
status = CyU3PDmaMultiChannelCommitBuffer (chHandle, (dmaBuffer.count + CY_FX_UVC_MAX_HEADER), 0);
if (status == CY_U3P_SUCCESS)
{
DMABufferWatermark++;
#ifdef DEBUG_PRINT_FRAME_COUNT
glDmaDone++;
#endif
}
else
{
if(glDmaResetFlag == CY_FX_UVC_DMA_RESET_EVENT_NOT_ACTIVE)
{
glDmaResetFlag = CY_FX_UVC_DMA_RESET_COMMIT_BUFFER_FAILURE;
CyU3PEventSet(&glFxUVCEvent, CY_FX_UVC_DMA_RESET_EVENT, CYU3P_EVENT_OR);
}
break;
}
/* Check if any more buffers are ready to go, and commit them here. */
status = CyU3PDmaMultiChannelGetBuffer (chHandle, &dmaBuffer, CYU3P_NO_WAIT);
}
}
else if (type == CY_U3P_DMA_CB_CONS_EVENT)
{
if (DMABufferWatermark > 0)
DMABufferWatermark--;
streamingStarted = CyTrue;
glCommitBufferFailureCount = 0; /* Reset the counter after data is consumed by USB */
}
DMABufferWatermark_Max = max(DMABufferWatermark_Max, DMABufferWatermark);
}
void UVCAppThread_Entry (uint32_t input)
{
for (;;)
{
apiRetStatus = CyU3PEventGet (&glFxUVCEvent, CY_FX_UVC_STREAM_ABORT_EVENT | CY_FX_UVC_STREAM_EVENT | CY_FX_UVC_DMA_RESET_EVENT | CY_FX_USB_SUSPEND_EVENT_HANDLER, CYU3P_EVENT_OR_CLEAR, &flag, LOOP_TIMEOUT);
...
CyFxUVCAppDebugPrint(4, "UVC: Completed %d frames and %d buffers, DMA(Watermark) = %d", glFrameCount, (glDmaDone != 0) ? (glDmaDone - 1) : 0, DMABufferWatermark_Max);
DMABufferWatermark_Max = 0;
}
}
It seems like DMABufferWatermark_Max is around pretty normal 2..3 even in case of -71 error and never comes close to all 8 available DMA Buffers which I expect as a pre-error criterion.
How to correctly implement the recommendations given in the article KBA231382 in the software code?
Regards,
Sergiy
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Rashi,
I greatly appreciate your support,
It seems that this problem lies only on the hardware side. I ended up replacing the USB cables, adding a small ceramic capacitor (22uF) between the VUSB/GND pins on the camera side, and adding a script to reset the USB camera from Linux before running the capture application. Also i disabled USB autosuspend on Linux side.
After all these changes, the -71 error in the logs is no longer observed, at least for now
Best Regards,
Sergiy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
It seems like DMABufferWatermark_Max is around pretty normal 2..3 even in case of -71 error and never comes close to all 8 available DMA Buffers which I expect as a pre-error criterion.
>> The no. of buffers committed will depend on the DMA buffer count configured in the firmware. Please let me know the DMA buffer count and no. of PIB sockets associated with the DMA channel.
As USB 2.0 is used, the maximum UVC throughput in bytes (frame height × frame width × pixel size ×
frames per second) should be less than 40-MBps ( mentioned in AN75779 app note)
Please let me know if you tried the below points mentioned in the KBA
"Do the following to avoid all DMA buffers from being filled up:
- Allocate the maximum possible memory to the DMA buffers.
- Configure horizontal blanking of the image sensor as high as possible (to do this vertical blanking can be reduced thereby maintaining same frame rate). While doing this make sure that vertical blanking does not go below ~350 us."
Rashi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Rashi,
The no. of buffers committed will depend on the DMA buffer count configured in the firmware. Please let me know the DMA buffer count and no. of PIB sockets associated with the DMA channel.
<uvc.h>
/* UVC Video Streaming Endpoint Packet Size */
#define CY_FX_EP_BULK_VIDEO_PKT_SIZE (0x400) /* 1024 Bytes */
/* UVC Video Streaming Endpoint Packet Count */
#define CY_FX_EP_BULK_VIDEO_PKTS_COUNT (0x10) /* 16 packets (burst of 16) per DMA buffer. */
/* DMA buffer size used for video streaming. */
#define CY_FX_UVC_STREAM_BUF_SIZE (CY_FX_EP_BULK_VIDEO_PKTS_COUNT * CY_FX_EP_BULK_VIDEO_PKT_SIZE) /* 16 KB */
/* Number of DMA buffers per GPIF DMA thread. */
#define CY_FX_UVC_STREAM_BUF_COUNT (4)
<uvc.c>
/* Create a DMA Manual channel for sending the video data to the USB host. */
dmaMultiConfig.size = CY_FX_UVC_STREAM_BUF_SIZE;
dmaMultiConfig.count = CY_FX_UVC_STREAM_BUF_COUNT;
dmaMultiConfig.validSckCount = 2;
dmaMultiConfig.prodSckId [0] = (CyU3PDmaSocketId_t)CY_U3P_PIB_SOCKET_0;
dmaMultiConfig.prodSckId [1] = (CyU3PDmaSocketId_t)CY_U3P_PIB_SOCKET_1;
dmaMultiConfig.consSckId [0] = (CyU3PDmaSocketId_t)(CY_U3P_UIB_SOCKET_CONS_0 | CY_FX_EP_VIDEO_CONS_SOCKET);
dmaMultiConfig.prodAvailCount = 0;
dmaMultiConfig.prodHeader = 12; /* 12 byte UVC header to be added. */
dmaMultiConfig.prodFooter = 4; /* 4 byte footer to compensate for the 12 byte header. */
dmaMultiConfig.consHeader = 0;
dmaMultiConfig.dmaMode = CY_U3P_DMA_MODE_BYTE;
dmaMultiConfig.notification = CY_U3P_DMA_CB_PROD_EVENT | CY_U3P_DMA_CB_CONS_EVENT;
dmaMultiConfig.cb = CyFxUvcApplnDmaCallback;
These buffers are left unchanged, as in the original example, with total DMA memory = 16Kb x 4 x 2 = 128Kb.
As USB 2.0 is used, the maximum UVC throughput in bytes (frame height × frame width × pixel size ×
frames per second) should be less than 40-MBps ( mentioned in AN75779 app note)
Yes, I've seen that mention. As for me, 32.7 MB/s definitely has a reasonable margin in relation to the ~40 MB/s maximum.
Please let me know if you tried the below points mentioned in the KBA
"Do the following to avoid all DMA buffers from being filled up:
1. Allocate the maximum possible memory to the DMA buffers.
Yes, but raising number of DMA buffers per GPIF DMA thread to 6 (total DMA memory 192Kb) changes nothing at all: error -71 still remains, and the variable DMABufferWatermark_Max still does not exceed 2-3.
2. Configure horizontal blanking of the image sensor as high as possible (to do this vertical blanking can be reduced thereby maintaining same frame rate). While doing this make sure that vertical blanking does not go below ~350 us."
Unforunatelly it is not possible to change sensor vertical / horizontal blanking configuration with current sensor.
In any case, I'm still trying to find a way to prevent the host from issuing CY_U3P_USB_SC_CLEAR_FEATURE.
I am starting to think that root case of error -71 (0x43) appears on host side might not be an device buffer overrun but it might be something related to physical connection interference.
Does the -71 (0x43) error appearing on the host side always mean an FX3 DMA overrun, or could this error be related to a physical connection issue that the FX3 firmware can do little about?
Regards,
Sergiy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Sergiy,
Yes, but raising number of DMA buffers per GPIF DMA thread to 6 (total DMA memory 192Kb) changes nothing at all: error -71 still remains, and the variable DMABufferWatermark_Max still does not exceed 2-3.
>> Please confirm if you are using the default GPIF II state machine as in AN75779. If yes, please confirm if you tried to increase the DMA buffer size to 32KB count 3. With this change, did you also changed the RX payload size in the probe control structure and counter values in GPIF state machine?
Does the -71 (0x43) error appearing on the host side always mean an FX3 DMA overrun, or could this error be related to a physical connection issue that the FX3 firmware can do little about?
>> The two reasons through which 0x47 is received, is mentioned in the KBA pointed above
Rashi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Rashi,
My bad, -71 hex is 0x47, not 0x43
Please confirm if you are using the default GPIF II state machine as in AN75779.
Yes, I only changed GPIF bus width to 16 bit with corresponding changes to DATA/ADDR_COUNT = 8183
If yes, please confirm if you tried to increase the DMA buffer size to 32KB count 3. With this change, did you also changed the RX payload size in the probe control structure and counter values in GPIF state machine?
Yes, i changed DMA buffer size to 32Kb with corresponding changes in glProbeCtrl20 and GPIF DATA/ADDR_COUNT modified to 16375. I am able to see video but i still can't get rid of sporadic error -71
If this error is caused by the device, and not by the host, I would like to find a way in the firmware to prevent this by restarting the GPIF state machine in advance with DMA buffer clearing, losing only 1-2 frames, instead of responding to the CLEAR_FEATURE issued by the host followed by restarting video capture with loss of 50 frames.
Regards,
Sergiy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Rashi,
My bad, error -71 should has hex 0x47 of course
Please confirm if you are using the default GPIF II state machine as in AN75779.
Yes, with 16-bit GPIF data bus width and ADDR/DATA_COUNT = 8183
If yes, please confirm if you tried to increase the DMA buffer size to 32KB count 3. With this change, did you also changed the RX payload size in the probe control structure and counter values in GPIF state machine?
I tested with DMA Buffer 32KB x 3 and glProbeCtrl20 changed accordingly, ADDR/DATA_COUNT = 16375. Video stream works same as before, with sporadic -71 error in host logs and CY_U3P_USB_SC_CLEAR_FEATURE request appears on device side.
How can I catch an event or a precondition for such an event in the firmware code, which is the cause of the -71 error? As I understand it, CLEAR_FEATURE is a consequence of the host's reaction to such an event.
Regards,
Sergiy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Rashi,
My bad, error -71 hex is 0x47, not 0x43.
Please confirm if you are using the default GPIF II state machine as in AN75779.
Yes, just changed the width of the GPIF bus to 16 bit with ADDR_COUNT /DATA_COUNT recalculations (8183 for 16-bit)
If yes, please confirm if you tried to increase the DMA buffer size to 32KB count 3. With this change, did you also changed the RX payload size in the probe control structure and counter values in GPIF state machine?
I changed DMA buffer size to 32KB x 3 with corresponding changes in glProbeCtrl and changed GPIF counters ADDR/DATA_COUNT to 16375. I can see the video exactly the same as before, but the sporadically repeated -71 error remains.
Host application log:
May 17 04:58:50 sla-alip kernel: [ 3032.960305] uvcvideo: Non-zero status (-71) in video completion handler.
May 17 04:58:50 sla-alip VTNext: V4L2_BUF_FLAG_ERROR: need to restart capture
May 17 04:58:51 sla-alip VTNext: USB Capture PixelFormat YUYV Width 640 Height 512 BytesPerLine 1280
FX3 Device Debug UART:
1586.430 - UVC: Completed 1299 frames and 15 buffers, DMA(WM) = 2
1587.430 - UVC: Completed 1349 frames and 15 buffers, DMA(WM) = 2
1587.685 - Clear feature request detected..., DMA(WM) = 2
1587.685 - Application Stopped
1587.686 - UVC: Completed 0 frames and 0 buffers, DMA(WM) = 2
1588.686 - UVC: Completed 0 frames and 0 buffers, DMA(WM) = 0
1588.689 - Application Started
1588.689 - UVC: Completed 0 frames and 0 buffers, DMA(WM) = 0
1589.689 - UVC: Completed 49 frames and 14 buffers, DMA(WM) = 2
If this error is caused by the device and not the host, I would like to find in the firmware the reason for which it is called, and restart the GPIF State Machine and clear the DMA buffers in advance, losing only 1-2 frames.
That's why I added watermark variable to log output and expected the difference between committed buffers and consumed buffers at the time of the error to be 12 (or 6, in the case of 32Kb DMA buffers). But for some reason this doesn't happen.
Instead, the host sends a CLEAR_FEATURE_REQUEST for some reason and restarts the video, resulting in a loss of 50 frames.
Simply put, the firmware still cannot detect the moment that precedes the occurrence of an error and prevent it, like it was advised in KBA231382.
Regards,
Sergiy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Sergiy,
Please let me know if you have any debug prints for the DMA overflow or commit buffer failure 0x47 on the firmware side.
I see -71 in host app debug logs, please let me know how is this information sent to the host.
To debug the reason for clear feature from host, please share the USB traces using Wireshark (.pcap) and kindly confirm if CyU3DebugPrint or any other blocking API is not called in DMA callback.
Also, please register for GPIF (CyU3PPibRegisterCallback) and Endpoint event (CyU3PUsbRegisterEpEvtCallback) callback and share the UART debug prints.
Rashi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Rashi,
No, there is no UART prints on DMA callback
Unforunatelly i'm not able to provide USB sniffer files as this host device provided with pre-compiled linux kernel with no USBMON support enabled.
I'm not sure i've got your idea right, but i did following:
1. Speeded up Debug UART with uartConfig.baudRate = CY_U3P_UART_BAUDRATE_2M;
2. Registered CyU3PPibRegisterCallback and CyU3PUsbRegisterEpEvtCallback to capture all events:
CyU3PPibRegisterCallback (CyFxUvcAppPibCallback, 0xFFFFFFFF);
CyU3PUsbRegisterEpEvtCallback(CyFxAppIntEpCb, 0xFFFFFFFF, 0xFF, 0xFF);
static void CyFxAppIntEpCb(
CyU3PUsbEpEvtType evType,
CyU3PUSBSpeed_t usbSpeed,
uint8_t ebNum)
{
if ((ebNum != CY_FX_EP_BULK_VIDEO) && (evType != CYU3P_USBEP_NAK_EVT))
CyU3PDebugPrintTime(4, "EP:%02X,%d", ebNum, evType);
else {
CyU3PDebugPrintTime(4, "NAK");
}
}
static void CyFxUvcAppPibCallback (
CyU3PPibIntrType cbType,
uint16_t cbArg)
{
if ((cbType == CYU3P_PIB_INTR_ERROR) && ((cbArg == 0x1005) || (cbArg == 0x1006)))
{
if (!back_flow_detected)
{
CyU3PDebugPrintTime (4, "Backflow detected..., DWM=%d", DMAWatermark_Max);
back_flow_detected = 1;
}
}
CyU3PDebugPrintTime (4, "PIB:Typ %d,%02X", cbType, cbArg);
}
Debug log output:
123.609 - NAK
123.609 - NAK
123.609 - NAK
123.610 - NAK
123.610 - NAK
123.611 - NAK
123.612 - NAK
123.613 - NAK
123.613 - NAK
123.614 - PIB:Typ 4,1012
123.615 - Clear feature request detected..., , DWM=3
123.615 - PIB:Typ 4,1012
123.615 - PIB:Typ 4,101A
123.616 - Application Stopped
123.616 - UVC: Completed 0 frames and 0 buffers, DWM=3
124.616 - UVC: Completed 0 frames and 0 buffers, DWM=0
124.619 - Application Started
124.619 - UVC: Completed 0 frames and 0 buffers, DWM=0
Position of PIB event could be random relatively to Clear Feature. DWM variable is my attempt to monitor the maximum number of occupied DMA buffers according to KBA231382 / Scenario 2. CyU3PDebugPrintTime is my wrapper for CyU3PDebugPrint which adds time in Seconds.Milliseconds format.
Anyway, it seems like this issue is more hadware-oriented, since the following two actions significantly reduce the number of errors:
1. Adding 100uF capacitor between +5V and GND close to USB connector pins on device side.
2. Reseting USB device and disabling autosuspend before capture application startup :
UVC_CAMERA=/sys/bus/usb/devices/usb1
sudo sh -c "echo -1 > /sys/module/usbcore/parameters/autosuspend"
sudo sh -c "echo 0 > $UVC_CAMERA/authorized"
sleep 0.5
sudo sh -c "echo 1 > $UVC_CAMERA/authorized"
We will try to make more changes to the camera power circuit, maybe we missed something.
Thanks,
Sergiy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Sergiy,
From the UART logs shared, I can see that there are multiple NAK events before the clear feature. It seems there is some issue with the USB interface.
Also, as you mentioned that doing some changes on hardware improved the issue, the issue could be due to bad USB link.
You can refer to AN70707 and refer to the Schematic and layout checklist
Rashi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Rashi,
I greatly appreciate your support,
It seems that this problem lies only on the hardware side. I ended up replacing the USB cables, adding a small ceramic capacitor (22uF) between the VUSB/GND pins on the camera side, and adding a script to reset the USB camera from Linux before running the capture application. Also i disabled USB autosuspend on Linux side.
After all these changes, the -71 error in the logs is no longer observed, at least for now
Best Regards,
Sergiy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Sergiy,
Glad to hear that the issue is resolved!
Rashi