In testing I came across a strange result. I have an FX3 application where data is streamed at a fixed rate from an ADC. While overall the application is reliable and can transfer data for a day or more without corruption, changing the size of the CyDmaBuffer to certain values will reliably cause less data to be received than requested. For example, using the modified driver that enables transfers > 4 MB, if I configure 5111808 bytes per buffer (4992 1024 byte USB packets), then transfer 12 buffers, buffer 11 is always smaller by 12288 bytes:
[Device 0, Thread 0] 0: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 0! (hEvent: 1176) start: 1 (expected: 1) end: 128 [Device 0, Thread 1] 1: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 1! (hEvent: 1232) start: 129 (expected: 129) end: 256 [Device 0, Thread 0] 2: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 2! (hEvent: 1248) start: 257 (expected: 257) end: 384 [Device 0, Thread 1] 3: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 3! (hEvent: 1244) start: 385 (expected: 385) end: 512 [Device 0, Thread 0] 4: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 4! (hEvent: 1236) start: 513 (expected: 513) end: 640 [Device 0, Thread 1] 5: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 5! (hEvent: 1260) start: 641 (expected: 641) end: 768 [Device 0, Thread 0] 6: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 6! (hEvent: 1268) start: 769 (expected: 769) end: 896 [Device 0, Thread 1] 7: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 7! (hEvent: 1272) start: 897 (expected: 897) end: 1024 [Device 0, Thread 0] 8: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 8! (hEvent: 1276) start: 1025 (expected: 1025) end: 1152 [Device 0, Thread 1] 9: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 9! (hEvent: 1280) start: 1153 (expected: 1153) end: 1280 ERROR: [Device 0, Thread 0] 10 SMALL BUFFER RECIEVED (missing 12288 bytes) !!! [Device 0, Thread 0] 10: Got 5099520 bytes (127.692308 FX3 DMA buffers) in buffer 10! (hEvent: 1220) start: 1281 (expected: 1281) end: 34516 DMA buffer wrong end [Device 0, Thread 1] 11: Got 5111808 bytes (128.000000 FX3 DMA buffers) in buffer 11! (hEvent: 1252) start: 30421 (expected: 1409) end: 34517 DMA BUFFERS DESYNCED! DMA buffer wrong end Overflows: 0 Total FX3 DMA transfers Produced: 1536
In this test, the hardware was attempting to generate 128*12 = 1536 DMA transfers, and the hardware reports sending exactly that many with zero overflows. However, buffer 10 is inexplicably missing 12288 bytes, which is about 31% of one DMA buffer, which causes all subsequent frames to be misaligned (the "BUFFERS DESYNCED" error). Buffer 11, although shifted 12288 bytes, is the correct size. This will happen every time when the CyUSB buffer is this size, but goes away if I change it.
Wireshark shows that the missing bytes are only sent after the transfer has been aborted:
My understanding is that CyUSB buffers need to be an integer number of USB packets (in this case 4992) and should hold an integer number of FX3 DMA buffers (in this case 128), but otherwise can be adjusted as makes sense for the application. I'm not sure how to interpret the smaller returned number of bytes. Is there a way to check why the device sent less than expected?
Please share the following details so that we can debug the issue:
1. Based on my understanding, the FX3 DMA buffer size is 39KB (i.e 4992/128 KB). Please correct me if my understanding is wrong.
2. What exactly is the CyDmaBuffer? Is it a buffer used in the host application?
3. Are you using a custom host application or streamer? If you are using a custom application, is it possible for you to test with streamer and see if the issue is reproducible or not?
4. What are the sizes for CyDmaBuffer that gives no failures?
5. What is the source of data? Is it an FPGA interfaced with FX3 over GPIF II interface?
6. Please share the complete UART debug logs and wireshark traces as zip files for us to check
7. What is the cause of Abort seen in the traces? Is it sent from the application? Please let me know why this abort is sent.
8. Is it possible to send test patterns from the FPGA? That is, the first 5111808 can be sent as one pattern, next 5111808 bytes can be sent as another pattern and so on. In this way, we can identify/confirm which buffer was sent late.
1. DMA buffer size is 39936 bytes.
2. By CyDmaBuffer I mean the buffer passed to BeginDataXfer. For example, using a buffer size of 5111808 bytes fails reproducibly, but 4*5111808 successfully transfers 500,000 consecutive buffers.
3. I am using a custom application. Testing with the streamer is not easy, but would be a good idea if I can hack something together.
4. I could not get 4*5111808 to fail even in days of trying.
5. Data is being copied from an ADC directly to the FX3. There is no FPGA or anything like that.
6. The logs are not very helpful. They show the transfer start, the expected number of buffers completing, and then successful end of the transfer. There are no errors and the sequence is similar to the larger buffer case in which no data is lost.
7. I call abort at the end of the transfer to clear any residual data. If I do not do this, subsequent runs of the test program sometimes are unreliable, especially if there is a program crash. I can test without if that is a problem.
8. Yes, sorry for being unclear. Each DMA buffer is prepended with its buffer number (since start of program). The "start" and "end" values above are the buffer numbers. Looking at the actual transfered data, buffer 11 has the missing 12288 bytes prepended to it. So actually no data is lost, except for the last 12288 bytes of the final packet, which arrive after the abort and are discarded.
One thing that is unclear to me is how the FinishDataXfer call actually decides to return less data than requested? Is this something on the host side, or is the FX3 somehow indicating that less data should be returned?
I'll try to get wireshark logs and also test on another PC in case it is some kind of OS/driver/hardware problem tomorrow.
Did a quick test on another PC, and side from which buffer in the sequence was small, still have the same problem. I've attached a wireshark log as well.
I think my next step would be to try and reproduce this in the streamer application. This will be complicated though since my app sends a bunch of vendor commands to configure the transfer. I'll have to see if I can send those manually.
My sincere apologies for the delayed response.
Based on my understanding, a start and end indexes are added to the data stream by FX3 after receiving the data from the ADC. Please correct me if my understanding is wrong.
If my understanding is correct, then can you please let me know where exactly in the data stream is the "start" and "end" sequences added? Also, can you share the relevant code in the firmware that adds these information? This is to understand why the "end" is different in the following print that was shared as part of your original description:
start: 1281 (expected: 1281) end: 34516
Please let me know if the Abort () is called after FinishDataXfer returns or not. Is it possible for you to share a trace captured using a hardware analyzer like lecroy? This should give us more information on the issue seen.
Also, can you please test using streamer and let us know the result?