CyU3PUsbSendEP0Data corrupts bulk DMA to host

[mgiacomelli]

In another thread I mentioned testing recovery from error conditions on the FX3.  To better test this, I made a short test driver that setups up a large number of buffers with BeginDataXfer and copies a fixed number of buffers from device to host.  It then checks for host errors, sends a vendor command to reset the GPIF to its initial state (rearmAcquisition()), and then repeats the cycle:

for (int arm = 0; arm < numRearm; arm++)
{
	for (int k = 0; k < devices; k++)
	{
		if(arm ==0)
			startAcquisition(USBDevices[k]);
		else
		{
			//check for erors on device - this corrupts bulk transfers
			//getRearmStatus(USBDevices[k]);
			
			Sleep(100);
			rearmAcquisition(USBDevices[k]);
		}
	}
	long rLen;
	for (i = 0; i < totalTransfers; i++)
	{
		for (int k = 0; k < devices; k++)
		{
			if (WaitForSingleObject(inOvLapList[k][nextBuffer].hEvent, 2000) != WAIT_OBJECT_0)
			{
				printf("WaitForXfer failed!\n");

				fail = true;
				goto End;
			}
			rLen = bufferSize;
			if (USBDevices[k]->BulkInEndPt->FinishDataXfer(bufferList[k][nextBuffer], rLen, &(inOvLapList[k][nextBuffer]), contextslist[k][nextBuffer]))
			{
				double triggers = rLen / (triggerLength *sampleSize);
				printf("%d: Got %d bytes (%f triggers) (hEvent: %d)!\n", i, rLen, triggers, inOvLapList[k][nextBuffer].hEvent);
				bytesReceivedList[k] += rLen;

				if (i + QueueSize < totalTransfers*numRearm)	
				{
					contextslist[k][nextBuffer] = USBDevices[k]->BulkInEndPt->BeginDataXfer(bufferList[k][nextBuffer], bufferSize, &(inOvLapList[k][nextBuffer]));
					if (USBDevices[k]->BulkInEndPt->NtStatus) // BeginDataXfer failed
					{
						printf("Xfer request rejected");
						return -1;
					}
				}
			}
			else
			{
				printf("FinishDataXfer failed on buffer %d!\n", i);
				goto End;
			}
		}
		//increment the circular buffer
		++nextBuffer;
		nextBuffer %= QueueSize;
	}
}

This test program can run for 10,000s of iterations without errors and seems reliable. 

Next I added a function called getRearmStatus to check if the device had any buffer overflows or other errors between iterations (see commented out call above):

int getRearmStatus(CCyUSBDevice *USBDevice)
{
	CCyControlEndPoint *ept = USBDevice->ControlEndPt;
	ept->Target = TGT_DEVICE;
	ept->ReqType = REQ_VENDOR;
	ept->Direction = DIR_FROM_DEVICE;
	ept->ReqCode = 0xA7;		//get device status
	ept->Value = 0;
	ept->Index = 0;		

	INT32 RearmStatus[2];  //number of overflows and DMA transfers in 4 byte words
	long len = 8;			
	ept->XferData((PUCHAR)RearmStatus, len);

	printf("Overflows since last rearm: %d\n",RearmStatus[0]);
	printf("Total FX3 DMA transfers Produced this rearm: %d\n",RearmStatus[1]);
	return RearmStatus[0];
}

And here is the firmware for that vendor command:

//poll to see what the status of the previous acquisition was, returns [overflows,DMA transfers]
case 0xA7:
CyU3PDebugPrint (4, "Poll overflow status\r\n");
if(wLength == 8)
{
	int *returnBuf = (int*) glEp0Buffer;
	*returnBuf = overflows;
	returnBuf = (int*) &glEp0Buffer[4];
	*returnBuf = dmacount;
	status = CyU3PUsbSendEP0Data (wLength, glEp0Buffer);
	if (status != CY_U3P_SUCCESS)
	{
	    CyU3PDebugPrint (2, "Send data failed\r\n");
	}
	else
	{

		CyU3PDebugPrint (2, "CyU3PUsbSendEP0Data success! \r\n");
	}
}
else
{
	CyU3PUsbAckSetup ();
}
isHandled = CyTrue;
break;

It takes the buffer overflow and total dma count, puts into an 8 byte buffer and sends it back to the host with CyU3PUsbSendEP0Data() .  This works and the host gets the correct values, but afterwards it is very likely that the bulk transfer will not be able to restart.  Usually within 1-5 cycles the transfers will error out.

For example, here is a log where it fails on the first time:

found 3 devices
USB-Serial (Dual Channel) Vendor 1 - 1204 - 7 -
USB-Serial (Dual Channel) Vendor MFG - 1204 - 7 -
Cypress FX3 USB StreamerExample Device - 1204 - 241 -

Configuring FX3 acquisition with parameters:
Number of triggers: 1024
Total acquisition size: 20447232 bytes (19 MB)
CYUSB bufferSize: 20447232 , 19968.000000 USB packets (16 total buffers)
Total CYUSB buffers per acquisition: 1 (1024.000000 triggers per buffer)

Beginning Transfer...

0: Got 20447232 bytes (1024.000000 triggers) (hEvent: 872)!
Overflows since last rearm: 0
Total FX3 DMA transfers Produced this rearm: 512
[above is the vendor command checking for device errors]

Rearming the ADC!
[reset the GPIF to the initial state]
FinishDataXfer failed on buffer 0!
NTSTATUS = c0000001
status string:  [state=STALLED status=UNKNOWN]
acquisition failed!

And here is the serial output:

Start ACQUIRE! 
Poll overflow status
CyU3PUsbSendEP0Data success!
Rearming acquisition 
CYU3P_PIB_ERR_THR1_WR_OVERRUN: 1 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR1_WR_OVERRUN: 2 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR0_WR_OVERRUN: 3 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR0_WR_OVERRUN: 4 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR0_WR_OVERRUN: 5 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR1_WR_OVERRUN: 6 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR1_WR_OVERRUN: 7 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR0_WR_OVERRUN: 8 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR0_WR_OVERRUN: 9 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
CYU3P_PIB_ERR_THR0_WR_OVERRUN: 10 (PROD: 937, CONS: 0) commit fail: 0 (code: 0) DMA: 937
Stopping acquisition! 
Overflows: 10 (PROD: 937, CONS: 0)
Received a clear feature request!

So CyU3PUsbSendEP0Data returns success, the host sends a vendor command to restart the acquisition, but the bulk endpoint never responds (status is stalled).  Commenting out the call to CyU3PUsbSendEP0Data completely fixed the error (can run for hours), but then I don't have any way to get the number of buffer overflows from the device, at least without resetting the bulk endpoint.  

Note that the GPIF state machine has reached its DONE state when the vendor command is issued (although DMA is still configured), so no bulk transfer is currently in progress.  The rearm command moves the GPIF back to its initial state to restart the next round of testing, which quickly fails.  

To summarize how to reproduce:

1) Copy data from GPIF with bulk endpoint to host using USB3.

2) Pause GPIF but leave DMA configured 

3)  Call CyU3PUsbSendEP0Data 

4) Restart GPIF (e.g. CyU3PGpifSMSwitch) and try to receive data on host through bulk endpoint.

After 4 the bulk endpoint will usually stop responding, although I have seen it work for as many as 6 cycles before terminating.  If 3) is skipped the program can transfer data forever. 

I do not understand this error.  Other threads mentioned not calling CyU3PUsbAckSetup after CyU3PUsbSendEP0Data, which I do not do.  I also saw issues with USB2 and the control endpoint, but I am not using USB2.  It seems like the control endpoint should not break transfers on the bulk end point?

If helpful, here is the entire firmware file: GpifToUsbFixedBuffer_EP0_fail.zip 

[Rashi_Vatsa]

Hello,

From the details shared I understand that the BULK endpoint stalls and EP0 transfer is not successful. Is my understanding correct?

If yes, please let me know the data (in bytes)  requested by the host on the BULK endpoint in a single transfer. Please confirm if the data requested in a single transfer is <= 4MB. Please refer to this thread  Solved: Maximum transfer size for BeginDataXfer() / Finish... - Infineon Developer Community  

Please check the NTSTATUS and USBDSTATUS values once the data transfer on the BULK endpoint is completed.

2) Pause GPIF but leave DMA configured

>> Please let me know when is this step done in the firmware.

Kindly let me know if the data sent by the device in response to the vendor command is received by the host successfully. If yes, how is the STEP 4 done

[mgiacomelli]

---

@Rashi_Vatsa wrote:

Hello,

From the details shared I understand that the BULK endpoint stalls and EP0 transfer is not successful. Is my understanding correct?

No, the EP0 transfer is successful:  

Overflows since last rearm: 0
Total FX3 DMA transfers Produced this rearm: 512

So the host gets the overflow count, but afterwards when the GPIF starts the bulk transfer times out.

---

If yes, please let me know the data (in bytes)  requested by the host on the BULK endpoint in a single transfer. Please confirm if the data requested in a single transfer is <= 4MB. Please refer to this thread  Solved: Maximum transfer size for BeginDataXfer() / Finish... - Infineon Developer Community  

---

Buffer size is 20447232 bytes (19 MB).  I am using the updated driver to allow up to 32MB per buffer on Win10.  All the CyUSB parameters are in my post if you need them, sorry if I made them unclear.

---

Please check the NTSTATUS and USBDSTATUS values once the data transfer on the BULK endpoint is completed.

Let me know if this is what you need:

printf("WaitForXfer failed!\n");
printf("NTSTATUS = %x\n", USBDevices[k]->BulkInEndPt->NtStatus);
char buff[1000];
PCHAR s = buff;
USBDevices[k]->UsbdStatusString(USBDevices[k]->BulkInEndPt->UsbdStatus, s);
printf("status string: %s\n", s);

NTSTATUS = c0000001
status string:  [state=STALLED status=UNKNOWN]

---

2) Pause GPIF but leave DMA configured

>> Please let me know when is this step done in the firmware.

Kindly let me know if the data sent by the device in response to the vendor command is received by the host successfully. If yes, how is the STEP 4 done

---

In this case the GPIF state machine advances until its end state and loops forever.  Once all of the data is received by the host, the host loop advances to the top of the code I posted, calls rearmAcquisition which has a simple vendor request hander that moves the GPIF back to its starting state.  I posted the firmware if you need to double check that, but this is the key part:

CyU3PDebugPrint (4, "Rearming acquisition (%d)...\r\n", DataSignature2*256+DataSignature);
	/* Jump directly to the start state, we do not check where we are currently (pass invalidState)... */
	uint16_t invalidState = 0xFFFF;
	apiRetStatus = CyU3PGpifSMSwitch(invalidState, START, invalidState, ALPHA_START, 0xFFFFFFFF);

	if (apiRetStatus != CY_U3P_SUCCESS)
	{
		CyU3PDebugPrint (4, "CyU3PGpifSMSwitch failed, error code = %d\r\n", apiRetStatus);
		CyFxAppErrorHandler (apiRetStatus);
	}

	//ACK the control transfer since we don't return data
	CyU3PUsbAckSetup();

This completes successfully ("Rearming acquisition" prints in the log above) and the GPIF does start (all the CYU3P_PIB_ERR_THR1_WR_OVERRUN messages indicate that it is generating data).   Note that the DMA hardware is left running (no new call  to CyU3PDmaMultiChannelSetXfer), but this seems very reliable, I have done over 1 million 19MB buffers in various tests and until I started adding the CyU3PUsbSendEP0Data it would appear to run forever without error or data corruption.

[mgiacomelli]

In the other thread you mentioned a problem with my hosting, here is the file directly.

[Rashi_Vatsa]

Hello,

I understand that you are trying to solve the problem as mentioned in this thread  Flush stale data after error conditions - Infineon Developer Community  

As asked in the thread, please provide the UART debug prints to understand the flow of the firmware

[mgiacomelli]

Hi Rashi,

I posted the debug logs above.  Let me know what else you would like to see and I can add them.  

Thank you for reminding me about the data flushing issue.  I got distracted once I had a workaround, but I would still like to understand that problem a little better as well.  I'll try to update that post later today.  

[Rashi_Vatsa · Accepted solution]

Hello,

From the shared UART debug prints, I understand that the following is the sequence of vendor commands. Please let me know if this is correct.

0x97 >> 0xA7 >>0xA1

If yes, please try the above sequence with the attached .c file. Please share the debug prints in case of failures

[mgiacomelli]

I tried adding CyU3PDmaMultiChannelReset followed by CyU3PDmaMultiChannelSetXfer immediately after each CyU3PUsbSendEP0Data, but the call to CyU3PUsbSendEP0Data still prevents bulk transfers until the next time I reset the end point.

Any ideas?  I can see no obvious reason why sending (but not receiving) data on a control end point should break pending bulk transfers but it is 100% reproducible.  Is this a known errata?