issue with HTTP client Header parser

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Hello,
I met an issue with the lib http and http_client with the function which get the Http answer received after sending a HTTP GET request:

The function callback "deferred_receive_handler" (in /libraries/protocols/HTTP_clent/http_client.c):
manage the copy of received TCP data into buffers. One buffer for the header part, and another for the payload. The function detect the separation of the header and the payload with a double "\r\n" sequence. Copy the part before into the header buffer, and the part after into the payload buffer. The double caracters "\r\n" is not saved in the header neither payload buffer.

The function "http_parse_header" (in /libraries/protocols/HTTP_clent/http.c):
it parse the keyword of the header by detecting the characters "\r\n".
=> So, the last keyword of the header is never detected has the buffer is not terminated by the "\r\n" (as removed by the deferred_receive_handler function).

Concretely, if I am looking for a keyword of the header which is positionned at the last keyword, it is never detected! And unfortunately, the keyword order is managed by the server and I can't modify it.

It is a known issue ? is there is patch of something for that ?
I can modify the lib function to avoid that but I am surprised that nobody meet this issue before.
In addition, I am very surprised as this method is the recommanded usage, found in the exemple Cypress Academy WW101 (CypressAcademy_WW101_Files-master\Projects\ww101key\07c\09_aws_get).

Thanks

0 Likes
1 Solution
SChaudeurge
Level 2
Level 2
5 replies posted 5 sign-ins First solution authored

I had the confirmation of Wiced technical support that this issue is a bug from the HTTP_client lib.

The following patch has been validated to correct this bug:

from the file http_client.c, just invert the position of lines 505-506 with lines 508-509:

/***** Patch lib :
* just invert the 2 lines of code
* => instead of removing the "HTTP_CRLF_CRLF" from the Header and Payload,
* keep one HTTP_CRLF in the Header size (and keep removing both HTTP_CRLF_CRLF from the payload). By this way, the http parser will be able to detect the last keyword
* as it need a \r\n to detect a new line
* */

/* Payload starts just after the header */
http_response.payload += strlen( HTTP_CRLF_CRLF );

/* This will have HTTP header length */
http_response.response_hdr_length = (http_response.payload - strlen(HTTP_CLRF) - data); // Keep only \r\n in the header (not double \r\n, otherwise the "http_parse_header" function remaining_length will have value 2. And it will continue parsing the next line and can read the data part as header, which is wrong.
/***** ENF OF THE PATCH  **************/

 

View solution in original post

15 Replies
Aditi_B
Moderator
Moderator
Moderator
500 replies posted 5 questions asked 250 replies posted

Hi,

Are you using any standard AnyCloud example or have developed a custom example? If custom then please provide us the project details. Also, provide us some pointers to reproduce this issue at our end.

Thanks

Aditi

0 Likes
lock attach
Attachments are accessible only for community members.

Hello,

I am using the hardware module 1LD 43438 with Wiced v6.4.

The communication is done with HTTP request to an Amazon S3 server. But I use only standard, so the result is the same with any http communication with any server.

You can have a look on the function "deferred_receive_handler"and you will clearly see that the double "\r\n\r\n" (defined by HTTP_CRLF_CRLF) is removed as it the header length stop before it and the payload pointer start after it:

 

 

 

[...]
if ( find_status_line( (char*) data, fragment_available_data_length, &parsed_data, &parsed_data_length ) == WICED_SUCCESS )
{
     /* Now extract packet payload info such as data, data length, data type and message length */
     http_response.payload = (uint8_t*) strnstrn( (char*)data, fragment_available_data_length, HTTP_CRLF_CRLF, sizeof( HTTP_CRLF_CRLF ) - 1 );

      /* This will have HTTP header length */
      http_response.response_hdr_length = (http_response.payload - data);

      /* Payload starts just after the header */
      http_response.payload += strlen( HTTP_CRLF_CRLF );
[...]

 

 

 

Now, if you look the "http_parse_header"function:

 

 

 

while ( http_get_next_line( (const char*)line, remaining_length, &next_line ) == WICED_SUCCESS )
{
   uint32_t a;

   for ( a = 0; a < number_of_header_fields; a++ )
   {
       if ( memcmp( header[a].field, line, header[a].field_length ) == 0 )
       {
           if ( http_get_next_string_token( (const char*) line, (uint16_t) ( next_line - line ), ':', &( header[ a ].value ) ) == WICED_SUCCESS )
            {
                header[a].value_length = (uint16_t)( next_line - header[ a ].value ) - ( sizeof( HTTP_CLRF ) - 1 );
             }
             result = WICED_SUCCESS;
        }
     }

     remaining_length -= (uint32_t)( next_line - line );
     line = next_line;
   }

 

 

 

the "htt_get_next_line" function is looking for a \r\n to detect each keyword of the HTTP header. As the double \r\n is removed, the last line is never detected.

 

 

 

wiced_result_t http_get_next_line( const char* line, uint16_t max_length, char** next_line )
{
    uint32_t a;

    for ( a = 0; a < max_length - 1; a++ )
    {
        if ( ( *( line + a ) == '\r' ) && ( *( line + a + 1 ) == '\n' ) )
        {
            *next_line = (char*)line + a + 2;
            return WICED_SUCCESS;
        }
    }

    return WICED_NOT_FOUND;
}

 

 

 

 

I think you can easily reproduce this issue with any platform, using  the exemple from Cypress Academy WW101 : CypressAcademy_WW101_Files-master\Projects\ww101key\07c\09_aws_get.

0 Likes
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Here is an exemple of what I receive from the server (http response):

 

HTTP/1.1 206 Partial Content[CR][LF]
x-amz-id-2: bv4B8TitzS84pSyIlzKRJoipeBz+5rhwYRb5FWTCrfRIKNwc1c/4R5hkSWldxtXko7DTBErJxz8=[CR][LF]
x-amz-request-id: D0MZVDT6MQVEVKK3[CR][LF]
Date: Mon, 12 Jul 2021 15:13:06 GMT[CR][LF]
Last-Modified: Mon, 07 Jun 2021 13:22:29 GMT[CR][LF]
ETag: "866bd984f7949ecc5d5a7b89576ff07f"[CR][LF]
x-amz-server-side-encryption: AES256[CR][LF]
x-amz-version-id: H41RnyEXAT0Dxwx8nGqnUz6wuJN0VRs2[CR][LF]
Accept-Ranges: bytes[CR][LF]
Content-Range: bytes 18839344-18839344/18839344[CR][LF]
Content-Type: binary/octet-stream[CR][LF]
Server: AmazonS3[CR][LF]
Content-Length: 1024[CR][LF]
[CR][LF]
0xaa,0xab,0xac, etc...

 

after the reception callback the message is splitted into Header buffer and Payload buffer:

Header:

 

HTTP/1.1 206 Partial Content[CR][LF]
x-amz-id-2: bv4B8TitzS84pSyIlzKRJoipeBz+5rhwYRb5FWTCrfRIKNwc1c/4R5hkSWldxtXko7DTBErJxz8=[CR][LF]
x-amz-request-id: D0MZVDT6MQVEVKK3[CR][LF]
Date: Mon, 12 Jul 2021 15:13:06 GMT[CR][LF]
Last-Modified: Mon, 07 Jun 2021 13:22:29 GMT[CR][LF]
ETag: "866bd984f7949ecc5d5a7b89576ff07f"[CR][LF]
x-amz-server-side-encryption: AES256[CR][LF]
x-amz-version-id: H41RnyEXAT0Dxwx8nGqnUz6wuJN0VRs2[CR][LF]
Accept-Ranges: bytes[CR][LF]
Content-Range: bytes 18839344-18839344/18839344[CR][LF]
Content-Type: binary/octet-stream[CR][LF]
Server: AmazonS3[CR][LF]
Content-Length: 1024

 

and payload:

 

0xaa,0xab,0xac, etc...

 

As you can see, the last header keyword is not terminated by a [CR][LF] so, if I execute the function http_parse_header to looking for the last keyword it return a WICED_NOT_FOUND error.

 

 

parse_header.field = "Content-Length: ";
parse_header.field_length = sizeof("Content-Length: ")-1;
parse_header.value = NULL;
parse_header.value_length = 0;
result = http_parse_header(response->response_hdr,response->response_hdr_length,parse_header, 1); // Parse headers to get the content length
if (result == WICED_SUCCESS)
{
   // never reach here !!!
}

 

 

 

0 Likes
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Does anyone have any answer ?

1 week after the ticket is opened ...

Thank you.

 

0 Likes
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Hello,

I am sorry, I must insist: this is a critical issue for my project as it broke the OTA functionnality and all the products developped and already sold to final customers are no more accessible via http communication.

At least could you tell me is someone is currently working on this point ?

If nothing move I will contact our sale distributor as it is abnormal to have a such poor technical support (more than 2 weeks without any answer!)  ...

 

Thank you in advance !

0 Likes
Aditi_B
Moderator
Moderator
Moderator
500 replies posted 5 questions asked 250 replies posted

Hi,

Apologies for the inconvenience caused due to the delay. I've raised this issue to the internal team and probably by the end of this week, we'll have the solution/pointers to your issue. I'll revert back to you as soon as I have the update. Thank You for your patience.

Thanks

Aditi

0 Likes

@Aditi_B wrote:

Hi,

Apologies for the inconvenience caused due to the delay. I've raised this issue to the internal team and probably by the end of this week, we'll have the solution/pointers to your issue. I'll revert back to you as soon as I have the update. Thank You for your patience.

Thanks

Aditi


This is a very good bug report, it not only reported the symptom but also explained why the issue happened.
Clearly the http_parse_header parses the keyword of the header by detecting the characters "\r\n" but the
response->response_hdr is not ended with "\r\n". So it failed to parse latest keyword.

This is clearly a bug!
You said by the end of this week, we'll have the solution/pointers to your issue on July 27?
I'm surprised there is no fix so far.

Aditi_B
Moderator
Moderator
Moderator
500 replies posted 5 questions asked 250 replies posted

Hi,

Yes, it is a corner case that got missed in the HTTP LIB implementation. The customer has raised a support case and the issue is being resolved there. More updates will be provided on the fix/resolution soon.

Thanks

Aditi

MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Hello. Any news ?

0 Likes
Aditi_B
Moderator
Moderator
Moderator
500 replies posted 5 questions asked 250 replies posted

Hi,

Can you check with the snip example "httpbin_org" to see whether you get the same issue as above? It's an HTTP Client application. Location: 43xxx_Wi-Fi\apps\snip\httpbin_org. I'm trying to reproduce your issue at my end. Let me know whether this example reproduces the issue at your end.

Thanks

Aditi

0 Likes
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Yes you should be able to reproduce it with the httpbin_org snip as the issue I observed seems to comes from the http and http_client lib

I will try to do the same on my side

0 Likes
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

You can use this snip. Once you print the header of the response, identify the last keyword of the header, just before and try to detect it using the function "http_parse_header".
it parse the keyword of the header by detecting the characters "\r\n".
=> So, the last keyword of the header is never detected has the buffer is not terminated by the "\r\n" (as removed by the deferred_receive_handler function).

You could be able to reproduce the issue.

0 Likes
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Hello.
Any news ? did you reproduce the issue ?

There is now 1 month and a half I reported the issue, and still nobody seems to seriously take this subject into consideration !

Thanks.

0 Likes
MaCa_2922036
Level 3
Level 3
10 replies posted 10 sign-ins 5 replies posted

Hello,

Does anybody reproduced the issue ?

Does anybody investigates ?

0 Likes
SChaudeurge
Level 2
Level 2
5 replies posted 5 sign-ins First solution authored

I had the confirmation of Wiced technical support that this issue is a bug from the HTTP_client lib.

The following patch has been validated to correct this bug:

from the file http_client.c, just invert the position of lines 505-506 with lines 508-509:

/***** Patch lib :
* just invert the 2 lines of code
* => instead of removing the "HTTP_CRLF_CRLF" from the Header and Payload,
* keep one HTTP_CRLF in the Header size (and keep removing both HTTP_CRLF_CRLF from the payload). By this way, the http parser will be able to detect the last keyword
* as it need a \r\n to detect a new line
* */

/* Payload starts just after the header */
http_response.payload += strlen( HTTP_CRLF_CRLF );

/* This will have HTTP header length */
http_response.response_hdr_length = (http_response.payload - strlen(HTTP_CLRF) - data); // Keep only \r\n in the header (not double \r\n, otherwise the "http_parse_header" function remaining_length will have value 2. And it will continue parsing the next line and can read the data part as header, which is wrong.
/***** ENF OF THE PATCH  **************/