SDK-3.7.0: Detect memory leak while testing https_client snip code

Tip / Sign in to post questions, reply, level up, and achieve exciting badges. Know more

cross mob
AxLi_1746341
Level 7
Level 7
10 comments on KBA 5 comments on KBA First comment on KBA

Add while (1) { } around the wiced_https_get() call, then the device will

run to out-of-memory in a few minutes. then wiced_https_get() always fails.

Any chance Cypress AE team can take a look? I think this is an important issue.

I'm not sure if the memory leak is in TLS library or other part.

Axel

0 Likes
55 Replies

mwf_mmfae wrote:

I spoke to the development team and we should be able to release SDK 3.7.0-7 next week prior to Thanksgiving.  This rev will have all of the patches which were applied in WICED Studio 4. 

Hi mwf_mmfae

The SDK bug is a gating issue for us and all our resources depend on the timing to get SDK fix.

Our h/w and application features are ready, the only missing bit is the SDK bug that we cannot fix it by ourselves.

And we need to rearrange full test once we got new SDK.

It is not acceptable that the SDK release delay again and again.

I have to ask again. When will SDK 3.7.0-7 available?

Note, I don't ask any customization for us. I'm asking for SDK bug fixes only.

0 Likes

I will continue to ask the development team for a time commitment on this release of the SDK.

I have asked twice today.

0 Likes

mwf_mmfae wrote:

I will continue to ask the development team for a time commitment on this release of the SDK.

I have asked twice today.

Thanks, I hope to get sdk release soon.

0 Likes
Anonymous
Not applicable

Hi mwf_mmfae,

What about the issue reported yesterday in the below mentioned discussion chain. Even I spend whole in testing my application on 4.0.1 hoping that you might have fixed the BESL issue but its so sad to see that we are still getting out_of_memory error after some no. of runs

Its annoying to see that after these much discussions also your team is not able to provide a good stable release for BESL Library without any memory leaks. Its something basic that a library you provide should not have any kind of memory leaks. Other bugs still its acceptable but not Memory leak

https://community.cypress.com/message/29721?et=watches.email.thread#29721

0 Likes

mwf_mmfae

I think something very wrong in WICED SDK release process!

If cypress team can release fixes only for the SDK bugs (including source code fix or library fix) on the forum by patch or by zip files,

People don't need to wait for new sdk release. People can continue use the SDK they are working on.

Otherwise cypress team needs to frequently update sdk release.

However, current situation is: No bug fix in any form for very long time.

I use the SDK so I do care about the quality of the SDK.

I have tried asking to improve the process, but I don't see too much improvement.

e.g.

Let the core team review user provided patches on the forum.

For the person familiar with the code, it's easy to tell if a fix is correct or not.

Follow up the pending issues. As I have pointed out many issues are still pending without follow up actions.

No response from cypress, it's so silence.

And most importantly, cypress needs to provide bug fix for the binary library in older SDK.

Evey new sdk can become old sdk some doay. You can not keep asking users to upgrade SDK.

For example, all older 3.x SDK has problem connecting IIS server using TLS-v1.2 due to a bug in BESL library.

This means all wiced devices on the market using https has this problem.

I just cannot believe there is no fix from cypress for such issue.

0 Likes
Anonymous
Not applicable

Hi mwf_mmfae,

If providing a memory leak-free TLS library is a real burden for your core team please think about alternative way to help customers --

providing alternative open-source TLS library adapter library as an option.

ex. mbed TLS (https://tls.mbed.org/ )


https://tls.mbed.org/

That customers can have another way to save their projects.

0 Likes

According to the SW Team:

1.) Fixes in 4.0.1 release

    1. a. MQTT reconnect logic fixed up to allow application to initiate a reconnection whenever the connection goes down. Of course, the app has to handle the reconnection but the library reliably sends an event on disconnection.
    2. b. BESL memory leaks – There were 2 leaks fixed relating to BESL and these 2 fixes should have been present since 4.0 release

2.) Fixes in 3.7.0-7 release

    1. a. BESL memory leaks – Both the BESL leaks have been fixed.
0 Likes

mwf_mmfae wrote:

2.) Fixes in 3.7.0-7 release

    1. a. BESL memory leaks – Both the BESL leaks have been fixed.

What about the BESL fix for sdk-3.1.2?

More and more of my customers are complaining the issue about not able to connect azure-iot (due to BESL bug).

mifo​ Tested latest-released WICED 4.1.   Out of the box, the heap leak still occurs when using MQTT.  

Despite closing the connection, my heap search algorithm still finds and frees quite a few instances of "bignum" "x509" and "tls".

Any thoughts?

0 Likes

DaveStude

I spoke with the SW team and far as we know, the memory leaks associated with BESL that is used by HTTPS/MQTT have been resolved. If you can share a short sample app which demonstrates the issue you are seeing, then vsha can check and get back to you.

djjw​  kask

0 Likes

HI Cypress Team,

Below you can find listing of calls for MQTT open and MQTT close routines (2 full cycles MQTT open-close) and info about mem allocation (malloc_info_command).

As you can see some MQTT close routine do not release all allocated memory by some MQTT open routine (probably mqtt_network_connect).

wiced_result_t mqtt_network_connect( mqtt_socket_t *socket )

{

    return wiced_tcp_connect( &socket->socket, &socket->server_ip_address, socket->portnumber, WICED_MQTT_CONNECTION_TIMEOUT );

}

==> allocates about 3kB of RAM but wiced_tcp_disconnect do not release this allocated memory, so each open-close cycle "earns" 3kB up to "out memory" event or hang-on system.

It is very easy to find and check, so why up to now you have not a solution of problem or some workaround ?

Robert

ARENAALLOCATEDNON-INUSETOP-MOSTdARENAdALLOCLEAK
wiced_hostname_lookup
###### mqtt_network_init / mqtt_network_init2775626900856856
###### mqtt_network_init / wiced_tcp_create_socket277562690085685600
###### mqtt_network_init / wiced_tls_init_root_ca_certificates31852291482704270440962248
###### mqtt_network_init / wiced_tls_init_identity40044317408304525681922592
###### mqtt_network_init / wiced_tls_init_context40044317408304525600
###### mqtt_network_init / wiced_tcp_enable_tls40044317408304525600
###### mqtt_network_init / mqtt_network_connect44140348449296473640963104
###### mqtt_network_init / wiced_tcp_register_callbacks44140348449296473600
MQTT connection opened44140348449296473600
###### mqtt_network_deinit / mqtt_network_deinit44140348449296473600
###### mqtt_network_deinit / wiced_tcp_unregister_callbacks44140348449296473600
###### mqtt_network_deinit / mqtt_network_disconnect44140348449296473600
###### mqtt_network_deinit / wiced_tls_reset_context44140327881135247360-2056
###### mqtt_network_deinit / wiced_tls_deinit_root_ca_certificates44140300201412081280-2768
###### mqtt_network_deinit / wiced_tcp_delete_socket441403002014120812800
MQTT connection closed441403002014120812800-3120
wiced_hostname_lookup441403002014120812800
###### mqtt_network_init / mqtt_network_init441403002014120812800
###### mqtt_network_init / wiced_tcp_create_socket441403002014120812800
###### mqtt_network_init / wiced_tls_init_root_ca_certificates441403226811872812802248
###### mqtt_network_init / wiced_tls_init_identity44140348609280601602592
###### mqtt_network_init / wiced_tls_init_context44140348609280601600
###### mqtt_network_init / wiced_tcp_enable_tls44140348609280601600
###### mqtt_network_init / mqtt_network_connect482363797210264549640963112
###### mqtt_network_init / wiced_tcp_register_callbacks482363797210264549600
MQTT connection opened482363797210264549600
###### mqtt_network_deinit / mqtt_network_deinit482363797210264549600
###### mqtt_network_deinit / wiced_tcp_unregister_callbacks482363797210264549600
###### mqtt_network_deinit / mqtt_network_disconnect482363797210264549600
###### mqtt_network_deinit / wiced_tls_reset_context48236359081232854960-2064
###### mqtt_network_deinit / wiced_tls_deinit_root_ca_certificates48236331401509688880-2768
###### mqtt_network_deinit / wiced_tcp_delete_socket482363314015096888800
MQTT connection closed482363314015096888800-3120
0 Likes

Hi Cypress,

The results of further investigation:

wiced_mqtt_connect -> mqtt_connection_init -> mqtt_network_init -> mqtt_network_connect -> wiced_tcp_connect (NetX_Duo) -> wiced_tcp_start_tls -> wiced_generic_start_tls_with_ciphers -> ssl_handshake_client_async

ssl_handshake_client_async( &tls_context->context ) allocates memory block but ssl_free(&tls_context->context) releases smaller memory block = MEMORY LEAK !

Beacuse SSL library is closed type, we can not correct that bug and another problems. It seems that BESL memory leak problems has long story (about 2 years) and up to now still not solved!

When you will provide useful BESL library?

Robert

0 Likes

Hi -

With a connected IP interface, we are calling the following to connect to aws iot:

wiced_mqtt_connect(mqtt_obj, address, interface, callbacks, security,

        &conninfo);

Then waiting some time and calling,

wiced_mqtt_disconnect(mqtt_obj);

and then after a little more time

wiced_mqtt_deinit(mqtt_object);

Standard stuff when compared to the secure_mqtt app.  I'm also ussing malloc_debug to note the allocated heap elements.   After this process, here's what I see:

----                     ----

bignum                   312    20034810

bignum                   312    200346d0

bignum                   180    20034fe8

bignum                   180    20034f30

bignum                   180    20034e78

bignum                   180    20034dc0

bignum                   180    20033828

bignum                   308    20034c88

bignum                   56     200337e8

bignum                   308    20034b50

tls                      72     20033f18

bignum                   56     20033ed8

bignum                   308    20033da0

pubkey                   212    20033cc8

x509                     940    20033918

0 Likes
Anonymous
Not applicable

Hi,

We were using SDK_3.5.2 in which MQTT reconnect has issues.

Based on the discussion in the forum we ported our application SDK_4.1.0 but the memory leak issues exist.

I am explaining the application logic on a connect, disconnect and reconnect. Please let me know if there is something which we may be missing.

Connect:

1. Shadow_init() does aws_app_init() subscribing to channels

2. aws_mqtt_conn_open() ->

3.resource_get_readonly_buffer() which does read the certificates ->

4. wiced_mqtt_connect()

Disconnect: On WICED_MQTT_EVENT_TYPE_DISCONNECTED event from mqtt_connection_event_cb()

1. Shadow_close() ->    

              mqtt_network_deinit(&(((mqtt_connection_t*)app_info.mqtt_object)->socket));

              mqtt_connection_deinit((mqtt_connection_t*) app_info.mqtt_object);

2. Then call the shadow_init  to trigger  aws_mqtt_conn_open()

In this scenario we observe that reconnect device hangs

Disconnect: On WICED_MQTT_EVENT_TYPE_DISCONNECTED event from mqtt_connection_event_cb()

1. Call the shadow_init  to trigger  aws_mqtt_conn_open()

In this scenario we observe that reconnect fails with error code (3036)

Looking forward for quick response.

Thanks in advance.

0 Likes

tested with WICED 5.0.   Closing MQTT connection no longer leaves the BESL items on the heap.   Issue appears resolved.

Thanks Dave.

0 Likes