Unable to transfer large payload over MQTT over TLS using mbedtls

NiDa_4773006 · ‎Sep 15, 2020

I am trying to transfer a large file (~100kB) from our server over MQTT using mbedtls as TLS library. Any incoming/outgoing payloads that are less than the MTU (1500 bytes) transfers without error. Any incoming payloads greater than MTU result in TCP receive error. However, the exact same code downloading 9kB payloads from server via broker will succeed if verbose logging is enabled. The logging significantly slows the application. Is there any special application considerations when downloading payload that are greater than MTU (ie require defragmentation buffer)? If required, I can provide wireshark captures.

Setup:

Custom board with CY8C6247BZI-D54

RTOS

Wiced SDK 6.4 running custom 43xx_Wi-Fi app

Netx Duo v5.10

Common Defines

#for reduction of RAM usage

WICED_CONFIG_DISABLE_SSL_SERVER

WICED_CONFIG_DISABLE_ENTERPRISE_SECURITY

WICED_CONFIG_DISABLE_ADVANCED_SECURITY_CURVES

#Error Logging

WPRINT_ENABLE_PLATFORM_ERROR

WPRINT_ENABLE_SECURITY_ERROR

WPRINT_ENABLE_NETWORK_ERROR

#TX/RX Pools / TCP

WICED_USE_COMMON_PKT_POOL

COM_PACKET_POOL_SIZE=20

WICED_TCP_WINDOW_SIZE=65535

Scenario 1 - Download 108kB file in multiple 9kB chunks, verbose logging off. Failed download of chunk

Following Error occurs:

mbedtls_ssl_read failed with error : [2]

[MQTT-LIB] TCP receive error 2

mbedtls_ssl_read failed with error : [2]

[MQTT-LIB] TCP receive error 2

mbedtls_ssl_read failed with error : [2]

[MQTT-LIB] TCP receive error 2

Scenario 2 - Download 108kB file in multiple 18kB chunks, verbose logging on. Failed download of chunk

Following Error occurs

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2798: => write record

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:1310: => encrypt buf

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:1615: <= encrypt buf

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2487: => flush output

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2539: <= flush output

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2960: <= write record

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:7215: => read

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:3834: => read record

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2263: => fetch input

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2429: in_left: 0, nb_want: 5

TLS library asked for [5] bytes

Received new TCP packet with length [1460]

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2459: in_left: 0, nb_want: 5

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2460: ssl->f_recv(_timeout)() returned 0 (-0x0000)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2474: <= fetch input

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2263: => fetch input

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2429: in_left: 5, nb_want: 16413

TLS library asked for [16408] bytes

Skip [5] no of bytes from TCP received packet with length : [1460]

Defragmentaion case : copying 1460 bytes to defrag buffer 0x0802aa30

Defrag buffer length [16413], [0] bytes processed, [0] skipped, [0] received

TLS record defragmentation failed with result : [2]

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2459: in_left: 5, nb_want: 16413

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2460: ssl->f_recv(_timeout)() returned 2 (-0xfffffffe)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:4046: mbedtls_ssl_fetch_input() returned 2 (-0xfffffffe)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:3842: mbedtls_ssl_read_record_layer() returned 2 (-0xfffffffe)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:7314: mbedtls_ssl_read_record() returned 2 (-0xfffffffe)

mbedtls_ssl_read failed with error : [2]

[MQTT-LIB] TCP receive error 2

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:7215: => read

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:3834: => read record

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2263: => fetch input

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2429: in_left: 0, nb_want: 5

TLS library asked for [5] bytes

halfway TLS defrag ## defragmentation buffer bytes received [13140] total defragmentaion bytes [16413]

Already have [13140] bytes in defragmentation buffer, asked for [5]

defragmentation buffer bytes skipped [0]

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2459: in_left: 0, nb_want: 5

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2460: ssl->f_recv(_timeout)() returned 0 (-0x0000)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2474: <= fetch input

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2263: => fetch input

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2429: in_left: 5, nb_want: 16413

TLS library asked for [16408] bytes

halfway TLS defrag ## defragmentation buffer bytes received [13140] total defragmentaion bytes [16413]

halfway TLS record defragmentation failed with result : [2]

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2459: in_left: 5, nb_want: 16413

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:2460: ssl->f_recv(_timeout)() returned 2 (-0xfffffffe)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:4046: mbedtls_ssl_fetch_input() returned 2 (-0xfffffffe)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:3842: mbedtls_ssl_read_record_layer() returned 2 (-0xfffffffe)

WICED/security/BESL/mbedtls_open/library/ssl_tls.c:7314: mbedtls_ssl_read_record() returned 2 (-0xfffffffe)

mbedtls_ssl_read failed with error : [2]

[MQTT-LIB] TCP receive error 2

Scenario 3 - Download 108kB file in multiple 9kB chunks, verbose logging on. Successful download chunks

Additional Defines:

MBEDTLS_MEMORY_DEBUG

MBEDTLS_DEBUG_C

MBEDTLS_SSL_DEBUG_ALL

MBEDTLS_DEBUG_LOG_LEVEL=2

WPRINT_ENABLE_SECURITY_INFO

WPRINT_ENABLE_SECURITY_DEBUG

WPRINT_ENABLE_NETWORK_DEBUG

Logs of successful chunk downloads: