I making a project with the CY8CMBR2016 controller, I followed the recommendations of datasheet, the button have 8 mm diameter and the more long trace have 85 mm, the Cpin with 2.2nF, the calculate parasite capacitance the bigger have 17pF, the serie resistor with 560R, the resistor for Out_Sel for select encoding 4-bit with resistor 1K5, ARST with resistor 1K5, CMOD Cap with 2.2nF, Scan with resistor 1K5 and Sensitivity resistor 1k5 for medium, in debug with resistor of 5K6 and power supply with 3V3, some time the PCB alone the pin INT going up alone without anybody touch, Some people have some idea what happened. On the pin 4-bit don't have signal.
My overlay have 2mm but doesn't works of 0 until 2 mm
OBS the other chip on the PCB are the NFC controller but are disable!
i too faced similar issue while working with cap_sence last year. all remembered is that Cmod capacitance and resistance value in series wih need to be troubleshooting.
i din't remembered exact values i have used but but the underlying fact is that when you increse the series resistance (5.4 k in your case) more drop will be across the resistance leading the config hyper-sensitive (unstable). i think 560E is ideal value you may vary it for tuning but it should not be in more than 1K .
try reducing the value of resistance. if this is making the it less sensitive or if you are not getting proper results you may try incread
sing the C_mod value arond 4.7nf (but this will increase the sensing time)
if you want details about the configuration used please let me know i will get that from my archives.