After 4 years of use we suddenly had many devices fail to reconnect. Had an err code “conn_handshake: err 0xffffd900”
Cesanta Support told us that is was a ca.pem that was out of date.
How can we verify that the mDash automatic certificate update from LetsEncrypt is successful?
How to retrieve the ca.pem file from LetsEncrypt and where do we put it?
Thanks for any help!!
Exact same issue here. Looking at the ESP32 console, we receive the following:
E event.c:18:mg_error 0x39 TLS handshake: -0x2700
Cesanta also advised that our ca.pem was out of date. What’s interesting is that FW built with mDash v1.2.13 + esp-idf v3.3.6 failed to reconnect, whereas FW with mDash v1.2.16 + esp-idf v4.4.6 could reconnect even though they had identical ca.pem files. Very odd.
It appears Cesanta has rolled back the mDash certificate today which allowed our units with old FW to reconnect, but that is likely just a temporary fix. We need to update ca.pem on the devices as you’ve noted.
Not yet sure how to do that, to be continued…
-AD
Thanks for your post. We were advised by Cesanta tech support that this was a temporary fix lasting less than 30 days. They also advised the following:
“Certificates from LetsEncrypt have life span of 3 months.
Our automatic updates trigger 1 month before the expiration day.
Meaning, that if your devices manage to connect, you have maximum 1 month to
update your devices. After that, NOTHING could help you reconnect your devices remotely.
You’d need to go and physically update the certificates.”
What is very troubling is that this has never been a problem for the past 4 years. I can find no documentation on the process of obtaining the ca.pem file and exactly how to update it on our Firmware.
this seems like a big deal. I’ve got devices with customers some of which are 3 years old.
@cesanta I’ve never seen any mention of keeping CA certs up to date on remote devices, is this something has been written up and we’ve missed?
What I’m most confused by is how ca.pem was created in the first place, does anyone know the details of this? Can it be fixed by flashing a new FW build?
ca.pem and its expiration is just a common TLS knowledge
Not mentioned explicitly in the documentation.
If you’re most confused how ca.pem was created in the first place, you should educate yourself on the build process and on the TLS essentials.
This is indeed a mystery.
Yes, one can Google how to manually create a ca.pem but it’s not clear how it is created in the ESP32/mDash flow.
It looks like in the 1.2.16 version of mdash (which is open source) there is a hardcoded certificate used to initialize the TLS connection.
Copied that s_pem, formatted it and running the openssl x509 -in xx.pem -text command on that certificate gives:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
82:10:cf:b0:d2:40:e3:59:44:63:e0:bb:63:82:8b:00
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Validity
Not Before: Jun 4 11:04:38 2015 GMT
Not After : Jun 4 11:04:38 2035 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:ad:e8:24:73:f4:14:37:f3:9b:9e:2b:57:28:1c:
87:be:dc:b7:df:38:90:8c:6e:3c:e6:57:a0:78:f7:
75:c2:a2:fe:f5:6a:6e:f6:00:4f:28:db:de:68:86:
6c:44:93:b6:b1:63:fd:14:12:6b:bf:1f:d2:ea:31:
9b:21:7e:d1:33:3c:ba:48:f5:dd:79:df:b3:b8:ff:
12:f1:21:9a:4b:c1:8a:86:71:69:4a:66:66:6c:8f:
7e:3c:70:bf:ad:29:22:06:f3:e4:c0:e6:80:ae:e2:
4b:8f:b7:99:7e:94:03:9f:d3:47:97:7c:99:48:23:
53:e8:38:ae:4f:0a:6f:83:2e:d1:49:57:8c:80:74:
b6:da:2f:d0:38:8d:7b:03:70:21:1b:75:f2:30:3c:
fa:8f:ae:dd:da:63:ab:eb:16:4f:c2:8e:11:4b:7e:
cf:0b:e8:ff:b5:77:2e:f4:b2:7b:4a:e0:4c:12:25:
0c:70:8d:03:29:a0:e1:53:24:ec:13:d9:ee:19:bf:
10:b3:4a:8c:3f:89:a3:61:51:de:ac:87:07:94:f4:
63:71:ec:2e:e2:6f:5b:98:81:e1:89:5c:34:79:6c:
76:ef:3b:90:62:79:e6:db:a4:9a:2f:26:c5:d0:10:
e1:0e:de:d9:10:8e:16:fb:b7:f7:a8:f7:c7:e5:02:
07:98:8f:36:08:95:e7:e2:37:96:0d:36:75:9e:fb:
0e:72:b1:1d:9b:bc:03:f9:49:05:d8:81:dd:05:b4:
2a:d6:41:e9:ac:01:76:95:0a:0f:d8:df:d5:bd:12:
1f:35:2f:28:17:6c:d2:98:c1:a8:09:64:77:6e:47:
37:ba:ce:ac:59:5e:68:9d:7f:72:d6:89:c5:06:41:
29:3e:59:3e:dd:26:f5:24:c9:11:a7:5a:a3:4c:40:
1f:46:a1:99:b5:a7:3a:51:6e:86:3b:9e:7d:72:a7:
12:05:78:59:ed:3e:51:78:15:0b:03:8f:8d:d0:2f:
05:b2:3e:7b:4a:1c:4b:73:05:12:fc:c6:ea:e0:50:
13:7c:43:93:74:b3:ca:74:e7:8e:1f:01:08:d0:30:
d4:5b:71:36:b4:07:ba:c1:30:30:5c:48:b7:82:3b:
98:a6:7d:60:8a:a2:a3:29:82:cc:ba:bd:83:04:1b:
a2:83:03:41:a1:d6:05:f1:1b:c2:b6:f0:a8:7c:86:
3b:46:a8:48:2a:88:dc:76:9a:76:bf:1f:6a:a5:3d:
19:8f:eb:38:f3:64:de:c8:2b:0d:0a:28:ff:f7:db:
e2:15:42:d4:22:d0:27:5d:e1:79:fe:18:e7:70:88:
ad:4e:e6:d9:8b:3a:c6:dd:27:51:6e:ff:bc:64:f5:
33:43:4f
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
55:1f:58:a9:bc:b2:a8:50:d0:0c:b1:d8:1a:69:20:27:29:08:
ac:61:75:5c:8a:6e:f8:82:e5:69:2f:d5:f6:56:4b:b9:b8:73:
10:59:d3:21:97:7e:e7:4c:71:fb:b2:d2:60:ad:39:a8:0b:ea:
17:21:56:85:f1:50:0e:59:eb:ce:e0:59:e9:ba:c9:15:ef:86:
9d:8f:84:80:f6:e4:e9:91:90:dc:17:9b:62:1b:45:f0:66:95:
d2:7c:6f:c2:ea:3b:ef:1f:cf:cb:d6:ae:27:f1:a9:b0:c8:ae:
fd:7d:7e:9a:fa:22:04:eb:ff:d9:7f:ea:91:2b:22:b1:17:0e:
8f:f2:8a:34:5b:58:d8:fc:01:c9:54:b9:b8:26:cc:8a:88:33:
89:4c:2d:84:3c:82:df:ee:96:57:05:ba:2c:bb:f7:c4:b7:c7:
4e:3b:82:be:31:c8:22:73:73:92:d1:c2:80:a4:39:39:10:33:
23:82:4c:3c:9f:86:b2:55:98:1d:be:29:86:8c:22:9b:9e:e2:
6b:3b:57:3a:82:70:4d:dc:09:c7:89:cb:0a:07:4d:6c:e8:5d:
8e:c9:ef:ce:ab:c7:bb:b5:2b:4e:45:d6:4a:d0:26:cc:e5:72:
ca:08:6a:a5:95:e3:15:a1:f7:a4:ed:c9:2c:5f:a5:fb:ff:ac:
28:02:2e:be:d7:7b:bb:e3:71:7b:90:16:d3:07:5e:46:53:7c:
37:07:42:8c:d3:c4:96:9c:d5:99:b5:2a:e0:95:1a:80:48:ae:
4c:39:07:ce:cc:47:a4:52:95:2b:ba:b8:fb:ad:d2:33:53:7d:
e5:1d:4d:6d:d5:a1:b1:c7:42:6f:e6:40:27:35:5c:a3:28:b7:
07:8d:e7:8d:33:90:e7:23:9f:fb:50:9c:79:6c:46:d5:b4:15:
b3:96:6e:7e:9b:0c:96:3a:b8:52:2d:3f:d6:5b:e1:fb:08:c2:
84:fe:24:a8:a3:89:da:ac:6a:e1:18:2a:b1:a8:43:61:5b:d3:
1f:dc:3b:8d:76:f2:2d:e8:8d:75:df:17:33:6c:3d:53:fb:7b:
cb:41:5f:ff:dc:a2:d0:61:38:e1:96:b8:ac:5d:8b:37:d7:75:
d5:33:c0:99:11:ae:9d:41:c1:72:75:84:be:02:41:42:5f:67:
24:48:94:d1:9b:27:be:07:3f:b9:b8:4f:81:74:51:e1:7a:b7:
ed:9d:23:e2:be:e0:d5:28:04:13:3c:31:03:9e:dd:7a:6c:8f:
c6:07:18:c6:7f:de:47:8e:3f:28:9e:04:06:cf:a5:54:34:77:
bd:ec:89:9b:e9:17:43:df:5b:db:5f:fe:8e:1e:57:a2:cd:40:
9d:7e:62:22:da:de:18:27
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----
I hope it helps…
That is very helpful! This explains why our devices running mDash1.2.16 are functional even though they have a discrete ca.pem file loaded that is expired.
FWIW, we are trying to roll out 1.2.16 images to our fleet. Unfortunately, this only works if devices were online before reckoning day. If not the devices are bricks 
Thanks for your post, very helpful! We are trying to update our fleet to 1.2.16 however we’re running into substantial issues. And as Autodog pointed out we will have to recall all units that are not online.
If you are able to build with 1.2.16 version of mdash think you’ll be okay. We took the code from mdash GitHub Examples minimal built it and it did successfully log on.
Yes - migrating from 1.2.13 to 1.2.16 was very painful for us, many breaking changes. Make sure you test thoroughly before deploying. There were many things that didn’t transition smoothly for us. Your mileage may vary.
-AD
I think this needs to be fixed, you have paying customers in this forum who have missed this detail and are being caused pain. Would be very beneficial to include this information in the mDash/Mongoose OS documentation somewhere.
@cesanta can you point me to some details on where the build process so I can educate myself?
What I don’t understand is why this is just happening now - my ca.pem that exists on all my devices seems to be the same which is:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
Signature Algorithm: sha1WithRSAEncryption
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Sep 30 21:12:19 2000 GMT
Not After : Sep 30 14:01:15 2021 GMT
Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
Looks like the cert expired a few years ago and never had an issue somehow?
From my limited testing, fortunately the fix seems to be as easy as creating a new FW that includes the ca.pem file I’ve created per the instructions in the Cesanta post and adding it to the filesystem.
Still trying to work out what I do for devices that have been offline for a little while and may come back online after the 30 day countdown…
** UPDATE ** Looks like this is the problem
The good news is that according to this post it sounds like @cesanta could make a special request to extend us out till Thursday, June 6th, 2024
This change appears to have fixed the problem a while back but was never released. Maybe someone at @cesanta could help us out with that?
If such an extension is possible, that would be a HUGE help to all of our customers. The logistical nightmare of swapping out bricks in the field makes my head (and wallet) hurt.
Would be interested in @cesanta’s view on this one.
I concur it would be a huge benefit to all of us! We have found anything built with 1.2.16 successfully logs in and remains online. We are considering developing a do-nothing app and deploying it, to keep our customers online past the deadline. In this way we would still be able to deploy the app once we get it tested.
@klimbot thank you for digging out the cause of the TLS failure.
It’s LetsEncrypt wants to migrate entirely to their own CAs, cause they are now bundled by browsers. Thus they shortened the expiration time of their original cert, signed by another company. That caused this issue.
A request has been made to LE to make an extension for mdash.net , Extend cross-signed by IdenTrust’s DST Root CA X3 signed certs - Help - Let's Encrypt Community Support
Thanks for making the request, hopefully we get a few more months to remote update our devices 
Yes, they made a hint on how to keep requesting old stuff.
We made this change, and let it run for a week.
Then we’ll revert to the new root - we want to make sure the update work, and catch those people who do not update.
This is necessary, cause if we keep running on the old cert and do not text until the doom day, it’ll be too late to fix then.
We very much dislike the “many more months to update” attitude, as experience tells that things get undone, and turn un-fixable then. So please do NOT expect this to continue for months. Update NOW.
Agree, we are working on an immediate fix and will roll out ASAP to online devices, but many of our fleet come online/offline periodically so we really need it rolled back after you test to ensure we can actually reach the majority of our fleet over the next couple of months.
If I can, I’d love to suggest something like the following timeline based on the Let’s Encrypt timeline(assuming it works for all):
April 14th - Cesanta tests any root ca changes that are required, and customers check that updated devices are connecting to mDash as expected without X3 cert in chain
April 15th or 16th - Cesanta move to new root ca that includes X3 in chain
April 16th – May 31st - Customers of mDash continue to update fleet devices that come online
June 1st - Cesanta request new cert from Let’s Encrypt valid till end of Sept
June 1st - New cert still allows customers to update remote devices that come online
Aug 1st - Cesanta auto-update cert process requests new cert 2 months after previous as per Cesanta automated process
Aug 2nd - Any issues that pop up can be resolved just as they have been by Cesanta rolling back cert, extending max window 1 month.
Sept 1st - No X3 based certs available anymore - remote devices on old cert will be bricked