Recently, several users experienced reconnection issue.
That is due to the out-of-date certificates.
Here is a quick description of why it is happening.
- When a device connects to mdash.net, it MUST have a certificate file, called
ca.pem
. This file is put on a device during firmware build. It is NOT created by the cloud.
- The
ca.pem
file is used during TLS handshake to verify the cloud
- So, both cloud and a device have certificates. Every certificate has a life span. They expire. If either a cloud cert or
ca.pem
cert is expired, a connection will fail.
- It is a USER responsibility to keep
ca.pem
files up-to-date. You can see the expiration date of any certificate by using an OpenSSL command openssl x509 -in CERT.pem -text
- It is Cesanta’s responsibility to keep mdash.net’s cert up-to-date. And Cesanta does it by using https://letsencrypt.org/ service. https://letsencrypt.org/ provides TLS certificates that are valid for 3 months.
- Cesanta setup an automatic auto-update procedure, where certificates are updated every 2 months - 1 month before it expires. And that goes on for years, with NO changes.
- A recent update happened automatically. NO changes was done to the update procedure. NO changes was made to mdash.net itself.
- Many users apparently have outdated
ca.pem
files. Why exactly this update caused failure, and why it did not fail before, is unclear at this point. One thing is clear: an outdated certificates must be updated.
Problem mitigation.
- Since Cesanta updates mdash.net’s certificate 1 month prior its expiry, it is possible to rollback to the previous one temporarily, since it is still valid.
- Cesanta did exactly that to make user’s devices to reconnect
- However this is a temporary measure that will work only for 1 month ahead.
- If user’s
ca.pem
files are not updated, then after 1 month it’ll be impossible to update them remotely, as they’d fail to connect to mdash. Only updating them physically would be possible.
5 Therefore during this time, all users must update their ca.pem
.
How to get ca.pem
file
- The
ca.pem
file is not made by Cesanta. It cannot be made by anyone but the issuer of the mdash’s certificate, which is https://letsencrypt.org/
- Therefore, it is https://letsencrypt.org/ which gives away
ca.pem
files for their customers.
-
https://letsencrypt.org/ has two different certtificate types: RSA and EC. They have two different corresponding CA files. By concatenating two of them together, it is possible to create
ca.pem
which supports both types that https://letsencrypt.org/ uses.
- The actions are:
a) Go to Chain of Trust - Let's Encrypt
b) Download RSA root CA, https://letsencrypt.org/certs/isrgrootx1.pem
c) Download EC root CA, https://letsencrypt.org/certs/isrg-root-x2.pem
d) Make an empty ca.pem
file, add EC root CA first, and then RSA root CA second
- Now you’ve got a
ca.pem
file which can work with ANY service that uses LetsEncrypt - and that includes mdash.
- You can use the OpenSSL command mentioned above to see the expiration times for your
ca.pem
We suggest to create ca.pem
and update certificates on your devices,
Questions about “How do I put ca.pem on my device” are of a different nature: if you own your fleet, you should know the answer. This post describes the cause of the issue, and the way to fix it.
In the coming days, the auto-update procedure will keep updating mdash.net certificate, which will cause an out-of-date devices fail.
WE WILL NOT STOP IT, for the simple reason to detect this condition and encourage everyone to update their devices. Once in two days, we’ll roll back certificates back to let disconnected devices connect again, and be accessible for updates.
NOTE - this is YOUR responsibility to take care about certificate management on your devices.
mdash.net did NO changes to either certificate update procedure, or to the service itself. So there is no apparent trigger for these failures, most probably a signature mechanism on LetsEncrypt changed - it is still unclear.
Hope that helps.
Please can you confirm whether a device built using mongoose-os and incorporating the ca-bundle at GitHub - mongoose-os-libs/ca-bundle will use the correct details in ca.pem? It appears as though this lib hasn’t been updated for around 2 years so may now include some expired / soon to expire LetsEncrypt certificates?
No, we can’t confirm that. Please make ca.pem file manually.
Alternatively, make a change to ca-bundle to add required CA certs.
Thanks for the post, having read it over a few times now and done some reading it’s starting to make more sense.
I’m sure I speak for all of your customers when I say please keep us informed of any developments. I know there are others in the same boat as me where sometimes customers do not turn on devices for a few months, so this will cause some pain. We’d appreciate any way possible to extend this period while trying to update certificates on devices.
This feels like it’s a process not without it’s risks, since a broken ca.pem
file could brick a remote device. @cesanta it would be great if you could comment on some best practices for doing this.
The ca.pem
file I get following the steps above is:
-----BEGIN CERTIFICATE-----
MIICGzCCAaGgAwIBAgIQQdKd0XLq7qeAwSxs6S+HUjAKBggqhkjOPQQDAzBPMQsw
CQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFyY2gg
R3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMjAeFw0yMDA5MDQwMDAwMDBaFw00
MDA5MTcxNjAwMDBaME8xCzAJBgNVBAYTAlVTMSkwJwYDVQQKEyBJbnRlcm5ldCBT
ZWN1cml0eSBSZXNlYXJjaCBHcm91cDEVMBMGA1UEAxMMSVNSRyBSb290IFgyMHYw
EAYHKoZIzj0CAQYFK4EEACIDYgAEzZvVn4CDCuwJSvMWSj5cz3es3mcFDR0HttwW
+1qLFNvicWDEukWVEYmO6gbf9yoWHKS5xcUy4APgHoIYOIvXRdgKam7mAHf7AlF9
ItgKbppbd9/w+kHsOdx1ymgHDB/qo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0T
AQH/BAUwAwEB/zAdBgNVHQ4EFgQUfEKWrt5LSDv6kviejM9ti6lyN5UwCgYIKoZI
zj0EAwMDaAAwZQIwe3lORlCEwkSHRhtFcP9Ymd70/aTSVaYgLXTWNLxBo1BfASdW
tL4ndQavEi51mI38AjEAi/V3bNTIZargCyzuFJ0nN6T5U6VR5CmD1/iQMVtCnwr1
/q4AaOeMSQ+2b1tbFfLn
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----
With the corresponding result from openssl x509 -in CERT.pem -text
being:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
41:d2:9d:d1:72:ea:ee:a7:80:c1:2c:6c:e9:2f:87:52
Signature Algorithm: ecdsa-with-SHA384
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X2
Validity
Not Before: Sep 4 00:00:00 2020 GMT
Not After : Sep 17 16:00:00 2040 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X2
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (384 bit)
pub:
04:cd:9b:d5:9f:80:83:0a:ec:09:4a:f3:16:4a:3e:
5c:cf:77:ac:de:67:05:0d:1d:07:b6:dc:16:fb:5a:
8b:14:db:e2:71:60:c4:ba:45:95:11:89:8e:ea:06:
df:f7:2a:16:1c:a4:b9:c5:c5:32:e0:03:e0:1e:82:
18:38:8b:d7:45:d8:0a:6a:6e:e6:00:77:fb:02:51:
7d:22:d8:0a:6e:9a:5b:77:df:f0:fa:41:ec:39:dc:
75:ca:68:07:0c:1f:ea
ASN1 OID: secp384r1
NIST CURVE: P-384
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
7C:42:96:AE:DE:4B:48:3B:FA:92:F8:9E:8C:CF:6D:8B:A9:72:37:95
Signature Algorithm: ecdsa-with-SHA384
Signature Value:
30:65:02:30:7b:79:4e:46:50:84:c2:44:87:46:1b:45:70:ff:
58:99:de:f4:fd:a4:d2:55:a6:20:2d:74:d6:34:bc:41:a3:50:
5f:01:27:56:b4:be:27:75:06:af:12:2e:75:98:8d:fc:02:31:
00:8b:f5:77:6c:d4:c8:65:aa:e0:0b:2c:ee:14:9d:27:37:a4:
f9:53:a5:51:e4:29:83:d7:f8:90:31:5b:42:9f:0a:f5:fe:ae:
00:68:e7:8c:49:0f:b6:6f:5b:5b:15:f2:e7
-----BEGIN CERTIFICATE-----
MIICGzCCAaGgAwIBAgIQQdKd0XLq7qeAwSxs6S+HUjAKBggqhkjOPQQDAzBPMQsw
CQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFyY2gg
R3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMjAeFw0yMDA5MDQwMDAwMDBaFw00
MDA5MTcxNjAwMDBaME8xCzAJBgNVBAYTAlVTMSkwJwYDVQQKEyBJbnRlcm5ldCBT
ZWN1cml0eSBSZXNlYXJjaCBHcm91cDEVMBMGA1UEAxMMSVNSRyBSb290IFgyMHYw
EAYHKoZIzj0CAQYFK4EEACIDYgAEzZvVn4CDCuwJSvMWSj5cz3es3mcFDR0HttwW
+1qLFNvicWDEukWVEYmO6gbf9yoWHKS5xcUy4APgHoIYOIvXRdgKam7mAHf7AlF9
ItgKbppbd9/w+kHsOdx1ymgHDB/qo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0T
AQH/BAUwAwEB/zAdBgNVHQ4EFgQUfEKWrt5LSDv6kviejM9ti6lyN5UwCgYIKoZI
zj0EAwMDaAAwZQIwe3lORlCEwkSHRhtFcP9Ymd70/aTSVaYgLXTWNLxBo1BfASdW
tL4ndQavEi51mI38AjEAi/V3bNTIZargCyzuFJ0nN6T5U6VR5CmD1/iQMVtCnwr1
/q4AaOeMSQ+2b1tbFfLn
-----END CERTIFICATE-----
If I understand this correctly, once updated our devices will be good till 2040?
Following the instructions in this post, the output of that file is:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:3a:f1:e6:a7:11:a9:a0:bb:28:64:b1:1d:09:fa:e5
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
Validity
Not Before: Aug 1 12:00:00 2013 GMT
Not After : Jan 15 12:00:00 2038 GMT
Subject: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:bb:37:cd:34:dc:7b:6b:c9:b2:68:90:ad:4a:75:
ff:46:ba:21:0a:08:8d:f5:19:54:c9:fb:88:db:f3:
ae:f2:3a:89:91:3c:7a:e6:ab:06:1a:6b:cf:ac:2d:
e8:5e:09:24:44:ba:62:9a:7e:d6:a3:a8:7e:e0:54:
75:20:05:ac:50:b7:9c:63:1a:6c:30:dc:da:1f:19:
b1:d7:1e:de:fd:d7:e0:cb:94:83:37:ae:ec:1f:43:
4e:dd:7b:2c:d2:bd:2e:a5:2f:e4:a9:b8:ad:3a:d4:
99:a4:b6:25:e9:9b:6b:00:60:92:60:ff:4f:21:49:
18:f7:67:90:ab:61:06:9c:8f:f2:ba:e9:b4:e9:92:
32:6b:b5:f3:57:e8:5d:1b:cd:8c:1d:ab:95:04:95:
49:f3:35:2d:96:e3:49:6d:dd:77:e3:fb:49:4b:b4:
ac:55:07:a9:8f:95:b3:b4:23:bb:4c:6d:45:f0:f6:
a9:b2:95:30:b4:fd:4c:55:8c:27:4a:57:14:7c:82:
9d:cd:73:92:d3:16:4a:06:0c:8c:50:d1:8f:1e:09:
be:17:a1:e6:21:ca:fd:83:e5:10:bc:83:a5:0a:c4:
67:28:f6:73:14:14:3d:46:76:c3:87:14:89:21:34:
4d:af:0f:45:0c:a6:49:a1:ba:bb:9c:c5:b1:33:83:
29:85
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage: critical
Digital Signature, Certificate Sign, CRL Sign
X509v3 Subject Key Identifier:
4E:22:54:20:18:95:E6:E3:6E:E6:0F:FA:FA:B9:12:ED:06:17:8F:39
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
60:67:28:94:6f:0e:48:63:eb:31:dd:ea:67:18:d5:89:7d:3c:
c5:8b:4a:7f:e9:be:db:2b:17:df:b0:5f:73:77:2a:32:13:39:
81:67:42:84:23:f2:45:67:35:ec:88:bf:f8:8f:b0:61:0c:34:
a4:ae:20:4c:84:c6:db:f8:35:e1:76:d9:df:a6:42:bb:c7:44:
08:86:7f:36:74:24:5a:da:6c:0d:14:59:35:bd:f2:49:dd:b6:
1f:c9:b3:0d:47:2a:3d:99:2f:bb:5c:bb:b5:d4:20:e1:99:5f:
53:46:15:db:68:9b:f0:f3:30:d5:3e:31:e2:8d:84:9e:e3:8a:
da:da:96:3e:35:13:a5:5f:f0:f9:70:50:70:47:41:11:57:19:
4e:c0:8f:ae:06:c4:95:13:17:2f:1b:25:9f:75:f2:b1:8e:99:
a1:6f:13:b1:41:71:fe:88:2a:c8:4f:10:20:55:d7:f3:14:45:
e5:e0:44:f4:ea:87:95:32:93:0e:fe:53:46:fa:2c:9d:ff:8b:
22:b9:4b:d9:09:45:a4:de:a4:b8:9a:58:dd:1b:7d:52:9f:8e:
59:43:88:81:a4:9e:26:d5:6f:ad:dd:0d:c6:37:7d:ed:03:92:
1b:e5:77:5f:76:ee:3c:8d:c4:5d:56:5b:a2:d9:66:6e:b3:35:
37:e5:32:b6
-----BEGIN CERTIFICATE-----
MIIDjjCCAnagAwIBAgIQAzrx5qcRqaC7KGSxHQn65TANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH
MjAeFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEcyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuzfNNNx7a8myaJCtSnX/RrohCgiN9RlUyfuI
2/Ou8jqJkTx65qsGGmvPrC3oXgkkRLpimn7Wo6h+4FR1IAWsULecYxpsMNzaHxmx
1x7e/dfgy5SDN67sH0NO3Xss0r0upS/kqbitOtSZpLYl6ZtrAGCSYP9PIUkY92eQ
q2EGnI/yuum06ZIya7XzV+hdG82MHauVBJVJ8zUtluNJbd134/tJS7SsVQepj5Wz
tCO7TG1F8PapspUwtP1MVYwnSlcUfIKdzXOS0xZKBgyMUNGPHgm+F6HmIcr9g+UQ
vIOlCsRnKPZzFBQ9RnbDhxSJITRNrw9FDKZJobq7nMWxM4MphQIDAQABo0IwQDAP
BgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUTiJUIBiV
5uNu5g/6+rkS7QYXjzkwDQYJKoZIhvcNAQELBQADggEBAGBnKJRvDkhj6zHd6mcY
1Yl9PMWLSn/pvtsrF9+wX3N3KjITOYFnQoQj8kVnNeyIv/iPsGEMNKSuIEyExtv4
NeF22d+mQrvHRAiGfzZ0JFrabA0UWTW98kndth/Jsw1HKj2ZL7tcu7XUIOGZX1NG
Fdtom/DzMNU+MeKNhJ7jitralj41E6Vf8PlwUHBHQRFXGU7Aj64GxJUTFy8bJZ91
8rGOmaFvE7FBcf6IKshPECBV1/MUReXgRPTqh5Uykw7+U0b6LJ3/iyK5S9kJRaTe
pLiaWN0bfVKfjllDiIGknibVb63dDcY3fe0Dkhvld1927jyNxF1WW6LZZm6zNTfl
MrY=
-----END CERTIFICATE-----
I use this package also, but when I look on my devices the ca.pem
I see in the filesystem doesn’t match - looks like the changes were never released
@cesanta considering I could build a new FW with update ca.pem
do you think it’s higher risk to attempt live push/overwrite of the ca.pem
file or to just flash a new firmware?
Since mDash has rolled back the certificate can you suggest a way for us to validate that any new certs pushed to devices will actually work once the certificate is rolled in ~30 days?
Is there another address we can try to hit or something that is using the new certs?
I’m reading about openssl
and curl
and potentially how to recreate a call using a specific ca-bundle.pem
to try and simulate this, any ideas would be much appreciated
You could try other our sites - they are all managed by LE.
For example, mongoose.ws, or dash.vcon.io
Devices won’t work obviously, but for testing TLS handshake should be enough.
Attention - X3-based certificates are active for a week from now.
Please use this time to update your fleet.
We’ll move to the new root in a week, to ensure that updates work as expected.
Understand you’ll need to test the new root in a week, but please if possible revert back after testing - we have customers who don’t turn devices on for weeks at a time and we won’t know when they are back online without the X3 based cert.
Can you confirm that sketch is built with the mDash 1.2.16 Library on sep32 will function after you move to the new root in a week?
We are at a DEAD STOP cannot get custom RPC functions to compile with 1.2.16 Library.
Please show an example or documentation.
Custom RPC function example rpc_gpio_write. Error “rpc_gpio_write not found”
@bclough please do not pollute this thread with irrelevant stuff - further on, all irrelevant posts will be removed
FYI - asked on letsencrypt about the right way of testing the clients:
Thanks for sharing @cesanta , interesting that the ca.pem file used in the ca-bundle library tagged 2.20.0
still connects for me using the method in your post.
Can you share the ca.pem
you were testing with?
Testing with the same ca.pem file - and yes, it connects OK.
Not sure yet how to interpret that, I am expecting to see a failure.
@cesanta checking in to hear the latest on this, what is the plan for mDash?
1 Like
UPDATE:
As of today Cesanta has implemented a workaround which allows old devices to connect. We confirmed this allowed some of our devices with mDash v1.2.13 to connect where they could not yesterday.
Cesanta advises this workaround will be in place for 3 months, so need to make sure your legacy devices are updated before then.
-AD