Recently, several users experienced reconnection issue.
That is due to the out-of-date certificates.
Here is a quick description of why it is happening.
- When a device connects to mdash.net, it MUST have a certificate file, called
ca.pem. This file is put on a device during firmware build. It is NOT created by the cloud. - The
ca.pemfile is used during TLS handshake to verify the cloud - So, both cloud and a device have certificates. Every certificate has a life span. They expire. If either a cloud cert or
ca.pemcert is expired, a connection will fail. - It is a USER responsibility to keep
ca.pemfiles up-to-date. You can see the expiration date of any certificate by using an OpenSSL commandopenssl x509 -in CERT.pem -text - It is Cesanta’s responsibility to keep mdash.net’s cert up-to-date. And Cesanta does it by using https://letsencrypt.org/ service. https://letsencrypt.org/ provides TLS certificates that are valid for 3 months.
- Cesanta setup an automatic auto-update procedure, where certificates are updated every 2 months - 1 month before it expires. And that goes on for years, with NO changes.
- A recent update happened automatically. NO changes was done to the update procedure. NO changes was made to mdash.net itself.
- Many users apparently have outdated
ca.pemfiles. Why exactly this update caused failure, and why it did not fail before, is unclear at this point. One thing is clear: an outdated certificates must be updated.
Problem mitigation.
- Since Cesanta updates mdash.net’s certificate 1 month prior its expiry, it is possible to rollback to the previous one temporarily, since it is still valid.
- Cesanta did exactly that to make user’s devices to reconnect
- However this is a temporary measure that will work only for 1 month ahead.
- If user’s
ca.pemfiles are not updated, then after 1 month it’ll be impossible to update them remotely, as they’d fail to connect to mdash. Only updating them physically would be possible.
5 Therefore during this time, all users must update theirca.pem.
How to get ca.pem file
- The
ca.pemfile is not made by Cesanta. It cannot be made by anyone but the issuer of the mdash’s certificate, which is https://letsencrypt.org/ - Therefore, it is https://letsencrypt.org/ which gives away
ca.pemfiles for their customers. -
https://letsencrypt.org/ has two different certtificate types: RSA and EC. They have two different corresponding CA files. By concatenating two of them together, it is possible to create
ca.pemwhich supports both types that https://letsencrypt.org/ uses. - The actions are:
a) Go to Chain of Trust - Let's Encrypt
b) Download RSA root CA, https://letsencrypt.org/certs/isrgrootx1.pem
c) Download EC root CA, https://letsencrypt.org/certs/isrg-root-x2.pem
d) Make an emptyca.pemfile, add EC root CA first, and then RSA root CA second - Now you’ve got a
ca.pemfile which can work with ANY service that uses LetsEncrypt - and that includes mdash. - You can use the OpenSSL command mentioned above to see the expiration times for your
ca.pem
We suggest to create ca.pem and update certificates on your devices,
Questions about “How do I put ca.pem on my device” are of a different nature: if you own your fleet, you should know the answer. This post describes the cause of the issue, and the way to fix it.
In the coming days, the auto-update procedure will keep updating mdash.net certificate, which will cause an out-of-date devices fail.
WE WILL NOT STOP IT, for the simple reason to detect this condition and encourage everyone to update their devices. Once in two days, we’ll roll back certificates back to let disconnected devices connect again, and be accessible for updates.
NOTE - this is YOUR responsibility to take care about certificate management on your devices.
mdash.net did NO changes to either certificate update procedure, or to the service itself. So there is no apparent trigger for these failures, most probably a signature mechanism on LetsEncrypt changed - it is still unclear.
Hope that helps.