Sectigo SSL Certificate Root Expiration Issue

It has come to our attention that some of our customers have experienced outages in their website’s services or errors caused by a Sectigo Root certificate expiration on May 30.

If you run a standard website or blog with an SSL certificate and you still see your padlock in the browser, you are likely not affected by this issue and you don’t need to take any further action. Your website should be functioning normally in current browsers. 

If your website or other online service uses other applications or integrations such as APIs, сURL, OpenSSL, etc. you may have experienced problems or outages. If you have had any service disruptions or errors or your visitors use browsers older than 2015 and report issues, you will need to take action to update the service. 

Follow this guide to check if your service is affected and how to fix it.

We would like to share more details with our customers about the incident.

What Happened? 

Let’s start with an explanation of how SSL certificates are used by various applications. Whenever an application (for example, a browser) contacts a web service over the SSL/TLS protocol, the web service provides a set of SSL certificates to the application. The application then verifies that they were issued for the service the application is accessing, that the expiration date of the certificates has not passed, and that the certificates were signed by a trusted Certificate Authority. 

In order to verify the latter, the application attempts to link the provided certificates to one of the certificates contained in its trust storage. This trust storage is distributed either together with the operating system, runtime system, browser, or the application itself. If all the checks are passed, the application continues communication over the secure protocol. If not, the application either drops the connection or informs the user about potential security threats.

Sectigo’s AddTrust External CA Root was valid for 20 years until May 30, 2020 and was considered to be legacy. Using cross-certification, the Certificate Authority issued a pair of new Root certificates in 2010, which are valid until 2038, to replace the legacy Root. The new Root certificates were distributed via security updates to the majority of software applications utilizing the SSL/TLS protocol by mid-2015.

They continued to provide these certificates with the CA-bundles that included the AddTrust External CA Root and either USERTrust RSA Certification Authority or USERTrust ECC Certification Authority Intermediate (that expired on May 30, 2020) until April 30, 2020, to ensure that the certificates have the widest possible ubiquity (supported by most devices and systems, including the legacy ones).

As a result, many customers installed their SSL certificates together with the CA-bundle that was due to expire before the end-entity certificate (the one issued for the domain name). 

Thanks to the new cross-signed Root certificates, all modern browsers have both the expired and new Roots and automatically switched to using the new Root certificates. Users with these browsers did not experience any issues due to the expiration. 

Apart from that, the expiration of the Root certificate did not put data transmitted over secured connections at risk. However, when it comes to applications other than browsers that use cURL, such as SSL/TLS libraries of various programming languages, and runtime systems, some were unprepared for the change. 

These applications often use custom methods to verify SSL certificates. They may not rely on the operating system’s way of building the chain of trust and could not switch to using the new Root certificates. These applications either did not have the new Root certificates, had a broken certificate path validation logic, or were set up to explicitly trust the expired Root. For more detailed information on the affected clients, please refer to the research of Carnegie Mellon University.

The expiration of the Root certificate affected:

• Legacy clients that did not receive security updates since before mid-2015
  • Apple Mac OS X 10.11 (El Capitan) or earlier
  • Apple iOS 9 or earlier
  • Google Android 5.0 or earlier
  • Microsoft Windows Vista & 7 if the Update Root Certificates Feature has been disabled since before June 2010
  • Microsoft Windows XP if an Automatic Root Update has not been received since before June 2010
  • Mozilla Firefox 35 or earlier
  • Oracle Java 8u50 or earlier
  • Embedded devices (especially copy machines) that have not installed a firmware update since before mid-2015
• Clients configured to explicitly trust one of the expired Roots and ignore the operating system’s or vendor’s managed truststore
• Client software based on OpenSSL library prior to version 1.1.1
• Some OpenLDAP clients
• Java applications that do not use the default truststore
• Clients using cURL tool
• Clients with configured alerting via Pingdom or OpsGenie
• Applications that are connected to by any of the affected clients via SSL/TLS protocol

As a consequence of the expiration, various services were not able to initiate secure connections with other services or could not be accessed by other services. 

How to Fix the Issue

If you would like to check whether your service is affected or need to rectify the issue, please follow this guide.

Why Clients Were Not Informed Beforehand

We did not expect the expiration of the Root certificate to impact our customers for several reasons.

Even if there is an expiring Root certificate installed on the server, the new Roots are already included in the trust store of modern browsers and operating systems. This means that when an end-user accesses the website, the certificate chain of trust will be built from the new Root certificates.

Because of this, we did not expect there to be any issues with modern systems (including OS, distributions etc.). 

As it turned out, we made a mistake and underestimated the importance of the issue. We did not conduct a thorough enough investigation of the subject. We did not realize that legacy systems and applications with custom certificate validation mechanisms are widely used.

Namecheap was among those who suffered from the issue. The expiration of the Root certificates affected some of our internal services and caused outages on May 30, 2020.

Why Did I Receive an Expired CA-bundle for SSL Issued in April? 

Sectigo changed the CA-bundle provided to clients 30 days before the expiration of the root. But because of the bug  in our system we continued providing the AddTrust External CA Root until the date of its expiration, May 30.

A description of the bug: to increase the speed at which we receive certificate details from the Sectigo system and to reduce the load to our system, we keep all the CA bundles in a cache and do not get them directly from the Certificate Authority when an SSL is downloaded from a user account.

This bug has already been taken care of and all customers will now receive a modern chain when downloading the SSL from their account. 

If you experience any issue caused by the chain downloaded from your account, please let us know at once. Alternatively, you can download the CA-bundle with the new Root here.

How Will We Ensure that the Issue Does Not Reoccur?

We understand that the issue has caused a tremendous inconvenience to many of our customers and we apologize for that. 

In order to prevent such issues in the future we will update the way our system receives the CA bundle from Sectigo to ensure that only the relevant chain is downloaded from a user account, with the same level of website performance. 

In future, we will also inform our customers about any news and changes from the Certificate Authority and the SSL industry, so that the risks can be mitigated before any impact actually occurs. 

We apologize for any inconvenience to you and your business. That’s why we would like to offer you up to 20% off on all SSL certificates. Just add any SSL certificate to your cart and use the coupon code RootIssue2020 at checkout. The coupon code is valid from June 1 to June 30, 2020.

If you still have any questions or issues after reviewing our guide, don’t hesitate to contact our support team.