Senior Software Engineer Nikos Tsouknidas is part of the 主播大秀 Digital Media Distribution team. He explains how the team discovered an undocumented change between versions of their web server software, which caused certificate verification to fail.
In the Online Technology Group (OTG) Media Distribution team, we use digital certificates for service-to-service and user-to-service secured communication. We rely on centralised policies and procedures for issuing and updating all our certificates.
Recently we updated our Transport Layer Security (TLS) enabled Apache server to version 2.4 and had issues accessing our secure services. When our personal certificates started failing, we saw this inscrutable message in our error logs:
Certificate Verification: Error (44): Different CRL scopeAnd thus began an odyssey, as this issue is not well (if at all) documented online.
Introduction
As always, a little is a good start. Some big organisations generate their own root Certificate Authority (CA) that is only renewed once every several years, whereas other organisations will use a trusted third-party root CA. From that root CA, intermediate CAs can be issued. Any of these CAs can issue certificates for servers and clients, but usually the root CAs don’t issue these kinds of certificates. So keep in mind, there is a tree of CAs.
The plot thickens
If people ever want to leave your organisation (unlikely :D ) or you want to decommission some machines, you will want to revoke these server or client certificates, and one of the ways to do that is to have a Certificate Revocation List (CRL) that includes these certificates. These CRLs can be deltas, or full CRLs, and they are usually updated on a regular basis.
Cue in Apache.
Apache () will need to know where your CRLs are, with the SSLCARevocationPath directive (e.g. /etc/crl/) and will also need to know how far up the CA tree to go with the SSLCARevocationCheck directive (none, leaf or chain). That last directive is important. If set to chain, Apache will go up the whole CA tree to the root, and make sure none of these intermediate CAs has been revoked by its issuing CA.
For now, we’ll keep the SSLCARevocationCheck directive as leaf, and use a revocation path, instead of just a revocation file.
In Apache 2.4 you will need to have a CRL for all CAs in your revocation path. One more thing about your CRLs is that Apache looks for a symlink whose filename is the hash of the CRL. A one liner to get this is:
$ ln -s ca.crl `openssl crl -hash -noout -in ca.crl`.r0().
Unravelling “Error (44): Different CRL scope”
So everything worked for your certificate and your Apache 2.2 server, and now that you are on Apache 2.4 you are getting this error. Search for a solution to that and you won’t get much joy, but hopefully we’ll change that with this post.
What is the scope of a CRL
Looking for an explanation for this error are not much help:
X509_V_ERR_DIFFERENT_CRL_SCOPE: Different CRL scope
The only CRLs that could be found did not match the scope of the certificate.
And the initial explanation from :
The CRL scope is the set of certificates that could appear on a given CRL. For example, the scope could be “all certificates issued by CA X”
But if you look into the openssl source code, and search for “crl scope“, eventually you will find the file crypto/x509/x509_vfy.c, and the get_crl_score() function, where this quite enlightening comment exists:
/* Check cert for matching CRL distribution points */…just before the call to the crl_crldp_check() function.
This is the eureka moment, and the first point where Distribution Points (DP) (4.2.1.13. in [rfc5280]() ) were connected to that problem. It is very difficult to find documentation connecting problems to CRL extensions and the Different CRL scope error.
Of course, once you know that an extension, in our case X509v3 Issuing Distrubution Point [sic] is the smoking gun…
The RFC is easy to read
From
For each distribution point (DP) in the certificate’s CRL
distribution points extension, for each corresponding CRL in the
local CRL cache, while ((reasons_mask is not all-reasons) and
(cert_status is UNREVOKED)) perform the following:
…
(b) Verify the issuer and scope of the complete CRL as follows:
…
(2) If the complete CRL includes an issuing distribution point
(IDP) CRL extension, check the following:
(i) If the distribution point name is present in the IDP
CRL extension and the distribution field is present in
the DP, then verify that one of the names in the IDP
matches one of the names in the DP. If the
distribution point name is present in the IDP CRL
extension and the distribution field is omitted from
the DP, then verify that one of the names in the IDP
matches one of the names in the cRLIssuer field of the
DP.
How we were caught up in this
The problem explained above should have been present in all versions of Apache, as it is part of the RFC. Apparently Apache 2.2 uses a more relaxed version of the openssl verify command, verifies differently, or just ignores this error. The only mention found in the Apache changelog is in version 2.3.15:
*) mod_ssl: revamp CRL-based revocation checking when validating
certificates of clients or proxied servers. Completely delegate
CRL processing to OpenSSL, and add a new [Proxy]CARevocationCheck
directive for controlling the revocation checking mode. [Kaspar Brand]
What happened with our certificates was that the issuing CA did not have a Distribution Point (DP) extension at all, and the CRL issued from that CA actually had an Issuing Distribution Point (IDP) extension. These are meant to match, but the check fails before that, since the CA does not have the relevant extension.
How to test with openssl
Ivan Ristic has an in which you can find instructions on how to create your root CA and intermediate CA, then issue server and clients certificates from these. You can recreate the problem by creating a CRL with the IDP extension, while the CA that issues that CRL does not have the DP extension. One tricky bit was how to get a config file that creates a CRL with an IDP extension, but the good people at MIT had for us.
Some helpful commands:
# look into your CRL
$ openssl crl -noout -text -in /etc/crl/<your_crl>.r0
# print the issuer of all the CRLs symlinked into your CRL path (e.g. /etc/crl/)
$ for f in /etc/crl/*.r?; do echo -n "$f: "; openssl crl -noout -issuer -in $f; done
# same as above, but printing the whole CRL
$ for f in /etc/crl/*.r?; do echo -n "$f: "; openssl crl -noout -text -in $f; done
# Use openssl to verify the validity of a certificate, by attempting to look up a valid CRL, where the CRL has been issued by a specific CA (you have to use the CA bundle, with the whole CA tree)
$ openssl verify -CAfile <your_ca_bundle>.crt -CRLfile <your_crl_file>.r0 -crl_check <your_certificate>.pemHints ‘n Tips ‘n Leftovers
When you run an openssl x509 command on certificate bundles, you’ll only get information about the first certificate in the bundle, and openssl doesn’t seem to have support for viewing the nth or all certificates in bundles yet. You can less(1) that file though, and look through it.
To conclude
This was quite a difficult problem to resolve and to test. As a happy coincidence, we had for a two day TLS course that week. Even though we couldn’t pin-point the problem together, he helped us with the right tools, knowledge and confidence to recreate this problem in a sandbox. We were then able to zero in on the cause.
Thanks go to our Operations team, Ivan Ristic, and all the 主播大秀 teams that collaborated to help us get to the bottom of this. The 主播大秀 engineering teams take pride in a deep understanding of the platforms we use, and always work together towards solutions. !