The Administrator’s TLS Certificate Hell

640_data-webNowadays, a responsible administrator should never run any service without TLS anymore. Unfortunately, this makes life even more difficult for several reasons. This article discusses some problems associated to certificates, specifically in respect to web applications which interactively connect to secondary services.

On one hand TLS configuration requires a lot of knowledge to create a secure configuration. Pages such as Bettercrypto.org give a good support.

Another problem is to get certificates. Although it is not difficult in theory, practically the world of certificates is commercially driven. Currently, the trust model is built on a certificate store which contains “the officially trusted” certificates. As was shown with CAcert‘s root certificate it seems to be a commercial issue to get accepted within this store. And this in turn creates some problems. One is that almost all certificate authorities require you to pay for a valid certificate. CAcert seems to have failed to be an alternative. Hopefully, EFF‘s new approach Letsencrypt.org will change this ­čÖé Another solution might be StartSSL who offers free certificates as well.

Certificate Validation

Server certificates are validated by a client by checking the certificate authority’s signature within the server’s certificate. The validation process requires the client to have the CA’s certificate. And this cannot securely be retrieved from the Internet without having an additional certificate to validate the CA’s certificate. This obviously creates an endless loop.

Thus, those CA certificates are locally installed in advance. These create the trust-anchor. In other words this means that all TLS (and SSL) clients require some kind of trust-anchor to be able to create really secure connections to servers. And this usually is the certificate store mentioned above which is nothing else than a file containing a set of CA certificates.1

A lot of clients today support TLS and it is an integral part of all modern operating systems. A package known as Network Security Services (NSS) is a set of tools and libraries which provide access to cryptography (such as TLS). It is typically installed and provides root CA certificates. Typically, OpenSSL is installed as well. It also provides tools and libraries to deal with cryptography and, of course, it provides root CA certificates.

Web Application Backends

Network services are highly integrated today. This is that e.g. social networks are capable of being integrated into each other. But not just these, also cloud services, collaboration tools, and lot of other services can interact.

Complex modern web applications do a lot of stuff in the background  to create this interaction with other services. Mostly, these are modules accessing other web services. If these remote services run TLS, the web application has to validate certificates in the background which in turn requires that the web apps have access to trust-anchors.

Typical examples are WordPress, one of the most popular blogging software, and ownCloud, a popular free cloud software. Both ship with several additional plugins for background access to remote web services. And unfortunately, some of them don’t use the system’s installed root CA file but have their own root CA files. And this is the beginning of the end.

Root CA File Chaos

I’m running a WordPress multi-site installation, as well as an ownCloud in a TLS-enabled Apache with SNI on a FreeBSD 10.1 system. A find `sudo find -E /etc /usr -regex ".*\.(crt|pem)"` reveals a lot of certificate files.

/etc/ssl/cert.pem
/usr/local/share/certs/ca-root-nss.crt
/usr/local/etc/ssl/cert.pem
/usr/local/www/apache24/owncloud/config/ca-bundle.crt
/usr/local/www/apache24/owncloud/3rdparty/guzzle/http/Guzzle/Http/Resources/cacert.pem
/usr/local/www/apache24/owncloud/3rdparty/rackspace/php-opencloud/tests/OpenCloud/Tests/Common/cacert.pem
/usr/local/www/apache24/owncloud/apps/files_external/3rdparty/google-api-php-client/src/Google/IO/cacerts.pem
/usr/local/www/apache24/owncloud/apps/files_external/3rdparty/Dropbox/OAuth/ca-bundle.pem
/usr/local/www/apache24/owncloud/apps/files_external/3rdparty/aws-sdk-php/Guzzle/Http/Resources/cacert.pem
/usr/local/www/apache24/owncloud/resources/config/ca-bundle.crt
/usr/local/www/data/wp-content/plugins/wordpress-seo/vendor/yoast/api-libs/google/io/cacerts.pem
/usr/local/www/data/wp-includes/certificates/ca-bundle.crt
/usr/local/openssl/cert.pem

Actually there are a lot more files but I shortened the list to those relevant in respect to this article. There is the system default file in /etc/ssl, then there is the file from NSS in /usr/local/share/certs and the file from OpenSSL in /usr/local/openssl.

And additionally, as you can see, there are seven (!) root CA files in the ownCloud installation directory and two in the WordPress directory. And because of that you get two big problems.

First, it is hard to maintain. How to keep all them up to date? On one hand you could trust that the files are updated regularly with software updates. But a golden rule of good security is to be paranoid and to trust nobody than yourself.

You can symlink all those files to a single central system file of your choice. Unfortunately this will break update functionality which means that the software cannot update itself without administrator interaction because of the file permissions.

Second, it is even more hard to maintain if you need CA certificates other than the “officially trusted” ones such as e.g. the CAcert root certificate or your own self-signed CA certificates. You can add them to the system default file and — as suggested before — symlink all others. But again you will have troubles with updates.

No Real Solution Yet

Unfortunately, I don’t have a real solution. On my system I maintain my own root CA file and I copy (not symlink) it to the web apps where necessary. Because of that the web app updates still work. But anyway, it always requires my interaction. One of the ownCloud modules accesses the Amazon cloud which has a certificate of a CA which was deleted from the FreeBSD CA bundle for security reasons. Because of that you cannot enable the “Contacts” and “Calendar” app of ownCloud. You first have to add this certificate. See this issue on Github, there I explained what to do. On WordPress you cannot upgrade the TLS-based network sites (“Upgrade Network”) before you add missing CA certificates. Additionally you have to modify the file wp-includes/class-wp-http-curl.php and add the command curl_setopt( $handle, CURLOPT_CAPATH, '/etc/ssl/certs'); directly after the call to curl_init().

If you have troubles with (TLS) clients always be aware that a root CA file is required which might not have all required CA certificates for your setup.

  1. All browsers e.g. have this “file” built-in.