Make Python Trust a Self-Signed SSL Certificate

I recently found that some customer environments use self-signed SSL certificates for internal services. As soon as our application code tried to call those services, certificate verification failures started appearing everywhere.

Since the project code did not disable certificate verification, I needed a way to make the environment trust the certificate without changing the application itself.

Build a test environment

The following test was done on Ubuntu 16.04.

I used Docker to start two containers: one server and one client.

The server container provided a simple HTTPS service. After installing Nginx and OpenSSL, I generated a key and certificate with:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/test.com.key -out /etc/ssl/certs/test.com.pem

Then I enabled SSL in Nginx by editing /etc/nginx/sites-enabled/default:

listen 443 ssl default_server;
listen [::]:443 ssl default_server;

ssl_certificate /etc/ssl/certs/test.com.pem;
ssl_certificate_key  /etc/ssl/private/test.com.key;

Reload Nginx:

nginx -s reload

In the client container, add a hosts entry for test.com and try a request with curl. As expected, it fails:

curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

So now we have a reproducible self-signed certificate problem.

Make the system trust the self-signed certificate

Copy /etc/ssl/certs/test.com.pem from the server container into the client container. You can place it anywhere, for example under /etc/ssl/certs/.

Then compute its certificate hash:

openssl x509 -noout -hash -in test.com.pem

Example output:

2ce47d04

Create a symlink in the certificate directory using that hash and the .0 suffix:

ln -s test.com.pem /etc/ssl/certs/2ce47d04.0

At this point the operating system now trusts the self-signed certificate.

Test it

Try again with curl:

curl https://test.com

Or with Python’s urllib:

import urllib.request

with urllib.request.urlopen('https://test.com') as response:
    print(response.read())

Both now succeed.

If you test with requests, however, you may still get certificate verify failed, even though the code does not use verify=False:

import requests

response = requests.get("https://test.com")
print(response.text)

Why requests still fails

The requests library uses certifi as its certificate store, so by default it does not verify against the system certificate directory.

According to the requests documentation:

https://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

you can point it to a certificate file or directory through REQUESTS_CA_BUNDLE. On Ubuntu, exporting the system certificate directory works:

export REQUESTS_CA_BUNDLE="/etc/ssl/certs"

After that, the same requests code no longer raises the certificate verification error. It may still show a warning, but the request succeeds.