TLS certificate validation in Golang: CRL & OCSP examples

Most applications use TLS for data-in-transit encryption and every programming language has a TLS support in its ecosystem. TLS was introduced in 1999 based on SSL 3.0. It's quite an old protocol, but, what is more important, it's very complex.

Apart from a simple “socket encryption” feature, TLS has dozens of various extensions. Dealing with all of them is pretty hard, even though they’re there for a reason. That’s why many TLS libraries have limited support for many of these extensions.

Unfortunately, even though Golang has native support for TLS, it has extremely limited support for OCSP and CRL. OCSP and CRL provide a way to verify whether the TLS certificate was revoked by CA before the application establishes secure communication with a service that uses this certificate.

At Cossack Labs, we provide security tools for developers to protect data in their apps. Mostly, we rely on cryptography, but it doesn’t live in a vacuum. Typical application and network security controls are necessary parts of surrounding infrastructure, and they, for various reasons, rely on OCSP and CRL to establish trust to remote parties.

We stumbled upon intricacies in OCSP and CRL when building Acra. Acra database security suite is an application that sits between the app and the database and encrypts / decrypts sensitive data.

Acra is written in Go and provides data security for applications exposed to elevated risks. The support of OCSP and CRL is crucial for Acra to prevent unauthorised connections from malicious or misconfigured apps to sensitive data.

To meet our security model, we had to implement OCSP / CRL verification in Golang ourselves. This is precisely what this post is about, but before diving into implementation details, let’s talk about OCSP and CRL from the bird’s eye.

1. The purpose of OCSP and CRL protocols
   • TLS and OCSP
   • TLS and CRL
2. OCSP and CRL in a PKI context
3. Configuring TLS for OCSP and CRL in Golang
   • Implementing OCSP in Golang
   • Implementing CRL in Golang
4. Live examples
5. How using OCSP and CRL could go wrong
6. Conclusions

1. The purpose of OCSP and CRL protocols #

OCSP and CRL help the certificate authority (CA) inform the application that a particular certificate has been invalidated so that the application should reject it.

For example, if a domain was hacked and its private TLS key was leaked. That stolen private key makes it possible to perform MitM attacks on visitors, redirecting them to the malicious website instead of the real one or intercepting sensitive data.

Another example would be identity stealing. When a server uses a client certificate to match it with a specific user, having someone’s private key means impersonating this user to a server. You can imagine the consequences.

Let’s examine OCSP and CRL as certificate validation approaches, TLS extensions that define them, and a simple Golang implementation.

TLS and OCSP #

OCSP (Online Certificate Status Protocol), RFC6960 is an interactive protocol that allows any party of a TLS handshake to ask the designated authority whether a provided certificate is still valid.

In the most simple scenario, the request contains only the serial number of the certificate we’re interested in.

The response from the authority is one of the following:

• Good—the certificate is still valid.
• Revoked—the certificate was revoked for some reason, and the app shouldn’t trust it. Possible reasons include: the private key was compromised, the CA key was compromised, the user owning the mentioned certificate no longer belongs to the company.
• Unknown—the OCSP server does not know about the existence of a certificate in question (about its serial number, to be precise). This may be caused by a misconfiguration or delays in OCSP server database updates.

The OCSP response is signed by the CA itself or by another certificate signed by CA and allowed to sign OCSP responses.

OCSP also has a non-interactive variant called OCSP stapling. The first party (server) adds OCSP response to a certificate presented to the other party (client). This way, the client does not need to perform additional network requests, only to validate the attached response. The standard library in Golang allows extracting such stapled OCSP responses, but they are not processed by default.

OCSP advantages: the application finds out the certificate was revoked as soon as possible (time depends only on how often the application performs OCSP requests).

OCSP disadvantages: increased handshake time because of additional network request(s). Imagine doing all the back-n-forth on every app startup for every TLS certificate. Be careful: if the OCSP stapling is used, the “Good” response may be cached for an extended time on the server-side. Long caching may allow the attackers to use revoked certificates while the application treats them as valid.

TLS and CRL #

CRL (Certificate Revocation List), RFC5280, is a non-interactive protocol. CRL is a file that contains a list of certificates revoked by a single CA–certificates' serial numbers and reasons why they were revoked. While the certificates might be still active (their expiration date has not come), they are revoked and shouldn’t be trusted.

As the CRL is always growing, the RFC describes “delta CRLs”–shorter files that contain only updates since the last version. CRL extensions can specify other fields of the certificate, not only serial numbers.

CRL has a “zero latency” compared to OCSP, but only if it was cached. On the other hand, applications don’t know whether certificates are revoked until they download the newer version of CRL.

CRL advantages: instant response (unless the cached CRL is outdated and should be downloaded again).

CRL disadvantages: there may be some time frame during which the application continues to use a revoked certificate before the new CRL is downloaded.

2. OCSP and CRL in a PKI context #

OCSP and CRL are necessary building blocks for enabling key rotation / revocation in PKI.

PKI consists of many moving parts: deployed services and admin procedures to manage them. PKI comes in all shapes and sorts: deploying self-signed homebrewed PKIs in Kubernetes, relying on openssl easy-ca, building something unique based on CFSSL, or putting together a bunch of bash scripts.

PKI’s architecture could be pretty fragile: implementing some of the key management procedures incorrectly decreases the security guarantees of the whole system in counterintuitive ways.

The simplest PKI could start with manual generation, signing and deployment of TLS cryptographic keys and corresponding certificates for every service to enable secure communications between them. OCSP and CRL are often considered “too advanced” to start with.

However, from a cryptographic perspective, once a cryptographic key is generated, there should be a procedure that manages its end of use: expiration and revocation.

OCSP and CRL are specifically designed to ensure that once the key becomes compromised, it is excluded from communications as soon as possible.

During a TLS handshake, each application validates the certificate (version, domain, expiration date, ciphersuites, etc.), and then runs additional checks to verify that the certificate was not revoked by a certificate authority.

Typically, setting up PKI components is done by the infrastructure engineers. Application developers rarely think about TLS certificates or how PKI works. However, implementing OCSP and CRL are a part of the application code. Developers should add code to perform network requests, validate the response, and handle the certificate revocation before continuing the connection to an untrusted service.

Unfortunately, many languages don’t come with OCSP / CRL batteries included, and developers need to homebrew them.

As application security is complex, mistakes in implementation could pass undetected and lead to a security compromise while preserving the illusion of security.

That’s why this post has a security overview, the implementation details, and the “what could go wrong” section.

3. Configuring TLS for OCSP and CRL in Golang #

Typically, in Golang, we create a TLS connection in two steps:

1. First, we create a tls.Config struct which may contain additional trusted certificates, requirements for TLS version and / or ciphers, and so on.
2. Second, we use this config for creating outgoing (tls.Dial()) or incoming (tls.Listen() / tls.NewListener()) encrypted stream.

As it was mentioned, the standard tls.Conn does not handle either OCSP or CRL. There is an ongoing discussion about supporting OCSP. Maybe there will be some attempts to implement it, but the Golang standard library currently doesn’t provide an OCSP verifier.

What can we use to implement it? The answer is TLS config. This is where we set up certificates, cipher suites, and other related things before creating outgoing connections or starting to listen for incoming ones.

Among all the fields of crypto.tls.Config, there is one called VerifyPeerCertificate:

VerifyPeerCertificate func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error

This callback will be called after a normal certificate chain validation is done (such as ensuring that the certificate belongs to a trusted CA and uses allowed ciphers). If this callback returns nil, a TLS handshake will be continued, otherwise, it will be aborted.

So, all our work from now on will be around this single callback: VerifyPeerCertificate. Remember the name.

How exactly will the arguments look like?

• rawCerts will contain chains of certificates in a raw ASN.1 format. Each chain starts from the leaf certificate and ends with a root self-signed CA certificate (if any).
• verifiedChains will contain a certificate chain that was verified in a sense that the signatures are valid and the last certificate in the chain is among trusted ones (installed in the system or added in TLS config). Compared to rawCerts, verifiedChains gives parsed and usable structs instead of raw byte arrays.

The Golang standard lib doesn’t support all TLS extensions; thus, if you need to use a particular extension, you may need to parse a certificate from rawCerts manually and then extract necessary properties.

But luckily that’s out of scope of this post as Go lib provides all what we need for Acra:

• OCSPServer — URL(s) of OCSP servers able to respond about current certificates.
• CRLDistributionPoints — URL(s) where we can download CRL(s) generated by current certificate’s CA.

Acra is a database encryption suite that helps you to encrypt sensitive fields and search through them. Works with SQL / NoSQL and validates TLS certificates

Check out Acra features

3.1. Implementing OCSP in Golang #

Suppose we are inside the VerifyPeerCertificate callback and got the OCSP service URL from verifiedChains[0][0].OCSPServer[0] (for simplicity, let’s ignore CA certificates and the rest of URLs if there are any).

Let’s start with crafting the OCSP request.

Module x/crypto/ocsp provides basic functions we will use to build a request and parse a response. Let’s take a look at ocsp.CreateRequest:

func CreateRequest(cert, issuer *x509.Certificate, opts *RequestOptions) ([]byte, error)

The first argument is the certificate we are validating. Just take verifiedChains[0][0]. The second argument is the issuer of this certificate, here it is verifiedChains[0][1].

If the verifiedChains[0][1] value is empty, it means that there’s no certificate’s issuer. Therefore, we are dealing with a self-signed certificate that passed previous checks (is allowed by TLS config), and OCSP is not relevant in this case.

The last arg, opts, currently only allows to set one thing: hash function used in the request. Let’s take SHA256 instead of default SHA1:

opts := &ocsp.RequestOptions{Hash: crypto.SHA256}

And create the request:

buffer, err := ocsp.CreateRequest(verifiedChains[0][0], verifiedChains[0][1], opts)
if err != nil {
    return nil, err
}

Now the buffer contains the request, serialized in a proper format, ready to be sent to the server. Let’s do it!

The request is often sent as an unencrypted HTTP because the response is signed, decreasing the chances of its tampering. The MIME types we are sending and expecting are application/ocsp-request and application/ocsp-response respectively.

httpRequest, err := http.NewRequest(http.MethodPost, verifiedChains[0][0].OCSPServer[0], bytes.NewBuffer(buffer))
if err != nil {
    return nil, err
}
ocspURL, err := url.Parse(verifiedChains[0][0].OCSPServer[0])
if err != nil {
    return nil, err
}
httpRequest.Header.Add("Content-Type", "application/ocsp-request")
httpRequest.Header.Add("Accept", "application/ocsp-response")
httpRequest.Header.Add("host", ocspURL.Host)

httpClient := &http.Client{}
httpResponse, err := httpClient.Do(httpRequest)
if err != nil {
    return nil, err
}
defer httpResponse.Body.Close()
output, err := ioutil.ReadAll(httpResponse.Body)
if err != nil {
    return nil, err
}

Then, response parsing looks like:

ocspResponse, err := ocsp.ParseResponseForCert(output, clientCert, issuerCert)

This ocspResponse is an ocsp.Response struct. If we get no error, the response helps us to decide whether the certificate is still valid.

The ocspResponse.Status represents one of the following:

• ocsp.Good means the certificate is still valid, and we can allow it.
• ocsp.Revoked says the certificate was revoked and cannot be trusted. Use ocspResponse.RevokedAt, ocspResponse.RevocationReason to get more info if needed.
• ocsp.Unknown states that the server does not know about the requested certificate, so, the response is neither “valid” nor “revoked”. You decide what to do in such cases. This may be a result of a server misconfiguration, or the OCSP responder did not get the latest updates after the questioned certificate was recently signed. From a security perspective, we’d recommend stopping trusting the certificate.

Now, based on ocspResponse.Status, you can return nil or some custom error from the callback.

3.2. Implementing CRL in Golang #

CRL does not require creating any particular request type. Instead, we can download the file and use the existing function to parse it into a struct. The CRL response is complex and should be cached somewhere in the app memory or on a disk.

One of the key advantages of CRL is that it’s non-interactive.

However, CRLs have a certain lifetime, so your cache should have an expiration feature. Whenever the app accesses a cached CRL, if it turns out to be outdated, the app has to download the whole CRL again (or download a delta CRLs).

Since the CRL is signed, the app can use a simple HTTP request for the download.

It’s essential to verify the CRL response signature to prevent potential attackers from tampering with the list.

Let’s get to code now. The first step is downloading a CRL (or reading it from a cache). Remember, the TLS certificate may contain CRL URL(s) in its metadata, and it’s the CRLDistributionPoints field of tls.Certificate.

After downloading, you parse the CRL (the ParseCRL function is part of the Golang standard library):

crl, err := x509.ParseCRL(rawCRL)
if err != nil {
    return nil, err
}

As description says, this function can parse both PEM-encoded and DER-encoded CRLs, while other libraries may have specific funtions for these two.

The next step is to check the signature. The CRL should be signed by the CA of the certificate you’re validating. This step is crucial to prevent tampering with CRLs.

err = issuerCert.CheckCRLSignature(crl)
if err != nil {
    return nil, err
}

Now, let’s make sure that CRL is not outdated. This is relatively straightforward as CRL contains an expiration timestamp called NextUpdate. This timestamp indicates a deadline when the CA will create a newer version of CRL, even if no certificates were revoked. However, if some certificates were revoked, the CRL will be regenerated as soon as possible.

if crl.TBSCertList.NextUpdate.Before(time.Now()) {
    return nil, ErrOutdatedCRL
}

Now we are sure that CRL is still valid. So let’s see how we can find out whether our certificate is among the revoked ones.

Well, in a simple case, the validation function looks like this:

func checkCertWithCRL(cert *x509.Certificate, crl *pkix.CertificateList) error {
    for _, revokedCertificate := range crl.TBSCertList.RevokedCertificates {
        if revokedCertificate.SerialNumber.Cmp(cert.SerialNumber) == 0 {
            return ErrCertWasRevoked
        }
    }
    return nil
}

As you can see, it’s just a list of serial numbers, and if you find a matching one in the list, this certificate should be rejected. However, for performance purposes, it would be better to build a map[bytes]pkix.RevokedCertificate after parsing the CRL for more efficient lookups.

4. Live examples #

To illustrate everything we’ve posted above, we created minimalistic OCSP and CRL implementations in Go.

Check out the GitHub repo cossacklabs/blogposts-examples with all the scripts to generate TLS certificates, OCSP responder based on OpenSSL, and Golang server-side and client-side apps. Feel free to review and run examples to see how the verification works with valid and revoked certificates.

Let’s run OCSP responder server using OpenSSL (see detailed instructions in Readme):

openssl ocsp \
     -index ca/index.txt \
     -CA ca/ca.crt.pem \
     -rsigner ocsp-responder/ocsp-responder.crt.pem \
     -rkey ocsp-responder/ocsp-responder.key.pem \
     -port 8888 \
     -ignore_err

ACCEPT [::]:8888 PID=34777
ocsp: waiting for OCSP client connections...

Let’s run a TLS server that uses a revoked certificate:

go run ./cmd/tls-server -key cert2-revoked/cert2-revoked.key.pem -cert cert2-revoked/cert2-revoked.crt.pem

Accepting connection...

The client-side application connects to the server, receives a TLS certificate, asks the OCSP responder for the OCSP response, and aborts connection because the certificate is revoked:

go run ./cmd/tls-client -ca_cert ca/ca.crt.pem -use_ocsp

Server certificate: Test leaf certificate (cert2-revoked)
Server certificate issuer: Test CA certificate
Validating server certificate with OCSP (serial: 113274642861313425704666455845030198894915511470)
[*] Crafting an OCSP request
[*] Preparing HTTP request to OCSP server
[*] Launching HTTP request to OCSP server
[*] Parsing OCSP server response
[-] Certificate status is Revoked
Cannot connect to server: The certificate was revoked!

Revoked certificates shall not pass!

(Interestingly, if you don’t provide the -use_ocsp flag, our client app won’t check if the certificate is revoked and will perform a connection to a server. Check out the Readme for details.)

Now, let’s try TLS certificate verification using CRL.

Let’s run a TLS server that uses a valid and non-revoked certificate:

go run ./cmd/tls-server -key cert1/cert1.key.pem -cert cert1/cert1.crt.pem 

Accepting connection...

The client-side application connects to the server, receives a TLS certificate, checks the local cache for the CRL file, and searches if the certificate serial ID is present there. As the server uses a non-revoked certificate, the client app continues the connection:

go run ./cmd/tls-client -ca_cert ca/ca.crt.pem -crl_file ca/crl.pem

Server certificate: Test leaf certificate (cert1)
Server certificate issuer: Test CA certificate
Validating server certificate with CRL (serial: 547114458127197346809836113712612368226057720033)
[*] Reading ca/crl.pem
[*] Parsing ca/crl.pem
[*] Checking CRL signature
[*] Checking CRL validity
[*] Searching for our certificate
[*] Revoked certificate serial: 113274642861313425704666455845030198894915511470
[+] Did not find validated certificate among revoked ones
Server certificate was allowed
Successfully connected to server

Nice, right?

Feel free to review the implementation, use it in your Go apps, or create pull requests.

We work with companies on demanding markets.
Read how we use Acra to protect data in critical infrastructure.

Critical infrastructure

Industrial

Protecting telemetry data of power grids

Protecting data signals transmitted over the air between power distribution stations and central dispatch system.

Read story

5. How using OCSP and CRL could go wrong #

Let’s talk about how things could go wrong in using OCSP and CRL.

Please refer to OWASP Transport Layer Protection Cheat Sheet to learn the best practices configuring TLS in general.

Establish session before validating the TLS certificate #

Both OCSP and CRL can cause visible delays in the application work, so, developers might be tempted to “optimise” things. The application might perform TLS revocation checks in the background while it continues communication with a potentially untrusted service.

From a security perspective, until the validation is fully finished (the OCSP / CRL response is downloaded, parsed, verified and a decision is made), the service-in-question is considered untrusted.

Only after performing all validation steps, the app should establish a TLS connection with the service. Developers should consider this delay as a part of a standard application flow and indicate that the application “is loading” to the users / other services.

Wrong sequence #

The sequence of the “checking local CRL cache”, “downloading new CRL”, and “checking OCSP response” matters.

The fastest way is to check the local CRL cache, but then the application won’t know if the certificate was revoked lately. The sequence of checks depends on the desired security and performance properties of the solution.

Unreachable CRL / OCSP responders #

The application tries to update the CRL or receive the OCSP response, but the responder servers are unreachable.

This could destabilise the whole solution, as applications can be stuck in limbo, not communicating with other services while validating the certificates. The solution design and the application code should handle this situation—using an exponential backoff for network requests or failing after N attempts.

Certificate revocation checks are too rare #

How often should the application check if the certificate was revoked: on every connection or once-in-a-while? It depends on a threat model and security requirements.

When talking to security-sensitive services, applications should validate certificates before every connection. In contrast, others might rely on cache’s TTL. For example, CRL cache TTL might differ from CRL’s “next update” date; thus, the application checks for a new CRL more often.

That’s a performance <> security tradeoff.

Lack of signature validation #

The CRL response is signed, and it’s essential to verify the signature to prevent potential attackers from tampering with the list.

See the signature validation in our example or refer to Go crypto/x509 docs.

Accepting “Unknown” status as valid #

The OCSP response could have “Good”, “Revoked”, or “Unknown” status for the certificate.

If the application treats “Unknown” status as “the certificate is still valid”, it opens a threat vector. If the OCSP infrastructure is under attack, the application will continue talking to the already malicious service, treating it as valid. Developers should be aware and treat “Unknown” status as “Revoked”, or re-send the OCSP request later.

CRL cache poisoning #

From the attacker’s perspective, poisoning (tampering with) the CRL cache is an “easy hack”.

Suppose CRL is cached in a local file without any integrity checks. In that case, the application won’t recognise that the file was changed.

If a CRL comes over the network, the DNS cache poisoning could result in the application talking to the malicious CRL responder and receiving malicious CRLs.

Delta CRLs #

Delta CRLs (downloading only CRL changes instead of the whole file every time) help speed up things.

But if implemented poorly, issues with network connectivity might result in the application not receiving some delta CRLs and continuing to trust the revoked certificates.

6. Conclusions #

Even though the standard Golang library provides easy-to-use and secure TLS 1.3, it doesn’t support OCSP and CRL protocol out-of-the-box. This discussion on GitHub may give you hints about how OCSP might be integrated into the standard library in future.

OCSP and CRL support is a must-have for applications that process sensitive data. OCSP / CRL are the small bits, but they’re the connection tissue in the trust fabric of PKI in distributed applications.

PKI shouldn’t be considered reliable and secure if the applications don’t have an automated way to validate revoked certificates.

OCSP and CRL help the application check whether the certificate was revoked by CA, preventing applications from connecting to potentially malicious or misconfigured servers. Take a look at the OCSP and CRL implementations in Golang we’ve built for this post using only standard modules.

Implementing OCSP and CRL in Go app takes no more than 100 LOCs for each, but the devil is in the details. Even with OCSP / CRL support, your solution might be susceptible to design and implementation mistakes, opening a wide attack surface.

To get a deeper understanding of other certificate verification procedures (not mentioned in this post), take a look at Acra code (network/ocsp.go and tls-client/tls-client.go). As Acra protects sensitive data, it uses strict TLS settings to provide strong security guarantees.

Who knows how many exciting findings wait for you deep inside all those TLS-related RFCs? Use encryption where possible, prefer ciphers with PFS, and may the force be with you.