Adding Certificates to Ubuntu and GitLab

Introduction

I recently set up GitLab and Mattermost on an Ubuntu 14.04 server at work using the GitLab Community Edition Omnibus installer. Overall the process was incredibly easy and I highly recommend these products. However, I ran into some hiccups when I tried to run the applications under SSL. Since it took me a little while to figure out how to work through the issues, I thought it might be helpful to document what I did to get it working properly.

Table of Contents

  1. Request an SSL Certificate
  2. Install the SSL Certificate
  3. Configure GitLab to Use HTTPS
  4. Problem: Token Errors When Authenticating to GitLab
  5. Solution Part 1: Importing the Certificate Authority’s Signing Certificate
  6. Solution Part 2: Telling GitLab to Use Ubuntu’s CA Certificate File

Request an SSL Certificate

The first step is to request an SSL certificate for the server. Because GitLab and Mattermost run on separate virtual hosts with their own fully qualified domain names (FQDNs), I requested two certificates. To do this I ran the openssl “req” command.

openssl req -new -newkey rsa:2048 -nodes -out gitlab.example.com.csr -keyout gitlab.example.com.key -subj "/C=US/ST=Texas/L=Austin/O=Example Company/OU=IT/CN=gitlab.example.com"

This produces two files:

  1. A private key with the extension .key
  2. A certificate signing request (CSR) with the extension .csr

Certificate Request Generation

To get an SSL certificate, you give the CSR file to one of the many Certificate Authorities (CAs) along with a fee for issuing you a new certificate. They return to you a certificate signed by their CA Signing (or Root) certificate. Their root certificate is signed by their private key so that it can be independently verified. The file you get back should have an extension of either .crt or .pem (they mean the same thing).

Install the SSL Certificate

Once you have received the signed certificate from the certificate authority, you need to install this certificate in a place where GitLab can use it. This is effectively the same process for installing certificate in Apache, Nginx, or another web server. To do this you need both the certificate file and the private key that was generated when you generated the certificate signing request.

Place both of these files into /etc/gitlab/ssl/:

sudo cp *.crt *.key /etc/gitlab/ssl/

Configure GitLab to Use HTTPS

Next we need to configure GitLab to use HTTPS.

To do this, you’ll need to edit /etc/gitlab/gitlab.rb using your favorite text editor (vi, emacs, nano, gedit, etc.). You’ll need to do this as root, so make sure to include the “sudo” command.  For example:

sudo vi /etc/gitlab/gitlab.rb

First, change the external_url value to include https:

external_url 'https://gitlab.example.com'

Next, tell the Nginx web server to redirect http requests to https:

nginx['redirect_http_to_https'] = true

Now we do the same thing for Mattermost:

mattermost_external_url 'https://mattermost.example.com'
...
mattermost_nginx['redirect_http_to_https'] = true

Finally, we tell Mattermost to use https when authenticating with GitLab:

mattermost['gitlab_auth_endpoint'] = "https://gitlab.example.com/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://gitlab.example.com/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://gitlab.example.com/api/v3/user"

We don’t need to specifically reference the certificate files we installed as long as their file names match the FQDNs of the servers.

Finally, we reconfigure GitLab:

sudo gitlab-ctl reconfigure

Now both GitLab and Mattermost will be using SSL. The last step is to configure GitHub’s authentication system to allow requests from Mattermost over HTTPs. This is done in the Admin section (GitLab Admin Icon) of GitLab under the Applications tab.
GitLab Applications Tab GitLab Edit Application

Problem: Token Errors When Authenticating to GitLab

Token Request Failed

By looking in the Mattermost logs, I saw that the problem was related to the SSL certificate:

sudo tail -f /var/log/gitlab/mattermost/mattermost.log

[2016/02/05 11:15:01 CST] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=1zh9orjzcfb1tfsmjtj7rekoya uid= ip=192.168.122.160 Token request failed [details: Post https://gitlab.example.com/oauth/token: x509: certificate signed by unknown authority]

The problem is that the GitLab Omnibus comes with its on “cacerts.pem” file, which contains the list of trusted CA root certificates. It probably does this so that it can easily run on a number of different Linux distributions without worrying about the location of those system’s cacerts file.

Solution Part 1: Importing the Certificate Authority’s Signing Certificate

The first step in solving this problem is to import the signing certificate for the certificate authority into the system’s list of trusted certificates.

To do this, we first need to know what root certificate was used to sign our certificates. We can discover this by examining the certificate with OpenSSL:

openssl x509 -in gitlab.example.com.crt -text -noout

We can look through the output of this command to find the signing certificate’s URL:

Authority Information Access: 
    OCSP - URI:http://ocsp.entrust.net
    CA Issuers - URI:http://aia.entrust.net/l1k-chain256.cer

We can then use wget (or curl) to download that certificate:

wget http://aia.entrust.net/l1k-chain256.cer

Then we use OpenSSL to convert it to PEM format:

openssl x509 -inform der -in l1k-chain256.cer -out l1k-chain256.pem

Next we need to place the PEM in /usr/local/share/ca-certificates/:

sudo cp l1k-chain256.pem /usr/local/share/ca-certificates/

Finally, we run the update-ca-certificates utility, which adds the new certificate to the system’s list of trusted certificates.

sudo update-ca-certificates

The output should look something like this:

Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....
done.
done.

Solution Part 2: Telling GitLab to Use Ubuntu’s CA Certificate File

The final piece of the puzzle is to tell GitLab to use the system’s trusted certificate file instead of its own. This is done by removing /opt/gitlab/embedded/ssl/certs/cacert.pem and replacing it with a symlink to /etc/ssl/certs/ca-certificates.crt.

sudo rm /opt/gitlab/embedded/ssl/certs/cacert.pem
sudo ln -s /etc/ssl/certs/ca-certificates.crt /opt/gitlab/embedded/ssl/certs/cacert.pem
sudo gitlab-ctl reconfigure

That’s it! You should now be able to run both GitLab and Mattermost over HTTPS without an issue.

4 thoughts on “Adding Certificates to Ubuntu and GitLab

  1. 1. Developer enforced https to gitlab and ran into an issue with connecting via git client since it was using an internal cert
    2. Fixed this by exporting the thqptslicense01 root cert from the certmgr.msc program to a pem file and configured git to accept those certs with code he wrote
    Question: should the windows admins have the pem file pushed out to all desktops to a specific directory or distribute to all devs to install manually?

  2. Sorry Fatima, I didn’t see your comment until just now. I would suggest having only developers install it, just to limit possible exposure, but both are workable solutions.

  3. I have this same issue, but wanted to document how I solved this issue since this is one of the top google search results regarding the `x509: certificate signed by unknown authority` issue.

    I read through all the other links / issues mentioned here and did run across a couple others

    * http://vaelen.org/2016/02/05/adding-certificates-to-ubuntu-and-gitlab/

    But ultimately, I would recommend reading the following blog post that describes why SSL certs are different on different systems and why this causes so much headache:

    * https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/

    In my specific use case, we run a RHEL / Centos 6 server.

    While I had already added my Internal CA root certificate to the `/etc/pki/ca-trust/source/anchors/` directory and then ran the `update-ca-trust` as root. This updates certificates in the `/etc/pki/ca-trust/extracted` directory.

    After updating this and forcing the GitLab CA certificate to point to this extracted certificate, I still could not get Mattermost to connect properly:

    “`
    [root@server /]# ls -la /opt/gitlab/embedded/ssl/certs/
    total 8
    drwxr-xr-x. 2 root root 4096 Jul 21 13:19 .
    drwxr-xr-x. 4 root root 4096 Jul 21 12:47 ..
    lrwxrwxrwx. 1 root root 55 Jul 21 13:02 cacert.pem -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
    “`

    It wasn’t until I saw that there was another certs directory from the HappyAssassin blog post that was not being updated. The `/etc/ssl/certs` is a sym link to `/etc/pki/tls/certs` and in here these certificates did not have my internal root CA certificate.

    So I removed these and pointed this these to use certificates from the extracted directory:

    “`
    [root@server /]# ls -la /etc/ssl/
    total 16
    drwxr-xr-x. 2 root root 4096 Jul 21 08:30 .
    drwxr-xr-x. 122 root root 12288 Jul 21 13:29 ..
    lrwxrwxrwx. 1 root root 16 Jul 21 08:30 certs -> ../pki/tls/cert
    [root@server /]# ls -la /etc/ssl/certs/
    total 20
    drwxr-xr-x. 2 root root 4096 Jul 21 13:27 .
    drwxr-xr-x. 5 root root 4096 Jul 21 08:30 ..
    lrwxrwxrwx. 1 root root 49 Jul 21 13:27 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
    lrwxrwxrwx. 1 root root 55 Jul 21 13:27 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
    -rwxr-xr-x. 1 root root 610 May 9 08:32 make-dummy-cert
    -rw-r–r–. 1 root root 2242 May 9 08:32 Makefile
    -rwxr-xr-x. 1 root root 829 May 9 08:32 renew-dummy-cert
    “`

    It wasn’t until I updated these then did one more gitlab reconfigure that Mattermost would finally recognize the certificates from GitLab.

    I’m not sure if this is something that is encoded into Mattermost or possibly one of its dependencies, but this was pretty difficult to track down especially since every system does SSL certificates differently and applications/software all look for these in different ways.

    Hopefully this helps someone else or at least provides them with background on the SSL issues since that blog post really helped me figure out the core issue here.

  4. Thank you very much. In my case I had to use a Windows based CA which required me to get the complete chain from the CA for your method to work. Maybe this is a hint for other users like me.

Leave a Reply

Your email address will not be published. Required fields are marked *