I recently set up GitLab and Mattermost on an Ubuntu 14.04 server at work using the GitLab Community Edition Omnibus installer. Overall the process was incredibly easy and I highly recommend these products. However, I ran into some hiccups when I tried to run the applications under SSL. Since it took me a little while to figure out how to work through the issues, I thought it might be helpful to document what I did to get it working properly.
Table of Contents
- Request an SSL Certificate
- Install the SSL Certificate
- Configure GitLab to Use HTTPS
- Problem: Token Errors When Authenticating to GitLab
- Solution Part 1: Importing the Certificate Authority’s Signing Certificate
- Solution Part 2: Telling GitLab to Use Ubuntu’s CA Certificate File
The first step is to request an SSL certificate for the server. Because GitLab and Mattermost run on separate virtual hosts with their own fully qualified domain names (FQDNs), I requested two certificates. To do this I ran the openssl “req” command.
openssl req -new -newkey rsa:2048 -nodes -out gitlab.example.com.csr -keyout gitlab.example.com.key -subj "/C=US/ST=Texas/L=Austin/O=Example Company/OU=IT/CN=gitlab.example.com"
This produces two files:
- A private key with the extension .key
- A certificate signing request (CSR) with the extension .csr
To get an SSL certificate, you give the CSR file to one of the many Certificate Authorities (CAs) along with a fee for issuing you a new certificate. They return to you a certificate signed by their CA Signing (or Root) certificate. Their root certificate is signed by their private key so that it can be independently verified. The file you get back should have an extension of either .crt or .pem (they mean the same thing).
Once you have received the signed certificate from the certificate authority, you need to install this certificate in a place where GitLab can use it. This is effectively the same process for installing certificate in Apache, Nginx, or another web server. To do this you need both the certificate file and the private key that was generated when you generated the certificate signing request.
Place both of these files into /etc/gitlab/ssl/:
sudo cp *.crt *.key /etc/gitlab/ssl/
Next we need to configure GitLab to use HTTPS.
To do this, you’ll need to edit /etc/gitlab/gitlab.rb using your favorite text editor (vi, emacs, nano, gedit, etc.). You’ll need to do this as root, so make sure to include the “sudo” command. For example:
sudo vi /etc/gitlab/gitlab.rb
First, change the external_url value to include https:
Next, tell the Nginx web server to redirect http requests to https:
nginx['redirect_http_to_https'] = true
Now we do the same thing for Mattermost:
mattermost_external_url 'https://mattermost.example.com' ... mattermost_nginx['redirect_http_to_https'] = true
Finally, we tell Mattermost to use https when authenticating with GitLab:
mattermost['gitlab_auth_endpoint'] = "https://gitlab.example.com/oauth/authorize" mattermost['gitlab_token_endpoint'] = "https://gitlab.example.com/oauth/token" mattermost['gitlab_user_api_endpoint'] = "https://gitlab.example.com/api/v3/user"
We don’t need to specifically reference the certificate files we installed as long as their file names match the FQDNs of the servers.
Finally, we reconfigure GitLab:
sudo gitlab-ctl reconfigure
Now both GitLab and Mattermost will be using SSL. The last step is to configure GitHub’s authentication system to allow requests from Mattermost over HTTPs. This is done in the Admin section () of GitLab under the Applications tab.
By looking in the Mattermost logs, I saw that the problem was related to the SSL certificate:
sudo tail -f /var/log/gitlab/mattermost/mattermost.log [2016/02/05 11:15:01 CST] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=1zh9orjzcfb1tfsmjtj7rekoya uid= ip=192.168.122.160 Token request failed [details: Post https://gitlab.example.com/oauth/token: x509: certificate signed by unknown authority]
The problem is that the GitLab Omnibus comes with its on “cacerts.pem” file, which contains the list of trusted CA root certificates. It probably does this so that it can easily run on a number of different Linux distributions without worrying about the location of those system’s cacerts file.
The first step in solving this problem is to import the signing certificate for the certificate authority into the system’s list of trusted certificates.
To do this, we first need to know what root certificate was used to sign our certificates. We can discover this by examining the certificate with OpenSSL:
openssl x509 -in gitlab.example.com.crt -text -noout
We can look through the output of this command to find the signing certificate’s URL:
Authority Information Access: OCSP - URI:http://ocsp.entrust.net CA Issuers - URI:http://aia.entrust.net/l1k-chain256.cer
We can then use wget (or curl) to download that certificate:
Then we use OpenSSL to convert it to PEM format:
openssl x509 -inform der -in l1k-chain256.cer -out l1k-chain256.pem
Next we need to place the PEM in /usr/local/share/ca-certificates/:
sudo cp l1k-chain256.pem /usr/local/share/ca-certificates/
Finally, we run the update-ca-certificates utility, which adds the new certificate to the system’s list of trusted certificates.
The output should look something like this:
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d.... done. done.
The final piece of the puzzle is to tell GitLab to use the system’s trusted certificate file instead of its own. This is done by removing /opt/gitlab/embedded/ssl/certs/cacert.pem and replacing it with a symlink to /etc/ssl/certs/ca-certificates.crt.
sudo rm /opt/gitlab/embedded/ssl/certs/cacert.pem sudo ln -s /etc/ssl/certs/ca-certificates.crt /opt/gitlab/embedded/ssl/certs/cacert.pem sudo gitlab-ctl reconfigure
That’s it! You should now be able to run both GitLab and Mattermost over HTTPS without an issue.