The Journal.

A technology blog run by Rowden team members

The unexpected journey of deploying a SonarQube Azure App Service

When starting a new project, it’s important to get your development workflow right from the outset.

When starting a new project, it’s important to get your development workflow right from the outset. Code smells, style issues, pitfalls and lack of test coverage are way more difficult to fix in a 100KLOC project than in a fresh one, after all, and there’s likely no better tool for ensuring code quality than SonarQube: it supports a whole swathe of languages, is very powerful and flexible, and best of all it’s free! (Well, the Community Edition at least, but there’s no size limits and you get almost the same feature set as the paid versions.)

The easiest way to get started with SonarQube is to use its companion cloud service, SonarCloud. While you can download SonarQube CE for free and install it on the local machine or VM of your choice, there’s little point in going to all that trouble when for just 10€/month (at time of writing) you can have a robust, high-availability, scalable solution all managed for you… right?

Well, unfortunately for some companies (we at Rowden among them), there are particular requirements to fulfill when it comes to security and traceability, and allowing Random Inc. to access all of one’s code and store it in a datacentre… somewhere, just won’t cut it. (Not to say that SonarCloud’s security isn’t stellar, it’s just a matter of compliance — for example, to ensure everything is stored in the same UK-based location.)

Enter the Azure Web App . It’s basically a managed web server that lets you run your own code, bringing with it a number of advantages over a traditional VM:

  • Fully containerised — Start, stop, and scale your instances without worry
  • CD support — Rebuild and deploy your app within seconds of pushing to a git repo
  • Managed web frontend — HTTPS, firewalls, and load balancing are all handled for you
  • High-availability — The container will restart if your app quits or stops responding, with diagnostics tools to help figure out the problem
  • Built-in management — Web SSH, file browsing, log streaming and more are available
  • Authentication — Control access to the app with AD, Google, Facebook etc.
  • Supports a variety of tech stacks, from .NET to PHP

(This isn’t to say you can’t set up a normal VM with all this stuff, but it would be a real pain, especially without a full-time devops engineer.)

There are, however, a couple of downsides to consider:

  • Limited configuration support — since you’re running in a container, some system-level settings are read-only (you do otherwise get root access, though)
  • Web services only — You can only serve HTTP over port 80 (with SSL on 443 managed for you). If you want to run some other TCP/UDP service, or you need to use a different HTTP port, you’re out of luck
  • Miscellaneous bugs/limitations — Since you’re not running on bare metal, there’s a few things that don’t work correctly, or have strange behaviour. I get the impression one only runs into this when trying to push the boundaries of the service (as I’m about to describe) but it’s worth bearing in mind that an App Service is not a VM, and you should try to work with that fact rather than against it

The plan

In this guide, we’ll be making an Azure App Service linked to an SQL Server database (for storing rules, project definitions and so on) and an Azure DevOps repository, which will contain a script to download and start up a SonarQube instance. This allows us to upgrade our SonarQube version or add any additional configuration quite seamlessly, with all the safety and tracability of a version control system.

This works because, while the App Service is mainly designed for running custom code, if the tech stack is supported (SonarQube runs on Java, so this is true in our case) there’s nothing to stop you downloading and running third-party binaries. And while there is support for auto-detecting the layout of a built app, you can also specify a command or a shell script to run instead to ensure everything’s set up how you want it.

Things that don’t work

The first thing you might be tempted to try is to slap a Docker image on a ‘Web App for Containers’. There’s a couple of options available here:

  • SonarQube Certified by Bitnami — Works, but only supports PostgreSQL databases, which is fine but these are much more expensive and less flexible in Azure than the standard SQL Server. But if you’ve got one already provisioned then go ahead and skip the rest of this guide 🙂
  • Official SonarQube images (using ‘Docker Hub’ with the ‘sonarqube’ image) — These don’t work due to limitations of the App Service, and can’t be guaranteed to work in future for reasons that will be explained shortly…

Furthermore, custom Docker images (essentially) only work on Linux, they’re a lot slower to start up, and they don’t necessarily have WebSSH integrated.

Resources you’ll need

Here’s all the Azure resources you’ll need to create to get a SonarQube setup working:

  • App Service Plan — Linux B2 or greater
  • SQL Server + Database — Basic (5 DTU) with 2GB storage is fine
  • Web App — using the aforementioned App Service Plan, running Code with the Java 11 SE stack

This guide will focus on Linux, since it’s cheaper and SonarQube runs well on it, but similar instructions can be followed for Windows as well. In addition if you’re really strapped for cash you can try using a B1 instance, but you’ll need to (manually) add more swap space or it’ll run out of memory and auto-reboot… and it’ll be really quite slow to run.

Feel free to use any supported database, but if you’re starting from scratch then the basic SQL Server resource is cheap, runs well, and can be scaled up in future.

As of version 7.9, SonarQube’s support for Java 8 has been dropped, so you’ll need to pick one of the Java 11 runtimes. I’ve gone with Java SE, but TomCat should work as well.

Setting up the database

When you first create the database, make sure you’re using a case-sensitive collation! The default, under Additional Settings, will be SQL_Latin1_General_CP1_CI_AS – change this to SQL_Latin1_General_CP1_CS_AS.

First, allow your DB server to be accessed from your client IP (for configuration) and from Azure services (for running it). Feel free to delete the former once you’re done.

Log in to your server’s ‘master’ database using SSMS, and run CREATE LOGIN SonarQube WITH PASSWORD = 'hunter2'; (you can use any username here). The password doesn’t in theory need to be secure since you’re restricting access to Azure services only, but it’s always good to be careful.

Then log in to the database you created (or open the Query Editor in Azure Portal) and run:

ALTER DATABASE YourSonarQubeDatabase SET READ_COMMITTED_SNAPSHOT ON WITH ROLLBACK IMMEDIATE;
CREATE USER SonarQube FOR LOGIN SonarQube WITH DEFAULT_SCHEMA = dbo; 
ALTER ROLE db_owner ADD MEMBER SonarQube;

Configuration

Web Apps have a nifty configuration system that allows passing environment variables in a secure manner, rather than setting them in code. We’ll need the following variables set:

  • SONARQUBE_JDBC_URL = jdbc:sqlserver://ServerName.database.windows.net;databaseName=YourSonarQubeDatabase
  • SONARQUBE_JDBC_USERNAME = SonarQube
  • SONARQUBE_JDBC_PASSWORD = hunter2
  • WEBSITES_PORT = 80

Note that the variable names can only contain letters, numbers and undersores. Dots, for example, are not allowed and will be silently converted to underscores. Dot-separated names are very commonly used in the Docker ecosystem, and the snake-case names shown here are deprecated in the official SonarQube images.

WEBSITES_PORT is a special variable that indicates to the service manager what port to bind the frontend to, and can in theory be any value. However at the time of writing this setting appears to be ignored, and thus only port 80 works correctly. YMMV.

You’ll also want to enable App Service Logs to make debugging easier.

Creating the scripts

Create a new repository in your Azure DevOps project; this is where we’ll put the scripts that will download and start the SonarQube server.

You could create PowerShell variants of this for Windows, on which you won’t need to do the ElasticSearch workaround since the setting it bypasses doesn’t exist

#!/bin/sh
# startup.sh
SONAR_VERSION=7.9
SONARQUBE_HOME=/opt/sonarqube
# Download SonarQube and put it into an ephemeral folder
wget -O /tmp/sonarqube.zip https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-$SONAR_VERSION.zip
mkdir /opt
unzip /tmp/sonarqube.zip -d /opt/
mv /opt/sonarqube-$SONAR_VERSION /opt/sonarqube
chmod 0777 -R $SONARQUBE_HOME
# Workaround for ElasticSearch
adduser -DH elasticsearch
echo "su - elasticsearch -c '/bin/sh /home/site/wwwroot/elasticsearch.sh'" > /opt/sonarqube/elasticsearch/bin/elasticsearch
# Install any plugins
cd $SONARQUBE_HOME/extensions/plugins
wget https://github.com/hkamel/sonar-auth-aad/releases/download/1.1/sonar-auth-aad-plugin-1.1.jar
# Start the server
cd $SONARQUBE_HOME
exec java -jar lib/sonar-application-$SONAR_VERSION.jar \
  -Dsonar.log.console=true \
  -Dsonar.jdbc.username="$SONARQUBE_JDBC_USERNAME" \
  -Dsonar.jdbc.password="$SONARQUBE_JDBC_PASSWORD" \
  -Dsonar.jdbc.url="$SONARQUBE_JDBC_URL" \
  -Dsonar.web.port="$WEBSITES_PORT" \
  -Dsonar.web.javaAdditionalOpts="$SONARQUBE_WEB_JVM_OPTS -Djava.security.egd=file:/dev/./urandom"

#!/bin/sh
# elasticsearch.sh
# Use the configuration file SonarQube provides, but keep everything else at the default
cp /opt/sonarqube/temp/conf/es/elasticsearch.yml /opt/sonarqube/elasticsearch/config
# Run the ElasticSearch node (without forcing the bootstrap checks)
exec java \
-XX:+UseConcMarkSweepGC \
-XX:CMSInitiatingOccupancyFraction=75 \
-XX:+UseCMSInitiatingOccupancyOnly \
-Des.networkaddress.cache.ttl=60 \
-Des.networkaddress.cache.negative.ttl=10 \
-XX:+AlwaysPreTouch \
-Xss1m \
-Djava.awt.headless=true \
-Dfile.encoding=UTF-8 \
-Djna.nosys=true \
-XX:-OmitStackTraceInFastThrow \
-Dio.netty.noUnsafe=true \
-Dio.netty.noKeySetOptimization=true \
-Dio.netty.recycler.maxCapacityPerThread=0 \
-Dlog4j.shutdownHookEnabled=false \
-Dlog4j2.disable.jmx=true \
-Djava.io.tmpdir=/opt/sonarqube/temp \
-XX:ErrorFile=../logs/es_hs_err_pid%p.log \
-Xms512m \
-Xmx512m \
-XX:+HeapDumpOnOutOfMemoryError \
-Des.path.home=/opt/sonarqube/elasticsearch \
-Des.path.conf=/opt/sonarqube/elasticsearch/config \
-Des.distribution.flavor=default \
-Des.distribution.type=tar \
-cp '/opt/sonarqube/elasticsearch/lib/*' \
org.elasticsearch.bootstrap.Elasticsearch

The main reason for this bit of chicanery with ElasticSearch is to avoid its bootstrap checks crashing the server. The one in particular we want to avoid is vm.max_map_count, which ought to be set to a much higher value than the App Service has by default (this is only a problem if you’re running a multi-node cluster), but since that part of the filesystem is read-only there’s not much else we can do about it other than overwriting the script with one that doesn’t force these checks. This is the main reason why the official Docker image doesn’t work. All the arcane-looking java options are just the ones SQ passes into ES by default.

You may also notice that we’re running ES in its own user, but running SQ as root. In truth we’d be perfectly fine to run the whole thing as root — it’s an isolated, containerised instance after all — but ElasticSearch will complain profusely if you try this, so we’ll indulge it rather than setting es.insecure.allow.root=true.

When you commit these files, make sure you’re using LF/Unix line endings or the App’s terminal won’t like your scripts!

If you want to install or update plugins, you’ll need to do it in startup.sh, otherwise they’ll be cleared out whenever your app restarts.

Continuous Deployment

Go to the Deployment Centre on your App, and follow the instructions for Azure DevOps, picking the repository you just created.

  • No need to define any build tasks; instead set the current working directory (./) as the archive root to zip up all the scripts.
  • Set the ‘Startup File’ to /home/site/wwwroot/startup.sh
  • Set the tech stack to Java 11 SE (NB at time of writing the tech stack list is a little glitched — just pick the first ‘Java SE’ from the top. You may also find it not present at all on the initial creation — in that case just pick anything, and change it later by editing the release pipeline)

With a bit of luck, you should see the ‘Starting up’ page within a few minutes, and everything working in around 10. If something’s gone wrong, or if you’d just like to see what it’s doing, head over to the Log Streaming Service. The one to look out for is ‘SonarQube up’. Possible causes of errors might be an incorrect DB configuration, permissions errors, or lack of disk space/memory.

When it’s time to upgrade, just change the value of SONAR_VERSION and push! (I’ve used this successfully to move from 7.8 to 7.9)

Authentication with Active Directory

If you’re already using Azure, then you’ll probably want to link in your AD infrastructure to provide a Single Sign-On experience. The author of the AAD SSO plugin (the one we already included) has made a great guide to setting it all up, check it out: https://github.com/hkamel/sonar-auth-aad/wiki/Setup

Remember to force authentication in the Security tab, and change the admin password to something strong!

Adding your projects

Now for the fun part, linking in your projects! There’s a SonarQube extension for Azure DevOps that makes this really easy. Once it’s set up, add the necessary steps to your CI pipeline from the toolbar. Here’s what ours looks like for a .NET Core project using xUnit and Coverlet:

steps:
- task: SonarQubePrepare@4
  inputs:
    SonarQube: 'My_Sonar_Connection'
    scannerMode: 'MSBuild'
    projectKey: 'My_Project_Key'
    extraProperties: >
      sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)\coverage.opencover.xml
- task: DotNetCoreCLI@2
  inputs:
    command: 'build'
- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    arguments: >
      /p:CollectCoverage=true
      /p:CoverletOutputFormat=opencover%2ccobertura
      /p:CoverletOutput=..\
- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: 'coverage.cobertura.xml'
- task: SonarQubeAnalyze@4

There’s a few things to note here:

  • Make sure you pass in an absolute path to the reportsPaths property
  • Some characters, like commas, require escaping to work in MSBuild: opencover,cobertura becomes opencover%2ccobertura
  • Azure’s built-in coverage tab supports the JaCoCo and Cobertura formats, regardless of language
  • SonarQube supports particular formats for coverage reports depending on the project language. In the case of C# these are Opencover and dotCover (JaCoCo is only supported for Java)
  • Hence, Coverlet is our tool of choice for this use case as it supports a number of different output formats, and allows writing out multiple copies in the same task.

All going well, you should see your SonarQube dashboard update with the latest analysis!