Sonar code analysis automation with bitbucket pipelines

Overview

Developers team often experiencing troubles with code smells and some of them are not well visible during code review and can be merged with target branch. That’s why we provide automated mechanism of code analysis which eliminates human factor regarding code smells. For this purposes we will use AWS remote instance, Bitbucket Pipelines and Sonarqube as static code analysis tool.

Advantages

Main advantage to use remote AWS instance for Sonar code analysis automation with bitbucket pipelines is that we don't need to use few clouds. For example if we used Sonarcloud and Bitbucket Cloud Pipelines for our task. Also, AWS is one of the common use remote engine, so more people will be able to use our solution without struggle.

Tools

Hybris, AWS, Docker
Shell Script, Curl
Bitbucket, Bitbucket Pipelines
Sonarqube with Branch plugin and Jacoco

Schema description

We configure our AWS remote instance as custom bitbucket pipelines runner and host for sonar. Our custom hybris image acts as base image in pipeline. During process of pipeline execution code is pulled from repo and then it is built and tests are executed to get coverage report. Then binaries and jacoco coverage report are analysed with sonarqube which is always up and running so that you can always have access to code analysis results. Results of sonarqube analysis is processed with sonarqube quality gate. If quality gate conditions are not satisfied, pipeline execution fails. If pipeline execution failed, then developer should go to sonarqube instance and see his code analysis results. There he can see code smells in exact places and he should fix them to satisfy quality gate conditions.

Steps to implement above mentioned schema:

AWS instance creation and configuration

Create EC2 instance
Select instance with needed performance parameters. Create ssh key pair to connect to instance via terminal
Add inbound rule to allow access to sonarqube on http://<your-instance-public-ip>:9000.You can also specify ip-addresses, from which this port can be reached

Connect to your running instance with downloaded ssh key:

ssh -i /path/key-pair-name.pem instance-user-name@instance-public-dns-name

Setup docker on your instance:

sudo apt update
sudo apt install docker.io
sudo groupadd docker
sudo usermod -aG docker ${USER}

Configure runner on EC2 instance:
- Go to your Bitbucket repo settings:
- Then to Runners:
- Click Add runner:
- Specify runner’s name and add custom label if needed:
- Run this command on your EC2 terminal, but change attribute ‘-it' on '-itd’ :

Run docker container with sonarqube_with_branch_plugin:

docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 mc1arke/sonarqube-with-community-branch-plugin

Configure Sonarqube:
- Create project and project key
- Generate token(global or project specific) - to connect to sonar host server
- Upload quality profile from $HYBRIS_HOME/build-tools/sonarqube/java-hybris-profile.xml and set it as default for java
- Use default quality gate or create one with preferred conditions and set it as defaultIf you prefer configuration over code, take a look at our sonar setup using curl and sonar web_api:

curl -u <CREDENTIALS> -X POST <SONAR_HOST_URL>/api/projects/create?name=<PROJECT_NAME>\&project=<PROJECT_KEY>
curl -u <CREDENTIALS> -X POST <SONAR_HOST_URL>/api/user_tokens/revoke?name=<TOKEN_NAME>
curl -u <CREDENTIALS> -X POST <SONAR_HOST_URL>/api/user_tokens/generate?name=<TOKEN_NAME>\&project_key=<PROJECT_KEY> | awk 'match($0,/"token":"([a-z_0-9]*)["]/) { print substr($0,RSTART+9,RLENGTH-10)}' > <CUSTOM_PATH>/token.txt
curl -u <CREDENTIALS> -X POST -F 'backup=<HYBRIS_HOME>/build-tools/sonarqube/java-hybris-profile-public.xml' <SONAR_HOST_URL>/api/qualityprofiles/restore
curl -u <CREDENTIALS> -X POST <SONAR_HOST_URL>/api/qualityprofiles/set_default?language=java\&qualityProfile=hybris+Official+Profile+1

# below we copy default quality gate and remove code coverage condition then set it as default (use if code coverage analysis is not needed)
curl -u <CREDENTIALS> -X POST <SONAR_HOST_URL>/api/qualitygates/copy?name=<CUSTOM_QUALITY_GATE_NAME>\&sourceName=Sonar+way
curl -u <CREDENTIALS> -X GET <SONAR_HOST_URL>/api/qualitygates/show?name=<CUSTOM_QUALITY_GATE_NAME> | awk 'match($0,/"id":"([A-z-0-9]*)","metric":"new_coverage"/) { print substr($0,RSTART+6,RLENGTH-31)}' > <CUSTOM_PATH>/cond_uuid.txt
curl -u <CREDENTIALS> -X POST <SONAR_HOST_URL>/api/qualitygates/delete_condition?id=$(cat <CUSTOM_PATH>/cond_uuid.txt)
curl -u <CREDENTIALS> -X POST <SONAR_HOST_URL>/api/qualitygates/set_as_default?name=<CUSTOM_QUALITY_GATE_NAME>

Example:

curl -u admin:admin -X POST http://172.17.0.1:9000/api/projects/create?name=Hybris\&project=hybris

Hybris docker image creation

Download hybris of desired version and unzip it in custom dir.

Remove sonar.branch.name property from:

<LOCAL_HYBRIS_FOLDER>/hybris/bin/platform/resources/ant/sonar.xml

It’s mandatory for pull-request analysis! But if you want to analyse on commits, you don’t need to remove it.

Download jacoco plugin, unzip it in custom dir.

Add this option to local.properties if you want to see code coverage:

standalone.javaoptions=-javaagent:apache-ant/lib/jacocoagent.jar=destfile=../../log/jacoco/jacoco.exec,append=true,excludes=com.google.*:com.sun.*:de.hybris.platform.*

Create Dockerfile:

# <LOCAL_HYBRIS_FOLDER> - folder where hybris is located on your local machine after extraction
# <LOCAL_JACOCO_FOLDER> - folder where jacoco is located on your local machine after extraction
# <CONTAINER_HYBRIS_FOLDER> - folder where you desire to locate hybris inside container

FROM sapmachine:<VERSION>
COPY <LOCAL_HYBRIS_FOLDER> <CONTAINER_HYBRIS_FOLDER>
ENV HYBRIS_HOME=<CONTAINER_HYBRIS_FOLDER>
ENV HYBRIS_PLATFORM=$HYBRIS_HOME/hybris/bin/platform
ENV ANT_HOME=$HYBRIS_PLATFORM/apache-ant
ENV PATH=$PATH:$ANT_HOME/bin
COPY <LOCAL_JACOCO_FOLDER>/lib/jacococli.jar <LOCAL_JACOCO_FOLDER>/lib/jacocoagent.jar $ANT_HOME/lib

Build image an push it to your repo in DockerHub:

docker build -t <preferred_name>:<preferred_version> .
docker push <preferred_name>:<preferred_version>

bitbucket-pipeline.yml (stored in project repo root)

image: <HYBRIS_IMAGE> # created above

options:
  docker: true
  size: 2x # doubled ram for docker container

pipelines:
  pull-requests: # to trigger analysis on PR
    '**':
      - step:
          runs-on: # mandatory when using custom runners(not bitbucket embedded runner)
            - linux
            - self.hosted
          name: Build & analyze
          script:
            - mv .git config data $HYBRIS_HOME/hybris && mv bin/custom $HYBRIS_HOME/hybris/bin # move cloned dirs to corresponding hybris folders

            - cd $HYBRIS_PLATFORM && ant clean all -Dprofile <CI_PROFILE> && ant alltests -Dprofile <CI_PROFILE> -Dtestclasses.all.custom.extensions=true -Dtestclasses.web=true # generate .class files for sonar and run tests for jacoco.exec report file

            - mkdir -p $HYBRIS_HOME/hybris/bin/target/site/jacoco  # create dir to store .xml report file
            - java -jar $ANT_HOME/lib/jacococli.jar report $HYBRIS_HOME/hybris/log/jacoco/jacoco.exec --classfiles $HYBRIS_HOME/hybris/bin/custom/**/classes --xml $HYBRIS_HOME/hybris/bin/target/site/jacoco/jacoco.xml

            - ant sonarcheck -Dsonar.host.url=<SONAR_HOST_URL>
                             -Dsonar.login=<SONAR_TOKEN>
                             -Dsonar.projectKey=<PROJECT_KEY>
                             -Dsonar.source=11
                             -Dsonar.extensions=<EXTENSIONS_TO_ANALYZE> # example: -Dsonar.extensions=extension1,extension2,extension3
                             -Dsonar.java.binaries=../**/classes
                             -Dsonar.java.libraries=../**/lib
                             -Dsonar.scm.provider=git 
                             -Dsonar.exclusions=**/*.css,**/*.js,**/*.json,**/*.jsp,**/*.xml,**/*.yml,**/*.less # project specific file types or dirs to be excluded from analysis
                             -Dsonar.java.coveragePlugin=jacoco
                             -Dsonar.coverage.jacoco.xmlReportPaths=$HYBRIS_HOME/hybris/bin/target/site/jacoco/jacoco.xml
                             # -Dsonar.coverage.exclusions=** # if you don't want sonar to analyse code coverage
                             # -Dsonar.log.level=DEBUG

            - cp $HYBRIS_HOME/hybris/bin/.scannerwork/report-task.txt $BITBUCKET_CLONE_DIR/tmp # copy report to $BITBUCKET_CLONE_DIR to use with "artifacts"
          artifacts: # should be stored in $BITBUCKET_CLONE_DIR
            - tmp/report-task.txt
      - step:
          runs-on: # mandatory when using custom runners(not bitbucket embedded runner)
            - linux
            - self.hosted
          name: Quality Gate
          script:
            - pipe: sonarsource/sonarqube-quality-gate:1.1.0
              variables:
                SONAR_TOKEN: <SONAR_TOKEN>
                REPORT_FILE: $BITBUCKET_CLONE_DIR/tmp/report-task.txt