ARTIFACTORY: Maven build and containerising it with Docker build using Gitlab and integration with Artifactory and scan with JFrog Xray as part of a JFrog Project
The intention of this KB article is to kick start a maven build and containerize it with a docker build using GitLab as a CI/CD tool and integrate it with Artifactory with a simple use case. We have used a SaaS version of the JFrog platform. However, the use case stands valid for the self-hosted JFrog Platform as well.
IntroductionGitLab can be used as a CI/CD tool along with many other features that it has. In order to integrate the GitLab with JFrog Artifactory, there is no such out-of-the-box plugin available. However, we can integrate it using our very own CLI tool called JFrog CLI. We can build with multiple package types using JFrog CLI. More details on JFrog CLI is available at this link.
In this KB article, we will discuss a maven build and then containerize it using a docker file and then do a docker build, and then push it to Artifactory. The build has the below steps –
- Create a GitLab project and upload all the required files to the GitLab repository.
- Pull a base image from Docker Hub.
- Build a docker image using the image that we have pulled. The step # 2 and # 3 will be done as part of a Dockerfile.
- As part of the docker build using the Docker file, it will do a “maven clean install” which will build the docker packages, and then will do the “maven clean deploy” which will push the maven packages to the Artifactory and then create a docker image along with those jar files that are created by the “maven clean install” command.
- Do a scan of the image before even pushing to the Artifactory. You can find more details about on-demand scanning here on this page - https://www.jfrog.com/confluence/display/JFROG/Xray+On-Demand+Binary+Scan. Please note that it will refer to the watch and policy and the policy has a rule to fail the build if the image contains any vulnerability as the “Fail build” option is checked in the Rules of the Policy as mentioned in the 4.B section of “JFrog Platform Setup”.
- Push the image to the Artifactory and publish the build information.
- Scan that build using the JFrog Xray.
- Scan the jars created as part of the maven build that are pushed to Artifactory.
JFrog Platform Setup:We have the below configurations in the JFrog platform. Please note that we will be pushing the build and scan it as part of a project. For more details on the project, kindly refer to this document - https://www.jfrog.com/confluence/display/JFROG/Projects.
1. We have first created a project and it will look like below -
2. We have created a virtual, a remote and a local docker repositories and we have clubbed the local and the remote repository to the virtual repository and we will use the virtual repository as an endpoint -
3. We have created a virtual repository and a local repository for release and a virtual repository and a local repository for snapshot and clubbed them in the virtual repository as shown below -
4. Xray configuration has to be done as below -
A. Create a watch and policy before we run the pipeline. Otherwise, the XRay scan will not succeed. Please follow the below links for creating the watch and policy –
Create the policy and the rules using this link – https://www.jfrog.com/confluence/display/JFROG/Creating+Xray+Policies+and+Rules
Create the watch and add the repository, build, policies and rules using this link – https://www.jfrog.com/confluence/display/JFROG/Configuring+Xray+Watches
B. Make sure to enable the below option in the rules of the policy when it is created –
C. Need to include the build in the Indexed Resources in the JFrog platform. However, when the build is first created with the Xray scan step, it will skip the Xray scan step as the build is not included in the Indexed Resources. That’s why it is recommended to include the builds using the Include or Exclude pattern. For this example, the Include pattern is “*/**”. This means the platform will add all the builds that are pushed to the Artifactory, to the Indexed Resources –
For more details on the Indexed Resources, it is also recommended to go through this KB article – https://jfrog.com/knowledge-base/xray-how-to-index-and-scan-all-builds-in-xray-in-the-unified-platform/
D. Make sure to add the maven repositories to the Indexed Resources as shown below -
Docker File:For a docker build, we need to have a docker file created. The docker file contents is as below -
FROM maven:latest RUN mkdir -p /usr/local/maven && mkdir -p /root/.m2 && mkdir /root/.m2/repository # Copy maven settings, containing repository configurations COPY settings.xml /root/.m2 WORKDIR /usr/local/maven ADD ./maven-example . RUN mvn clean install RUN mvn clean deploy
The docker file is actually performing the below items -
- Pull a maven image
- Create directories as “/usr/local/maven” and “/root/.m2” and “/root/.m2/repository”
- Copy the “settings.xml” to the “/root/.m2” directory.
- Declare “/usr/local/maven” as the working directory.
- Copy the entire “maven-example” folder along with its content from the GitLab repository to the defined working directory.
- Run the command “mvn clean install”
- Run the command “mvn clean deploy”
GitLab setup:We need to have the below GitLab setup -
1. First create a project in the GitLab. You can find the details on how to create the project using the below links -
A. GitLab official link - https://docs.gitlab.com/ee/user/project/working_with_projects.html
B. Youtube link - https://www.youtube.com/watch?v=DGuMvGYZ7lY
2. Upload all the required files to GitLab repository as mentioned in the links in step # 1. It will look like this -
As you can see from the above screenshot that there is a “maven-example” folder created. This folder contains the maven source code and the “pom.xml” for the maven build. This is a sample maven project that I have taken from this page - https://github.com/jfrog/project-examples/tree/master/maven-examples/maven-example. The contents looks like below -
3. Make sure the “pom.xml” file has the distribution management with the repository configuration as mentioned below. The example “pom.xml” looks like below -
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0""
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">;
<modelVersion>4.0.0</modelVersion>
<groupId>org.jfrog.test</groupId>
<artifactId>multi</artifactId>
<version>3.7-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Simple Multi Modules Build</name>
<modules>
<module>multi1</module>
<module>multi2</module>
<module>multi3</module>
</modules>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.2.2</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>3.3.2</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.2.1</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<addMavenDescriptor>false</addMavenDescriptor>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<archive>
<addMavenDescriptor>false</addMavenDescriptor>
</archive>
</configuration>
</plugin>
</plugins>
</build>
<distributionManagement>
<repository>
<id>releases</id>
<url>https://test.jfrog.io/artifactory/dct-libs-release</url>;
</repository>
<snapshotRepository>
<id>snapshots</id>
<url>https://test.jfrog.io/artifactory/dct-libs-snapshot</url>;
</snapshotRepository>
</distributionManagement>
</project>
4. Another important file you need to have is the “settings.xml” file, which has to be generated from the “Set Me Up” button from Artifactory UI. Kindly refer to this page on how to generate the “settings.xml” file - https://www.jfrog.com/confluence/display/JFROG/Maven+Repository#MavenRepository-AutomaticallyGeneratingSettings. The example “settings.xml” looks like this -
<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 http://maven.apache.org/xsd/settings-1.2.0.xsd"" xmlns="http://maven.apache.org/SETTINGS/1.2.0""
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">;
<servers>
<server>
<username><username></username>
<password><password></password>
<id>central</id>
</server>
<server>
<username><username></username>
<password><password></password>
<id>snapshots</id>
</server>
</servers>
<profiles>
<profile>
<repositories>
<repository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>dct-libs-release</name>
<url>https://customer.jfrog.io/artifactory/dct-libs-release</url>;
</repository>
<repository>
<snapshots />
<id>snapshots</id>
<name>dct-libs-snapshot</name>
<url>https://customer.jfrog.io/artifactory/dct-libs-snapshot</url>;
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>dct-libs-release</name>
<url>https://customer.jfrog.io/artifactory/dct-libs-release</url>;
</pluginRepository>
<pluginRepository>
<snapshots />
<id>snapshots</id>
<name>dct-libs-snapshot</name>
<url>https://customer.jfrog.io/artifactory/dct-libs-snapshot</url>;
</pluginRepository>
</pluginRepositories>
<id>artifactory</id>
</profile>
</profiles>
<activeProfiles>
<activeProfile>artifactory</activeProfile>
</activeProfiles>
</settings>
5. Click on the settings button and make sure that you have the CI/CD option enabled as shown in the screenshot below else you will not see the CI/CD option where you have to check the pipeline run status -
6. Now create the variables in the settings as shown below. We will use these variables in the pipeline script -
The defined variables are as follows -
ART_DOCKER_PASS: <Password to access Artifactory docker repository. Generally it is the same with which you login to the JFrog UI and view docker repositories>
ART_DOCKER_REG: test.jfrog.io
ART_DOCKER_USER: <Username to access Artifactory docker repository. Generally it is the same with which you login to the JFrog UI and view docker repositories>
ART_PLAT_PASS: <Password to login to JFrog UI>
ART_PLAT_URL: https://test.jfrog.io
ART_PLAT_USER: <Username to login to JFrog UI>
ARTIFACTORY_PROJECT_KEY: <Project Key defined in the project as shown in step # 1 under JFrog Platform Setup>
7. Now add a file called “.gitlab-ci.yml” in the GitLab project. The details on how to add the file is mentioned in this document - https://docs.gitlab.com/ee/ci/quick_start/
8. Now add the below content in the yaml file that is just added -
docker-build:
# Use the official docker image.
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
# docker login to Artifactory
- docker login -u "$ART_DOCKER_USER" -p "$ART_DOCKER_PASS" $ART_DOCKER_REG
# Installing JFrog CLI
- curl -fL https://install-cli.jfrog.io | sh
- chmod +x /usr/local/bin/jf
# Configure Artifactory instance with JFrog CLI
- pwd
- cd /usr/local/bin/
- ./jf --version
- ./jf c add artifactory-server --url=$ART_PLAT_URL --user=$ART_PLAT_USER --password=$ART_PLAT_PASS
script:
# docker build
- docker build --pull -t "$ART_DOCKER_REG/dct-docker/test:$CI_JOB_ID" /builds/swarnendukayal/maven-docker-test
- cd /usr/local/bin/
# On-deman scanning
- ./jf docker scan "$ART_DOCKER_REG/dct-docker/test:$CI_JOB_ID" --project=$ARTIFACTORY_PROJECT_KEY
# Docker push
- ./jf rt docker-push "$ART_DOCKER_REG/dct-docker/test:$CI_JOB_ID" dct-docker --build-name=maven-docker-jfrog --build-number=$CI_JOB_ID --project=$ARTIFACTORY_PROJECT_KEY
# Collect the environment variables
- ./jf rt bce maven-docker-jfrog $CI_JOB_ID --project=$ARTIFACTORY_PROJECT_KEY
# Pass the build information to Artifactory
- ./jf rt bp maven-docker-jfrog $CI_JOB_ID --project=$ARTIFACTORY_PROJECT_KEY
# Scanning the docker build
- ./jf bs maven-docker-jfrog $CI_JOB_ID --project=$ARTIFACTORY_PROJECT_KEY
# Run this job in a branch where a Dockerfile exists
rules:
- if: $CI_COMMIT_BRANCH
exists:
- DockerfilePlease note that the location “/builds/swarnendukayal/maven-docker-test” marked in bold above, is configured to define the GitLab repository path from where the Dockerfile will refer the files while doing the docker build. This location will change project to project and profile to profile.
For example: If the project URL(sample only) is as below - https://gitlab.com/abc/test-jfrog/-/blob/master/.gitlab-ci.yml, then the location would be “/builds/abc/test-jfrog”. Kindly change as per your project and repository location.
Once you save this file, it will immediately, trigger a build and you can find the build status as shown below -
The build shows as failed as the Xray policy has been defined to fail the build if any vulnerabilities are caught.
How does the build look in the JFrog platform:1. The build will appear in the builds section as below -
2. Once you click on the build, it will show you the build numbers as shown below -
As you can see that the Xray status is also showing as the builds are scanned and the violations are given. If you would like to check the XRay status specific to a build number, then click on the build number and then go to the “XRay Data” as below -
The scan which was done for the image which was pushed to Artifactory, can be seen here -
The maven packages that we have pushed to the Artifactory are deployed to the repository and the packages have also been scanned as we can see the violations in the Xray tab as shown below -
Please note that the steps that we are doing here, except the docker build steps and the maven build steps from Dockerfile, are basically the steps as mentioned in this blog – https://jfrog.com/blog/manage-your-docker-builds-with-jfrog-cli-in-5-easy-steps/. Kindly go through it as well for more clarity on how the JFrog CLI works with docker.