ARTIFACTORY: Artifactory Migration using JFrog CLI Transfer Tool

Note:
This is a dynamic article. JFrog will update it when new information becomes available. Save this page and check back periodically for the latest information.
_Note:
This article is for migrating between 2 self hosted servers. To migrate from self hosted to the cloud - please visit this doc

Overview
The JFrog CLI transfer tool command allows transferring (copying) all the Artifacts stored in one Artifactory instance to a different Artifactory instance. However, this is only half the picture. If you wish to make a full replica of your existing Artifactory you will need to transfer all configuration and access related data as well. This is useful for migrating between different DB types or if you need to move Artifactory to a new Datacenter. 
This guide will help you create a full replica of your existing Artifactory utilizing the JFrog CLI for the Artifact transfer. 

This process requires 2 steps:

Import Export - Transfer of configurations (including repository structure), from source to target.
File Transfer - Transfer of Artifacts from source to target.

This article covers this process step by step.

 

Tip
This is one of many methods for migrating JFrog Artifactory. 
For a complete understanding of the overall migration process, we recommend reviewing our main article: 

Comprehensive Guide to Migrating JFrog Artifactory

Prerequisites and initial setup:

Setting up the Target server:

For the migration, you will need to create a new Artifactory server which will be set up and connected to a new database.
The source and target Artifactories should be running the same version, to avoid potential issues. (For the import/export compatibility).
Master.key & Encryption:
Ensure that the master.key (located at $JFROG_HOME/artifactory/var/etc/security/master.key) is copied from the source Artifactory to target before starting the new instance.

Note:

If a target instance was set up and started without the same master key of the source Artifactory, the DB must be dropped, recreated and then master key inserted and then restarted.
(Since initial connection of Artifactory to a fresh DB encrypts its data with the existing master key).
Target Artifactory must be provisioned with a license.

Note:

If you have an Enterprise + license on both Source and Target Artifactories, if desired you can save time by first manually copying the Filestore from one Artifactory to another (or connecting to the same filestore) and then use the --filestore=true flag to copy the metadata only using the tool. More information can be found in the appendix.

Step 1 - Transferring configurations:

Note:

This migration method is zero downtime. However, after performing this step of exporting the configuration, any future changes to the config will not be reflected in the new instance and will need to be made there. This includes any changes to users, groups, permissions and creation of new repositories.

Perform a System Export with both content and metadata excluded. This will create a small export file which includes the skeleton of the repositories including all configuration, as well as access entities like users and permissions.
```
 
```
Copy it to the Target Artifactory. ( rsync or scp can be used, or any other compatible copying method).

Target Artifactory

Ensure Artifactory is installed and running properly, UI is accessible.
Perform a System Import, and leave all checkboxes empty.

Note (Icon) :The binarystore.xml file is not overwritten when importing the system configurations and therefore should be set manually.
You should be able to see the repositories from the source Artifactory as well as user, groups and other configurations that have been created, yet no artifacts will be presented in each repository.
To login to the target Artifactory you will need to use a known user from the source Artifactory, as the target instance users will have been replaced as part of the import.

Step 2 - Artifact Transfer Preparations & Plugin installation

The following section will go through the steps to ensure the following 2 components are in place:

Install and set up JFrog CLI on a machine with a connection to both Source and Target Artifactory.
Install the transfer plugin on the source Artifactory.

JFrog CLI

On a machine with access to both the source and target Artifactory (this can be the source instance itself). Download and Install the latest JFrog CLI using one of the JFrog CLI installers:
For example:
```
curl -fL https://install-cli.jfrog.io | sh
```
Configure the connection details of the source Artifactory instance with your admin credentials. Run the following command and follow the instructions.
When asked about an Authentication method please use the Access Token Method.
```
 jf c add source-server
```
Please make sure to create and use an Access Token for the authentication method for both the source and target Artifactories.

You can validate the configuration addition to the file with the following command:
```
jf config show
jf rt ping --server-id source-server
```
Configure the connection details of the Target Artifactory instance. (Use an access token for authentication as well)
```
jf c add target-server
```
Ensure that the JFROG_HOME environment variable is set and holds the value of the JFrog Artifactory installation directory. It usually points to the /opt/jfrog directory.
Add the following line to your shell configuration file, e.g., $HOME/.bashrc or more globally in /etc/environment.
```
export JFROG_HOME=/opt/jfrog
```
```
 
```

Install Transfer plugin on source instance

If you have internet access and have the CLI directly installed on your Artifactory source instance which is not a Kubernetes or Docker install.
Then you can install the data-transfer user plugin automatically by running the following command:

jf rt transfer-plugin-install source-server

If the source instance does not have internet access or is a Kubernetes or Docker based install, then we will load the plugin manually:

We will need to load the plugin files onto the source instance in this directory:
```
/opt/jfrog/artifactory/var/etc/artifactory/plugins
```
a) For an airgapped instance, follow these instructions on a machine with internet access then copy the files to the correct location in the airgapped instance.

b) For K8s or docker, exec into the container and download the Plugin using CURL (The only tool possible in this container):

The plugin is composed of 2 files
- Groovy file, that should be inserted to the plugins directory
- Jar file, that should be inserted in a lib directory as its sub folder. It must be manually created if it does not exist.
```
cd /opt/jfrog/artifactory/var/etc/artifactory/plugins

curl -O "https://releases.jfrog.io/artifactory/jfrog-releases/data-transfer/[RELEASE]/dataTransfer.groovy" --globoff

#Then create a lib directory inside the plugins directory if not alreadt present
mkdir lib
cd lib

#Now download the jar file:

curl -O "https://releases.jfrog.io/artifactory/jfrog-releases/data-transfer/[RELEASE]/lib/data-transfer.jar" --globoff
```
Once this has been added, we should run the following API command to Reload Plugins:
https://docs.jfrog.com/integrations/reference/reloadplugins
```
curl -X POST -uadmin:<pass> <Artifactory-server-address>/artifactory/api/plugins/reload
```

Note:
Step 1 and 2 can be done in any order, just note that the target-server connection to the CLI must be done after the import as the import will wipe/replace all access tokens on the target server.

Step 3 - Transferring the Artifacts

This step will transfer the Artifacts from the source to target Artifactory. For more information on this step you can see the full documentation here:  
https://docs.jfrog.com/artifactory/docs/transferring-files-between-artifactory-servers

Key Points for File Transfer:

Remote repository cached artifacts are not transferred. If you wish to include cached remote repository content you can first manually copy the content of the remote cache to a local repository using the UI, API or CLI. This will then be transferred to the new Artifactory.
Files created or modified on the source after the transfer starts are transferred, along with their custom properties if the file content also changed.
Files deleted on the source after the transfer starts are not deleted on the target.
If only a file's custom properties change on the source (not its content), these property changes are not updated on the target.
Federated repositories are transferred without their federation members. After the transfer, you'll need to reconfigure the federation members as described in the Federated Repositories documentation.
Since the files are pushed from the source to the target instance, the source instance must have a network connection to the target.

In order to transfer the artifacts we need to run the following commands:

Checking the connectivity between the two servers by using the --prechecks=true flag:
```
jf rt transfer-files source-server target-server --prechecks=true
```
If the above prechecks command completes successfully, we can run the full command
```
jf rt transfer-files source-server target-server 
```
Example of the output you should be seeing when running the transfer-files command:
We can use the following command to get the status of the transfer
```
jf rt transfer-files --status
```
Example output:
You can also control the number of working threads in the migration by running the jf rt transfer-settings command, themore threads used, the more load there will be on the source Artifactory, you must monitor your usage to see what is the best balance in your situation:
```
jf rt transfer-settings
```
The default value is 8 but can be changed even during the file transfer.
Once the process is completed, you should get a message it was completed successfully:

User-added image

For more information about Controlling the transfer speed with the its considerations, please head to Controlling the File Transfer Speed documentation

Manually Copying the Filestore to Reduce the Transfer Time

The jf rt transfer-files command transfers the binaries to your new filestore as well as the metadata of the binaries to the database (file paths, file names, properties, and statistics). For Enterprise + customers there is an option to manually copy the filestore (or reuse the existing one) and just use the tool for the metadata, this greatly speeds up the process, for more information see the appendix.

Step 3 - Testing and Tuning:

Finally, once all of the data and metadata have been moved into the new Artifactory server, you can follow the following steps to test and tune the new instance:

Try downloading any files and see if you can.
Federated repositories are transferred without their federation members. After the transfer, you'll need to reconfigure the federation as described in the Federated Repositories documentation.

Step 4 - Cutover:

Syncing the deltas can be done by re-running the transfer-files process. The data transfer plugin is saving the status of the transfer and therefore, if the process is triggered again, it will only move the deltas. You can also use the --exclude-repos flag to skip specific repositories or --include-repos, to include only a few specific ones.(To achieve this, the JFrog CLI stores the current state of the file transfer process in a directory named transfer located under the JFrog CLI home directory (usually ~/.jfrog/transfer).
Changing the DNS to point to the new Artifactory and start receiving traffic.

Appendix

Manually Copying the Filestore to Reduce the Transfer Time

For Enterprise + customers where Artifactory is above version 7.55.2, there is an option to manually copy the filestore (or reuse the existing one) and just use this tool to transfer the metadata. This can greatly speed up the transfer process. 

To use this option please follow the following steps:

Ensure you have an Enterprise+ license on source and target Artifactory.
Manually transfer/sync the content of the source and targets filestores. (This can be done using any external copy method based on your filestore).
Any binaires not synced will be manually brought across by the transfer tool.

(Optional) In theory the same filestore can be connected to both the target and source to achieve this goal. For example both Artifactory instances can be pointed to the same s3 bucket.

If this is done then you must disable Garbage Collection on both instances by setting its cron expression to run a long time from now. You can refer to this KB article for more info. (0 0 0 ? * * 2030/1)

Set the checkBinaryExistenceAllowed value to true, in order to allow the target server to accept the metadata only. It can be done by running this REST API:

curl -u<USERNAME>:<PASSWORD> -d @checkBinaryExistenceAllowed.json -H "Content-Type: application/json" -XPUT
http://<ARTIAFCTORY-URL>/artifactory/api/config/storage/checksumReplication

The content of the checkBinaryExistenceAllowed.json file is:

{
  "checkBinaryExistenceAllowed" : "true",
  "daysToTrust" : "30"
}

You can also run the following REST API to get the status after changing the value to true:

REST API:
curl -u<USERNAME>:<PASSWORD> -XGET http://<ARTIAFCTORY-URL>/artifactory/api/config/storage/checksumReplication

Output:
{
  "checkBinaryExistenceAllowed" : true,
  "trustUntil" : "2024-03-08 14:17:05 +0000"
}

Now when running the transfer-files command we can add the flag --filestore=true to attempt to move only the metadata. If there is no corresponding binary, the tool will manually transfer the binary as well, even with this flag set.
```
jf rt transfer-files source-server target-server --filestore=true
```

Note: 
Without specifying this flag, Artifactory will actively move the binaries even if they are already there, which will take longer to complete.