ARTIFACTORY: How to copy a large repository or folder from one location to the other

You may have the use case of needing to copy a particularly large repository or folder from one location to another. At first, you may have just tried a copy command from the Artifactory UI or rest API but with large folders/repositories with many items, this may cause issues either due to the copying taking too long or you may run into a resource bottleneck. 

If your concern is just with how long the copy command is taking you can use the JFrog CLI which is multithreaded and performs many copy calls by calling a copy command on each item. You can read more about the JFrog CLI and copying large repositories in this other Knowledge Base article.

But if your concern is with the resource bottleneck go ahead and take a look at the rest of this article for an example to workaround this. This example will demonstrate how you can instead perform a copy on smaller batches of items at a period of time. The following will guide you through a possible way to do this with the rest API and some scripts.

Example

First, we will generate a list of items. This example will just be generating a root-level list of items. If needed you can head deeper into the folder structure.
This is using the item properties rest API and using the jq utility to get a list of items, url encode them, and output them into a text file.

curl -uadmin "<ART_URL>/artifactory/api/storage/<REPO_NAME>/" | jq -r '.children[].uri' | jq -Rr @uri > listOfItems.txt

Next, we will run the copy call on each item generated from the list.
The following is a simple bash loop that reads each line from the text file and performs the copy rest call. I had previously exported the TOKEN variable before running the loop with my user’s token. This example is copying from one repository to another. Go ahead and replace the <REPO_NAME*> sections according to your needs. You may also adjust the sleep parameter as needed.

while read -r line; do echo -e "\n\n `date` - Copying $line\n\n";  curl -XPOST -u<USER>:$TOKEN "<ART_URL>/artifactory/api/copy/<REPO_NAME_1>$line?to=<REPO_NAME_2>"; echo -e "\n\n`date` - Sleeping\n\n"; sleep 3; done < listOfItems.txt 

This will print out each copy command and the results. If there are any issues you can spot the particular command/item and further troubleshoot.

One thing to note is that Artifactory copies/moves items according to Linux convention which can be slightly unintuitive. The copy command will create a folder within a folder for a folder already existing as seen in the following example (note how ‘folder2’ when copied over is moved within the existing ‘folder2’).

Example of copying a folder in Linux:
Original Structure:

$ tree
.
├── folder1
│   └── folder2
│       └── test.txt
└── newlocation
    └── folder2

4 directories, 1 file

Running a copy command will result in a ‘folder2’ being created inside the destination folder not overwriting the existing ‘folder2’ in the ‘newlocation’ location.

$ cp -r folder1/folder2/ newlocation/folder2/
$ tree
.
├── folder1
│   └── folder2
│       └── test.txt
└── newlocation
    └── folder2
        └── folder2
            └── test.txt

5 directories, 2 files

So if you are copying to a specific path you should take care to not include the particular folder name at the end of the path of the destination unless you want to copy the directory structure into that particular location. 

An example of the potentially unintended nested folder can be seen in action if you add the “$line” variable to the destination section of the curl and run the loop twice. The second loop will attempt to copy the items into the second repository which will already have the folders copied over from the first loop so they will be placed into the folder instead of overwriting the folder
I.e. Adjusted script which can lead to the unintended nested folder structure

while read -r line; do echo -e "\n\n `date` - Copying $line\n\n";  curl -XPOST -u<USER>:$TOKEN "<ART_URL>/artifactory/api/copy/<REPO_NAME_1>$line?to=<REPO_NAME_2>$line"; echo -e "\n\n`date` - Sleeping\n\n"; sleep 3; done < listOfItems.txt

Loop 1:



Loop 2: