ARTIFACTORY: Fine-Tuning AWS SDK v2

Fine-Tuning AWS SDK v2 with Artifactory

Starting from Artifactory 7.125.4, Artifactory allows the use of AWS SDK v2 for more info please see here. 
The default values that are set for Artifactory out-of-the-box are designed to optimize the behavior for the vast majority of users. The scenarios below are a guide for users who find that some of the default values do not meet their system needs. All the parameters below should be adjusted in the s3 provider inside the $JFROG_HOME/artifactory/var/etc/artifactory/binarystore.xml

Slow Upload from the Client

AWS SDK v2 uses asynchronous non-blocking mechanisms to perform its operations. To adapt it with blocking operations, a thread pool exists that facilitates these operations. The thread pool has 15 threads by default. However, if reading bytes from the client is slow, these threads may become overloaded and degrade Artifactory's upload performance.
To deal with this possibility, Artifactory allows for increasing the numbers of threads using the blockingDataReadThreads parameter.

Saturated Network Bandwidth

AWS SDK v2, and especially the CRT client, aim to provide high throughput. However, it is possible that if the traffic on Artifactory surpasses the capabilities of the network, some components like the network card or the provisioned network level, may hit system overload. If this happens, JFrog recommends reducing the value of the parameter crtTargetThroughputInGbps, which sets the throughput measured in Gbps that the CRT client aims for. When reduced, this results in less use of the network.
This will allow Artifactory to run slower on overloaded networks. However, it will also cause degradation in system performance.

Optimize Multipart Uploads

Artifactory’s S3 provider offers the same multipart upload mechanism for both AWS SDK v1 and v2. Multipart upload is configured by the parameters multiPartLimit and multipartElementSize.
When using AWS SDK v2, the multipart upload operation is performed using the CRT client. This client aims for high throughput. This means that the CRT client with SDK v2 may trigger a high amount of requests to upload parts to the S3 storage.
If you need to upload less parts, consider decreasing multiPartLimit to have fewer multipart uploads, or increasing multipartElementSize for uploading parts with a bigger size. If your use case involves generally uploading large binaries, consider increasing multipartElementSize.
For more information on multipart upload, click here.

Connection Timeouts with High Load

When Artifactory uses the AWS SDK v2, operations like multipart upload might quickly open a significant number of connections to S3 storage. This may cause the storage solution to take longer than expected to open a connection.
To deal with this situation, you can use the parameters connectionTimeout and connectionAcquisitionTimeout to set these thresholds for both the Apache HTTP Client and Netty, used behind the scenes.

Upload Failures for Non-AWS Users

AWS SDK v2 provides mechanisms for checksum validation, both during upload and download. These mechanisms are enabled by default in Artifactory. Some S3-compatible storage solutions might not fully implement the checksum validation mechanisms that the SDK uses. If this is the case, this could cause uploads to fail.
If you are using an S3-compatible storage solution (not AWS S3), you experience failures during uploads, and you find in the Artifactory logs exceptions related to XAmzContentSHA256Mismatch, consider setting disableChecksumValidation = true (default is false) to solve this problem.
However, before doing this, you should consider the implications of disabling the checksum validation mechanism. For more information on AWS’ checksum validation mechanism, see the AWS documentation.

Logging

By enabling debug logging, you can get more verbose data about the operations performed by Artifactory’s S3 provider and the SDK it uses.

Artifactory S3 Provider

Set the package org.jfrog.type.s3 to debug as follows:

<logger name="org.jfrog.type.s3" level="DEBUG"/>

AWS SDK v2

For request/response summary logging for AWS SDK v2, enable as follows:

<logger name="software.amazon.awssdk.request" level="DEBUG"/>

usePresigning Parameter Not Supported in AWS SDK v2

The use of presigned URLs for executing all operations against S3 is not supported in AWS SDK v2. If the parameter usePresigning is enabled when using AWS SDK v2, the following warning message will appear in logs when Artifactory starts up:
“The property usePresigning is not used by the current S3 provider and will be deprecated.”
Note: 

Downloading via presigned URLs using direct download with the enableSignedUrlRedirect parameter is supported in the S3 provider regardless of the AWS SDK version.