ARTIFACTORY: How to Resolve Artifactory Timeouts

What is server timeout?

Timeouts typically occur when the upstream server exceeds the expected response time, leading to request failures. This article focuses on timeout scenarios using JFrog Artifactory as an example. Large-sized artifacts, such as docker images, often encounter timeout issues. However, it's important to note that increasing the timeout value may not be the initial solution to address the problem.



This general architecture below can be referred to throughout this article.

How to identify which layer/device is timing out?

When faced with timeouts in a network configuration, it's important to identify which layer or device is causing the issue. One way to do this is by conducting tests that isolate specific devices. For instance, if docker upload failures are observed, the first device to check would be the JFrog Artifactory Server. You can attempt to upload a docker image directly from the local machine to Artifactory to see if it succeeds. If it does, then proceed to test the Reverse proxy and the load balancer.

Load balancers, like any other device in a network, have configurable timeouts. These timeouts are typically set to terminate idle connections. If you encounter timeouts at the load balancer, you can adjust the timeout values accordingly.

Similarly, in a high-availability (HA) environment, the Reverse proxy serves as the initial point of contact for incoming requests, playing a crucial role in maintaining the connection until a response is received from the upstream JFrog Artifactory Server.

Specifically, in Nginx, you can tweak timeout settings to address potential issues. The proxy read timeout defines the duration to keep the connection alive with the client, while the proxy sends timeout determines the timeout value for transferring requests to the upstream server. Adjusting these values can be helpful, especially when dealing with large artifact uploads. For example, setting the proxy_read_timeout, proxy_connection_timeout and proxy_send_timeout to 180 seconds extends the timeout duration(as the default is set to 60 seconds).

Snippet for reference:-

http {
  proxy_read_timeout 180;
  proxy_connect_timeout 180;
  proxy_send_timeout 180;
}

Timeouts in JFrog Artifactory Server:

Usecase1: Timeout occurring while downloading a package from an external site – this can occur on any of the following devices involved in the request flow.


Timeout issues can also arise in JFrog Artifactory - Tomcat when the server lacks capacity. Ensure sufficient resources are available for Tomcat to handle the load and fine-tune the Artifactory server for high loads (Here is the link to the KB to tune Artifactory for heavy loads). The default timeout is 60 seconds but can be increased by modifying the server.xml file by adding the below parameter to the tomcat/conf/server.xml file(a restart is required to take the changes into effect).

shared:

 extraJavaOpts: “connectionTimeout=“60000””

Additionally, consider increasing the socket timeout in the remote repository advanced settings if upstream responses are slow. If the issue persists, it could be due to the network proxy terminating connections. Remove the proxy and collaborate with the appropriate teams to address the problem.

Usecase2: Timeout occurs while downloading a private package from the backend filestore like S3. The usual flow may look like the diagram below:-



Check 1: Validate if there are issues with the S3 provider by downloading binaries directly from S3 using tools like s3cmd or AWS CLI on the local server of the JFrog Artifactory.

Check 2: If S3 performance is fine, bypass the network proxy and attempt to download the binary. To increase the timeout between JFrog Artifactory Server and backend S3, modify the Artifactory binarystoreconfig.xml file with the provided snippet, specifying the desired connection timeout.

<connectionTimeout>120000><connectionTimeout>

An example snippet of Artifactory Binary store config would look like this:

<config version=”2”>
   <chain template=”google-storage-v2″/>
   <provider id=”google-storage-v2″ type=”google-storage-v2″>
       <bucketName>my-bucket</bucketName>
       <path>myPath</path>
       <rootFoldersNameLength>3</rootFoldersNameLength>
       <useInstanceCredentials>false</useInstanceCredentials>
       <signatureExpirySeconds>10</signatureExpirySeconds>
       <proxyHost>127.0.0.1</proxyHost>
       <proxyPort>8888</proxyPort>
       <proxyIdentity>username</proxyIdentity>
       <proxyCredential>password</proxyCredential>
       <maxConnections>50</maxConnections>
       <connectionTimeout>120000</connectionTimeout>
   </provider>
</config>

Timeout during the replication: 

Push replication representation:

Checkpoints:

• JFrog Artifactory1’s Server Tomcat
• JFrog Artifactory1’s Server network proxy
• Reverse proxy in front of JFrog Artifactory2 Server
• Tomcat of JFrog Artifactory2 Server

Pull replication representation:

Checkpoints:

• JFrog Artifactory1’s Server Tomcat timeout
• JFrog Artifactory1’s Server network proxy
• Remote repository socket timeout
• Reverse proxy in front of JFrog Artifactory2 Server
• Tomcat of JFrog Artifactory2 Server

Please note that we can set the replication timeout at Artifactory configurations as well. To get more information about it, you can also refer to this KB article.