Introduction
You may occasionally encounter errors such as "Timeout waiting for connection from pool" when the HTTP outgoing connection pool is exhausted. An example of this error message is shown below:
20XX-08-XXT23:48:03.018Z [jfrt ] [WARN ] [2afccf3bcc1b249d] [.s.b.p.RetryBinaryProvider:153] [7.0.0.1-8040-exec-52] - Failed to fetch blob 'b97ae2defa791700910e53c05cb61f8d49d7f5ac' from next binary provider, retry number 0
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
The default connection pool limit is set to 50, but you might need to adjust this limit to prevent such errors. First, it's essential to monitor the pool's actual usage.
Resolution
Monitoring the HTTP Outgoing Connection Pool
To monitor the HTTP outgoing connection pool, you can enable debug logging by adding the following configuration to the
/var/opt/jfrog/artifactory/etc/artifactory/logback.xml file:
<appender name="connectionpool" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${log.dir}/artifactory-connectionpool.log</File>
<rollingPolicy class="org.jfrog.common.logging.logback.rolling.FixedWindowWithDateRollingPolicy">
<FileNamePattern>${log.dir.archived}/artifactory-connectionpool.%i.log.gz</FileNamePattern>
<maxIndex>10</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>25MB</MaxFileSize>
</triggeringPolicy>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="org.jfrog.common.logging.logback.layout.BackTracePatternLayout">
<pattern>%date{yyyy-MM-ddTHH:mm:ss.SSS, UTC}Z [jfrt ] [%-5p] [%-16X{uber-trace-id}] [%-30.30(%c{3}:%L)] [%-20.20thread] - %m%n</pattern>
</layout>
</encoder>
</appender>
<logger name="org.apache.http.impl.conn.PoolingHttpClientConnectionManager" additivity="false">
<level value="Debug"/>
<appender-ref ref="connectionpool"/>
</logger>
Once the logging configuration has been set, you can monitor the connection releases by executing the following command:
tail -f artifactory-connectionpool.log | grep 'Connection leased'
2023-03-23 17:45:11,445Z [jfrt ] [DEBUG] [af1054d3aa215b3 ] [ttpClientConnectionManager:351] [http-nio-8081-exec-9] - Connection leased: [id: 99][route: {s}->https://repo1.maven.org:443][total available: 1; route allocated: 1 of 50; total allocated: 1 of 50]
Tuning the Remote Repository Connection Pool
For remote repositories (e.g., "repo1.maven.org"), you can adjust the connection pool by setting the following parameter in the artifactory.system.properties file:
artifactory.http.client.max.total.connections=100
Tuning the S3 Connection Pool
To monitor the S3 connection pool, you can run:
tail -f artifactory-connectionpool.log | grep ‘amazonaws’
Example output:
2024-09-03 23:06:08,524Z [jfrt ] [DEBUG] [532bbe8b40fa5fef] [ttpClientConnectionManager:267] [tp-nio-12000-exec-59] - Connection request: [route: {s}->https://XXXXXXXXX.s3.us-east-1.amazonaws.com:443][total available: 0; route allocated: 40 of 50; total allocated: 40 of 50]
To increase the maximum connections for S3, set the <maxConnections> parameter in the binarystore.xml file within the s3-storage-v3 section:
<provider type="s3-storage-v3" id="s3-storage-v3">
<maxConnections>200</maxConnections>
</provider>
Conclusion
By monitoring and adjusting these connection pool parameters, you can prevent errors like “Timeout waiting for connection from pool” and ensure that your applications run smoothly.
Regularly reviewing the connection utilization will help you maintain optimal performance and avoid potential bottlenecks.