Introduction
Artifactory users encounter this critical error in the service logs, particularly during high-load operations such as YUM metadata recalculation or heavy concurrent usage:
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
This error results in transaction failures and service instability.
In this article, I will explain what the error means and how to resolve it.
Why This Error Occurs
This specific OutOfMemoryError is not a Java Heap memory issue (i.e., it is not fixed by increasing the -Xmx flag). Instead, it is a resource constraint imposed by the underlying Operating System (OS), usually Linux.
1. Threads are OS Resources
Every Java thread created by the Artifactory JVM (Light Weight Process, or LWP, in Linux terminology) requires the OS to allocate two things:
2. Hitting the nproc Limit
Linux enforces a maximum number of processes and threads a single user can own, defined by the nproc (maximum user processes) limit.
-
When the JVM's number of threads (including its own internal threads and all worker threads) exceeds the configured Soft Limit (or Hard Limit) for nproc, the OS rejects the creation of any new thread.
-
The system may have tens or hundreds of gigabytes of free RAM, but it refuses the request because it hits a count-based quota, leading directly to the unable to create native thread error.
3. Hitting the total virtual memory limit
The OS fails to allocate the necessary Native Memory for the new thread's stack. This occurs when the total committed memory in the system nears its limit.
-
Scenario A: Disabled Swap: If swap is disabled (or too small), the total virtual memory available is capped at a value close to the physical RAM size. The combination of the Java Heap, Metaspace, Direct Buffers, and the stacks of all existing threads consumes the available physical memory.
-
Scenario B: Fragmentation: Even with free RAM, the OS may be unable to find a large enough contiguous block of virtual memory (e.g., 1 MB for the thread stack) due to heavy memory fragmentation, leading to the same allocation failure.
Resolution
The fix requires increasing the Soft and Hard nproc limits for the user running the Artifactory service (e.g., artifactory, tomcat, or a dedicated user).
Step 1: Verify the Current Limit
Run the following command as the Artifactory service user to confirm the current enforced limit:
ulimit -u
If the output is low, for example, 8192 or less, this is the bottleneck.
Limit Resource Value
soft nproc The enforced limit. Set to a high value.
hard nproc The absolute ceiling. Must be ≥ the soft limit.
Step 2: Modify the OS Configuration
You must modify the Linux Pluggable Authentication Modules (PAM) limit configuration file. This requires root access.
-
Open the /etc/security/limits.conf file or create a specific file in the /etc/security/limits.d/ directory (e.g., /etc/security/limits.d/artifactory.conf).
-
Add or modify the following lines, replacing ARTIFACTORY_USER with the actual user running the Artifactory service:
We recommend setting the limit to 65536 or higher:
# Set the nproc limit for the Artifactory service user
ARTIFACTORY_USER soft nproc 65536
ARTIFACTORY_USER hard nproc 65536
If you are unsure of the user, using the wildcard * will apply it to all non-root users, but it is better to be specific.
Step 3: Restart the Service and Verify
-
Restart the Entire Linux Server (or at minimum, log out and back in as the ARTIFACTORY_USER if you can).
-
Restart the Artifactory Service. This is crucial to ensure the process inherits the new resource limits.
-
Verify the new limit by running ulimit -u again as the service user. The output should now be 65536.
Once the service is restarted with the higher thread quota, the system will allow Artifactory to create the necessary worker threads without throwing the OOM error.
If the error message persists, you might try enabling or increasing the SWAP memory, although be mindful of the total memory limit of the VM.