ARTIFACTORY: Why am I getting intermittent 500 errors when uploading large files?

Sometimes you may observe the following behavior, where you are uploading a large file using REST or UI when running Artifactory behind Nginx, but it fails with a 500 Internal Server error:



Taking a look at the request within the artifactory-request.log files, the request seems to be received by the node. The status code might be 200, 201, or 202:

*****|user|PATCH|/api/docker/docker-local/v2/Path/To/Artifact/111111111111111111|202|xxxxxxx|0|xxx|xxxxx

First we need to confirm if the issue is actually from the Nginx,by bypassing Nginx and uploading the same file directly through Artifactory’s node IP.
If this works, we can conclude that the issue is happening at Nginx’s end.

As a next step, we need to look into the logs and configuration of Nginx. 

Parsing Nginx’s error.log:

In the error.log, you may find error messages below:

2023/01/28 18:06:10 [crit] #####: *2369986 pwrite() "/var/cache/nginx/client_temp/0000001111" failed (28: No space left on device), client:****, server: ******, request: "PATCH /v2/ocker-local/Path/To/Artifact HTTP/1.1", host: "******"

2023/01/28 18:06:10 [crit] #####: *2369986 pwritev() "/var/cache/nginx/client_temp/0000001111" has written only 3737 of 8184, client: ****, server: ****, request: "PATCH /v2/docker-local/Path/To/Artifact HTTP/1.1", host: "****"

This error indicates that there is no space left on your device. 
Based on the above error, we can understand that Nginx has the buffering feature enabled. As this error indicates that there is no space left for the caching process. 

There are 2 approaches to fix the issue:

Check the available disk space:

Navigate to the path that appears in the error message (/var/cache/nginx/client_temp) and confirm there’s no left available storage.  If it is a possible option, increase the storage.  

Disable buffering feature:

The buffering feature means that every coming request is being read before it is passed to the proxied server (Artifactory). 
Up-to-date versions of Nginx have proxy_request_buffering enabled by default. 
Therefore, we might observe 500 internal errors while having this setting. 

So what is happening is then: 
While uploading large files, firstly the entire body request is being read and buffered, and since there’s not enough space it failed.  

Resolution:

In such a case, and based on our Nginx wiki page, it is recommended to turn off the request buffering option and enable the HTTP/1.1.
In order to apply the mentioned and required changes, the Nginx configuration should include the values below:

proxy_request_buffering off;
proxy_http_version 1.1;

Lastly, I’d like to share the recommended Nginx configuration for your reference to review.

** In this article we are showing an example of an issue with lack of space on the device. Please note that the 500 internal error could occur due to other reasons that lead to observing it which requires resolving it in different ways.