ARTIFACTORY: How to identify the root cause of high disk I/O and throughput that caused failed download with closed connections

Introduction 

High disk I/O and throughput on local disk can create significant impact on artifactory performance which may result in download failure with closed connections or even system crash.
Even with direct upload binary providers, artifactory do still have load on local disk because of cache-fs/logs and other tmp files. 
Additionally, applications like reverse proxy that run on the same vm as artifactory might also create high disk I/O when they are not configured correctly. 
Here we will show some common causes and ways to identify the issues. 


Identified the root cause and potential fix

Identify the Culprit Process
The fastest way to see "who is doing it" is iotop. It works exactly like the standard top command but for disk usage.

Command:

sudo iotop -oPa

-o: Only show processes actually doing I/O (hides idle ones).
-P: Show processes instead of individual threads (easier to read).
-a: Accumulate I/O (useful to catch processes that write in short, fast bursts).

Look at the DISK WRITE and DISK READ columns. The top entry is your culprit. Note the PID and the Command name.
  If iotop change too fast, try pidstat:

# Print IO stats every 1 second, 10 times
pidstat -d 1 10

Identify what is getting read/write. Once you have the PID from iotop (e.g., PID 1234), you need to know what file it is writing to. Is it a log file? A temp file?

# List open files for a specific PID, looking for regular files (REG)
sudo lsof -p 1234 | grep REG


Look at the file paths on the right.
If you see a log file (e.g., catalina.out or access.log) growing rapidly, you have a "log spam" issue.

```
Deep Dive: "Why are you writing?"
```

If lsof shows the files but you don't understand why the process is touching them so aggressively, you can snoop on the system calls.
Command: (Warning: This slows down the process slightly. Use for short bursts, e.g., 5 seconds).

# Trace write operations for the PID
sudo strace -f -e trace=write -p 1234


You will see a stream of data being written.
write(3, "Error: Connection refused...", 28) -> The app is in an error loop filling up logs.
write(5, <binary data>, 4096) -> It is dumping binary data (likely database or file copy).

Rule out Hardware Failure
Sometimes high "throughput" is actually retries. If a disk sector is bad, the drive controller might try to read it 50 times in a row, which looks like high activity but zero actual data transfer.

dmesg -T | grep -iE 'disk|scsi|error' | tail -n 20

Look for: "I/O error", "sector", or "reset". If you see these, your physical disk (or SAN volume) is failing.

Based on the above output, we might identify a few common heavy i/o:

Nginx that is on the same vm as artifactory or other log monitoring tools

```
Heavy write to log files
```
```
Heavy write to eventual folder
```
```
Heavy write to cache folder
```

For case 1, you will want to check your nginx config and it’s out of the scope for this articles

For case 2, check if you have enabled a very verbose logger and turn it off if not needed.

For case 3, eventual binary provider is no longer recommended. Consider switching to a direct provider. https://jfrog.com/help/r/artifactory-how-to-migrate-from-eventual-to-direct-upload-mechanism/artifactory-how-to-migrate-from-eventual-to-direct-upload-mechanism

For case 4, The direct upload provider still utilizes write to disk during put request because of cache-fs. Artifactory will save stream to cache _pre, validate integrity and copy to cache folder so that the upload is immediately available locally.

If you have already tuned up the cache fs size to a reasonably large number and your disk i/o or throughput limit is already at a high level that can’t be lifted, you might consider turning off this feature.
You can disable this by setting skipDuringUpload to true for cache-fs provider in binarystore.xml