ARTIFACTORY: How to recover from “No space left on device” error from Artifactory pods

In a few scenarios, we have seen the storage of the Artifactory PVC is fully occupied and due to which the Artifactory pod will go to crashloop state. In this situation either we might need to increase the storage size of the PVC or we need to access the PVC to which the Artifactory pod is attached to and remove the contents which are not required..

Below steps are the detailed instructions for creating a new helper pod with the problematic mount attached to get some space. 

Step 1: Firstly, it is required to scale down the statefulset of all the Artifactory, so that all the pods will be terminated. The reason for this is because by default the PVC’s will be with RWO Access modes so that only one pod can be attached to the PVC which is having high disk utilization.

Step 2: Next, we would need to create a helper pod by using the claim name of the PVC, so that on applying this, we can see a new pod attached with the Artifactory pvc which can help to access the new pod and access the content
 

apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: volume-afazure-artifactory-ha-primary-0 
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/tmp/"
          name: task-pv-storage

Step 3: To apply the above yaml, use the command kubectl apply -f pv-pod.yaml in the same namespace where Artifactory is installed.

Once after creation, we will see a pod like below when performing kubectl get pods -n <namespace>

task-pv-pod                            1/1     Running   0          19h

Step 4: Exec into the new pod "task-pv-pod" and go to "/tmp" directory to see which path is taking so much usage(du -sh *).

$ kubectl exec -it task-pv-pod sh

Since we have used the mountPath to point to /tmp on the above mentioned yaml configuration file, navigate to /tmp folder to access the contents stored on the PVC.

Step 5: Remove unnecessary files and delete the temporary "task-pv-pod" after the process

Step 6: Finally scale up the statefulset of Artifactory to initialize the Artifactory pods.

Note: Since we are directly performing a deletion on the PVC, it is recommended to perform the above steps on a lower environment prior to performing on the production server.