XRAY: What to Do If Xray Stops Indexing Artifacts

Relevant versions: This information pertains to Xray versions 2.x.

To troubleshoot, analyze, and understand the indexing process in Xray, we highly recommend that you begin by opening your Xray UI and navigating to the Admin tab > System monitoring > System messages. These sections may give you a useful indication as to whether something has gone wrong with any of Xray's services (e.g., if requests have failed, if your disk space is running low, etc.). You can also check to see if all of Xray's services are up and running by using the xray.sh script by adding the status all or ps flags to it:
 

$ xray.sh status all
$ xray.sh ps

Note: Which of the two you'll need to use will depend upon the Xray installation type you're running.

If you identify an issue with any Xray service, check your Xray server.log to see what caused the issue. If, however, Xray is in a healthy state, with all of its services up and running, you may need to dig into your RabbitMQ queues to determine what's causing Xray to cease indexing artifacts. RabbitMQ stores and manages all of the messages generated from Xray queues. These include events, artifact indexing, persisting and analyzing messages. Events can also be events that are being sent from Artifactory instances.

To check your RabbitMQ queues, do the following:

1. Access the queues through the RabbitMQ console using: https://localhost:15672/#/queues. If you're unable to access the RabbitMQ UI, try creating an SSH tunnel using: 

   ssh -L15672:127.0.0.1:15672 root@<machine ip>

2. Check your RabbitMQ event, index, and persist queues for messages (refer to the example screenshot below). When you're connected, it will allow you to understand if you have a bottleneck in one of your services or if you have any operation failure messages. According to the queue details, you will be able to narrow down the search to the relevant log and will have better understanding in which part of the process Xray failed.

Finally, you can investigate problems that may be occurring with Xray indexing from the Artifactory side by adding the following loggers to your ARTIFACTORY_HOME/etc/logback.xml file:

<appender name="xray" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${artifactory.home}/logs/xray.log</File>
<encoder>
<pattern>%date ${artifactory.contextId}[%thread] [%-5p] (%-20c{3}:%L) – %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<FileNamePattern>${artifactory.home}/logs/xray.%i.log</FileNamePattern>
<maxIndex>13</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>10MB</MaxFileSize>
</triggeringPolicy>
</appender>

<appender name="xrayDao" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${artifactory.home}/logs/xrayDao.log</File>
<encoder>
<pattern>%date ${artifactory.contextId}[%thread] [%-5p] (%-20c{3}:%L) – %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<FileNamePattern>${artifactory.home}/logs/xrayDao.%i.log</FileNamePattern>
<maxIndex>13</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>10MB</MaxFileSize>
</triggeringPolicy>
</appender>

<logger name="org.artifactory.addon.xray">
<level value="trace" />
<appender-ref ref="xray"/>
</logger>

<logger name="org.artifactory.storage.db.xray.dao">
<level value="trace" />
<appender-ref ref="xrayDao"/>
</logger>

*no restart is required for the loggers to apply.

Among the several triggers that can cause Xray to stop indexing artifacts, a common occurrence is when Xray has reached the configured limit for disk usage (default=80%). To resolve this issue, you'll need to increase that limit by increasing the maxDiskDataUsage value in Xray's xray_config.yaml configuration file.

If nothing above has been helpful to you, contact JFrog Support for further assistance.