When installing Xray for the first time, there is an initial DB sync that downloads and stores all the public data that is known to Xray. The data includes information on vulnerabilities, licenses, and third-party components.
This article will cover potential leads which might help you to solve common issues with the Xray database sync.
The database sync involves 2 of the Xray services:
- Server - This service is responsible for downloading and monitoring the DB sync progress.
- Persist - After the vulnerabilities and components have been downloaded locally by the server, the persist service is responsible for extracting and inserting them to the database.
The DB sync process
Downloading the update files. The files will be downloaded to $JFROG_HOME/var/work/server/updates, where the data will be saved in two folders: components, and vulnerabilities.
Once Xray has finished downloading all update files, it will start extracting each archive, which contains many json files, each containing lots of information. For each archive it has finished extracting, Xray will rename the file and add a '.Completed' suffix - <archive-name>.Completed.
Once the '.Completed' suffix is present, Xray will start to perform an Impact analysis process on the extracted information (this is done after the DB sync).
Once Impact analysis is completed, Xray will remove the '.Completed' files.
Note
In case of HA, one of the nodes will be chosen randomly to perform the DB sync.
The Xray DB sync is a database IOPS-intensive operation. This is because we write, update, and delete information in the database, which is why it is important to assure that the Xray DB is meeting the Xray system requirements.
Troubleshooting why the Xray Database sync doesn’t work
For these kinds of issues, there are several leads we should check:
- Proxy or a firewall - When there is a proxy such as a default system proxy configured on the JFrog platform, any outgoing request as part of the DB sync will be sent through this proxy. Therefore, it’s important to check that the proxy can serve the files and wouldn't block the requests. Here are the URLs which are required to be whitelisted on your proxy or firewall:
- https://releases.jfrog.io/
- https://releases-cdn.jfrog.io
- https://jxray.jfrog.io
- Expired trial license - An expired trial license in Xray might interfere with the Xray DB sync. Please refer to this article to check if that’s the case and for instructions on how to remove it.
- Expired JFrog license - In case this is not a trial license, it is possible that your JFrog platform is using an expired license or a license that doesn’t include Xray. If you suspect that this is the case, I suggest checking the license information using this REST API and check if it’s valid. The DB sync won't work with an expired license, and an error of such can be seen in the xray-server-service.log where the JXray server is failing to validate your license:
[2020/10/06 11:25:38 CEST] [EROR] (jfrog.com/xray/internal/jobs.DownloadOnlineUpdates:584) Updates worker id 0 failed to download update from https://jxray.jfrog.io/api/v1/updates/bundles?from=1593080483014&to=1601976328671: Failed to access :https://jxray.jfrog.io/api/v1/updates/bundles?from=1593080483014&to=1601976328671 return status code : 401
- Storage - Make sure Xray has enough storage to download all the files related to the DB sync. The recommendation is to have at least 300 GB of free space for the DB sync.
- Check Xray services connectivity - If one of the services is down, then it’s probably not a DB sync issue, but rather a wider one. HA node connectivity might also not be working properly. One way to monitor Xray is by using the UI Service status page (under Administration tab -> Monitoring -> Service Status) and checking that all services are healthy. Another way is to use a combination of the metrics and the Xray Router health check API.
- Issue: DB Sync Fails if Artifacts are Indexed Before Initial Sync
In some cases, if users start indexing repositories in Xray before the initial, first-time DB sync has completed, the system can get into a state where the DB sync process crashes or times out. This happens because the indexing workers and the DB sync workers compete for resources. To overcome this, you need to temporarily stop the indexing workers to allow the DB sync to complete successfully.
To resolve this, follow these steps:
Decrease the number of indexer workers to zero. This prevents Xray from trying to index new content while the DB sync is running. Run the following REST API call:
curl -u <USERNAME>:<PASSWORD> -X PUT '<ARTIFACTORY_URL>/xray/api/v1/configuration/workersCount' \
-H 'Content-Type: application/json' \
-d '{
"index": {
"new_content": 0,
"existing_content": 0
},
"persist": { "new_content": 8, "existing_content": 4 },
"analysis": { "new_content": 8, "existing_content": 4 },
"policy_enforcer": { "new_content": 8, "existing_content": 8 },
"sbom": { "new_content": 8, "existing_content": 4 },
"usercatalog": { "new_content": 8, "existing_content": 4 },
"sbomimpactanalysis": { "new_content": 8, "existing_content": 4 },
"migrationsbom": { "new_content": 4, "existing_content": 0 },
"impact_analysis": { "new_content": 8 },
"notification": { "new_content": 8 },
"panoramic": { "new_content": 8 }
}'
- Temporarily disable Impact Analysis. This further reduces the load during the DB sync. Add the following configuration to your $JFROG_HOME/xray/var/etc/system.yaml file:
server:
dbSync:
impactAnalysis:
enabled: false
- Restart Xray and start the DB sync.
- Important: Once the DB sync is complete, do not forget to revert the changes. Re-run the Indexer workersCount API call with the original values
and remove the impactAnalysis: enabled: false block from your system.yaml, then restart Xray again.
curl -u <USERNAME>:<PASSWORD> -X PUT '<ARTIFACTORY_URL>/xray/api/v1/configuration/workersCount' \
-H 'Content-Type: application/json' \
-d '{
"index": {
"new_content": 8,
"existing_content": 4
},
"persist": {
"new_content": 8,
"existing_content": 4
},
"analysis": {
"new_content": 8,
"existing_content": 4
},
"policy_enforcer": {
"new_content": 8,
"existing_content": 8
},
"sbom": {
"new_content": 8,
"existing_content": 4
},
"usercatalog": {
"new_content": 8,
"existing_content": 4
},
"sbomimpactanalysis": {
"new_content": 8,
"existing_content": 4
},
"migrationsbom": {
"new_content": 4,
"existing_content": 0
},
"impact_analysis": {
"new_content": 8
},
"notification": {
"new_content": 8
},
"panoramic": {
"new_content": 8
}
}'
Troubleshooting why the Xray Database sync is slow
As previously mentioned, the DB sync performance is mostly coupled with your database IOPS, which is why it’s important to meet the system requirements.
- Database connections - By default, Xray requires the following amount of DB connections:
- For Xray server service - 65 connections (60 active connection + 5 idle)
- Indexer, persist and analyze each require 35 connections (30 active + 5 idle). If we combine all three, a total of 105 connections.
- This means that a single Xray node requires around a total of 170 DB connections. For HA we may need to multiply the above with the number of nodes.
- As a formula, we summarize it as follows: Total # of connections = (number of nodes) * (maxOpenConnServer + maxOpenConnPersist + maxOpenConnAnalysis + maxOpenConnIndexer) + 50;
In case of any other issue with the DB sync, we may want to increase the log level for the Server and Persist services to DEBUG through the system.yaml.
For example:
server:
logging:
application:
level: debug
See relevant documentation for additional information.
Note
In case you are using an old version of Xray, we recommend upgrading Xray to version 3.38.2 or above since these versions contain major performance improvements for the DB sync process.