Resolution

On the Artifactory server, set these environment variables:

DNAME="cn=App, ou=Java, o=Zabbix, c=LV"
DAYS=3650
PASSWORD=changeit
CACERTS="/etc/ssl/certs/java/cacerts" #Used to create a new keystore

First, create the keystore and truststore for the application, Apache Tomcat in this case.

Use the $CACERTS variable as the basis for the new keystore and truststore. This is so all public CAs are used as well as the new certificates. Make sure to set the variable to an existing cacerts file, the above path is an example.

Generate the new keystore using these commands:

#Initialize the keystore
keytool -genkey -alias tomcat -keyalg RSA -validity ${DAYS} -keystore tomcat.keystore -storepass ${PASSWORD} -keypass ${PASSWORD} -dname "${DNAME}"

#Update the Truststore with the new Tomcat Keystore
cp ${CACERTS} tomcat.truststore

keytool -storepasswd -keystore tomcat.truststore -storepass changeit -new ${PASSWORD}

keytool -genkey -alias tomcat -keyalg RSA -validity ${DAYS} -keystore tomcat.truststore -storepass ${PASSWORD} -keypass ${PASSWORD} -dname "${DNAME}"

On the the JMX client, such as jconsole, do the same steps to create a new keystore:

DNAME="cn=App, ou=Java, o=Zabbix, c=LV"
DAYS=3650
PASSWORD=changeit
CACERTS="/etc/ssl/certs/java/cacerts"

keytool -genkey -alias jconsole -keyalg RSA -validity ${DAYS} -keystore jconsole.keystore -storepass ${PASSWORD} -keypass ${PASSWORD} -dname "${DNAME}"

cp ${CACERTS} jconsole.truststore

keytool -storepasswd -keystore jconsole.truststore -storepass changeit -new ${PASSWORD}

keytool -genkey -alias jconsole -keyalg RSA -validity ${DAYS} -keystore jconsole.truststore -storepass ${PASSWORD} -keypass ${PASSWORD} -dname "${DNAME}"

Then, export the public certificates from the keystores:

keytool -export -alias tomcat -keystore tomcat.keystore -file tomcat.cer -storepass ${PASSWORD}

keytool -export -alias jconsole -keystore jconsole.keystore -file jconsole.cer -storepass ${PASSWORD}

Finally, import the certificates into each other's truststores.This allows the application (Tomcat) to trust the client (JConsole), and vice versa:

keytool -import -alias jconsole -file jconsole.cer -keystore tomcat.truststore -storepass ${PASSWORD} -noprompt

keytool -import -alias tomcat -file tomcat.cer -keystore jconsole.truststore -storepass ${PASSWORD} -noprompt

#Get rid of any remaining CER certificate files
rm -f *.cer

On the Artifactory instance, add the JMX Remote properties to the artifactory.system.properties file:

vim /var/opt/jfrog/artifactory/etc/artifactory/artifactory.system.properties

com.sun.management.jmxremote=true
com.sun.management.jmxremote.port=9010
com.sun.management.jmxremote.authenticate=true
com.sun.management.jmxremote.password.file=/etc/opt/jmxremote.password
com.sun.management.jmxremote.access.file=/etc/opt/jmxremote.access
com.sun.management.jmxremote.ssl=false
javax.net.ssl.keyStore=/etc/opt/tomcat.keystore
javax.net.ssl.keyStorePassword=changeit
javax.net.ssl.trustStore=/etc/opt/tomcat.truststore
javax.net.ssl.trustStorePassword=changeit
com.sun.management.jmxremote.ssl.need.client.auth=true

#Restart the application
$ systemctl restart artifactory

On Zabbix Java Gateway (Exact steps will differ depending on the JMX monitoring application):

$ vim /usr/sbin/zabbix_java_gateway

ZABBIX_OPTIONS="$ZABBIX_OPTIONS -Dsun.rmi.transport.tcp.responseTimeout=$tcp_timeout -Djavax.net.ssl.keyStore=/etc/opt/jconsole.keystore -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=/etc/opt/jconsole.truststore -Djavax.net.ssl.trustStorePassword=changeit"

$ systemctl restart zabbix-java-gateway # Verify in the UI if it works