This guide details how to collect diagnostics for JFrog Artifactory's Go-based microservices: Router, Metadata, Event, and Observability.
Understanding Goroutine Dumps
A goroutine dump is the Go runtime’s equivalent of a Java thread dump, tailored to the Go concurrency model. It provides a snapshot of all goroutines at a specific moment, including:
- Goroutine ID and execution state.
- Full stack trace.
- Blocking reasons (e.g., waiting on a lock, channel, or syscall).
Enabling Profiling
Profiling must be enabled in the ‘system.yaml’ file to expose the debug endpoints.
1. Open ‘$JFROG_HOME/artifactory/var/etc/system.yaml’.
2. Add the following configuration to the respective service blocks:
metadata:
profiling:
enabled: true
router:
profiling:
enabled: true
event:
profiling:
enabled: true
observability:
profiling:
enabled: true
3. Restart Artifactory for the changes to take effect.
Note:
For security, the profiling endpoints are only accessible via ‘localhost’.
Collecting Goroutine Dumps
To diagnose issues like stuck processes or high concurrency, capture multiple snapshots (e.g., every 5–30 seconds).
Run the following commands on the Artifactory host:
Router
curl -vv "http://localhost:8049/debug/pprof/goroutine?debug=2" -o router_goroutine_dump.txt
Metadata
curl -vv "http://localhost:8046/metadata/debug/pprof/goroutine?debug=2" -o metadata_goroutine_dump.txt
Event
curl -vv "http://localhost:8046/event/debug/pprof/goroutine?debug=2" -o event_goroutine_dump.txt
Observability
curl -vv "http://localhost:8046/observability/debug/pprof/goroutine?debug=2" -o obs_goroutine_dump.txt
Collecting Machine-Readable Profiles
In addition to text dumps, you can collect binary profiles (CPU, Heap, Allocation) to be analyzed using the ‘go tool pprof <file>’ command.
Replace ‘<seconds>’ with your desired sample window (e.g., ‘30’).
Router
curl -vv "http://localhost:8049/debug/pprof/profile?seconds=30" -o router_cpu.prof
curl -vv "http://localhost:8049/debug/pprof/heap" -o router_heap.prof
curl -vv "http://localhost:8049/debug/pprof/allocs" -o router_allocs.prof
Metadata
curl -vv "http://localhost:8046/metadata/debug/pprof/profile?seconds=30" -o metadata_cpu.prof
curl -vv "http://localhost:8046/metadata/debug/pprof/heap" -o metadata_heap.prof
curl -vv "http://localhost:8046/metadata/debug/pprof/allocs" -o metadata_allocs.prof
Event
curl -vv "http://localhost:8046/event/debug/pprof/profile?seconds=30" -o event_cpu.prof
curl -vv "http://localhost:8046/event/debug/pprof/heap" -o event_heap.prof
curl -vv "http://localhost:8046/event/debug/pprof/allocs" -o event_allocs.prof
Observability
curl -vv "http://localhost:8046/observability/debug/pprof/profile?seconds=30" -o obs_cpu.prof
curl -vv "http://localhost:8046/observability/debug/pprof/heap" -o obs_heap.prof
curl -vv "http://localhost:8046/observability/debug/pprof/allocs" -o obs_allocs.prof
Collecting Execution Traces
The Go Execution Trace is the most detailed diagnostic tool available. It visualizes scheduler events, GC pauses, and Goroutine scheduling latency.
Warning:
Execution tracing has a high performance overhead. It captures every scheduler event and syscall. Keep trace durations short (under 10 seconds) to avoid impacting production traffic.
Router
curl -vv "http://localhost:8049/debug/pprof/trace?seconds=5" -o router_trace.out
Metadata
curl -vv "http://localhost:8046/metadata/debug/pprof/trace?seconds=5" -o metadata_trace.out
Event
curl -vv "http://localhost:8046/event/debug/pprof/trace?seconds=5" -o event_trace.out
Observability
curl -vv "http://localhost:8046/observability/debug/pprof/trace?seconds=5" -o obs_trace.out
To view the trace, use the command: ‘go tool trace <trace.out>’ (requires a local Go installation).
Related Articles:
XRAY: How to Collect Xray Thread Dumps
ARTIFACTORY: How to collect Artifactory thread dumps