Troubleshooting and Debugging
This article covers some common tips and tricks for debugging and troubleshooting GridGain and Ignite deployments.
Debugging Tools: Consistency Check Command
./control.sh|bat utility includes a set of consistency check commands that help with verifying internal data consistency invariants.
Persistence Files Disappear on Restart
On some systems, the default location for Ignite persistence files might be under a
temp folder. This can lead to situations when persistence files are removed by an operating system whenever a node process is restarted. To avoid this:
WARNlogging level is enabled for GridGain. You will see a warning if the persistence files are written to the temporary directory.
Change the location of all persistence files using the
DataStorageConfigurationAPIs, such as
Cluster Doesn’t Start After Field Type Changes
When developing your application, you may need to change the type of a custom
object’s field. For instance, let’s say you have object
A with field
int type and then you decide to change the type of
long right in
the source code. When you do this, the cluster or the application will fail to
restart because GridGain doesn’t support field/column type changes.
You can use the experimental
meta command to remove metadata that normally stores type affinity from the cluster. If you are not sure what to remove specifically, use the
list subcommand to list all metadata types. You can specify the type to remove by ID or by name, and also specify the output folder to store a backup in. After you do so, GridGain will treat the column as a new type and continue to operate normally.
control.sh --meta remove [--typeId <typeId>] [--typeName <typeName>] [--out <fileName>]
control.bat --meta remove [--typeId <typeId>] [--typeName <typeName>] [--out <fileName>]
meta command is not intended for normal cluster operation and the user is responsible for fulfilling conditions for proper execution:
Data of the removed type must not be stored in caches;
No other operations should be performed with the type. For example, you should not delete metadata while creating a new object.
You can also, in development, go into the
file system and remove the following directories:
located in the GridGain working directory (
wal might be located in other
places if you have redefined their location). This achieves a similar result to performing a
meta command, but is less targeted.
In production, we still recommend adding a
new field with a different name to your object model and removing the old one. This operation is fully
supported. At the same time, the
ALTER TABLE command can be used to add new
columns or remove existing ones at run time.
Debugging GC Issues
The section contains information that may be helpful when you need to debug and troubleshoot issues related to Java heap usage or GC pauses.
You can configure JVM to dump the heap automatically when the
OutOfMemoryException exception occurs.
This helps if the root cause of this exception is not clear as the dump provides a deeper look at the heap state at the moment of failure:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump -XX:+ExitOnOutOfMemoryError
Detailed GC Logs
In order to capture detailed information about GC related activities, make sure you have the settings below configured in the JVM settings of your cluster nodes:
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Xloggc:/path/to/gc/logs/log.txt
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -Xlog:gc:/path/to/gc/logs/log.txt -XX:+PrintFlagsFinal -XX:+UnlockDiagnosticVMOptions -XX:LogFile=/path/to/gc/logs/safepoint.log -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1
/path/to/gc/logs/ with an actual path on your file system.
In addition, for G1 collector set the property below. It provides many additional details that are
purposefully not included in the
Performance Analysis With Flight Recorder
In cases when you need to debug performance or memory issues you can use Java Flight Recorder to continuously collect low level runtime statistics, enabling after-the-fact incident analysis. To enable Java Flight Recorder use the following settings:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints
To start recording the state on a particular GridGain node use the following command:
jcmd <PID> JFR.start name=<recordcing_name> duration=60s filename=/var/recording/recording.jfr settings=profile
For Flight Recorder related details refer to Oracle’s official documentation.
Occasionally you may see an warning message about the JVM being paused for too long. It can happen during bulk loading, for example.
IGNITE_JVM_PAUSE_DETECTOR_THRESHOLD timeout setting may give the process time to finish without generating the warning. You can set the threshold via an environment variable, or pass it as a JVM argument (
-DIGNITE_JVM_PAUSE_DETECTOR_THRESHOLD=5000) or as a parameter to ignite.sh (
The value is in milliseconds.