Overview

This article provides details on how to Generate and Analyze Diagnostics File, as well as the definitions from each item involved in this process for quick reference.

Information

 Quick Steps

1. Navigate to Configuration > System > Diagnostics, "Diagnostics" tab.
2. Click on "Generate Diagnostics"
   
3. Download it when finished.

 Alternative (CLI):

1. In CLI, type:
   

   debug generate dump

2. FTP the file once generated.

A diagnostic file is a file that contains all the basic information about the system all packaged up into one zipped file structure. These files contain all pertinent information for diagnostic purposes, related to processes, general system behavior, configuration, and collection statistics possible, outside of the database. When troubleshooting an issue, looking at basic files in the diagnostic is a good place to start. Troubleshooting specific process problems can also use the more specific log files in the diagnostic in specific folders. Diagnostics can be generated ad-hoc by using either the web UI or CLI or can be generated automatically depending on process crashes or kernel crashes.

The file is in .tgz format, and by default is generated with the name of sysdump-[hostname]-[timestamp] for cases of kernel crashes or manual generation. If a process crashes and triggers a diagnostic, it is generated with the name sysdump-[hostname]-[process]-[timestamp].

Generation

Generating through the web UI can be done through the Configuration > System > Diagnostics, page, under the "Diagnostics" tab. This page will also list all diagnostics located on the Exinda that can be downloaded. Files that appear on the Diagnostic web UI page are located in /var/opt/tms/sysdumps. (As a side note, any file located in /var/opt/sysdumps will be able to be downloaded, regardless of the type of file or its purpose. That can be used to easily grab other files for required diagnostics if FTP is not available).

Generating a sysdump through the CLI can be done from any elevation and can be done via the command:

debug generate dump 

This places the sysdump in the same folder as where the web UI generates them.

Extracting the .tgz allows for viewing individual files and folders. Each primary process has its folder, and in the root are a set of miscellaneous diagnostics that are specific to general system behavior. The Diagnostic Viewer can be used to view everything in a single location, nicely laid out, but it does not offer search functionality like extracting the files and putting them into Notepad++ would.

Common Folders/Files To Examine

The following list is a set of the most common files and their contents.

• kdump - this folder will contain the contents of kernel dumps that are generated if there is a kernel panic or crash. This is one of the first places to look if there is an unexpected shutdown. A kernel panic triggers the kernel dump to work, where it copies the kernel to a reserved portion of memory to cleanly record the last information possible before the device reboots. The files contained in the folder are a kernel-level log of everything from time of boot to time of the crash, including memory dumps, CPU dumps, exceptions thrown, etc.
• jboss - this folder contains logs, errors, and other information regarding the operation of the web UI through jboss. The folder includes pdf_error (errors rendering graphs or PDF report requests) and server logs (showing requests for pages done by whichever hosts).
• optimizerd - this folder contains information regarding the optimizer and underlying Traffic Control statistics. It is an excellent place to look if standard pieces of information are not enough to diagnose a connectivity or throughput issue. This includes a copy of:
• tcdiag (an easily readable copy of EXTC)
• TC (real-time information about the TC tree at that moment in time)
• iptables
• tcpad - this folder contains logs and information about TCP acceleration that is going through the 'tcpad' process. This will keep a detailed list of every connection being accelerated, the type of traffic, source, destination, protocol, even the SMB path if supported.
• wmd - this folder contains logs and information about TCP acceleration that is being put into WAN Memory. This has information about the connections that are using WAN memory but has 'no information' about the specific files located in WAN memory itself.
• web_cache - this folder contains logs relating specifically to squid, consisting of three files
• access.log (a list of HIT/MISS for every request). 
• cache.log (internal logging of the squid process, including information relating to process crashes)
• rewrite.log (shows errors relating to writing files to squid). This is a good place to look for squid crashes and problems. General squid information is copied from the shell command:
  

  squidclient mgr:info

• dmesg - a file that is part of kernel logging. This shows all kernel-level messages that are logged, along with a timestamp of 'seconds since last boot'. These messages can be anything from the conntrack notifying that it is full (no new connections can be established), blocked processes, NIC ports going up and down, CPU dumps, memory dumps, kernel panics, etc. dmesg is wiped clean on a reboot of the device. If a device is having trouble, attempt to get a diagnostic before rebooting the system, otherwise dmesg will be empty, and a lot of kernel-level info will be lost.
• dmesg.boot - a portion of dmesg that shows only the last boot process of the system. While the normal dmesg will continue with all events, dmesg.boot stops after the boot process is complete.
• messages - the most critical file. This is the output of /var/log/messages, where every process in the Exinda process writes to. Kernel level messages are also written to messages. The messages file gives a good overall look at what is going on at the system in any given time. The timestamps in the log are sorted by local Exinda time (though certain messages taken from kernel-level processes might include the second timestamp in a different format, such as epoch time, time since boot or GMT).
• nf_conntrack - the 'connection tracker' that is located in RAM and written to the hard drive. This keeps track of every single connection going through the Exinda. It tracks the source/destination IPs, application ID, and the status of the TCP connection, as well as a timeout time. The time starts at 1 hour (3600 seconds) and counts down. If a connection reaches a time of 0 without any new traffic, it is dropped from the conntrack. If new traffic comes in, the timer is reset to 3600.
• running-config - If this is populated, it will show the full output of the 'show run' command and will list the entire running configuration of the device. This will occasionally time out and will show an error message instead of the system is under too much stress.
• sysinfo - A file that contains many basic traits of the system - shows hardware information (NIC stats using ethtool, process lists using ps, hard drive space using df, memory using free), software information (IP configuration, listening ports, license configuration, IP routes, netstat), a list of diagnostics that exist, a list of current logs. This shows a general system overview in both a hardware and software sense that can reveal basic information.
• systemlog - a file that shows permanently recorded information in the system with timestamps. Very few types of events go in the systemlog:
  • NTP time changes
  • First boots of new firmware
  • Installation of restricted licenses
  • When entering the shell

While there are many other files to work with, depending on the specific situation, the above are commonly used.