Overview

Exinda's firmware, as all programs, requires the ability to parse text, i.e., to read through a text file and find the information necessary for its operations (stripping out what is not required or needed and discarding it). Occasionally, the parsing mechanism in the Exinda fails, which may lead to performance issues and ultimately, cause the appliance to reboot.

When dealing with this problem, the following messages will appear in the logs:

Mar 9 09:38:55 exinda kernel: [494.714272] Command error before '(null)' (line=1981 field=4): <28> Command is truncated
Mar 9 09:39:06 exinda kernel: [505.374703] Unable to find command for 'ad <undecipherable> line=2527 field=0)
Mar 9 09:39:06 exinda kernel: [505.470005] Command error before '(null)' (line=970 field=3): <28> Command is truncated

Where the number in brackets, e.g., [494.714272], is the number of seconds since the last power on, and the undecipherable characters can appear as a mixture of anything, i.e., characters, symbols, question marks, etc.

Additionally, the following  trace will appear in a kdump:

[ 4648.627863] Unable to find command for 'a \xffffff90\xffffffd7\xffffff88\xffffffff\xffffffffX\xffffffdb\xffffff90\xffffffd7\xffffff88\xffffffff\xffffffff' (line=1147 field=0)
[23147.409005] Command error before '(null)' (line=2139 field=3): <22> Failed to parse IPv4 address with mask
[23750.249561] Command error before '(null)' (line=1159 field=3): <28> Command is truncated
[43081.553456] Unable to find command for 'ad +\x04\xffffff88\xffffffff\xffffffff' (line=2353 field=0)

If the device reboots, the trace in the kdump is:

[57985.651444] Call Trace:
[57985.654646] [<ffffffffa01e8d80>] ? cmd_parser_write+0x140/0x230 [proc_utils]
[57985.663092] [<ffffffffa01e8d80>] cmd_parser_write+0x140/0x230 [proc_utils]
[57985.671347] [<ffffffff811c5a13>] proc_reg_write+0x43/0x70
[57985.677946] [<ffffffff8116ae3f>] vfs_write+0xdf/0x1c0
[57985.684158] [<ffffffff8116b02f>] SyS_write+0x5f/0xa0
[57985.690272] [<ffffffff816aaaf9>] system_call_fastpath+0x16/0x1b

Note: it is possible that the call trace could include cmd_parse at the top, ahead of cmd_parser_write.

Root Cause

In the background, the parser is working to obtain the information it needs to operate and perform its functions from the input information. If the Exinda device cannot parse the information correctly, regardless of the reason, it goes into an errored state. Although the device tries to recover by itself, sometimes it is not possible. If system resources are already allocated and expecting a result from what has failed to be parsed, it will be empty or null. If that memory space is accessed or attempted to be deallocated, it can run into a fatal error, which will cause the device to reboot.

Resolution

In ExOS7.4.2, this bug has been fixed. Upgrading to ExOS7.4.2 will prevent this from occurring.

It is believed that the ad command that it is trying to parse affects the Active Directory (AD) Integration feature of the appliance. If you are running into problems with the AD Connector on the ExOS7.4.1 firmware, and these errors are observed, upgrade to ExOS7.4.2 to rule out this as the root cause of those problems. 

If the issue persists occurs in ExOS7.4.2, contact Exinda Support for further troubleshooting.