Exinda Scheduled/PDF Reports Stuck Loading and Diagnostics “Internal error, code 1003” (Root Cause: <code>/var</code> Disk Full)

Solution

Error / Symptoms

You may see one or more of the following:

• Scheduled report download never completes (stuck loading); the browser may show continuous polling requests like status?pollWait... returning HTTP 200 but no file is delivered (often accompanied by a broken file icon in the UI).
• Diagnostics generation/download fails in the GUI with “Internal error, code 1003 (see logs for details)”.
• Diagnostics UI shows “No Diagnostics Files.” and/or “No System Snapshot Files.”
• Dashboard/alarms show “System Disk Full - Full (or almost full) disk partitions: /var” (TRIGGERED).
• Dashboard shows “Database Status: Error (see logging)”.

Environment

• Product: Exinda
• Version observed: v7.5.3 (0012) (issue is not necessarily version-specific)

Root cause

Scheduled/PDF report exports and diagnostic archives require free space on the OS filesystem /var for temporary/output files and indexing.

If /var is critically full (commonly due to large log growth under /var/log/datad, or accumulated dumps/snapshots/tmp files), Exinda may:

• Fail to generate diagnostics in the GUI (code 1003).
• Hang scheduled/PDF report generation (the front-end polls indefinitely).
• Print “Generated dump … .tgz” from CLI but not actually produce a .tgz file (or fail to list it in the GUI).

Important: The Storage page can show plenty of free space on service/data volumes while /var remains full, because they are different filesystems.

Step-by-step remediation

1) Access the OS shell for disk checks (_shell)

Linux commands like df, du, and ls are not Exinda CLI commands; they must be run from _shell.

In Exinda CLI:

en
_shell

If _shell prompts for “License key:” and returns “% Permission denied.”, install a temporary restricted-shell license (request one through your normal licensing/support process), then re-try _shell.

2) Confirm whether /var is full

From _shell:

df -h

Check the % Used for /var. As a practical guideline, keep /var well below 90% used (several GB free) to ensure reporting and diagnostics can reliably write temporary/output files.

3) Identify what is consuming space under /var

From _shell:

du -x /var | sort -rn | head -20

A common offender is /var/log/datad. To inspect large files there:

du -x /var/log/datad | sort -rn | head -20
ls -lhS /var/log/datad | head -50

In many cases, one or more datad-server*.log files can grow extremely large (for example, ~10–12GB), pushing /var near full.

4) Free space on /var

Cleanup actions depend on what du/ls reveals, but typical safe cleanup targets include:

• Oversized/old log files under /var/log/datad/ (for example, remove older/rotated logs and unusually large historical logs).
• Leftover temporary sysdump directories (if present):

rm -rf /var/tmp/sysdump*

Re-check free space:

df -h

5) Re-generate diagnostics (sysdump) after freeing space

From Exinda CLI:

en
conf t
debug generate dump

Then confirm the file exists from _shell:

ls -lah /var/opt/tms/sysdumps/

If the CLI reports success but no .tgz exists

This commonly indicates /var is still critically full/near-full, preventing the sysdump archive from being written successfully (even if the command prints a “Generated dump … .tgz” message).

Use targeted searches to confirm whether the file was written elsewhere or not written at all:

find /var /tmp -xdev -type f -name "sysdump-*.tgz" -print 2>/dev/null
find /var -xdev -type d -name "sysdump*" -print 2>/dev/null

If needed, confirm the directory is writable:

touch /var/opt/tms/sysdumps/test_write && ls -l /var/opt/tms/sysdumps/test_write && rm -f /var/opt/tms/sysdumps/test_write

6) Clear stuck report/diagnostics generators by restarting services (maintenance window)

After recovering disk space, restart reporting/monitoring-related services from the Exinda CLI:

en
conf t
service collector restart
service statistics restart
service monitor restart

Wait 10–15 minutes for services to stabilize and for the monitoring pipeline to resume.

7) Validate

1. Diagnostics: In the GUI, re-try Diagnostics generation.
   • Confirm the GUI no longer shows “Internal error, code 1003 (see logs for details)”.
   • Confirm a new diagnostics file is listed and downloadable.
2. Reporting: Re-test reporting exports.
   • Generate/download a small PDF report range first (example: last 24 hours).
   • If successful, retry the original larger date range.
3. Dashboard → System:
   • Confirm the “System Disk Full … /var” alarm clears after space is recovered.
   • If Database Status remains “Error (see logging)”, additional database recovery may be required. Do not clear/rebuild databases without a maintenance window and a data-retention plan.

If sysdump download fails due to network timeout (SCP/SSH)

If you see an SCP/SSH error such as:

• ssh: connect to host <public_ip> port 22: Connection timed out

This is typically a network/firewall/NAT/access issue (not a sysdump creation issue). Validate:

• SSH is enabled under the appliance access settings.
• Firewalls/NAT allow TCP/22 to the appliance management IP (or retrieve the file from an internal host on the same LAN).
• If inbound SSH is blocked by policy, use an approved internal transfer method and then upload the file using your organization’s supported secure process.

Prevention

• Monitor /var usage regularly.
• Pay special attention to /var/log/datad/ growth; keep /var with a safety buffer (several GB free, and well below ~90% used) so report and diagnostics generation can write temporary/output files reliably.