Concurrent Connections Seen Permanently Increasing

Summary

In the Exinda appliance, the 'Concurrent Connections' graph show the number of total connections through the device. This number increases and decreases by using internal stats of the number of connections being kept track of by the Exinda. If the number of concurrent connections rises, it implies that there are more new connections than those being closed in a specific time period

Overview

The Exinda appliance uses tables and information to hold the number of connections in the device between sources and destinations. It uses this table to determine the type of information flowing through - whether new connections are part of existing conversations, whether it's a brand new connection between hosts or continuing ones. This connections table in the Exinda is what a lot of the statistics use in order to determine information for reporting purposes. This is the case with the Concurrent Connections and New Connections Per Second graph.

The concurrent connections are calculated based on this table at various points in time. If the number of concurrent connections rise, it implies that there are more connections coming in than there are being let go by the system.

Cause

In the connections table, each 'connection' (a single link between a source IP and port, destination IP and port and application) is kept for an hour. The timer that is kept ticks down every second. When new traffic comes in, it is compared against this table. If it matches a connection that currently exists, the timer is reset for an hour, and if not, it is added to the table as a new connection. When the timer for a connection reaches to zero (implying there has been no traffic matching that connection in an hour) it times out of the system and is dropped.

If the concurrent connections permanently keeps increasing, it is implying that there is traffic on the network which is constantly opening new connections to a greater degree than connections are leaving the table. This can be from legitimate traffic on the system, or it could indicate that the Exinda itself is harboring connections and refusing to release them.

Resolution

To investigate, examine the number of flows for each host under the Monitor > Hosts graph, or per application on the Monitor > Applications graphs. It will give an image of how many individual flows belong to a host or application. It is possible to pick out anomalous number of flows for specific hosts through this.

A key note is to look at any SQL servers that are instanced and having traffic flowing through the Exinda - SQL has an option to never terminate connections between servers, so it is possible that a busy SQL server will open and keep open a large number of connections over time; these will never disconnect so from the Exinda's perspective, new connections will continually be opened that will never time out.

If there is some worry that the Exinda itself is not releasing connections, please contact Exinda TAC.

Internal Notes

The table is held in nf_conntrack located in /proc/[folder of numbers]/net
It updates continually with the new information. Take a look at it to look for the expiry time (5th column, a number with a max value of 3600 - seconds in an hour - that counts down). We do not measure the new connections per second anywhere easily, so a way to take a look would be to create a copy of nf_conntrack and diff the copy with the original immediately - it will show the difference in the few seconds. Will still take some time to go over exactly what is 'new' and what is not - the last entries will be new.