When the attack fits in 0.01% of the packets: IP flow entropy in the age of encrypted traffic

01The median is small. The tail is enormous.

There is a known distortion in how network attacks get discussed. Headlines record the records, and the records are genuinely striking: in the first half of 2026, Cloudflare reported mitigating 935 network-layer attacks above 1 Tbps, a 519% rise between the first and second quarters^[10].

The same report carries the number almost nobody quotes. Over that period, 96.62% of network-layer attacks stayed below 500 Mbps and 90.60% ended in under ten minutes^[10]. The distribution is aggressively bimodal: a tail that bends a provider's infrastructure, and a central mass that never shows up on a link utilisation graph.

That central mass is the hard problem. A detector calibrated for the tail steps over it without recording anything. A detector calibrated for it floods the operator with alarms once the tail arrives. What a detection system needs, therefore, is not sensitivity but adjustable sensitivity, a point this series returns to in Part 4.

A small median does not mean a harmless event. A port scan brings nothing down on its own, though it is usually the opening move of someone looking for a vulnerability to exploit^[12]. The event that is cheap to generate is precisely the one that precedes the expensive one.

02What encryption left visible

Deep packet inspection, which examines the transported content, has lost ground irreversibly. With TLS 1.3^[8] and QUIC^[9], not only the payload but much of the session metadata that used to travel in the clear is now encrypted by default.

There is, however, one layer encryption cannot erase, because it is the condition for the network to function at all. Routers and switches must know where to forward each packet, so they group traffic into flows and export those records through protocols such as IPFIX^[7].

Encryption protects what was said. It does not hide who spoke to whom, when, how often, in what volume, and for how long. A flow record carries source and destination addresses, ports, protocol, packet, byte and flow counts, and timestamps. None of that requires reading a single byte of content.

It is worth being honest about the reach of this choice. A system observing only flow metadata does not detect viruses, Trojan horses, buffer overflows or privilege escalation, precisely because that evidence lives in the content^[1]. The scope is narrower, although it is a scope that remains available when everything else closes.

03Entropy, in plain terms

Entropy is a measure physicists invented to quantify disorder and information theory adopted to quantify uncertainty^[2]. In traffic analysis it takes on a more direct reading: entropy measures how far a traffic property is concentrated or dispersed.

Given a probability distribution p₁, p₂, …, p_n, Shannon entropy is defined as:

H_S = −∑ p_i log p_i

The value is minimal when nearly all probability piles onto a single point and maximal when it spreads evenly across many. Applied to destination ports, the measure separates a healthy web server, where almost everything arrives at port 443, from a target under scan, where tens of thousands of ports appear once each.

04Concentration and dispersion are the event's signature

What makes entropy interesting is not detecting that something changed. It is that the direction of the change, property by property, describes the type of event.

A TCP flood against a server concentrates the destination address, the destination port, the packet count and the flow count, while dispersing the source port, since the attack tool draws a random high port for each connection. A network scan does nearly the opposite, dispersing addresses and counts with tiny packets.

In ^[1] this reading was formalised as a textual signature in which each position corresponds to a flow property labelled concentrated (C), dispersed (D) or normal (N). The real distributed attack recorded in that study produced TCCDCCCC, the alpha flow event produced TCCCCCDC and the network scan produced IDDDDD.

These are fingerprints in practice. Two anomalies whose disturbances differ by orders of magnitude remain distinguishable by shape rather than by size.

05Why measuring the whole network is not enough

The classical application of entropy to anomaly detection works at the network level^[3][4]. A single distribution is computed by aggregating all traffic observed in the interval, and its deviation from habitual behaviour is checked.

The model performs well as long as there is one event at a time. When several occur simultaneously, however, a dominance effect appears: the attack moving gigabytes rewrites the aggregate distribution, and the discreet event vanishes inside it^[4].

The network scan figures in ^[1] show the scale of the problem. The scanning device probed 254 destinations and accounted, on average, for 0.01% of the packets, 0.001% of the bytes and 0.15% of the flows in the interval. The event was in the traffic. It simply was not in the average.

In the same study, when four distinct anomalies were combined into a single dataset, Shannon entropy applied at the network level identified only one of the four^[1]. This is not an implementation defect but a direct consequence of aggregating before measuring.

The proposed way out was to change the unit of observation. Instead of one distribution for the network, entropy is computed over the flows exchanged between each pair of devices, which requires representing the network as a directed graph and inspecting every edge. That is deep IP flow inspection, the subject of Part 3.

The second decision was to replace Shannon entropy with Tsallis entropy^[6], whose generalisation introduces an entropic parameter q controlling how much high and low probabilities weigh in the result^[5]. That parameter is the sensitivity dial mentioned in Section 1, and Part 4 shows what it costs and what it delivers.

06What comes in the next nine parts

The series walks the full path, from the raw flow record to autonomous operation, always testing the original design against the current landscape.

1. IP flows as a data source. NetFlow, IPFIX and what is given up along with the payload.
2. The network as a graph. Deep flow inspection and moving the unit of observation to the device.
3. The entropic parameter q. Tuning the detector between the tail and the median.
4. Adaptive thresholds. Why percentiles replaced mean and standard deviation.
5. Signatures and rule-based classification. From detection to diagnosing the event type.
6. Simultaneous anomalies. The case where the detector must see more than one thing at once.
7. Packet sampling. The cost of seeing 1 packet in 2048.
8. Locating source and destination. Heuristics for bidirectional flows with no initiator mark.
9. From detection to autonomy. Metrics, key performance indicators and the autonomic control loop.