Network Detection and Response in Encrypted Environments: What You Can Still See

Roughly 95% of enterprise network traffic is encrypted, and the share keeps climbing as TLS 1.3, QUIC, and Encrypted Client Hello move from edge cases to defaults. For a generation of security tools built around payload inspection, that should be a death sentence. It isn’t — but only because Network Detection and Response (NDR) platforms quietly stopped reading payloads years ago. They read the shape of the conversation instead: who’s talking to whom, with what TLS stack, on what cadence, in what volume, for how long.

This piece is about what’s actually observable on a modern wire. Not the marketing version where AI sees through encryption — it doesn’t — but the working version where defenders triangulate identity, intent, and anomaly from metadata that encryption was never designed to hide. We’ll walk the visible surface, the analytic techniques that exploit it, the reference fingerprints worth knowing, and the pieces that genuinely have gone dark.

What Encryption Actually Hides — and What It Doesn’t

Encryption protects payload confidentiality. It does not protect connection metadata, and that distinction is the entire foundation of NDR in the post-decryption era.

When a TLS session opens, the ClientHello message is sent in plaintext before any encrypted data flows. It advertises the cipher suites the client supports, TLS extensions in a specific order, supported elliptic curves, ALPN values declaring HTTP/2 or HTTP/3, and — historically — the Server Name Indication (SNI) field naming the destination host. The ServerHello that follows is also clear, exposing the chosen cipher and extensions. Both messages are observable to any sensor with packet visibility, even when subsequent application data is fully encrypted.

Beyond the handshake, every session leaks structural data that no encryption layer covers: source and destination IPs, ports, packet sizes, inter-arrival timing, session duration, byte counts in each direction, TLS certificate details (issuer, subject, validity, signing algorithm), and DNS queries that preceded the connection. Since 95% of network traffic is encrypted, traditional deep packet inspection struggles to see threats. Encrypted traffic analysis solutions identify threats without decryption, maintaining data confidentiality.

NDR platforms turn this metadata into detection signal through four broad techniques: TLS fingerprinting, flow and behavioral analysis, certificate analysis, and correlation across the DNS and HTTP layers that surround encrypted sessions. None of them read a single byte of payload.

TLS Fingerprinting: Identity Without Decryption

The ClientHello varies enough between TLS implementations — Chrome’s BoringSSL, Firefox’s NSS, Go’s crypto/tls, Python’s OpenSSL bindings, malware’s bespoke libraries — that hashing its contents produces a stable identifier for the calling application. This is the lineage that runs from JARM (server-side) through JA3 (client-side, 2017) to the current JA4+ suite released by FoxIO in 2023.

JA3 hashes a comma-joined string of TLS version, cipher suites, extensions, elliptic curves, and EC point formats — typically into a 32-character MD5. It worked well for years, but TLS 1.3 forced a redesign. JA4 is a successor set of fingerprints designed to address practical limitations of JA3, especially in modern TLS 1.3 environments where implementations can look less distinct or where over-uniqueness creates noise. Browsers also began randomizing extension order to break JA3, producing different hashes per connection from the same client.

JA4 fixes this by sorting extensions before hashing and structuring the output as three readable segments. A typical JA4 looks like t13d1516h2_8daaf6152771_e5627906d626. Segment A holds metadata — protocol (TCP or QUIC), TLS version, SNI presence, and counts of ciphers and extensions. Segment B is a truncated SHA-256 hash of the sorted cipher list. Segment C hashes the sorted extensions and signature algorithms, filtering out non-identifying GREASE values. The sorting is what neutralizes randomization evasion.

The full JA4+ family extends fingerprinting beyond the TLS client itself. JA4S fingerprints server responses, JA4H covers HTTP requests, JA4X profiles X.509 certificates, JA4L measures light-distance latency to detect proxies, JA4T fingerprints raw TCP behavior, and JA4SSH covers SSH sessions. Combining them is where detection fidelity sharpens — a JA4 that claims to be Chrome 138 paired with a JA4T that says Linux server is the kind of cross-layer contradiction that’s hard to forge.

The practical detection move is correlation, not lookup. Modern WAFs no longer block on a static hash. They use signals intelligence: User-Agent rarity (does this JA4 typically claim to be Chrome 134?), light distance measuring round-trip time, and protocol consistency — if the TCP fingerprint says Linux but the JA4 says iOS, it’s a bot.

Behavioral and Flow Analysis: Beaconing, Volume, and Cadence

Fingerprints identify the stack. Behavioral analysis identifies what the stack is doing.

C2 channels betray themselves through rhythm. A compromised host beaconing to its operator produces session intervals that are too regular to be human (jittered 60-second check-ins, for example), session durations that are consistently short, and byte volumes weighted heavily toward the upload direction during exfiltration. None of this requires payload visibility. A global e-commerce company detected encrypted C2 traffic masquerading as HTTPS using TLS fingerprinting and anomaly-based analysis (unusual session intervals, packet burst patterns) to flag the threat without ever touching the payload.

DNS tunneling — using DNS queries to smuggle data out of an environment — produces a similar tell. Even when the queries themselves are encrypted via DNS-over-HTTPS or DNS-over-QUIC, the volume of queries to a single domain, the entropy of the query strings in plaintext DNS, and the response timing fall outside normal distributions. A healthcare organization can identify irregular DNS tunneling activity from a compromised endpoint by analyzing session frequency and payload entropy, cutting dwell time from weeks to hours.

Lateral movement inside an environment leaves the most observable trail of all. East-west traffic between workstations, sudden SMB or RDP sessions to hosts that have never communicated before, anomalous service account behavior across subnets — these patterns appear in NetFlow and packet metadata regardless of whether the underlying protocols are encrypted. Behavioral baselining over weeks lets NDR flag the deviations.

Certificate and DNS Analysis: The Ecosystem Around the Tunnel

Certificates exposed during the handshake are an underused detection surface. Self-signed certificates on internet-facing services, certificates issued moments before first use, certificates whose Common Name doesn’t match the SNI being requested, and certificates from CAs that legitimate browsers wouldn’t trust — these are all visible without decryption and all correlated with command-and-control infrastructure. Tools flag suspicious SSL behaviors like self-signed or expired certificates and weak ciphers through SSL anomaly detection, and offer deep visibility via JA3, JA3S, and JARM fingerprinting.

DNS provides the other half of the picture. Even with DoH and DoQ encrypting the queries themselves, the DNS resolver chosen, the volume of lookups, the lifetime of resolved domains, and patterns matching domain generation algorithms (DGAs) remain detectable. Newly registered domains receiving traffic minutes after registration are statistically anomalous. Communication pattern analysis detects connections to newly registered domains, identifies covert command-and-control channels via ML-powered DNS tunneling detection, and flags anonymized communications with Tor activity detection.

What QUIC and ECH Actually Take Away

The picture above isn’t static. Two protocols are actively shrinking the visible surface in 2026.

QUIC runs over UDP, integrates TLS 1.3 directly into its transport handshake, and encrypts almost everything. Unlike TLS over TCP, where TCP headers, sequence numbers, and acknowledgments are visible in plaintext, QUIC encrypts packet numbers, acknowledgments, and even connection close frames. This encryption-by-default approach significantly reduces the metadata available for traffic analysis. Connection migration — the feature that lets a QUIC session survive a switch from Wi-Fi to cellular — also breaks the assumption that a session corresponds to a single source IP. The four-tuple that flow analysis depends on becomes unreliable.

QUIC traffic is still tractable, just harder. JA4 explicitly covers QUIC with a q prefix and h3 ALPN, and academic work has shown that the number of HTTP/3 request-response pairs in a QUIC connection can be estimated using only encrypted traffic with up to 92% accuracy. But the analytic effort is meaningfully higher and the resolution lower than what TCP-based TLS provided.

Encrypted Client Hello (ECH) is the bigger structural change. ECH wraps the sensitive parts of the ClientHello — the SNI most importantly — inside an encrypted inner ClientHello, leaving only a generic outer ClientHello visible to passive observers. Passive sensors lose visibility: if you compute JA4 from a passive tap, you may see “outer” fingerprints with less SNI and ALPN detail. Expect fewer distinct JA4 values in passive data.

For NDR sensors deployed on a SPAN port watching east-west or north-south traffic, ECH means the destination domain — historically one of the strongest detection signals — disappears. The Enea team has noted that the arrival of TLS 1.3 ECH challenges all known approaches to visibility into encrypted enterprise and network traffic, and many industry professionals are asking how they will be able to carry out essential network operations when TLS 1.3 ECH becomes the encryption standard.

The workaround is to terminate TLS at the edge — at a forward proxy, a TLS-terminating load balancer, or a SASE gateway — and compute fingerprints from the inner ClientHello there. This works but trades passive visibility for active inline insertion, with all the latency, certificate management, and privacy tradeoffs that decryption entails.

The Decryption Question

Plenty of NDR deployments still decrypt. Selective TLS interception via a controlled man-in-the-middle (the polite term is “TLS forward proxy”) gives full payload visibility, at the cost of certificate pinning conflicts with mobile banking apps and certain cloud services, regulatory exposure under HIPAA and PCI-DSS for sensitive categories, and the operational burden of CA distribution to every endpoint.

The industry consensus has shifted toward selective decryption — decrypt high-risk categories, bypass sensitive ones — paired with metadata-based detection on the rest. Decryption can be tailored using URL categories, custom domain or SNI objects, certificate validation rules, and SNI/CN/SAN mismatch detection. Strategic exceptions allow for certificate pinning cases like mobile banking applications and legal compliance considerations like HIPAA, PCI-DSS, and GDPR.

The economic case for full decryption keeps weakening as encrypted traffic analysis matures. The case for never decrypting anything is also weak — some attacks are visible only in payload. Most mature programs land on a hybrid: decrypt where law and pinning permit, fingerprint everywhere else.

Vendor Landscape and Tooling

The commercial NDR market in 2026 organizes around how vendors blend encrypted traffic analysis with broader detection. Cisco upgrades Talos models for encrypted-traffic heuristics, Fortinet fuses NDR into unified SASE, and Juniper applies Mist AI to telemetry baselining. Specialized players such as Darktrace, Vectra AI, and ExtraHop differentiate via self-learning models trained on proprietary behavior graphs.

Open-source remains foundational. Zeek (formerly Bro) and Suricata still underpin a substantial fraction of commercial NDR data pipelines. Corelight Open Network Detection and Response is a security platform powered by open-source technologies such as Zeek and Suricata, integrating network security monitoring, intrusion detection, packet capture, and AI-driven analytics. For teams building rather than buying, Zeek’s protocol parsing combined with JA4 plugins and Suricata’s signature engine produces a credible NDR foundation at a fraction of commercial licensing cost.

The Forrester and Gartner evaluations in late 2025 placed ExtraHop, Vectra AI, Darktrace, Corelight, and Cisco in leader positions, with encrypted traffic analysis specifically called out as a differentiating capability. ExtraHop received the highest scores possible in 10 criteria including Encrypted Traffic Analysis, Threat Hunting, Framework Mapping, and Detection Accuracy in The Forrester Wave: Network Analysis And Visibility Solutions, Q4 2025.

Pitfalls and Limitations

Fingerprint-based detection has real failure modes worth knowing.

JA4 values change with browser updates — Chrome’s roughly four-week release cycle means allow-lists go stale fast. FoxIO explicitly notes that JA4 fingerprints change as application TLS libraries are updated, roughly on the order of yearly update cycles for major apps. Cloudflare publishes guidance specifically warning that customers sometimes block new fingerprints that turn out to be benign browser updates.

Fingerprints aren’t cryptographic identity. A determined actor can produce a ClientHello that is byte-identical to a legitimate browser, making the resulting JA4 indistinguishable from real traffic. The defense isn’t any single fingerprint but the combination of JA4, JA4T, JA4H, and behavioral signals — the Cloudflare and DataDome approach of looking for cross-layer contradictions rather than single-hash matches.

Adversaries also actively obfuscate flow patterns. Recent academic work on obfuscated malicious traffic detection covers techniques like adding dummy packets, randomized inter-arrival delays, and padding to break statistical baselines. Defenders respond by training models on adversarial examples — an arms race with no finish line.

Encrypted internal east-west traffic remains underinstrumented at most enterprises. NDR sensors deployed only at the perimeter miss lateral movement entirely. Cloud-native deployments need explicit configuration of VPC Traffic Mirroring (AWS), vTAP (Azure), or Packet Mirroring (Google Cloud) to get the equivalent visibility — and container-to-container traffic inside service meshes like Istio often requires sidecar telemetry rather than network capture.

FAQ

Can NDR detect threats in TLS 1.3 traffic without decryption? Yes, for many threat categories. JA4 fingerprinting, certificate analysis, beaconing detection, and DNS correlation all work on TLS 1.3 metadata. What’s lost is content-based detection — payload signatures and embedded malware in transit require decryption.

What happens to NDR when ECH is widely deployed? Passive visibility into SNI and inner ClientHello details disappears. Defenders shift toward edge termination (proxy, SASE, TLS-terminating load balancer) where the inner ClientHello can be recovered, plus heavier reliance on behavioral and DNS-side signals. Treat passive sensor data and edge-terminated data as separate baselines.

Is JA3 still useful or should we move entirely to JA4? JA3 is widely deployed and still produces signal, but it’s brittle in TLS 1.3 environments and easily randomized. New deployments should default to JA4+. Existing JA3 infrastructure can run alongside JA4 — most NDR platforms emit both.

How does NDR coexist with EDR and XDR? NDR catches what EDR misses: unmanaged devices, IoT, OT, and the network-layer evidence of compromise that doesn’t touch a managed endpoint. NDR fills endpoint visibility gaps by detecting threats on unmanaged or stealthy hosts, and integrates with SIEM, SOAR, and EDR/XDR for correlation and automated response. The strongest detection programs feed NDR telemetry into XDR for correlation rather than treating them as alternatives.

Where This Goes Next

The trajectory is clear and not particularly comforting for defenders who liked deep packet inspection. More traffic encrypts. More protocols hide more metadata. Post-quantum TLS will introduce new cipher suites and new fingerprint dimensions. ECH adoption will accelerate as Cloudflare and Mozilla push it. QUIC will continue replacing TCP for high-volume HTTP traffic.

The compensating trajectory is also clear. Behavioral analytics on flow telemetry get sharper as training data accumulates. Cross-layer fingerprint correlation (TCP + TLS + HTTP + behavior) becomes harder to forge in totality even as individual layers get easier to spoof. Edge termination at SASE gateways centralizes decryption in places where certificate pinning isn’t an issue.

The practical move for security teams is to stop treating “encrypted” as synonymous with “invisible.” The wire still talks. It says less than it used to, but it says enough — if you’ve instrumented the right collection points, kept your fingerprint databases current, baselined behavior at sufficient depth, and accepted that no single signal is identity. The threats that defeat this approach are the ones using legitimate stacks, legitimate protocols, and patient cadences. Those were always going to be hard. Encryption didn’t change that. It just made the rest of detection more interesting.