The Role of Encrypted Traffic Analysis for Threat Detection [Q&A]

The Role of Encrypted Traffic Analysis for Threat Detection [Q&A]

Everyone is striving to make their systems more secure, and in many cases that means embracing encryption in order to protect data.

But using encrypted traffic on networks poses a headache for security teams because malicious content can be harder to detect. We spoke to Thomas Pore, director of security products at live actionto learn more about the problem and how to solve it.

BN: How does encrypted traffic impact network threat detection today?

TP: The growing adoption of encrypted network protocols is causing network visibility to deteriorate for security teams, and legacy tools are becoming less and less effective. In the fourth quarter of 2021 alone, 78% of malware distributed over encrypted connections was evasive, according to a recent report, highlighting the growing threat of advanced malware attacks. Additionally, the growing acceptance of HTTPS, the rapid deployment of encrypted protocols such as DNS over HTTPS and TLS 1.3 dramatically reduce visibility into server identity and content inspection, making threat detection more difficult, and in many cases nearly impossible, for network advocates. Once inside an organization’s network, hackers use encrypted sessions to move laterally from east to west. Traditional detection tools only inspect north-south traffic. This gives attackers the edge they need to perform advanced actions, like a ransomware attack.

BN: What is encrypted traffic analysis and why is it important for threat detection and response?

TP: Encrypted traffic analysis is a type of side channel analysis that allows network defenders to do their job while maintaining the privacy and network integrity provided by a fully encrypted system. Encrypted traffic analysis, combined with machine learning capabilities, assesses complex data patterns over time and differentiates normal from abnormal activity, all without requiring access to the data content. It enables security teams to leverage different types of C2 activities (such as tagging, TLS fingerprinting, and packet length sequence) to quickly uncover malicious behavior and network anomalies, which are essential for a effective detection and response to threats. Indeed, ETA enables visibility into network transactions, which provides valuable insight into encrypted traffic to aid network defenders.

BN: What is encryption blindness and how can it impact organizational security?

TP: Encryption blindness is caused by a lack of visibility into encrypted traffic, resulting in missed (hidden) threats in the network. Since most modern computer network traffic is now concealed in encryption, hackers can take advantage of this security gap to conceal their actions in encrypted traffic. In other words, much of the traffic in organizations today is not inspected simply because it is encrypted, which opens the door to attacks. As threats become more sophisticated and the attack surface increases, the effectiveness of many traditional strategies decreases, such as IDS, IPS, and forcible decryption. This calls into question the effectiveness of organizational security more than ever.

BN: What is the difference between Deep Packet Inspection (DPI) and Deep Packet Dynamics (DPD) for ETA?

TP: Deep Packet Dynamics (DPD) is a new approach to network packet assessment that eliminates the need for payload inspection. By analyzing more than 150 packet characteristics and behaviors in multi-vendor, multi-domain, and multi-cloud network environments, it can more reliably assess encrypted and unencrypted traffic.

When DPD is combined with machine learning and ETA, it provides unique capabilities for regaining visibility into encrypted traffic and offers some of the most advanced network detection and response capabilities available today. This includes a variety of benefits such as threat and anomaly detection that others miss; detect threats in real time; eliminate encryption blindness; reduce the time a SOC needs to investigate and respond to threats; validate end-to-end encryption compliance; providing visibility from core to edge to cloud; and enabling the security team to create a coordinated and cohesive response through other security tools such as SIEM, SOAR, etc.

In contrast, Deep Packet Inspection (DPI) is an older legacy approach that works primarily over unencrypted or plain text protocols such as HTTP. But encryption undermines DPI and allows malicious payloads to hide in encrypted traffic. In short, DPD gives network defenders a much clearer view of encrypted network traffic than DPI does.

BN: What role does ETA play in broader network detection and response solutions?

TP: Encrypted traffic analysis is a way to restore network visibility for defenders while preserving user privacy by combining DPD and advanced behavioral analysis combined with machine learning. Malicious actors and malicious system operators communicate with infected target systems using a set of techniques called Command and Control (C2). Threat actors use C2 techniques to mimic expected benign traffic by using common ports and standard encryption protocols to evade detection. Despite these precautions, ETA with machine learning effectively identifies malicious C2 activity on the network so you can stop an attack. Even without any visibility into the contents of the connection, ETA can tell a lot about the behavior of encrypted traffic and helps network defenders prioritize their network detection and response activities.

BN: What’s next – or on the horizon – with ETA?

TP: Analyzing encrypted traffic will further strengthen organizations’ long-term security strategies, through ongoing characterization of encrypted flows and recognition of behavior patterns. This extends to endpoints, assets and end-to-end encryption, mapping benign and expected traffic against malicious anomalies. Phishing and Remote Access Protocols (RDP/VPN) continue to be the primary infection vectors for ransomware and state-sponsored APT actors. ETA’s high-fidelity detection of anomalous characterization will make the difference in stopping the attack in the future.

Photo credit: / Shutterstock

Charles J. Kaplan