Finding Needles in Haystacks (The Size of Countries)

The lament of security analysts is often a limitation in the amount of data they can process, and the ensuing loss of data fidelity as size increases. As data sets grow they become unwieldy, making it difficult to add context through correlating security event data with other relevant data sets.

Full packet capture provides a method for maintaining a forensic copy of all network conversations. However the reality up until now is that full packet capture and analysis has been bounded by the size of the data, the time to process it and the ability of applications and tools to encode key attack, deviations, mis-use and anomaly data into visualizations.

When you can store all of your network data the issue then becomes how do you analyze it. How do you find the single conversation you are looking for in trillions of conversations?

Big Data has supplied both a method for parallel computation and at the same time the cost of storing all network data (full packet capture) is within reach of all organizations. At the same time threats are becoming more blended, complex and difficult to find. Big Data tools such as Apache Hadoop, PIG and NoSQL databases provide the ability to perform complex network traffic analysis at petabyte scale. These tools can be leveraged using the Amazon Cloud (Elastic Map Reduce) to process, query and persist packet capture data.

With these tools there is no time-cost trade off to analyzing every single conversation on a network, enriching the data, intersecting data sets and sharing anonymised data sets.

Allowing you to answer questions that not many tools can:

• How can I find Zero Day attacks in past traffic?
• How can I better detect attacks at greater confidence?
• What is normal?
• What is new (never seen before)?
• What attackers are similar to other attacks?
• What is the operating system and patch level of my attackers?
• What protocols are strongly correlated in relation to sessions, bandwidth and payloads?
• What sessions are tunnels?
• After each attack how did the victim's sessions and protocols change?
• What is a normal HTTP payload for each of my web servers? - - How does an attack differ?
• What are attackers doing within HTTPS sessions to my websites.
• How can I intersect white and blacklists with my network packet captures?