Description: FINDING NEEDLES IN HAYSTACKS (THE SIZE OF COUNTRIES)
The lament of security analysts is often a limitation in the amount of data they can process, and the ensuing loss of data fidelity as size increases. As data sets grow they become unwieldy, making it difficult to add context through correlating security event data with other relevant data sets.Full packet capture provides a method for maintaining a forensic copy of all network conversations. However the reality up until now is that full packet capture and analysis has been bounded by the size of the data, the time to process it and the ability of applications and tools to encode key attack, deviations, mis-use and anomaly data into visualizations.When you can store all of your network data the issue then becomes how do you analyze it. How do you find the single conversation you are looking for in trillions of conversations?Big Data has supplied both a method for parallel computation and at the same time the cost of storing all network data (full packet capture) is within reach of all organizations. At the same time threats are becoming more blended, complex and difficult to find. Big Data tools such as Apache Hadoop, PIG and NoSQL databases provide the ability to perform complex network traffic analysis at petabyte scale. These tools can be leveraged using the Amazon Cloud (Elastic Map Reduce) to process, query and persist packet capture data.With these tools there is no time-cost trade off to analyzing every single conversation on a network, enriching the data, intersecting data sets and sharing anonymised data sets.Allowing you to answer questions that not many tools can:How can I find Zero Day attacks in past traffic?* How can I better detect attacks at greater confidence?* What is normal?* What is new (never seen before)?* What attackers are similar to other attacks?* What is the operating system and patch level of my attackers?* What protocols are strongly correlated in relation to sessions, bandwidth and payloads?* What sessions are tunnels?* After each attack how did the victim's sessions and protocols change?* What is a normal HTTP payload for each of my web servers? - - How does an attack differ?* What are attackers doing within HTTPS sessions to my websites.* How can I intersect white and blacklists with my network packet captures?
The lament of security analysts is often a limitation in the amount of data they can process, and the ensuing loss of data fidelity as size increases. As data sets grow they become unwieldy, making it difficult to add context through correlating security event data with other relevant data sets.
Full packet capture provides a method for maintaining a forensic copy of all network conversations. However the reality up until now is that full packet capture and analysis has been bounded by the size of the data, the time to process it and the ability of applications and tools to encode key attack, deviations, mis-use and anomaly data into visualizations.
When you can store all of your network data the issue then becomes how do you analyze it. How do you find the single conversation you are looking for in trillions of conversations?
Big Data has supplied both a method for parallel computation and at the same time the cost of storing all network data (full packet capture) is within reach of all organizations. At the same time threats are becoming more blended, complex and difficult to find. Big Data tools such as Apache Hadoop, PIG and NoSQL databases provide the ability to perform complex network traffic analysis at petabyte scale. These tools can be leveraged using the Amazon Cloud (Elastic Map Reduce) to process, query and persist packet capture data.
With these tools there is no time-cost trade off to analyzing every single conversation on a network, enriching the data, intersecting data sets and sharing anonymised data sets.
Allowing you to answer questions that not many tools can:
How can I find Zero Day attacks in past traffic?
How can I better detect attacks at greater confidence?
What is normal?
What is new (never seen before)?
What attackers are similar to other attacks?
What is the operating system and patch level of my attackers?
What protocols are strongly correlated in relation to sessions, bandwidth and payloads?
What sessions are tunnels?
After each attack how did the victim's sessions and protocols change?
What is a normal HTTP payload for each of my web servers? - - How does an attack differ?
What are attackers doing within HTTPS sessions to my websites.
How can I intersect white and blacklists with my network packet captures?
MICHAEL BAKER BIO
Michael Baker is a technologist focused on information security and pushing the boundaries of software. Recently he has been using Big Data and NoSQL tools to pioneer new ways to collect, analyse and make security decisions on network data. A devotee of Network Security Monitoring (NSM) he looks to deliver on real potential of NSM using parallel processing, map/reduce and alternative databases.
Michael is a noted expert in Perimeter Security Architecture and Implementation having spent the majority of his 15 year security career designing and implementing Banking perimeters in Australia and Asia.
Michael has built and sold a security consulting company and designed and built a Managed Security Provider and Private Cloud platform. As the leader of an application development team he also built a construction collaboration platform that manages around $40 Billion AUD of construction projects.
Michael is currently CTO of Packetloop, a cloud-based security analytics and analysis platform and leading the security consulting firm, Black Foundry.
For More Information please visit : - http://2012.ruxcon.org.au/speakers
Tags: securitytube , hacking , hackers , information security , convention , computer security , ruxcon-2012 ,
Disclaimer: We are a infosec video aggregator and this video is linked from an external website. The original author may be different from the user re-posting/linking it here. Please do not assume the authors to be same without verifying.