Open Network Insight Project Builds On Big Data To Improve Security

Using big data to collect network events in an effort to help improve security is not a new concept, but it is evolving. Part of the evolution is coming by way of the open-source Open Network Insight (ONI) project, which has been generally available for just over a month and is already being used by Intel.

Helping to lead the ONI effort is Cloudera, one of the leading vendors and participants in the development of the Hadoop big data platform. ONI isn’t just Hadoop though, as simply having big data is not the same as understanding the data and making sense of it to improve security outcomes. In addition to Hadoop, ONI includes the open-source Wireshark project, which is a widely used packet sniffing and analysis technology; nfdump, which is a netflow network packet capture tool; and the D3 JavaScript visualization library and the Jupyter project for reporting.

“Hadoop is a really great platform for storing cyber-security information, and this is a use case that we see across industries,” Eddie Garcia, chief security architect at Cloudera, told eWEEK.

By making use of Hadoop as the back end for storing data, Garcia noted that that an organization can look at and analyze more data than by using a non-big data approach. From a community perspective over the last month, ONI has seen adoption and contributions by eBay and Accenture, among other organizations. Garcia noted that eBay is using ONI for its own security needs, while Accenture is now running ONI as a managed service for clients.

Standard model

One of the challenges of using Hadoop as the basis of a security platform is that many organizations have built their own approaches and there hasn’t been a standard model, but that’s something ONI is aiming to address, according to Garcia. When dealing with network security, among the challenges that ONI helps to solve is how to represent network data in Hadoop in a common model and format, regardless of the network device or vendor that the data comes from.

“What we’re enabling is a platform to store network data and do analytics on top,” he said. “In the future, what we see is an open model to enable other threat and security analysis, including users and servers.”

While the current iteration of ONI has a focus on network events, the platform is also useful for performing User Behavior Analytics (UBA), which is an increasingly common security activity for detecting anomalous user activity.

“A user is going to have an IP address,” Garcia said. “From that perspective, you can already do some user analytics with ONI today.”

What ONI is missing, he noted, is the direct correlation to the user and all the different devices that a particular user might be using to access a network. Work to enable more robust UBA is ongoing in ONI.

From a reporting perspective, Garcia explained that the Jupyter component helps to power the ONI dashboard. Currently in the ONI dashboard, it’s possible for an administrator to drill down into specific events and IP addresses, as well as filter by date and packet header information. As ONI community partners and users continue to embrace the platform, Garcia expects that vendors will build their own tools on top of ONI for even more functionality.

Despite only having been publicly available for a month, Garcia said the ONI project is already in production at Intel, which is also one of the core contributors and founders of the project.

“This technology grew out of a necessity that the number of events Intel was receiving outpaced the technology that they had,” he said. “Now with Hadoop behind it, Intel is able to capture and analyze data across 20 billion events they receive daily.”

Originally published on eWeek.

Take our hacking and viruses quiz here!