Drill has been established as an Apache Incubator Project, opening its continued development up to software engineers around the world, according to Tomer Shiran, director of product management for Hadoop vendor, MapR Technologies, which is one of the backers of the Apache Drill project.
The Drill project will work to create an open-source version of Google’s Dremel Hadoop tools, which Google uses to speed up its internal use of its Hadoop data analysis tools.
Drill aids Hadoop users by enabling vastly quicker queries of huge data sets, said Shiran.
“With Drill, you’ll be able to get really fast responses,” he said. Users will be able to get responses within one second, which is a key difference from other tools that are available today, he added.
As it presently works as it was designed, Hadoop does batch processing of large data sets. Drill will improve on that method by doing “interactive analysis” that can find the required answers in the data more quickly, said Shiran. “Interactive analysis is much faster than batch processing.”
The need for tools like Drill has been inspired by always-increasing user requirements, he said. “People have been doing queries in Hadoop, but since it doesn’t return answers to you within a few seconds, it has limitations.”
Using Drill, users will be able to do ad hoc analysis and get faster responses, whether they are seeking anomalies, data trends or even network intrusions, according to Shiran. “With all of those things, you’re going to have to get a pretty fast response or by the time you do figure it out, it’s going to be old news.”
The nascent Drill open-source project is currently in development and includes a variety of companies and individuals who are working on it right now. “A broad-based effort will be working on this,” said Shiran. “There’s quite a few people actively developing on the project now, so I don’t think it will be a long time before we have an early version released.”
Drill was inspired by Google’s Dremel project, which helps Google perform data analyses on its huge data sets such as analysing crawled web documents, tracking install data for applications on the Android Market, analysing spam, analysing test results on Google’s distributed build system and more, according to Shiran.
By developing Drill as an Apache open-source project, organisers will be able to establish Drill’s own APIs and establish a flexible and robust architecture that will support a broad range of data sources, data formats and query languages, according to the group.
MapR offers two versions of its Hadoop products: MapR M3, which is free; and MapR M5, which is a commercial version of the product with advanced features, including high availability, the ability to make data snapshots and do mirroring of datasets, and 24/7 support.
How well do you know your webcam? Take our quiz.
Consumer group Choice files with privacy regulator to stop three major Australian retailers from collecting…
Danish competitor files fresh antitrust complaint against Google for Jobs, saying its search traffic fell…
Toshiba board expected to approve appointment of two external directors from hedge fund investors, making…
Shanghai electric vehicle maker Nio says it is investigating deaths of two people after one…