Apache ‘Drill’ Speeds Hadoop Queries

Finding much faster ways to complete Hadoop queries for enterprise users is the aim of “Drill”, the latest open-source project being undertaken by the Apache Software Foundation.

Drill has been established as an Apache Incubator Project, opening its continued development up to software engineers around the world, according to Tomer Shiran, director of product management for Hadoop vendor, MapR Technologies, which is one of the backers of the Apache Drill project.

Open source tools

The Drill project will work to create an open-source version of Google’s Dremel Hadoop tools, which Google uses to speed up its internal use of its Hadoop data analysis tools.

“We’ve spent quite a few months talking to lots of organisations and potential users of Drill and to our customer base as well,” said Shiran, who is a founding member of the Drill project. “We wanted to put this out there as an open-source project, rather than just keep it within MapR for our use alone.”

Drill aids Hadoop users by enabling vastly quicker queries of huge data sets, said Shiran.

“With Drill, you’ll be able to get really fast responses,” he said. Users will be able to get responses within one second, which is a key difference from other tools that are available today, he added.

As it presently works as it was designed, Hadoop does batch processing of large data sets. Drill will improve on that method by doing “interactive analysis” that can find the required answers in the data more quickly, said Shiran. “Interactive analysis is much faster than batch processing.”

The need for tools like Drill has been inspired by always-increasing user requirements, he said. “People have been doing queries in Hadoop, but since it doesn’t return answers to you within a few seconds, it has limitations.”

Ad hoc analysis

Using Drill, users will be able to do ad hoc analysis and get faster responses, whether they are seeking anomalies, data trends or even network intrusions, according to Shiran. “With all of those things, you’re going to have to get a pretty fast response or by the time you do figure it out, it’s going to be old news.”

The nascent Drill open-source project is currently in development and includes a variety of companies and individuals who are working on it right now. “A broad-based effort will be working on this,” said Shiran. “There’s quite a few people actively developing on the project now, so I don’t think it will be a long time before we have an early version released.”

Drill was inspired by Google’s Dremel project, which helps Google perform data analyses on its huge data sets such as analysing crawled web documents, tracking install data for applications on the Android Market, analysing spam, analysing test results on Google’s distributed build system and more, according to Shiran.

By developing Drill as an Apache open-source project, organisers will be able to establish Drill’s own APIs and establish a flexible and robust architecture that will support a broad range of data sources, data formats and query languages, according to the group.

MapR offers two versions of its Hadoop products: MapR M3, which is free; and MapR M5, which is a commercial version of the product with advanced features, including high availability, the ability to make data snapshots and do mirroring of datasets, and 24/7 support.

How well do you know your webcam? Take our quiz.

Todd R. Weiss

Freelance Technology Reporter for TechWeekEurope and eWeek

Recent Posts

Consumer Group Says Major Australian Retailers Using Facial Recognition

Consumer group Choice files with privacy regulator to stop three major Australian retailers from collecting…

18 mins ago

Google Jobs Service Hit By Fresh Antitrust Complaint In Denmark

Danish competitor files fresh antitrust complaint against Google for Jobs, saying its search traffic fell…

1 hour ago

Toshiba Board Set For Vote That Would Back Buyout Strategy

Toshiba board expected to approve appointment of two external directors from hedge fund investors, making…

2 hours ago

Nio ‘Investigating’ Deaths Of Two After EV Plunges From Building

Shanghai electric vehicle maker Nio says it is investigating deaths of two people after one…

3 hours ago

Google Warns Of Italian Spyware On Apple, Android Phones

Italian company's hacking tools have been used to spy on Apple, Android smartphones in Italy…

3 days ago