Splunk Gets High Availability Update

Big data integration firm Splunk has updated its software to make it faster and increase its reliability.

The firm, which specialises in collecting and monitoring large amounts of machine generated data, believes that reliability will cut users’ costs, by obviating the need to keep multiple copies of huge datasets, and encourage more people to give Splunk a bigger role in their Big Data projects.

Do ya feel lucky, Splunk?

“If you can turn it into data, you can put it into Splunk,” said D J Skillman, Splunk’s European technical director. “We are very flexible on the inject side, and solve the massive problem of data collection.”

Splunk connects to other Big Data tools, but wants to expand its role beyond collection and, where possible, become the basic system used to hold and manipulate the data. Splunk 5 includes new “platform” style features to encourage developers to do more within the tool, such as software development kits (SDKs) for Python, JavaScript and PHP.

The new version of Splunk moves faster, so reports can be produced up to 1000 times more quickly, but Skillman believes the increase in resilience is a more important factor.

“Resilience used to be built in by making copies and using a SAN [storage area network], but if you are collecting 5Tbyte a day in a SAN environment, the cost is astronomical,” he said. Building in resiliency in the software means the same performance and reliability can be done on commodity hardware, he said.

Splunk is not trying to take on relational databases – Skillman says they are still best for old-style transactional data like customer records, and it also leaves a lot of the “unstructured” data such as voice and video to other players, but concentrates on users who generate masses of text data from machine sources.

Splunk also now has a bi-directional connector to Hadoop, which Skillman says is, again, about allowing for use of the best tool for the  job. “Hadoop is very inexpensive, and massively flexible for large dataset batch processing, but you have to program it.” Users need to have big data scientists to get the most out of Hadoop, he said.

Splunk is not open source, but operates on a “freemium” model whereby users can download and use the software for free, until the amount of data stored each day gets larger than 400MB. Above that, detailed pricing isn’t published, but is understood to start at $10,000 for a licence to store 1GB a day, falling rapidly as volumes rise.

Do you know about Europe’s role in tech history? Try our quiz!