Machines generate a lot of data. A single Linux or Windows server alone generates more logs than anyone has the time to examine manually. But within that data lies important insights and action items that can save your organization from the next cyber attack, prevent a mission-critical system from crashing, make predictions about future capacity needs, or increase sales by understanding user behavior on your website.
Sadly, much of the time this valuable lode goes unmined. It is only after a serious event that IT pros feverishly search through this data in an attempt to figure out what exactly happened.
Splunk makes mining this data easy and automatic. By using its powerful engine, intuitive user interface, and robust query language, IT professionals can create reports, dashboards, and alerts based on data generated from machines, databases, IoT sensors, HTTP outputs, syslogs, or almost any other source. Instead of being reactive, IT pros can now be proactive and even predictive. Splunk makes machine data work for IT pros, and in turn, is helping make IT a better, more capable business function.
Here are the key benefits of Splunk for IT:
Splunk is a veritable “Swiss Army knife” of the IT world. Splunk can ingest data from nearly any source imaginable. (I haven’t encountered a data source that Splunk cannot consume, but it might exist.)
Structured data from relational databases, Hadoop clusters, CSV files and the like are straightforward and easily consumed by Splunk. But any common business intelligence tool can capture and analyze structured data (I’m looking at you, Tableau).
Where Splunk excels is in the capturing and analyzing of unstructured, machine-generated data—the type of data that is immensely important to IT professionals. This data comes from many sources, and much of it from logs generated by information systems: servers, hypervisors, containers, firewalls, routers, sensors, databases, files/directories, and many more.
Machines are great at documenting everything that happens to them or by them. They are not great at generating actionable knowledge from that data, or even making that data readable to humans.
Splunk can use machine learning to look for patterns in the data. This means that Splunk can help IT pros predict an imminent cyber attack, alert them when an action will make the disk reach its maximum storage capacity, or predict when a backup will fail. It can also, of course, help you make standard business intelligence predictions based on data from databases and other business sources.
Part of the appeal of Splunk is its highly active community, which includes Splunk employees, customers, and enthusiasts. I visited their trendy headquarters in San Francisco in 2017, and it felt like I walked into an episode of HBO’s Silicon Valley. The headquarters had a quirky, laid-back feel, and the people were focused on making a great product and having fun while doing it.
Because of this attitude, Splunk has enabled a great and highly participative user community (a “fanbase” would perhaps be more accurate). The open-source extensibility of Splunk through APIs and SDKs means that thousands of add-ons and apps are available (most of them are free), with more being added frequently. These apps are created by vendors, users, and Splunk engineers. They extend the ability of Splunk by offering pre-defined data models, reports, dashboards, and much more. Anyone can make an app, and therefore Splunk is much more than just its core products—it’s an essential and evolving tool for any IT organization.
Splunk is not a mere business intelligence (BI) tool like Tableau. BI tools rely on mostly structured data and particular types of data sources to derive business insights. With Splunk, IT professionals can combine BI functionality with the powerful insight of unstructured machine data to get a more holistic view of the organization. This mix of structured and unstructured data is, in my opinion, the future of data. Splunk can be the single data analytics tool or complement the existing BI tools, and like most products, Splunk is not the only solution that does this.
Here’s how Spunk compares to existing alternatives out there:
A popular, open source alternative to Splunk is Elastic Stack (formerly ELK). Elastic Stack is really four open source products combined: Kibana, Elasticsearch, Beats, and Logstash. When these products are working together, they can replicate much of the functionality of Splunk, but Elastic Stack requires installation, configuration, and integration of a modular system, whereas Splunk bundles its core functionality as a complete package.
Sumo Logic is another popular alternative and leader of the “unseat Splunk’s throne” club. Sumo Logic has the advantage of being born in the cloud. Splunk’s cloud-native offering is a fairly recent development. Sumo Logic also has the advantage of being less expensive than Splunk.
Sumo Logic is still new, so the availability of plugins and apps do not compare with the size of Splunk’s “Splunkbase” (app store). In addition, as a newer startup, they seem to be struggling with their support options. With Splunk, IT pros can take advantage of Splunk Answers (A Stack Overflow-like Q and A site), and best-of-class premium support.
Planning and design can go a long way in making sure you have a successful and productive experience with Splunk. I recommend getting started with a trial of Splunk cloud and analyzing a small subset of data that represents the type of data for which you intend to use Splunk.
The next step is to consider your use case, and plan the appropriate resources and architecture. Splunk has best practice deployment models depending on the size of the user-base and the volume of expected data. Since Splunk is high performing, each search head (primary interface machine) will need 16 CPU cores and 16GB RAM. A small business may only have one search head, but large firms can have dozens.
Splunk is not cheap. For a medium enterprise deployment, in addition to licensing costs, to realize ROI from Splunk, you will likely need the equivalent resources of a full-time employee. This is because of the changing nature of data—cyber-attacks, systems, log file formats, and other variables. The needs of the IT organization and business will change as well and require new dashboards, alerts, and reports.
Training is essential for any Splunk implementation. Splunk is one of those “easy to get started but challenging to master” products. It’s critical that your IT team learns the ins and outs of Splunk. Find out how Udemy for Business can help train your IT team for a Spunk deployment. Request demo.
As a first step, my Udemy for Business course, The Complete Splunk Beginner Course, can help get your IT team ready for a Splunk deployment. This course covers the basics of SPL, which is Splunk’s “Search Processing Language.” SPL is a basic query and manipulation language that is like a cross between SQL and Linux shell, though not nearly as complicated as either. You don’t need to be a software developer to learn and understand SPL, and once you learn the basics, you will likely realize how simple and straightforward it is.
Architecture planning is important because there are many different ways to deploy a Splunk environment. In the course, we will deploy Splunk in a few different environments, analyze data from a provided dataset, build data models, design dashboards, and create reports and alerts.
An understanding of Linux is helpful, even if you are working in a Windows environment. Splunk was written in C/C++ and uses many of the conventions of Linux (SPL uses the pipe | function frequently, for example).
Splunk is one of the most powerful and useful tools I have encountered in my travels, and I’ve been in IT for a long time. By planning ahead, instituting the proper training curriculum, and employing your background of basic IT infrastructure knowledge, you can become the IT hero.
Udemy for Business adds hundreds of new courses each month on the latest technical and soft skills to help employees...
The AWS Certified Developer Associate exam is for developers who wish to demonstrate their ability to develop applications native to...
The world we work in today is changing rapidly. In response to this new norm, companies are shifting to a...