Finding the Right Business Intelligence Tool is Not So Easy

Which is the best business intelligence (BI) tool? What makes it the best one? What do existing customers think of their BI tools? Is the best business intelligence software also the right one for my business? Which criteria are the most important when comparing BI tools? Is superior functionality the main factor to consider? Or are aspects like vendor and implementer support equally important? And how do independent analysts rate different business intelligence solutions?

Comparing BI tools is no easy task. Especially since there are so many business intelligence products on the market, all promising they will deliver all sorts of business benefits while at the same time solving all manner of problems.

Naturally, companies searching for a new business intelligence software tool can become overwhelmed with all the choices and promises from vendors.

To make the comparison and search for the best BI tool for your company easier, this article sets out to provide you with a balanced view of what BI users – as well as BARC analysts – have to say about the leading business intelligence products on the market.

By combining our in-depth knowledge of current BI software tools with one of the largest databases of user reviews of business intelligence software, we aim to guide you through the first steps to finding the best BI tool for your company.

We believe a combined view of user feedback and in-depth analyst perspective is necessary in order to fully appreciate how business intelligence software tools compare against each other, and to understand which are the best BI products for your company.

In this article you will find out:

  • which are the best BI software tools as rated by users in 2017 (BI User Review Matrix)
  • how BARC analysts compare the products (BARC Score Business Intelligence)
  • the most important lessons learned from surveying more than 30,000 respondents about business intelligence software comparison and selection over the last 15 years
BI tool evaluation

Table of Content

User Comparison of Business Intelligence Software

The BI Survey: The User Comparison

No one knows more about how a business intelligence tool performs in the real world than the customers already using it. All too often, they find that products don’t live up to expectations, or that the business intelligence vendor does not support the BI tool properly. Therefore, when comparing BI software, there is almost nothing more valuable than user reviews.

This is where The BI Survey comes in. Collecting feedback from business intelligence software users every year for the last 15 years, The BI Survey provides detailed analysis of which BI software products get the best user ratings. This year’s BI Survey is based on the analysis of the real-world experience of 3,137 respondents with 2,611 answering a series of detailed questions about their usage of a named business intelligence software tool.

It is the largest and most thorough fact-based analysis of the BI market and software currently available, using over 15 years of experience to analyze market trends and challenge some of the myths surrounding the business intelligence software industry.

This edition features a broad range of BI software tools, not only from the well-known business intelligence giants, but also specialist products from much smaller vendors and open source vendors, that ordinarily don’t get much press but which, in many cases, offer outstanding value to their customers.

Altogether, the BI Survey compares 37 business intelligence tools (or groups of products) in detail. To be included in the detailed analysis a BI tool needs at least 30 user reviews.

Since not all business intelligence tools are alike, we use peer groups to ensure similar products are compared against each other both in fairness to the vendor and for the benefit of readers (see below for more information on the peer groups). The groups are essential to allow fair and useful comparisons of BI tools that are likely to compete.

The BI User Review Matrix rates BI software based on the business value the software provides (x-axis), customer/user experience (y-axis), customer satisfaction (color) and competitiveness (circle size). To see the comparison in action, please select a peer group.

The KPIs

THE BI Survey 16 examines BI product selection and usage among users in categories (KPIs) including business benefits, project success, business value, recommendation, customer satisfaction, customer/user experience, innovation and agility. There are 32 KPIs in total.
Different readers will have their own views on which of these KPIs are important to them. For example, some people will regard fast query performance as critical, whereas others may consider recommendation or innovation to be more important.
Consequently, we think reducing the KPIs to only one aggregated score is too simplistic to be helpful when seeking out the best business intelligence software to match your needs. Reducing all KPIs to just one overall score doesn’t provide much value for the reader. In our view, there are at least four crucial KPIs when it comes to comparing BI tools from a user perspective. They are made up from 23 individual root KPIs:
‘Business value’ is possibly the most important KPI, focusing on bottom-line benefits of BI projects. Business benefits are the real reason for carrying out any BI project. Business intelligence that does not deliver broad business value is superfluous.

The ‘Business value‘ KPI shows how a successful BI software product can provide these benefits in the real world. The KPI combines the ‘Business benefits’, ‘Project success’, ‘Price-to-value perception’, ‘Project length’ and ‘Innovation’ KPIs.

Delivering a superior user experience is more important than ever. BI professionals don’t want to have to spend a lot of time figuring out how a BI product works, attempting to learn interfaces or waiting around for a query to finish. With the current vogue for agility and BI self-service capabilities and the increasing need for users to be able to access a variety of data sources, the user experience of a BI tool is an important consideration for many organizations.

To calculate the quality of user experience of a BI tool, we combine the ‘Ease of use’, ‘Self-service’, ‘Performance satisfaction’, ‘Flexibility for users’ and ‘Recommendation’ KPIs.

Happy customers are a strong indicator of a superior BI software. We combine the ‘Product satisfaction’, ‘Vendor support’ and ‘Implementer support’ KPIs to calculate an aggregated ‘Customer satisfaction’ KPI. These three factors are clearly related: If one is lacking, then the importance of the others is accentuated.
The ‘Competitiveness’ KPI gives insights into how BI tools perform in a competitive selection process as well as the strength of a product’s market presence. It combines the ‘Considered for purchase’, ‘Competitive win rate’ and ‘Chosen as standard’ KPIs.

Recognizing which business intelligence software to compare entails understanding which BI tools have fared well in other organizations’ product selections. This enables users to eliminate ‘losers’ at an early stage in the selection process.

The KPI rules

Only measures that have a clear good/bad trend are used as the basis for KPIs.
KPIs may be based on one or more measures from THE BI Survey.
Only products with samples of at least 20 – 30 (depending on the KPI) for each of the questions that feeds into the KPI are included.
For quantitative data, KPIs are converted to a scale of 1 to 10 (worst to best). A linear min-max transformation is applied, which preserves the order of, and the relative distance between, products‘ scores.

The BI Survey BI User Review Matrix

Check Out the Planning User Review Matrix

Comparing BI Software with the help of peer groups

Not all BI tools are alike. Some are strong on reporting, while others are more suitable for dashboards or analytics.

Results from The BI Survey show the types of capabilities respondents use with their BI tools. Standard and enterprise reporting is top of the list followed by ad hoc query. The least popular BI capability this year is advanced analysis (e.g. predictive analytics or data mining), although it has the highest ‘planned’ rates, with 45 percent of respondents intending to start using advanced analytics in the future.

The peer groups are primarily based on the results from The Survey. They take into account how customers say they use the business intelligence software, which can vary widely across the different products. But we also include the experience and judgment of the BARC analysts in deciding on the groupings.

Peer groups are simply a guide to the reader to help make BI tools easier to compare and to show why individual products return such disparate results. They are not intended to be a judgment of the quality of the BI tools.

Some products appear in more than one peer group. We use these peer groups in a consistent way in our analysis, as well as in The BI Survey Analyzer.

The point to the peer groups is to make sure that the comparisons of the BI tools we make in The Survey make sense. The products are grouped together as we would expect them to appear in a short list for business intelligence software selection.

To make a proper choice, a buyer should first segment the market into the BI tool types that fit his requirements. The peer groups are intended to help with this task.

The segmentation of the peer groups is based on two key factors:

  • Usage scenario – functional peer groups – mainly data-driven and based on how customers say they use the product.
  • Regional focus – is the vendor a large international vendor with a truly global presence or does it focus on a particular region? We also take into account the location of BI Survey respondents.

The regional focus is of particular importance since organizations require local contacts, resources and knowledge from implementers to conduct projects in their own country. Furthermore, locally competing vendors and markets are better reflected and easier to compare with this method of market segmentation. To compare vendors with global coverage, a peer group featuring only large international vendors was created.

The peer groups

Includes products from vendors that have a significant presence in - and focus on - the Americas region.

Includes products that provide integrated functionality for BI and performance management, especially planning and budgeting.

Tasks where BI is used

Tasks Business Intelligence Software is used for

Tasks where BI is used (n=2618)

Want to rate your BI software?

Participate in the world‘s largest and most comprehensive survey of BI software users

Take part in The BI Survey
Analyst Comparison of Business Intelligence Software

BARC Score Business Intelligence: The Analyst Comparison of leading BI tools

BARC Score Business Intelligence compares the strengths and challenges of all the leading vendors in the business intelligence market as well as many smaller vendors that often have less visibility but equally offer outstanding value to their customers.

Based on countless data points and many analyst interactions, business intelligence software vendors are rated on a variety of criteria, from product capabilities and architecture to sales and marketing strategy, financial performance and customer feedback

Calculating the individual ratings for all criteria and all vendors produces two scores per company: the ‘portfolio capabilities’ score and the ‘market execution’ score, which are plotted on each axis resulting in the vendor’s dot on the BARC Score graphic.

‘Portfolio Capabilities’ includes a series of weighted sub-criteria based on functional and technological capabilities, as well as the overall architecture and product satisfaction from an end-user point of view.

‘Market Execution’ takes into account a blend of product, sales, and marketing strategy as well as certain organizational, financial and geographical considerations.

There are two separate inclusion criteria to be reviewed in the BARC Score: the first is associated with a business intelligence software vendor’s products and the other is linked to the financial results relating to those BI tools. To be evaluated in the BARC Score Business Intelligence, a vendor has to supply four out of five technologies from the following platform portfolio:

  • Formatted Reporting
  • Dashboarding
  • Ad hoc Query and Reporting
  • Analysis
  • Data Mining

In addition, the vendor has to generate a minimum of 15 million EUR in license revenue per year with the above product set, spread across at least two separate geographies. As individual geographies we consider:

  • Europe, Middle East and Africa
  • North America
  • Latin America
  • Asia/Pacific

Vendors with an open source business model are compared by their total revenue because those companies do not charge a license for their products, but an annual subscription fee.

The BARC Score Business Intelligence

Business Intelligence Software Comparison by BARC Analysts

The categories

Market leaders are vendors that drive both technology and market adoption in a highly influential manner. They possess both a broad portfolio of market-leading products with a strong brand as well as a robust commercial prowess through best-in-class sales and marketing programs, an extensive ecosystem of business partners and alliances, and a rock-solid financial position. Market leaders are considered a contender in virtually every planned implementation.
Trendsetters are well established vendors that drive strong market adoption, supported by technology innovation and strategic acquisitions and by leveraging robust account management and a solid track record. Their portfolio enjoys high brand awareness in the market, covers an extensive range of technologies and services with only few gaps. Trendsetters typically have a large market share, making them a viable contender in almost all implementation scenarios.
Challengers come in various shapes and sizes. They can be large vendors tapping into a new market by acquisition and pushing their way in with force, small innovative companies with a promising portfolio but limited sales and marketing resources, or vendors that attempt to disrupt a market with a new technology approach or different business model.
Specialists are smaller vendors with a portfolio focused on a specific market segment. Vendors can be either limited in their technical capabilities by concentrating on certain features and functions, or the company isn’t a global enterprise and focuses on select geographic regions.
Entrants are usually startups that have limited reach and visibility in the market. Their product capabilities are incomplete when compared to the respective competition, and the vendor’s long-term market potential is still unproven.
Don‘t miss out – Get the latest BI product insights, research, surveys and more!
Lessons Learned for BI Comparison

What We’ve Learned About Comparing Business Intelligence Software

Why the choice and comparison of business intelligence products is important when it comes to finding the best BI tool

Business intelligence projects are about creating business value by enabling better business decisions. An important – though not the only – part of this value creation is derived from the BI software products that gather, enrich, store and present data to end users, business analysts, or even data scientists.

Choosing a software product that doesn’t match your requirements can pose a serious threat to the success of your project. Survey results show major differences in the business benefits achieved with different BI tools.

The Business Benefits Index (BBI) is a unique measure used in The BI Survey to determine the value of BI projects. It takes both quantitative and qualitative benefits into account as well as different levels of possible benefit achievement. Here you can find more detailed information on how we calculate the BBI and how we classify best-in-class companies.

Business Benefits of the best and worst BI products
Percentage of businesses doing a competitive BI software comparison

Percentage of businesses doing a competitive BI software evaluation (n=2189)

Business benefits by selection type

Business benefits by selection type (n=2189)

How to select a BI product

Selecting the best business intelligence software is key to a successful project, and many software problems can be traced back to poor software selection processes.

The software is often used for ten years or more so the total cost of ownership (TCO) can be substantial. Investing time in choosing the best business intelligence software tool will have positive long-term effects.

Before purchasing and committing to any BI tool, we always recommend that organizations conduct detailed, formal evaluations of multiple BI tools on the market.

However, we are aware that not everyone follows this advice. The data collected in The BI Survey allows us to draw some definite conclusions on this topic.

The findings from this year’s Survey reflect those stretching back to 2002 in that business benefits improve when a company performs a product evaluation and benefits are further improved if enterprises extend the formal assessment process to a multi-vendor comparison.

What are the right criteria for evaluating BI software

The criteria best-in-class companies use for a selecting their BI software confirm that successful projects are based on selecting the tool rather than the vendor.

This is one of the most consistent findings over The BI Survey’s fourteen-year history.

  • Best-in-class companies take more selection criteria into account than laggards.
  • Best-in-class companies are much more interested in functionality, ease of use and fast query performance; all key considerations in enabling self-service BI.
  • Laggards, on the other hand, are more concerned with non product-related criteria such as ‘vendor listed as corporate standard’ or ‘bundled with another product’.
Reasons to buy BI software

Reasons for buying BI software tool (n=2434)

Implementation time for BI software

Median implementation time (n=2260)

Why implementation speed matters for a BI tool

Countless IT projects experience delays and data warehousing and business intelligence projects are by no means immune to this perennial issue.

Implementation time depends largely on the product being implemented. Naturally, the complexity of the BI solution and/or the project will impact on project length and certain BI products lend themselves better to more complex projects, thus increasing their average implementation time.

Clearly, other factors such as the number of data sources, the volume of data and the number of users or departments served also have an influence here.

Fast implementation times are key to a successful BI project. Projects taking more than three months to complete lead to significantly lower business benefits. Project implementation times appear to have a direct impact on the level of all business benefits; with benefits decreasing as project length increases.

Aim for a three months implementation window for the first application. Best-in-Class companies prove that it can be done in just two months on average.

If possible, try an incremental approach by breaking the project into a series of smaller projects.

Why good support is especially important

The standard of service provided by consultants varies considerably. The BI Survey asks business intelligence users about their experiences with external implementer support as well as vendor support. We found a clear correlation between customer ratings of implementer and vendor support and business benefits achieved.

Seeing such a big difference in business benefits between different types of consultants does not come as a surprise. Good support from an implementation consultant is an important aspect of project success.

Finding the right vendor in combination with the right implementer is crucial to project success. In previous editions of The BI Survey, we used to ask about the type of consultants companies used. Results revealed that small, specialized consulting firms frequently scored better than large general purpose consulting companies.

The success of business intelligence projects depends upon organizations’ and vendors’ abilities to deliver successful and relatively rapid implementations, along with supportive follow-up and maintenance.

Many BI solutions end up as shelfware within enterprises, due to difficulties in setting up software, difficulty in using tools, or lack of ability to communicate the benefits of the tools and platforms to end-users.

The ability to seamlessly incorporate business intelligence software and platforms into corporate decision-making processes and to closely align those solutions into business results is crucial.

Support for BI tool analyzed by business benefits
Percent of users having no problems with BI tool

No significant problems with the BI tool by
best-in-class companies (n=2586)

Avoiding problems

Survey results clearly reveal that a carefully conducted comparison and selection of business intelligence tools pays off. The organizations that achieve the greatest level of business benefits with business intelligence (Best-in-class companies) are much more likely to have conducted a competitive BI tool evaluation, and they experience far fewer problems with their business intelligence software.