The BI Survey 16 follows on from 15 successful editions of The BI – and former OLAP – Surveys.

Based on a sample of over 3,000 survey responses, The BI Survey 16 offers an unsurpassed level of user feedback on 37 leading Business Intelligence solutions.

Our research covers issues ranging from the purchase cycle right through to deployment, including critical information on performance levels, scalability and problems encountered.

Click the links below for in-depth descriptions of our survey methodology, the sample, the KPIs, the peer groups and much more:

Methodology & Sample
KPIs & Dashboards
0
Respondents
0
BI Products
0
Countries
0
Years

Components of The BI Survey

Instead of delivering one long document covering all aspects of The BI Survey, we have split the information into several smaller documents. By providing the raw data via a Web-based tool – The BI Survey Analyzer – users have the ability to carry out their own analysis of The Survey results. The BI Survey 16 is divided into several documents, as listed below.

The Survey documents do not need to be read in sequence. ‘The Results’, ‘Best Practices’ and the ‘Vendor Performance Summaries’ can be read independently.

The Results

An overview and analysis of the most important product-related findings and topical results from The BI Survey 16.

The Analyzer

Our powerful interactive online tool, enabling you to perform your own custom analysis of the full survey data set.

Best Practices

Provides advice to buyers of BI software as well as users and administrators of existing BI solutions based on the results of our analysis.

Vendor Performance Summaries

A series of executive reports on each product featured in The BI Survey 16. Each report contains an in-depth product review by BARC’s analyst team plus all of the relevant product-related results from The BI Survey.

Want to buy The BI Survey?

Are you interested in seeing the whole picture?

Contact us

Sample & Methodology

The value of a survey like this depends on having a sufficiently large, well-distributed and unbiased sample. This section describes the characteristics of the people who took part in the study.

The BI Survey 16 has the largest sample of any survey of business intelligence users available on the market. The BI Survey has a rule that, as far as possible, only sub-samples containing 30 or more data points should be reported.

We apply increasingly stringent data cleansing rules, using a number of different tests. We remove all suspect data that purports to be from user sites.

Most surveys are conducted or sponsored by an organization based in, and focused on, one country. However, Business Intelligence and Analytics is a worldwide market and we wanted, as far as possible, to capture a large international sample. This not only presents a more accurate global picture but also allows international variations to be analyzed.

The largest Business Intelligence and Analytics markets are the United States, Germany, France and the United Kingdom, so The BI Survey 16 was produced as a collaboration between organizations in each of these countries, and in partnership with publishers and vendors in these and other countries. It features not just the well-known US products, but also products from other regions including Europe and Australia.

The sample of the BI Survey

Sample size and make-up

Hundreds of thousands of people around the world were invited to participate in The BI Survey 16, using dozens of email lists, magazines and Web sites.

As in previous years, the questionnaire offered different sets of questions for vendors and users (or consultants answering on behalf of users). This seems to produce better quality data as in the past some vendors pretended to be users when they saw they were not being asked relevant questions.

Participants from last year who indicated that they would like to be part of our panel received a pre-filled questionnaire with answers from last year’s questions. They were asked to update their responses, and then to answer the new questions in this year’s Survey.

The results of the online data collected are shown in the table, with the numbers of responses removed also displayed.

The number of responses is split between users, consultants and vendors. Vendors answered a different set of questions to those answered by users.

Responses removed from the samplesResponses
Total responses3,137
Filtered during data cleansing70
Remaining after data cleansing3,067
Not yet considered buying-111
Total answering questions2,956
Total responses analyzedResponses
Users2,262
Consultants389
All users2,651
Vendors/Resellers305

Geographical distribution

One of the key objectives of The BI Survey is to achieve a geographically balanced sample that reflects the current global market for BI products. Therefore the online questionnaire was published in three languages: English, German, and French.

Having a geographically balanced sample has a major benefit: Results of The BI Survey are more closely representative of the world market, rather than being largely based on US experience, as is the case with many other surveys.

In regions where knowledge of English is sparse, such as South America and much of Asia and southern Europe, it is difficult to obtain a good level of feedback and the BI market is less mature in these countries.

Since the fourth edition of The BI Survey, we have significantly boosted the German sample by specifically targeting users in German-speaking countries, using a fully translated online questionnaire.

We also used a French questionnaire, further increasing our European coverage.

The net result is an extraordinarily international panel. Respondents were located in 90 countries. Five countries had 100 or more respondents, and twelve had 50 or more; 26 countries had ten or more respondents.

The regional distribution of the BI Survey

Respondents analyzed by region (n=3056)

Organization sizes by headcount

Business Intelligence and Analytics products are most commonly found in large organizations and a high percentage of the responses we receive are from users in companies with more than 2,500 employees.

Nevertheless, responses from small organizations have been catching up over the years.

The split between respondents from small and large enterprises is well balanced this year.

The distribution of the BI Survey by company size

Frequency of employee count in respondent organization (n= 2879)

Vertical markets

We asked all respondents their company’s industry sector. The chart shows the results of this question and only includes data from respondents who answered product-related questions in The Survey.

Manufacturing dominates the list, as it has in previous years.

BI Survey respondents analyzed by industry

Respondents analyzed by industry (n=2758)

BI Products in The BI Survey

At least 30 user reviews are necessary to be included in the detailed BI product analysis. In this year’s BI Survey we analyze 37 BI tools (or groups of products) in detail.

When grouping and describing the Business Intelligence solutions featured in The BI Survey, we did not strictly follow the naming conventions that the vendors use. In some cases, we combined various BI products to make analysis more convenient. Note that the names we use in The BI Survey are our own and are not always the official product names used by the vendors.

One of the key reasons for this is that the products we analyze are not necessarily the latest version of the tool. Vendors will often change a product’s name between versions, making it difficult to have a single official name for several versions of the same product.

Another reason is that we sometimes bundle related products into a single group to increase the sample size, even if the vendor prefers to view them as distinct for marketing reasons. In both these cases, the point is not to challenge the naming conventions of the vendor, but simply to reduce the complexity of the Survey findings for the convenience of the reader. In some cases, we also shorten the names of the products to improve the formatting of the charts.

We asked respondents explicitly about their experiences with products from a pre-defined list, with the option to nominate other products. This list is updated each year and is based on the sample size of the products in the previous year, as well as additional new products in the BI market. In cases where respondents said they were using an ‘other’ product, but from the context it was clear that they were actually using one of the listed products, we reclassified their data accordingly.

We solicited responses on all surviving products with more than a minimal response in the last BI Survey, plus a few others whose numbers have potentially grown to the point where there is enough data to be analyzed.

The table shows the products included in the detailed analysis.

The last few years have seen an increase in the proportion of German respondents. This is partly due to cooperation with German vendors and the presence of strong German subsidiaries of international vendors, reflected through products like arcplan, Bissantz, BOARD, Cubeware and SAP.

This year a few vendors appear in The BI Survey for the first time. These include Sisense, Zoho and CXO-Cockpit.

ProductNumber of reviews
arcplan (Longview)36
Bissantz94
BOARD58
CALUMO38
Cubeware77
cubus89
CXO-Cockpit36
Cyberscience79
Dimensional Insight37
Dundas31
IBM Cognos BI103
IBM Cognos TM133
Infor56
Information Builders31
Jedox44
Logi Analytics32
MS Excel200
MS Power BI44
MS SSRS88
MicroStrategy107
Oracle BI43
Phocas35
prevero32
Pyramid Analytics36
Qlik Sense43
QlikView239
SAP BEx92
SAP BO Analysis37
SAP BO Design St.32
SAP BO WebI127
SAS37
Sisense50
Tableau111
Targit34
Tibco28
Yellowfin38
Zoho Reports39

Want to rate your BI software?

Participate in the world‘s largest and most comprehensive survey of BI software users

Take part in The BI Survey

The peer groups

THE BI Survey 16 features a range of different types of BI tools so we use peer groups to help identify competing Business Intelligence products. They are used to ensure similar BI tools are compared against each other both in fairness to the vendor and for the benefit of the reader.

The groups are essential to allow fair and useful comparisons of BI tools that are likely to compete. The peer groups are primarily based on the results from The Survey, how customers say they use their products and our knowledge of those products.

Segmentation is based on two key factors:

  • Usage scenario – functional peer groups which are mainly data-driven and based on how customers say they use their products.
  • Regional focus – is the vendor a large international vendor with a truly global presence or does it focus on a particular region? We also take into account the location of BI Survey respondents.

Peer groups act as a guide to the reader to help make the products easier to understand and to show why individual products return such disparate results. They are not intended to be a judgment of the quality of the products. Most products appear in more than one peer group.

These peer groups are used in a consistent way in our analysis as well as in The BI Survey Analyzer.

Includes products from vendors that have a significant presence in - and focus on - the Americas region.

Includes products that provide integrated functionality for BI and performance management, especially planning and budgeting.

The KPIs

The goal of the KPIs is to help the reader spot winners and losers in The BI Survey 16 using well-designed dashboards packed with concise information. The Survey includes 7 aggregated KPIs, which can be absorbed at a glance. It also includes a set of 25 normalized KPIs, which we refer to as ‘root’ KPIs for each of the 37 products. The ‘aggregated’ KPIs are aggregations of these root KPIs.

This year we have calculated a set of KPIs for each of the nine peer groups. The values are normalized on the whole sample. Peer groups are used to enable fair and useful comparisons of products that are likely to compete.

The KPIs all follow these simple rules:

  • Only measures that have a clear good/bad trend are used as the basis for KPIs.
  • KPIs may be based on one or more measures from The BI Survey.
  • Only products with samples of at least 20 – 30 (depending on the KPI) for each of the questions that feeds into the KPI are included.
  • For quantitative data, KPIs are converted to a scale of 1 to 10 (worst to best).
  • A linear min-max transformation is applied, which preserves the order of, and the relative distance between, products‘ scores.
  • In some instances, adjustments are made to account for extreme outliers.

KPIs are only calculated if the samples have at least 15 – 30 data points (this varies depending on the KPI) and if the KPI in question is applicable to a product. Therefore some products do not have a full set of root KPIs. It is important to exclude KPIs based on small (and therefore not representative) samples to ensure that the graph scales are not distorted by outlier KPIs.

Measuring business benefits – the BBI and best-in-class companies

Business benefits are the real reason for carrying out any BI project and The BI Survey has been studying them directly for years. We ask respondents the extent to which they realize a list of benefits with their BI tools.

For each potential benefit, respondents are asked to indicate the level of achievement, if any, with five levels. We use a weighted scoring system to derive a composite score for each of the possible benefits, based on the level of benefit achieved. We call this the BBI (Business Benefits Index).

Participants were asked to rate each benefit. Business benefits are calculated by counting the number of each reported level of benefit and multiplying this number by the corresponding weighting. The products are then divided by the number of responses for that particular benefit to find the average response.

This rating system is the basis of the most important index in The BI Survey. It is a dimensionless number with an arbitrary value, but as long as the weighting system remains constant it can be used for comparisons between segments of the sample, such as the sample for individual products or regions, to name just two.

Throughout The BI Survey we use a classification of best-in-class companies and laggards. Best-in-class companies comprise the top 10 percent of companies, based on their achievement of business benefits, while laggards are defined as the 10 percent of companies achieving the lowest level of business benefits. This classification enables us to examine correlations between best-in-class/laggard companies and high (or low) KPI scores, and to identify what the most successful companies do differently to laggards.

For a more detailed description on the KPIs and how we calculated them see our KPIs & Dashboards PDF.

Aggregated KPIsRoot KPIs
Business ValueBusiness benefits
Project success
Price-to-value perception
Innovation (Aggregated)
Big Data Analytics
Cloud BI
Collaboration
Data discovery/visualization
Mobile BI
Visual design standards
Location intelligence
Operational BI
Project length
CompetitivenessConsidered for purchase
Competitive win rate
Chosen as standard
Customer SatisfactionProduct satisfaction
Vendor support
Implementer support
Customer ExperiencePerformance satisfaction
Ease of use
Recommendation
Self-service
Flexibility for users
Data volume
Query performance
PerformanceAggregation of performance satisfaction & query peformance
AgilityAggregation of project length, self-service & flexibility for users