The goal of the KPIs is to help the reader spot winners and losers in The BI & Analytics Survey 21 using well designed dashboards packed with concise information. The BI & Analytics Survey includes 6 aggregated KPIs, which can be absorbed at a glance. It also includes a set of 30 normalized KPIs, which we refer to as ‘root’ KPIs for each of the 33 products. The ‘aggregated’ KPIs are aggregations of these root KPIs.
This year we have calculated a set of KPIs for each of the ten peer groups. The values are normalized on the whole sample. Peer groups are used to enable fair and useful comparisons of products that are likely to compete.
The KPIs all follow these simple rules:
Only measures that have a clear good/bad trend are used as the basis for KPIs.
KPIs may be based on one or more measures from The BI & Analytics Survey.
Only products with samples of at least 20 – 30 (depending on the KPI) for each of the questions that feeds into the KPI are included.
For quantitative data, KPIs are converted to a scale of 1 to 10 (worst to best).
A linear min-max transformation is applied, which preserves the order of, and the relative distance between, products‘ scores.
In some instances, adjustments are made to account for extreme outliers.
KPIs are only calculated if the samples have at least 15-30 data points (this varies depending on the KPI) and if the KPI in question is applicable to a product. Therefore some products do not have a full set of root KPIs. It is important to exclude KPIs based on small (and therefore not representative) samples to ensure that the graph scales are not distorted by outlier KPIs.
Measuring business benefits – the BBI and best-in-class companies
Business benefits are the real reason for carrying out any BI project and The BI & Analytics Survey has been studying them directly for years. We ask respondents the extent to which they realize a list of benefits with their BI tools.
For each potential benefit, respondents are asked to indicate the level of achievement, if any, with five levels. We use a weighted scoring system to derive a composite score for each of the possible benefits, based on the level of benefit achieved. We call this the BBI (Business Benefits Index).
This rating system is the basis of the most important index in The BI & Analytics Survey. It is a dimensionless number with an arbitrary value, but as long as the weighting system remains constant it can be used for comparisons between segments of the sample, such as the sample for individual products or regions, to name just two.
Throughout The BI & Analytics Survey we use a classification of best-in-class companies and laggards. Best-in-class companies comprise the top 10 percent of companies, based on their achievement of business benefits, while laggards are defined as the 10 percent of companies achieving the lowest level of business benefits. This classification enables us to examine correlations between best-in-class/laggard companies and high (or low) KPI scores, and to identify what the most successful companies do differently to laggards.
For a more detailed description on the KPIs and how we calculated them see our Sample, KPIs and Methodology PDF.