The goal of the KPIs is to help the reader spot winners and losers in The BI Survey 16 using well-designed dashboards packed with concise information. The Survey includes 7 aggregated KPIs, which can be absorbed at a glance. It also includes a set of 25 normalized KPIs, which we refer to as ‘root’ KPIs for each of the 37 products. The ‘aggregated’ KPIs are aggregations of these root KPIs.
This year we have calculated a set of KPIs for each of the nine peer groups. The values are normalized on the whole sample. Peer groups are used to enable fair and useful comparisons of products that are likely to compete.
The KPIs all follow these simple rules:
Only measures that have a clear good/bad trend are used as the basis for KPIs.
KPIs may be based on one or more measures from The BI Survey.
Only products with samples of at least 20 – 30 (depending on the KPI) for each of the questions that feeds into the KPI are included.
For quantitative data, KPIs are converted to a scale of 1 to 10 (worst to best).
A linear min-max transformation is applied, which preserves the order of, and the relative distance between, products‘ scores.
In some instances, adjustments are made to account for extreme outliers.
KPIs are only calculated if the samples have at least 15 – 30 data points (this varies depending on the KPI) and if the KPI in question is applicable to a product. Therefore some products do not have a full set of root KPIs. It is important to exclude KPIs based on small (and therefore not representative) samples to ensure that the graph scales are not distorted by outlier KPIs.
Measuring business benefits – the BBI and best-in-class companies
Business benefits are the real reason for carrying out any BI project and The BI Survey has been studying them directly for years. We ask respondents the extent to which they realize a list of benefits with their BI tools.
For each potential benefit, respondents are asked to indicate the level of achievement, if any, with five levels. We use a weighted scoring system to derive a composite score for each of the possible benefits, based on the level of benefit achieved. We call this the BBI (Business Benefits Index).
Participants were asked to rate each benefit. Business benefits are calculated by counting the number of each reported level of benefit and multiplying this number by the corresponding weighting. The products are then divided by the number of responses for that particular benefit to find the average response.
This rating system is the basis of the most important index in The BI Survey. It is a dimensionless number with an arbitrary value, but as long as the weighting system remains constant it can be used for comparisons between segments of the sample, such as the sample for individual products or regions, to name just two.
Throughout The BI Survey we use a classification of best-in-class companies and laggards. Best-in-class companies comprise the top 10 percent of companies, based on their achievement of business benefits, while laggards are defined as the 10 percent of companies achieving the lowest level of business benefits. This classification enables us to examine correlations between best-in-class/laggard companies and high (or low) KPI scores, and to identify what the most successful companies do differently to laggards.
For a more detailed description on the KPIs and how we calculated them see our KPIs & Dashboards PDF.