How To Evaluate Software

Success is reflected in the vast differences in business benefits achieved using various software solutions. However, not all organizations pay due attention to the software selection process. Ideally, any decision to buy a tool should be preceded by a detailed, formal assessment of multiple tools.

Software selection processes have important strategic effects on companies. In large companies, the selection process itself can become very complex because there are many different issues to deal with. In companies with a wide geographical spread it can be hard to coordinate discussions.

From a vendor’s point of view, it is in their interest not to have to go through a formal competitive evaluation process.

Buyers also think of software selection as a confusing, time-consuming and possibly expensive process. It might therefore be tempting to skip this process and opt directly for a product that is:

Data collected in The BI Survey reveals that to dispense with any kind of formal software selection process is detrimental to achieving business benefits. We therefore recommend that organizations should carry out detailed, formal evaluations of multiple products before committing to the purchase of any tool.

However, The BI Survey found that 18 percent of respondents looked at only one product while another 20 percent conducted no formal evaluation at all. This figure has gone up slightly in recent years, and is still at a high level.

Frequency of evaluation method (n=2189)

Further analysis reveals the varying approaches taken in different countries/regions. The following chart shows the distribution of these evaluation methods in important IT markets with high sample sizes in The BI Survey 16.

Frequency of evaluation method by country or region (n=2189)

Overall, we see a correlation between evaluation methods and the realization of business benefits. Best-in-class companies carry out competitive evaluations much more regularly than laggards, while laggards are more than three times more likely than best-in-class companies to carry out no formal evaluation at all

Best-in-class companies by evaluation method (n=2189)

The Survey reveals the positive impact of a formal tool evaluation on improving business benefits.

The following chart compares business benefits by evaluation method showing which benefits are most likely to be achieved from BI projects. These benefits include faster and more accurate reporting, analysis or planning; better business decisions; and improved employee satisfaction and data quality.

Those benefits achieved less regularly tend to relate to quantifiable effects such as reduced costs, increased revenues and saved headcount.

Our analysts have found that business intelligence users are rarely able to calculate the ‘hard return’ of their projects. Instead, they notice qualitative improvements in the speed and accuracy of decision support.

Business benefits analyzed by evaluation method (n=2189)

Firstly, a formal evaluation helps organizations choose products that are well suited to their needs.

Also, the evaluation process itself provides a deeper insight into organizational requirements.

And finally, the evaluation process sets a realistic level of expectation among stakeholders as to what can be achieved with a particular solution.

Results from The BI Survey show that projects incorporating a formal multi-product evaluation deliver more benefits than those evaluating just one product. In turn, those projects evaluating a single tool deliver more benefits than projects where a favored product is purchased with no formal evaluation.

A contributory factor here could be that well-run, high profile projects — which are more likely to succeed anyway — have a greater tendency to include a formal, professionally conducted software selection phase.

The advantage of conducting a formal evaluation is not a recent finding. The following is a quote from The OLAP Survey 2 in 2002:

“Very strikingly, those who had performed a formal multi-product evaluation reported higher achievements of every single one of our listed possible benefits, as well as slightly higher goal fulfillment.”