How important are analyst evaluations of software vendor solutions and are there any blind spots?

I came across an article the other day regarding NetScout, a provider of network performance management products that is suing Gartner Research over their magic quadrant placement, which they were obviously unhappy with. NetScout is claiming that the vendors that pay for Gartner Research services are ranked above those that do not. This article brought to mind a couple of questions regarding research analysts. The first is just how important is the magic quadrant placement and ratings in general? The second one is one that I have thought about for a while, do analysts have a blind spot with regards to broader market needs?

There are some organizations out there that believe that analyst reports and rankings are the gold standard, and will not accept a vendor unless they are on the list. However, my experience has been that the majority take a more pragmatic approach. Many construct a fairly comprehensive vendor evaluation process, which include request for proposals (RFPs). Every organization is different and a solution that makes sense for one organization may not make sense for another. It is imperative that organizations understand their environments and connect the dots between their needs and the available solutions on the market.

There are many elements that need to be considered in a software purchase decision, such as price, functionality, implementation and training costs, enhancements and overall maintainability. Each element will have a different weight from organization to organization. Some will be more price sensitive than others, while others will require different functional capabilities. Measuring and matching all of these elements against the needs of an organization can be a very tedious and time consuming process, but it is time well spent to ensure that you end with the best solution for your organization.

While analysts firms provide good quality analysis of software vendor solutions, there is one area that is typically a blind spot for them. One of the metrics that analysts typically rate vendors on is their product functionality. While a vendor can get dinged for having a difficult or cluttered interface, more functionality in one direction or another generally translates to a positive. The blind spot in this is that not every organization wants or needs all of the functionality that a solution has. It’s the old 80/20 rule in software that states 80% of the users will only use 20% of the features of a solution. So the question is, who are the analysts targeting their reports at, the 80% or the 20? And, is it made clear in their analysis which one it is? If a software vendor attempts to target the 80% with less functionality that are much more pertinent to them, they are very likely to get dinged for not being a visionary. The unintended consequence of all of this is a functionality arms race among vendors vying to be positively rated by analysts. In many cases, they are essentially developing features for 20% of organizations that the other 80% don’t need and will likely never use. Another unintended consequence is that more functionality generally translates into a larger and more complicated product. This complexity generally leads to complicated implementations, higher end hardware, longer learning curves, which leads to a larger price tag.

Is Gartner Research operating a pay-to-play business model? While I have watched the marketing people in the software companies that I have worked for fall all over themselves to wine and dine analysts, I have seen no indication that it bought us any favor other than a more attentive ear. Are companies really blindly outsourcing their technology decisions to third party analysts? I don’t think so, but NetScout seems to think so or maybe there is an unseen agenda behind all of this.

I hope this information has been useful to you and as always, I welcome any comments. Please check out Vallum and our partner the GMI-Foundation.

About the Author:

Lance Edelman is a technology professional with 25+ years of experience in enterprise software, security, document management and network management. He is co-founder and CEO at Vallum Software and currently lives in Atlanta, GA.