We had a certain well-paid consultancy firm in the other day to tell a group of us credit unions what we’re doing right and wrong with our Internet web sites. It’s an annual process we’ve done the past two years. They rank us and the banks and everyone concerned gets to wave a flag when they win.
I’ve never been a big fan of heurestic reviews, so-called ‘experts’, or of well-meaning consultants who bury you in reports and best practices. In the end we scored well, though not as good as last year (not bad for a team of one with a small budget). It was surprising to see some peers already planning change requests as though this feedback was from God himself and would solve all their web woes. Ridiculous. It’s like driving your web strategy for the wrong customer.
Anyhow, when I got to the office today I spent some time going through the detailed methodology and scoring and did see some issues worth keeping in mind should you undertake similar reviews in the future. For fun, I’m going to phrase these like this consultancy titles some of its reports.
Three things reviewers need to do when providing heuristic feedback/reviews
- Reviewers need to be consistent. I couldn’t believe the number of contradictions in the detailed scorecard. In some cases we received contrary marks for the same piece of functionality in two different places. In other cases features were outright missed. Get your facts straight. It adds to the aura of credibility.
- Reviewers need to ensure neutrality. Heuristic reviews based on objective criteria that are applied subjectively provide little value. Subjectivity is critically tied to both the reviewer and the reviewer relationship to those being reviewed. If you’re being paid to consult on one hand and measuring on the other there will be bias.
- State the benefit of me listening and taking action on your recommendations A good consultancy will back up their recommendations if they are interested in a mutually successful outcome. Will it get me somewhere I’m not now? (e.g. 1st place vs. 3rd in subsequent reviews, a chance for publicity?).
2 replies on “Fire the consultants (some of them anyway)”
I’ve been guilty of conducting paid expert reviews in the past. They’re fun (and greatly inflate your ego) but I don’t see much value in them anymore, given the extreme level of subjectivity. These days I concentrate on producing and evaluating statistical results rather than quoting some off the cuff “best practice” ideal.
awesome post.
“get your facts straight. it adds to the aura of credibility.”
I’m stealing that line.
And since, I, like Geof, am guilty of having performed these types of reviews in the past, I think the value of the review really boils down to two things: 1) the soundness of the methodology (assuming there was one), and 2) the personal credibility of the reviewer. Often, its the consulting firm offering up its methodology. But its usually just one (or two) individuals delivering — so consistency is not a given.