Measurement · 5.5

Engagement surveys

Engagement surveys. What is actually the case, and how it compares with what is repeated.

A practical software reference for this part of the discussion is Monitask's overview of remote employee monitoring software.

Most large employers run an annual survey producing a score called engagement. The construct is proprietary, the benchmark is proprietary, and the most quoted statistic in the field comes from a vendor's own index.

What engagement is supposed to be

Something between satisfaction, commitment and discretionary effort. Academic work distinguishes these carefully; commercial instruments blend them, and different vendors blend them differently.

Which means two organisations with engagement scores of seventy have measured different things, and neither can be compared with the other except through the vendor whose norms they both bought.

The figure everybody quotes

A widely repeated statistic holds that only a small minority of the world's workers are engaged. It comes from one consultancy's proprietary index, based on a fixed set of items, with a threshold set by that consultancy, and it is not peer reviewed.

It may well capture something real. It is quoted as though it were an official statistic, and it is a commercial product measuring a proprietary construct against a proprietary threshold.

Does it predict anything

Meta-analyses report positive associations between unit-level engagement scores and outcomes including retention, safety incidents and customer measures. The associations are real and modest.

Two cautions. Most of the meta-analytic work has been conducted by the vendors whose instruments are being evaluated. And the direction is ambiguous: well-run units with reasonable workloads produce both good outcomes and good survey responses, so the score may be a symptom rather than a cause.

The anonymity problem

Reports broken down to team level in small teams are not anonymous in practice, and employees know it. A team of six with a demographic breakdown identifies everybody.

Response bias follows: the people most dissatisfied are the least likely to answer honestly, which biases the score upward in exactly the places where it matters.

The action gap

The best-established finding in this area is the least commercially interesting: surveying and then doing nothing is worse than not surveying. It converts a general grievance into a specific one about having been asked.

Employers that run a survey without a mechanism for acting on it are paying to reduce trust, and they are common.

Benchmarks

Vendors supply norms so a score can be compared with an industry average. The norm is the average of that vendor's clients, which is a self-selected group, and it changes as the client base changes.

Comparing this year with last year in the same organisation with the same instrument is the only comparison that means much, and even that requires nothing to have changed in the questions.

What to ask instead

Small numbers of specific questions about things the organisation could actually change, asked frequently, with the results and the resulting actions published to the people who answered.

That produces less impressive reporting and more usable information, which is the trade in almost every entry in this part.

The one question with a track record

Whether the person would recommend the organisation as a place to work. It is simple, it correlates reasonably with retention, and it resists the construct ambiguity that afflicts the composite scores.

It is also a single item, which makes it noisy at small scale, and it should be read as a trend rather than as a level.

The manager effect

Engagement scores vary more between teams within an organisation than between organisations, which points at the immediate manager as the dominant variable.

That is the most actionable finding the instruments produce and it is frequently buried under the organisation-level headline that the board asked for.

Frequency

Annual surveys ask people to summarise a year they cannot recall accurately, and the answer is dominated by the last month, in the same way performance ratings are. Short frequent instruments avoid this and produce a trend, which is the more useful object.

What to publish afterwards

The results, including the unflattering ones, and the specific actions with named owners and dates. Organisations that publish only a summary and a promise to act teach their staff that the survey is a formality, and the following year's response rate records that lesson.

Who does not respond

Response rates below about seventy per cent make a score hard to interpret, because non-response is not random. Reporting the score without the response rate is common and conceals the more informative number.

The exit interview

Collected almost universally and acted on almost never, for a structural reason: the person answering has no stake in the outcome and the person receiving it has no incentive to escalate criticism of a colleague who remains.

Asking six months after departure produces markedly more candid answers and almost nobody does it.

For broader context, consult Gallup workplace report.