Can "numbers" be good doc performance metrics? Is there a way to meaningfully interpret the quantitative user data we gather?

Question

I work on developer documentation at a tech startup. As of now, we implement the following feedback mechanisms:

• We have a thumbs-up/down feedback system on each page of the docs site. If a user clicks thumbs-down, we show a pop-up with more granular options about why they found the doc unhelpful and how we can improve it. This is invaluable feedback, however, it is qualitative.
• We also track the number of visitors to our docs site and the average time they spent. This is a more quantitative metric, and I usually feel better when the number of visitors goes up. However, I am not sure that's always a good thing.

I want to understand the techniques other tech writers use to know if their docs are serving the purpose. Do you track the number of visitors? Do you map the visitor trends to certain events like product releases? In short, how do you make sense of the numbers?

Answer 1

It is extremely difficult to measure the performance of a technical document because it is hard to gather the data and hard to interpret the data when you have it.

Let's start with the aim of technical communication. The aim is to make the user of a product productive by enabling them to use the product confidently and correctly. The logical measure of performance, therefore, is user's mean time to productivity.

The problem is, measuring user's mean time to productivity is very difficult. Virtually impossible in many cases. You simply cannot be there to observe them at work, nor can you instrument them or their work or the docs to gather the relevant data.

The Web does let us measure how often a document is read and how long a reader spends on it. The problem is, neither of these is an indication of document performance.

• A technical document gets read when the problem it describes occurs. This has nothing to do with the quality of the document and everything to do with the quality of the product it describes.

• The amount of time that the reader spends reading the document is no measure of its quality, since a good document could give the reader the information they need quickly, while a bad one might force the reader to read to the end and still not tell them what they need to know.

Finally, there is the issue of the relative value of a document. If the client's business loses a million dollars a minute when the server goes down, then the topic on how to restore the server after a crash is the most valuable topic in your doc set. But if your product is reliable, it will also be one of the least read topics in your doc set. Other commonly read topics may be worth only a few bucks in revenue each time they are read. They will score a lot higher in your metrics, but they deliver far less value in reality.

The best you can really do in many cases is to measure how well your docs ahere to known-good principles of design and rhetoric. It is a very imprecise measure and there will always be debates about which design principles and rhetorical practice best fit the current circumstances. (This is why answers on this board can never be provable in the way answers on SO are provable.)

A number of people have suggested performance measurements over the years but they are all either too expensive or too indirect to be certain. Better than nothing, perhaps, but certainly not definitive, and potentially quite misleading. (The problem with all indirect measurement is that it tempts you to optimize for the metric rather than for actual performance.)

Answer 2

You can't measure doc quality (or doc "performance") until you know what readers really want from the docs.

I've recently completed a study (to be published in STC's journal, Technical Communication, in 2019) that proposes a preliminary, focused, clearly defined, and reader-oriented model for defining documentation quality. This model can be used as a starting point for technical communicators and their managers who need to have reliable methods and metrics for measuring documentation quality.

In this study, I asked customer support groups from different companies around the world send a survey to their customers asking them to rate 15 quality dimensions. These 15 dimensions are based on empirical research done by Wang & Strong (1996), who designed a comprehensive, hierarchical framework of quality attributes, based on four categories that cover all aspects of quality:

• Intrinsic quality
• Representational quality
• Contextual quality
• Accessibility quality

Applying this to documentation, we can say that high-quality docs must be:

• Intrinsically good
• Clearly represented
• Contextually appropriate for the task
• Accessible to the reader

To properly define documentation quality, we really need to meet the following criteria:

• The definition must be from the readers' point of view: Because it is the readers alone who determine if the document we give them is high quality or not, any definition of documentation quality must come from the readers' perspective. Writers can come up with any number of quality attributes that they think are important, but at the end of the day, what they think is not as important as what the readers think.
• The definition must be clear and unequivocal: Both readers and writers have to "be on the same page" when it comes to what makes a document high quality. Misunderstandings of what readers actually want from the documentation are a recipe for unhappy readers.
• The definition must cover all possible aspects of quality: Quality is a multidimensional concept, and we must be sure that any attempt to define it is as comprehensive as possible. A definition that emphasizes one dimension over another, or leaves one out altogether, cannot be considered to be a usable definition.
• The definition must have solid empirical backing: To be considered a valid definition of documentation quality, serious research must be done to give it the proper theoretical underpinnings. Years of experience or anecdotal evidence can act as a starting point, but if we are serious about our professionalism and our documentation, we need more.

Wang & Strong's categories meet all of these criteria.

The point of this model is to prevent us from "asking the wrong questions". Because we need to capture feedback about documentation in context, we must ask only the minimal number of questions that provide us with the maximum useful information – and that's what I think my model does. When the study is published, and people start to try it out, we'll know more.

For more details, see:

• Wang R. & Strong, D. (1996). Beyond accuracy: what data quality means to data consumers. J. of Man. Info. Sys., 12 (4), p.5-34)
• Results from an offshoot of the current study, presented at STC 2017 in Washington DC (http://sched.co/8thX)

Answer 3

Qualities of the documentation itself, even quantitive ones, usually have little intrinsic value.

However, quantitive impact of docs on other areas can be often precisely measured and meaningfully interpreted.

Some of the companies I worked with used the following quality metrics for documentation:

• Number of support tickets. If customer support is overloaded with questions concerning a single topic, then maybe the documentation on this topic is not perfect.

• Number of failed deployments. If a documented process/task regularly gets done in an incorrect way, then maybe it is not documented properly

• Number of questions on a particular topic inside the team. If newly hired engineers tend to ask the senior staff the same set of questions concerning a single topic, then maybe this topic is not covered well in technical onboarding docs.

The best thing about this kind of metrics is that they can be clearly communicated to business.

– Our new user documentation has decreased the number of support tickets by X per cent? Great, so we've saved Y dollars we'd otherwise spend on outsourced technical support!

Answer 4

I'm not sure what you mean by "doc performance" - do you mean if the docs helped the readers do or know what they needed to do or know?

It’s hard to say that "success" (that is, "did the reader succeed in what s/he was trying to accomplish by consuming the content") is a measure of doc quality:

• For conceptual topics, you could define "success" as "the content helped readers understand the concepts they needed to know".
• For task topics, you could define "success" as "the content helped readers do the tasks they needed to do".

But this wouldn't take into account how easy it was for the reader to understand what you were saying, how many times it took the reader to do the task until they got it right, what things would've made the reader do the job better/faster/more efficiently or understand it better/faster, and so on.

Using "success" alone as a measure of doc quality is akin to using only Juran's "fitness for use" to measure general quality – it’s too vague and doesn’t really address all of the nuances (both objective and subjective, as per Pirsig's classic Zen and the Art of Motorcycle Maintenance) of what quality really is. Quality can only be defined by the user, of course (and for docs, this means the reader) – but the definition of quality must be multi-dimensional.