Mistakes to Avoid when Evaluating Technical Content

How do you evaluate the quality of your technical documents in a way that’s impartial, accurate, and easy to implement?

Recently we looked at how to create scoring frameworks (usually in Excel) to evaluate and score technical documents.

From one angle, the scoring is the easy part. What’s more difficult is getting people onside.

Encouraging them to use the framework, score correctly, and most important not see it as a tool which will be used to show up their weaknesses.

That’s the usual reaction.

Mistakes to Avoid when Evaluating Technical Content

If I score low, will my position be threatened or my performance reviews be affected?

This is a valid question as ultimately writers will be assessed partly on how they score. However, it should only be one factor in their overall assessment. With that in mind, let’s look at how to implement a scoring system that gets accepted, improves quality, and can be scaled.

• Give fair warning. No one likes to be caught unawares. So, let the writers know in advance that this is on the agenda and you and the team will be working on it in the coming weeks or months. This allows people to see what’s coming and prepare questions in advance.
• State your intentions. Once the project starts, explain why you’re doing this. Is it because you’ve been told to (not my idea but…) or because you feel the team will learn and benefit from it. Explain why. Give examples.
• Define the scope. Is everything going to be assessed? Probably not. Identify the scope of the project, ie what content will be evaluated first, for example, user guides, release notes, online or help, and use this as a baseline.
• Start small. The first phase is really to get the wheels in motion and find where it can be improve. Don’t expect too much from the numbers. It’s also the phase where you try to get the scoring system embedded into day to day operations, so later it’s just another task to be completed.
• Describe how it works. Walk the team through the process. Show them how scores are awarded, how (and why) weights are applied, and how this will be tracked over time. Why? It reduces fear of the unknowns and neutralizes some of the gossip and Chinese whispers.
• Don’t work in isolation. Instead of developing this by yourself – and then trying to launch it by yourself – get others involved early. Look for writers who are keen to better themselves, want to learn more, and are interested in expanded their skillsets. These will usually have the drive, energy, and attitude to help you get the system up and running.
• Dealing with naysayers. Expect resistance. There will always be team members who’ll try to undermine your position, find flaws in your plan (answer: so, how would you suggest we improve it?) and encourage others to do likewise. While this is to be expected, at least to some degree, you may need to talk to team members whose attitude is influencing the team negatively.
• Explain consequences. Be honest. Scores reflect your performance at work. However, they should only reflect part of your overall performance. For that reason, clarify how this will affect writers, for example, as bonuses are often determined by performance reviews, and what may occur if writers are consistently getting low scores.

Summary

Your team is more likely to accept and use the new scoring system if they’re involved from the get go. Let them know why you feel this will help their careers and if they have suggestions on ways to judge the quality of the content.

Delegate tasks so everyone is involved to some degree, for example, creating templates in Excel, adding best practices to the wiki, and updating style guides if necessary.