In human observation data, a person records an action or behavior. Generally, the recorder follows a specific set of rules to ensure consistency in the data. When humans deviate from the script of specific set of rules, though, data generally become corrupted. Kenn Tomasch details how a probable lack of training creates such error in data. The money observation quote:
My theory is that Chicago’s stats crew is counting everything that looks like a shot as a shot on goal, and the software is coming up with abnormally high totals for shots and saves because of it....
In a league where the variance between home and road saves per game is less than one per game, Jeff Richey grabs more than twice as many saves per game at home than on the road. In fact, the rest of the league’s starting goalies, on average, grab about one more save per game on the road than at home, yet Richey is getting almost 13 more at home than on the road.
Tomasch constructs a detailed and solid argument of why human observation, in this particular instance, leads to a faulty conclusion. He also provides a followup to the initial post.
As researchers, we need consistency checks to make sure that the data remains uncorrupted. Furthermore, we must train field personnel to avoid inconsistent data.