And now for something completely different. Well, not that different. But not the usual things. I've been working in human resources/education since February. It's quite enjoyable work. My job consists of being asked how to examine the effects of our company training program. Ultimately, what we really want is a return on investment (ROI) calculation, hopefully positive but you never know. Providing this value is tough! Myriad of problems: 1) job performance really hard to measure, 2) most data aren't easy RCT form, so causal inference is difficult, 3) job performance metrics aren't in dollar units, 4) there's a lot of different simultaneous efforts for different skills and mindsets etc. About the first issue, job performance measurement or job performance appraisal. One might think this is just a matter of designing some objective test of job tasks (work sample test) and testing people. Well, that's one way but these tests have low reliability (too few items/tasks), and are very expensive. For instance, giving a programmer a few tasks to complete might take all day. Consider the salary of a programmer, and then multiply that by the number of people you want to test, say, a few hundred, and you can see it's not a good idea. Most companies take a very pragmatic approach: they just have a single supervisor rate each underling on job performance. The question then becomes: how accurate are these things? The answer is that question is... complicated. We don't have any gold standard measurement of job performance. We only have a bunch of different suboptimal ones:
Interrater reliability in job performance ratings
Interrater reliability in job performance…
Interrater reliability in job performance ratings
And now for something completely different. Well, not that different. But not the usual things. I've been working in human resources/education since February. It's quite enjoyable work. My job consists of being asked how to examine the effects of our company training program. Ultimately, what we really want is a return on investment (ROI) calculation, hopefully positive but you never know. Providing this value is tough! Myriad of problems: 1) job performance really hard to measure, 2) most data aren't easy RCT form, so causal inference is difficult, 3) job performance metrics aren't in dollar units, 4) there's a lot of different simultaneous efforts for different skills and mindsets etc. About the first issue, job performance measurement or job performance appraisal. One might think this is just a matter of designing some objective test of job tasks (work sample test) and testing people. Well, that's one way but these tests have low reliability (too few items/tasks), and are very expensive. For instance, giving a programmer a few tasks to complete might take all day. Consider the salary of a programmer, and then multiply that by the number of people you want to test, say, a few hundred, and you can see it's not a good idea. Most companies take a very pragmatic approach: they just have a single supervisor rate each underling on job performance. The question then becomes: how accurate are these things? The answer is that question is... complicated. We don't have any gold standard measurement of job performance. We only have a bunch of different suboptimal ones: