Yeah, one of the paradoxes I found was on the one hand talking about being responsible for projects making large sums of money, but, well...it doesn't seem like he's seen much of it.
But indeed I feel like at some level there's been a pendulum swing from, let's say "stories" to "data" - indeed he touches on sabermetrics/McNamara. I like how he torches this: data is important but it's not enough (the map is not the territory) -- I started wondering about this a lot for example with Windows. "Oh, we removed this feature because telemetry showed it wasn't used." Well, why? Because it wasn't useful? Because it wasn't discoverable? Because it wasn't intuitive?
And that assumes the numbers are even any good: I remember from one of Sinofsky's Windows engineering blog posts, in which he talked about some feature and the percent of sessions in which it was used. And I thought, well, hold up. I hibernate and rarely restart, whereas many less sophisticated users shut down their computer entirely. So does that mean they effectively are counting those non-power users a lot more than me because they have more sessions?
And then there are other second order effects. If you lose your power users, do they then stop recommending your product, and what then? I see your net promoter score and raise you Goodhart's law (once a measure becomes a target it ceases to become a useful measure)
But indeed I feel like at some level there's been a pendulum swing from, let's say "stories" to "data" - indeed he touches on sabermetrics/McNamara. I like how he torches this: data is important but it's not enough (the map is not the territory) -- I started wondering about this a lot for example with Windows. "Oh, we removed this feature because telemetry showed it wasn't used." Well, why? Because it wasn't useful? Because it wasn't discoverable? Because it wasn't intuitive?
And that assumes the numbers are even any good: I remember from one of Sinofsky's Windows engineering blog posts, in which he talked about some feature and the percent of sessions in which it was used. And I thought, well, hold up. I hibernate and rarely restart, whereas many less sophisticated users shut down their computer entirely. So does that mean they effectively are counting those non-power users a lot more than me because they have more sessions?
And then there are other second order effects. If you lose your power users, do they then stop recommending your product, and what then? I see your net promoter score and raise you Goodhart's law (once a measure becomes a target it ceases to become a useful measure)