Recap
We continue from the previous thread, where we looked at one month’s race data in cat C to see if we could create an explanatory model from standard stats on ZwiftPower that could explain the race results of the participants. The conclusion was that we could… somewhat.
Specifically, we could explain 42% of it, meaning most of what actually explains race results is not hidden in the ZP stats but rather elsewhere, in unknown or inaccessible “variables”. But 42% is still something, and in that something WKG turned out to be the most important variable, followed by 1 min power.
NP/kg?
Then in replies to that thread some of you suggested that NP/kg might actually be a better variable than good old W/kg. I must admit, in all these years it never crossed my mind to consider NP/kg. Intuitively, it seems like a very good ratio, though, one that really tells you something about your effort in e.g. a race. So let’s have a look at it.
First we can look at how strongly NP/kg (or NPKG) correlates with our measure of success in races. If you remember, since placing 3rd in a race with 5 participants is not comparable to placing 3rd among 20 comptetitors, we couldn’t use finish position as a measure of success but had to normalize position into a value between 0 and 1, where values close to 0 is the top end of the results table. We called this normalized position rank percent. So if we look at how strongly correlated NPKG is with rank percent and then compare it to the correlation between WKG and rank percent, what gives?
rank percent vs WKG: -0.56
rank percent vs NPKG: -0.59
Wow! So NPKG does seem more strongly tied to results than WKG? However, replacing WKG with NPKG in the model in the previous post, the explanatory value of the model, the so called R2 measure, actually drops from 0.42 to 0.39! So NPKG is not a good variable for the model, as the model can now only explain 39% of why results come out as they do.
What if we add both WKG and NPKG as input in the model then? They are strongly correlated with each other though, 0.95, and that is not desireable. But we try anyway. Now the R2 drops even a tiny bit more. It stays at 0.39 but rounded up this time rather than down. Conclusion? Let’s shelf NPKG for now. I still kind of like it though…
Predictive Modeling
I finished the previous post by saying that the model we created isn’t all that interesting. Well, it is. But it is only an explanatory model that looks at each race in isolation. It would be even more interesting with a truly predictive model, one that tried to predict a race outcome based on the participants’ race histories. Let’s look at that.
To begin with, how can we consider race history? I have taken a straight average of every rider in the sample’s last five races. Now, since the data set was so small, covering only a month, not every rider will have raced that many times within that time frame. So if a rider has only raced three times, we take the average of those three instead. Average of what? Of the same variables we used last time, i.e. avg WKG, avg W, NP, 20 min W, 5 min W, 1 min W, kg, avg HR, max HR, and finally the variable we want to try to predict, rank percent. We will accept the average of previous rank percent values (normalized finish position) as one of the input variables too.
How did it go? How good is the model? Using a so called Random Forest Regression as model type:
R2 = 0.30
This means the model, with its variables, averages of historical standard ZP data, can only explain future results to a degree of 30%, so to speak. Most of what actually would have explained the future results is unknown. Using a different model type, one with the fancy name Gradient Boosting Regressor, we can get the R2 up to 0.32, but that’s it.
Bear with me for a minute. The results are actually not as bad as they seem. We will come back to that. But first, I want a fancier model.
A Fancier Predictive Model
What we did in the first purely explanatory model was to look at each race in isolation. Given the performance of each rider in each race then and there, can we explain the outcome with plain ZwiftPower stats? To some degree we could.
Then we created a predictive model. Can each rider’s previous race performances predict his or her next race performance? At first glance the model seems a little dodgy. It seems to say something but not much.
However, a rider doesn’t race in a vacuum. You know exactly what it can be like. It is early summer now, but imagine high Zwift season. Assume Zwift HQ has just released a new race series this week. And it’s Monday, primetime, and you have just signed up for the first race stage. We all know it. It is going to be complete hell. All the best and most frequent racers are going to be there, in your race, eager to try out the new race, and you are going to have to work for it. Then you do the same race again later in the week, on a Friday. Lower attendance, yes, but also quite a bit more forgiving. In a nutshell, who is in the race matters. So how do we capture this in a model?
We could in theory consider each rider’s race history to every other single rider’s history in every race, but I suspect it would melt my computer. It is a heavy calculation. So we simplify a bit. We look at the average “starting field strength” of each race instead.
(Specifically, I have calculated mean and standard deviation of the field’s avg WKG, the field’s avg rank percent and the field size, and also the difference between the field’s mean avg WKG and each rider’s historical avg WKG and call it relative WKG, plus the difference between the field’s avg rank percent and each rider’s historical avg rank percent and call it relative rank. To those we add the same historical averages as in the previous model.)
Then we model it, using a Random Forest Regression. Results?
R2 = 0.37
Interestingly, this turns out to be a somewhat better predictive model than the first one, which came out as 0.32 only. The interpretation would be that weighing in starting field strength matters. Of course it does. But still, only 0.37? Disappointing, no?
Anyway, in the very first model, in the previous post, we looked at the importance of the input variables. So which variables contributed the most to making accurate (or not so accurate) predictions of all the 17 variables included in this the latest model? I list the top 4 below (all 17 sum up to 100%).
- A rider’s previous average rank percent: 30.2%
- The difference between a rider’s historical avg WKG and the field’s mean avg WKG: 8.4%
- The field’s avg previous rank percent: 8.0%
- A rider’s previous average 1 min W: 6.1%
Interesting. Not surprising, and like in any human study, the one thing that tends to predict future performance the best is simply past performance. After that, if you have a WKG advantage against the field average, then that speaks for you. The field’s average previous rank percent in absolute terms matters too. If you line up against a bunch of podium takers, then of course it is going to be tough. Then finally, a rider’s sprint matters, like we have seen earlier in the previous post.
Brown turns to gold… or at least silver
Well, I was a bit disappointed myself with the model at first. But then I realized that R2 , usually a good measure, is actually horrible at evaluating this model. It is here a measure of how well the model can predict an exact value of rank percent. But rank percent is a normalized value. To illustrate the problem with an example, we said 0 was the best in rank percent, which went from 0 to 1. The rank percent you get from placing 5th varies depending on field size. Thus rank percent in itself is not an interesting measure, because it doesn’t tell you all that much. It was just a necessity in order to be able to compare races with varying starting fields at all. What we are really interested in is not this contrived value but rather the order in which riders cross the finish line. In other words, we need to denormalize rank percent and then use a different measure than R2.
There is a more suitable measure. It is called Spearman rank correlation, after a guy, a statistician, named… yeah yeah, you guessed it. It can tell us how well the model predicts the order of riders. In other words, the finish position. Just like R2, the so called Spearman correlation coefficient, ρ (that’s a rho for all non-Greeks), takes on values between 0 and 1, where a 1 means the model is perfect and can predict with total accuracy every time in the data sample for any rider. Anyway, the measure turns out:
ρ = 0.88
This is actually not bad at all! The model isn’t perfect but still rather accurate. And think about it, what if we could create a model out of ZwiftPower data that was perfectly accurate? That would kill racing. There would be no point in participating. Instead of wasting calories for 45 min you could just match up your stats against the others at the start line and decide who is the winner, because you know who is going to win, and you know that you are going to finish 23rd yourself. No, we need races to be just a bit unpredictable to be fun and exciting. An 0.88, though, doesn’t that hit a sweet spot?
But it is all probably too abstract for you yet. So let’s have a look at the model at work. What does its predictions look like? I have taken a race from the sample at random, a race the model has not yet “seen” and been trained on. I have anonymized the names and exact race, but it is from a Zwift race series back in April with 26 entries in cat C, here showing only the top 10. I am not sure how representative this race prediction is, but generally the predictions (I have looked at a few) look something like this.
Important note: Mr Silver has not been predicted. The reason is this was his first race in the sample period, so there is no previous history on him, which makes the model filter him out. Adjusting for this, the other predictions are the black little numbers where the names I erased should have been.
The model predicts an Aussie woman to win, but she must have had bad legs because she “only” made it just outside the podium. Instead, the predicted runner-up won. The other predictions are close, if not spot on, except the American guy who did a better race than the model thought he would. Well fought!
The model isn’t perfect. But I played with the thought of using it on myself before a race. You can’t really do that since you don’t know exactly who will show up at the start line until the very last minute. But if you could, would I want to run this not quite perfect model to see how I would likely do in the race? I came to the conclusion that I wouldn’t. Currently, as a mid-in-cat, this would only be depressing and discouraging, because it would spell it out for me and spoil the fun: “Dude, stop dreaming! You are going to finish 23rd whether you like it or not.” And then I might come in 21st or 24th, but essentially the model would be right.
But that isn’t really the point. The point is you can, apparently, predict race outcome, roughly, with simple and readily available stats. I am actually surprised.
So what’s up next? I haven’t decided yet but will think about it while trying to build a bigger data set. I will post something soonish. If you have any suggestions on what you would like to have a look at, feel free to share them here.
