How Good Is Recruiting Data at Predicting End of Season Power Ranking Performance?

[GTNavyNuke]

Three years ago for discussion on another site, I looked at how all the D1 teams at the time performed and correlated that to their four year Scout average star recruiting averages. Since it had been a while, I updated the data.

Not surprisingly, again what I found is that teams that have higher ranked recruiting classes tend to have higher Power Rankings. Specifically, I took the four year recruiting averages by Scout average stars and compared it to the JHowell end of Year Power Rankings. http://www.jhowell.net/cf/cfindex.htm

Back for the 2006 to 2010 time period, that there was a linear correlation of 28% to 43% depending on the year. What I found when I recently updated the data for 2010 to 2013 was a correlation of 27% to 42%, again depending on the year. {Geek note: Power and exponential curve fitting gives about the same correlation.}

So about 1/3 to a bit better of a team’s performance is correlated to the recruiting rankings.

I attached three charts.

· Two scatter charts for two time periods that plot recruiting classes against power rankings. The two time periods are 2006-2010 and 2010 to 2013. You can see the linear correlation coefficient (R2) in the left sidebar.

· A decile chart. This chart shows that if a team recruited in a certain decile (e.g. top 10 %), what percentage of that decile finished with a certain power ranking. This is the clearest chart that shows to me that recruiting rankings matter.

Now correlation is not causation. But there is a very good reason to think that better athletes will win more games. Also, there is 60-70% that is not correlated to recruiting rankings and is due to other factors – coaching, luck, offensive or defensive scheme, errors in recruiting rankings, individual growth, team chemistry, whatever.

But recruiting does matter. In a later thread, I’ll talk about GT’s probable performance based on recruiting through this year.

 

[Whiskey_Clear]

Better athletes win more games???....Naaaah....

One thing to consider here. Do the top teams get ranked higher in stars because they recruited better...or because they were ranked high with W/L?

Kinda where I'm going with this is our 2009 senior class. Very under rated class. Was that because all of the recruiting services just missed that year on us? Or did they just give GT the typical GT rating normally assigned from our prior W/L.

I for one remain skeptical on the difference between "2 star" and "four star" prospects and the gurus ability to predict future success.

This is an endless debate though. Damn I'm ready for spring ball at least to start.

 

[GTRanj]

· A decile chart. This chart shows that if a team recruited in a certain decile (e.g. top 10 %), what percentage of that decile finished with a certain power ranking. This is the clearest chart that shows to me that recruiting rankings matter.

Yes there is a clear correlation here because the top 10% teams have more kids rated at 5* or close to it. Many people agree, including cpj, that 5* are most of the time going to be a hit and not a miss. So this is probably the only chart I agree with. Regardless awesome info man. I appreciate the hard work.

 

[IEEEWreck]

Three years ago for discussion on another site, I looked at how all the D1 teams at the time performed and correlated that to their four year Scout average star recruiting averages. Since it had been a while, I updated the data.

Not surprisingly, again what I found is that teams that have higher ranked recruiting classes tend to have higher Power Rankings. Specifically, I took the four year recruiting averages by Scout average stars and compared it to the JHowell end of Year Power Rankings. http://www.jhowell.net/cf/cfindex.htm

Back for the 2006 to 2010 time period, that there was a linear correlation of 28% to 43% depending on the year. What I found when I recently updated the data for 2010 to 2013 was a correlation of 27% to 42%, again depending on the year. {Geek note: Power and exponential curve fitting gives about the same correlation.}

So about 1/3 to a bit better of a team’s performance is correlated to the recruiting rankings.

I attached three charts.

· Two scatter charts for two time periods that plot recruiting classes against power rankings. The two time periods are 2006-2010 and 2010 to 2013. You can see the linear correlation coefficient (R2) in the left sidebar.

· A decile chart. This chart shows that if a team recruited in a certain decile (e.g. top 10 %), what percentage of that decile finished with a certain power ranking. This is the clearest chart that shows to me that recruiting rankings matter.

Now correlation is not causation. But there is a very good reason to think that better athletes will win more games. Also, there is 60-70% that is not correlated to recruiting rankings and is due to other factors – coaching, luck, offensive or defensive scheme, errors in recruiting rankings, individual growth, team chemistry, whatever.

But recruiting does matter. In a later thread, I’ll talk about GT’s probable performance based on recruiting through this year.

Would you mind/ can you share the raw data? Looking at R^2 may not be the most informative thing here, and I kind of want to look for markers of backcasting in the data.

 

[yellojello]

Would you mind/ can you share the raw data? Looking at R^2 may not be the most informative thing here, and I kind of want to look for markers of backcasting in the data.

Mind explaining what markers of backcasting means?

 

[GTNavyNuke]

Would you mind/ can you share the raw data? Looking at R^2 may not be the most informative thing here, and I kind of want to look for markers of backcasting in the data.

NP. I uploaded the file I'm working with to googledocs https://drive.google.com/file/d/0B4mNRDaHIGn7YkdkVmx6MjQ5aFU/edit?usp=sharing

I just let Excel do the calcs of R2 (correlation coefficient) when I did a linear estimate of a data set.

If you can't see the formulas, I can e-mail / PM the file.

As I said before, the next post is going to be about GT ...... you'd never guess what the conclusion is (tic).

 

[IEEEWreck]

^^^ Thanks dude. I'll play around with it in some statistics packages that the Institute lets us fool with.

Mind explaining what markers of backcasting means?

Backcasting is the (informal, I think) term in econometrics for arbitrarily changing your model to fit the trend. It's frowned on because it produces picture perfect results for the period in question by decreasing the value of the model for predicting future values in the series. It's something close to lying with regression.

There are statistical measures that indicate when the overall trend has changed in the numbers. If the regression changes to fit and there's not 5 pages up front explaining why, its a good indication that the regression has something fishy going on.

I'm not sure, but I think that if the star ranks have been ****ed with individually based on what a 'good' school is, there should be noticeable patterns in some statistical criteria, and unexplained correlation in the data.

Uh... does that help?

 

[GTNavyNuke]

LOL. I understand what you are saying. I think you'll have a hard time figuring out whether Scout increased the star rankings of individual players since there are so many independent variables and no way to determine how an individual performed. The thing is that Scout gives the stars when the kids are juniors and seniors in high school. We don't get to see the team performance till 3 or 4 years later on and the Scout data hasn't been changed.

Do I think Scout pumps up star ratings when a "good" school offers a player? Yes, since it is another person's (school) validation of the player's ability. It is much better to be wrong in a group than wrong alone. So go with the group and coast on through {:sarcasm}.

 

[yellojello]

^^^ Thanks dude. I'll play around with it in some statistics packages that the Institute lets us fool with.


Backcasting is the (informal, I think) term in econometrics for arbitrarily changing your model to fit the trend. It's frowned on because it produces picture perfect results for the period in question by decreasing the value of the model for predicting future values in the series. It's something close to lying with regression.

There are statistical measures that indicate when the overall trend has changed in the numbers. If the regression changes to fit and there's not 5 pages up front explaining why, its a good indication that the regression has something fishy going on.

I'm not sure, but I think that if the star ranks have been ****ed with individually based on what a 'good' school is, there should be noticeable patterns in some statistical criteria, and unexplained correlation in the data.

Uh... does that help?

Yup, definitely helps. That's one reason I've always wanted to investigated as a hypothesis for the star ranking, but could not figure out a good way except tracking the star rating of a player over time ie. how much does the star ranking of a player change when he commits to Alabama?

 

[GTNavyNuke]

Yup, definitely helps. That's one reason I've always wanted to investigated as a hypothesis for the star ranking, but could not figure out a good way except tracking the star rating of a player over time ie. how much does the star ranking of a player change when he commits to Alabama?

But if the change turns out to correlate better with future team performance, is it a good change or a bad change?

Three years ago when I did this study, the correlations had improved to a high point in 2009 of 43%. So one reason I wanted to do this was to see if the correlations had gotten even better. I was surprised to find that they hadn't. The correlation between recruiting ranking and subsequent team performance has erratically declined in the last few years. So if they are cooking the books, it ain't working.

 

[takethepoints]

Since we know that Scout adjusts "stars" in relation to who offers a kid and where he signs over the recruiting season, the error terms here are irretrievably intercorrelated with the predictors. And that in turn means that the coefficients are no longer BLUE and the correlations are unreliable.

Sorry. This brought out the stats nerd. If you cured the problems, the results probably would – of course – be pretty much the same. They don't call OLS robust for nothing.

 

[Jean-Baptiste Rochambeau]

Better athletes win more games???....Naaaah....

One thing to consider here. Do the top teams get ranked higher in stars because they recruited better...or because they were ranked high with W/L?

Kinda where I'm going with this is our 2009 senior class. Very under rated class. Was that because all of the recruiting services just missed that year on us? Or did they just give GT the typical GT rating normally assigned from our prior W/L.


I for one remain skeptical on the difference between "2 star" and "four star" prospects and the gurus ability to predict future success.

This is an endless debate though. Damn I'm ready for spring ball at least to start.

Beano Cook is purported to have declared that he could predict 75% of the top twenty teams 20 years into the future, that most of the ranked programs of the present would still be the ranked programs in 20 years. Just sayin'.

 

[GTNavyNuke]

Yes there is a clear correlation here because the top 10% teams have more kids rated at 5* or close to it. Many people agree, including cpj, that 5* are most of the time going to be a hit and not a miss. So this is probably the only chart I agree with. Regardless awesome info man. I appreciate the hard work.

This is where I have ended up too. The 4 & 5 star recruits make a difference in overall results, but with 3 stars it is a crap shoot how they will mature.

The top recruiting teams have do consistently better in end of year Power Ranking (have less dispersion). Once you get down to our level of recruiting, there is a lot more dispersion of the results. The attached three charts hopefully show what I'm trying to say.

 

[forensicbuzz]

I think quantity matters as well as quality (also goes into rankings). If you have a 1st Team backed by a 1B Team instead of a 2nd Team, then you're going to perform better at the end of the game, i.e. more likely to win the close ones or pull away.

Most of the top teams not only have excellent starters, but most of their back ups are starter-quality too. The next recruiting level down (where I think we fall), there tends to be a drop-off from 1st to 2nd string.

 

[GTNavyNuke]

I agree with the depth on the field. I think it also extends to the team which the first team sees the most - the scout team. I would bet that Alabama's scout team would probably rank around 40th on their own. By practicing against better players, you get better.

 

[Boomergump]

Better yet, we should do a study that correlates the last 4 years of on-field performance to the upcoming year recruiting rankings. I'm willing to bet the correlation will be stronger. If I was an understaffed recruiting service who wanted to make a splash, that is the metric I would use. That is, along with grade inflation to the schools with the highest number of subscriptions too. Gotta keep stoking the fire.

 

[GTNavyNuke]

Beano Cook is purported to have declared that he could predict 75% of the top twenty teams 20 years into the future, that most of the ranked programs of the present would still be the ranked programs in 20 years. Just sayin'.

What Beano is saying is not true. What I could easily find is that the correlation of a teams 4 year prior Power Ranking to the next year's Power Ranking was correlated about 35-40% for the last 4 years. Just eyeball the attached the top 20 teams averaged over the last four years - about 1/2 of them finished above top 20 in any given year by my eyeball. If you get that low a result in 4 years, it'll be a lot less in 20 years. Of course, I wasn't looking at the top 20 teams and I'm not about ready to do a 20 year study.

So I rate Beano's statement as hyperbole - but I agree that the top teams stay strong in general. They have the resources, reputation and recruiting. It's just hard to stay in the top 20 with all the other teams who want to get there too.

 

[GTNavyNuke]

Better yet, we should do a study that correlates the last 4 years of on-field performance to the upcoming year recruiting rankings. I'm willing to bet the correlation will be stronger. If I was an understaffed recruiting service who wanted to make a splash, that is the metric I would use. That is, along with grade inflation to the schools with the highest number of subscriptions too. Gotta keep stoking the fire.

Boomer, for you, here is the study. You are correct.

There is about a 50%+ correlation between the average of the 4 previous years team Power Ranking and the subsequent year recruiting class ranking. So if you take the average of the Power Ranking of each one of the D1 schools for 2010 through 2013, that average is 56% correlated with the 2014 recruiting class ranking for the particular school. {About the same correlation for linear or exponential curve fitting.}

But all that really shows is that the better players want to play in the better football programs. No one can determine the motivation of how recruit rankings are generated from the level of data I have. To do that, you have to look at individual recruit rankings.

I believe that it is entirely rational and justified to reevaluate a recruits ranking when a successful program offers him. "Well when events change, I change my mind. What do you do?"

 

[Boomergump]

Boomer, for you, here is the study. You are correct.

There is about a 50%+ correlation between the average of the 4 previous years team Power Ranking and the subsequent year recruiting class ranking. So if you take the average of the Power Ranking of each one of the D1 schools for 2010 through 2013, that average is 56% correlated with the 2014 recruiting class ranking for the particular school. {About the same correlation for linear or exponential curve fitting.}

But all that really shows is that the better players want to play in the better football programs. No one can determine the motivation of how recruit rankings are generated from the level of data I have. To do that, you have to look at individual recruit rankings.

I believe that it is entirely rational and justified to reevaluate a recruits ranking when a successful program offers him. "Well when events change, I change my mind. What do you do?"

Wow, thanks for doing that. I agree with you about the motivation part. It is really hard to pin down the underlying reason for the correlation. You make some valid points. Is it players migrating to play for winners, or is it recruitniks just pretending to know? Perhaps it is a combination of both.

 

[augustabuzz]

I think quantity matters as well as quality (also goes into rankings). If you have a 1st Team backed by a 1B Team instead of a 2nd Team, then you're going to perform better at the end of the game, i.e. more likely to win the close ones or pull away.

Most of the top teams not only have excellent starters, but most of their back ups are starter-quality too. The next recruiting level down (where I think we fall), there tends to be a drop-off from 1st to 2nd string.

This is a major reason for UG wanting to keep our game in November instead of September.