Analytics post mortem

[slugboy]

Connelly picked FSU by 14.

Fremeau picked FSU by 14. https://www.bcftoys.com/2024-gp

The preseason projections were for FSU to have a top 25 offense and a top 15 defense. Here’s the F+ composite: https://www.bcftoys.com/2024-fplus. We were supposed to have the #40 offense and the #92 defense. The defense was obviously better than that.

Kelly Ford is already updating some of his model. We’re moving up. Not too much-we’re up to number 39 from

KFord Ratings

kfordratings.com

The models are going to take a while to come around. It’s pretty even between moving FSU down and moving us up.



Here is the Beta Rank ACC previews


Again, people thought we were a double digit underdog. They thought we’d score 20 and FSU would score about 34. We scored a little more than expected (and took the air out of the game but limiting possessions). We held FSU way below projections, even with two amazing field goals by their kicker.



We are still underestimated

 

[roadkill]

If anyone can share some of the OL and DL PFF grades after they come out, I'm interested to see how we graded out. Expecting they will be better than last year. especially for the DL.

 

[slugboy]

CFB Geek has one of the better stat models out there, and had some nice opinions on the game. We probably should have won by more.

 

[UgaBlows]

Didn’t we beat hell outta UCF last year? How are they ranked ahead of us?

 

[roadkill]

Didn’t we beat hell outta UCF last year? How are they ranked ahead of us?

Good question.
They ended last season ranked ahead of us in F+ and in ESPN's Power Rankings even after we beat them soundly and had a better record. I thought power rankings were supposed to be predictive.

 

[roadkill]

CFB Geek has one of the better stat models out there, and had some nice opinions on the game. We probably should have won by more.

Special teams. Ours weren't bad, but 9 of FSU's 21 points (43%) were from special teams. Another factor was they weren't that efficient on their last scoring drive (especially on third down) except for the 2 fourth-down clutch throws. Otherwise, the game wasn't as close as the score.

 

[TechPhi97]

All of this is interesting for me, as I have a Masters in Analytics and have been doing predictive modeling and statistical analysis work for 15 years. But the reason I throw my hands up is because (1) it's impossible to predict things with such a small number of occurrences with (2) no historical data that is transferable because of player/coach changes year over year and (3) where there is such inherent variability anyway.

Even late in the season picking the spread using a lot of these "use for predictions" models get you no uplift over random guessing. The only think I found interesting with the BCF Toys / FEI stuff last year was a surprising ability to get PAC-12 games right. Which doesn't matter anymore!

Doesn't mean that I don't want to look at the numbers, I appreciate you posting all of this because I love reading through it. But I hate how these guys use a ton of numbers to add the veneer of certainty to their predictions, when in fact they are as clueless as one of those people that picks their March Madness based off of their favorite colors or which mascot would win in a fight.

 

[TechPhi97]

Good question.
They ended last season ranked ahead of us in F+ and in ESPN's Power Rankings even after we beat them soundly and had a better record. I thought power rankings were supposed to be predictive.

They're not. I have a spreadsheet with all of the picks from 2023 (based on FEI each week) and they aren't.

 

[roadkill]

They're not. I have a spreadsheet with all of the picks from 2023 (based on FEI each week) and they aren't.

Interesting. Have you done the same spreadsheet based on FPI?
I would expect some routine variance from actual results because upsets happen in football. But if there's a wide variance overall, that points to a serious issue with the underlying structure of the indexes.

 

[CuseJacket]

@slugboy and others,

Not much time to research myself. Hoping you can explain this by way of analytics you've already observed.

In the category of "it's never as good or as bad as it seems", can you help me understand how happy we should be about our defense's performance?

We held FSU to 21 points in part because they had 7 drives. That is 3 points per drive. Generally, that's not considered good.

Per BCF Toys (https://www.bcftoys.com/2023-ppd/), last year:

• FSU O averaged 2.64 points per drive, good for #34 in the country (this includes post-Jordan Travis games incl. bowl)
• GT D averaged 2.65 points per drive, good for #107 in the country

That is before weighting for competition. Either way, while the eye test said we improved, the numbers suggest reason for caution. Thoughts?

 

[MtnWasp]

All of this is interesting for me, as I have a Masters in Analytics and have been doing predictive modeling and statistical analysis work for 15 years. But the reason I throw my hands up is because (1) it's impossible to predict things with such a small number of occurrences with (2) no historical data that is transferable because of player/coach changes year over year and (3) where there is such inherent variability anyway.

Thank you for these three points. The utility of models are that they are supposed to be effectively predictive, not just persuasive.

Modeling is not a black-box algorithm washing machine where crap input data magically comes out insightful objective science.

Faith in this kind of thing should not be blind. Confidence without an equal measure of interpretive and critical skills leaves one open to chicanery. And there is lots of that out there. Doesn't matter for football so much, but in other things...

Okay, sorry for that....

 

[cpf2001]

The first drive definitely drags down the overall defensive numbers in a way most of us are partially discounting in terms of how it feels.

The rest of the game, numerically - 13 points in 6 drives, 2.167 - was a substantial improvement compared to last year and in some ways quite possibly better than it looks - consider that those totals did not benefit from any turnovers and the scores that did happen were long FGs or needed multiple 4th down conversions. Lot of score effects on that last FSU TD vs FG decision. On the flip side, new QB and first game of the season. So it could go either way. But with our offense that could be a very good top 25 or better net point per drive if we can play like most of the day.

And while some of the numbers may look more “competent” than “great” for the D, the offensive numbers are in the “way better than they look” especially if you think FSU’s D is going to be similar to last year. They allowed 1.47 ppd, 11th best. GT put up 3.42 yesterday. That would beat out Michigan for fifth best in the country last year even without accounting for opponent. And it wasn’t fluky! Scoring drives of 6, 14, 11, and 12 plays for 79, 75, 89, 49 yards.

The biggest stat that jumps out from these charts at me - at least until we’re more sure of the level of competition - is that we didn’t force 3-and-outs and we also had a couple. Definitely some stuff to clean up.

 

[stinger78]

Didn’t we beat hell outta UCF last year? How are they ranked ahead of us?

 

[ramblineck]

Connelly picked FSU by 14.

Fremeau picked FSU by 14. https://www.bcftoys.com/2024-gp

The preseason projections were for FSU to have a top 25 offense and a top 15 defense. Here’s the F+ composite: https://www.bcftoys.com/2024-fplus. We were supposed to have the #40 offense and the #92 defense. The defense was obviously better than that.

Kelly Ford is already updating some of his model. We’re moving up. Not too much-we’re up to number 39 from

KFord Ratings

kfordratings.com

The models are going to take a while to come around. It’s pretty even between moving FSU down and moving us up.



Here is the Beta Rank ACC previews


Again, people thought we were a double digit underdog. They thought we’d score 20 and FSU would score about 34. We scored a little more than expected (and took the air out of the game but limiting possessions). We held FSU way below projections, even with two amazing field goals by their kicker.



We are still underestimated


View attachment 16643

View attachment 16644

Thanks for sharing. Not familiar with the other models but I have been following beta rank pretty closely the past couple years and he says his model takes 5-6 weeks to really get up to speed. I wonder if the other models are the same way.

I think there's a decent amount of nuance that these models don't capture such as player transfers/continuity accurately upfront (as other have pointed out) especially with limited data. There's no data points from last year that could really point you to what our Run D did yesterday which should give us some cause for excitement!

 

[takethepoints]

Modeling is not a black-box algorithm washing machine where crap input data magically comes out insightful objective science.

Faith in this kind of thing should not be blind. Confidence without an equal measure of interpretive and critical skills leaves one open to chicanery. And there is lots of that out there. Doesn't matter for football so much, but in other things...

1) Well, yes it is. One reason to use modeling is that you don't have any idea what's going on in thebox; all you have is inputs and outputs crudely measured. The nice thing about models is that allows you to make predictions anyway. And … sure enough, sometimes those predictions are wrong.

2) I go with this:

 

[slugboy]

Hi @CuseJacket

It’s just one game. I’m seeing a lot of people say that FSU was trash, and that our defense looked terrible. I’ve seen some others say that it was a hard fought game. We need a couple more games to tell.

We blocked well. We tackled well. Coverage was less good than I’d like. DJU doesn’t read the field well, and that probably saved us. FSU started picking up the blitz better in the second half. We need to blitz better, and get to the QB.

Our OL played well. DL played well, too. FSU picked up big yardage on fourth down twice.

Stats don’t show that our offense dictated the second half, and controlled the last six minutes of the game.

We really need to play well against Georgia State, Syracuse, and VMI. If we gel over the next four weeks, we’ll be a tough team. We can athletically hang with anyone, and shouldn’t get pushed around like we were against Bowling Green or BC. Next two games are huge.

From Game on Paper. It’s obviously wrong about the last drive. It’s missing two field goals-one from us and one from FSU. I think on a per-drive basis, we did well. FSU had a 50 and a 59 yard field goal-how often does that happen?

Thank you for these three points. The utility of models are that they are supposed to be effectively predictive, not just persuasive.

Modeling is not a black-box algorithm washing machine where crap input data magically comes out insightful objective science.

Faith in this kind of thing should not be blind. Confidence without an equal measure of interpretive and critical skills leaves one open to chicanery. And there is lots of that out there. Doesn't matter for football so much, but in other things...

Okay, sorry for that....

The models are predictive. People explicitly grade the models on how well they predict, both straight up and against the spread. They do very well straight up, and are about even with the point spread in predicting the final outcome.

Yesterday was a bloodbath for all the predictions. Over the course of the season, the good ones predict well.

https://public.tableau.com/app/profile/andrew.percival/viz/CFBPicker/Standings

 

[stinger78]

This team really needs to focus one game at a time and get past these games. They need to hone out some rough spots, and play players. But they have to win. A 4-0 start would do wonders for the program and would set us up for a potentially very nice season.

 

[wreckrod]

It’s not very analytical but one thing that stood out to me was it seemed like every 50/50 call went FSU’s way. They benefitted tremendously from a few critically timed penalties. Without the face mask, that’s a stop with a punt. Same thing later for the weak sauce PI call. And I really thought FSU benefitted from a ton of holding non-calls.

 

[gville_jacket]

Has anyone found the full participation list for the game? Cant find it. I’m curious how snap counts looked

 

[Augusta_Jacket]

Has anyone found the full participation list for the game? Cant find it. I’m curious how snap counts looked