Good news for sports broadcasters and fans who’d prefer their play-by-plays to have that human-touch: AI doesn’t know ball.
A new study by researchers at the University of North Carolina at Chapel Hill and Northeastern University found that top AI models are horrible at analyzing professional sports. The yet-to-be-peer-reviewed study sought to analyze how capable the most popular AI models are in the fields of perception, reasoning, simulation, and agency — four traits which are difficult to evaluate with existing testing methods.
To probe how AI performs in these areas, researchers turned to the wide world of sports to create a new kind of AI test. Called strategic video intelligence, or “SVI-bench,” the novel test comprised 35,000 hours of sports footage from basketball, soccer, and hockey, as well as 15 million annotated plays, 15,000 hours of professional analysis, 23,000 post-game reports, and 103,000 statistical records.
Where AI performed the best was in perception: identifying which player performs which action at a given point in the match. But even there, they struggled badly. The models, which included ChatGPT, Google’s Gemini, and the open-source model Qwen, successfully eyeballed which player was doing what roughly 74 percent of the time — a rate which would get even a volunteer Little League announcer sacked.
The AI models did far worse on causal reasoning, or explaining why certain plays went down the way they did, with success rates falling near 40 percent on average. For example, when researchers asked the models to identify what was unusual about a Cody Martin three-pointer — which bounced off the top of the backboard before landing in the bucket — ChatGPT replied that it was “his first made three of the game.”
Simulation, or asking AI to find evidence to predict things like where a player would physically go based on their trajectory, was also dismal. During these tests, the best-performing model was functionally flipping a coin in order to guess a player’s next steps, and performance dropped even further when models were asked to plot out longer motion toward a goal or basket.
As computer science researcher at Northeastern and study co-author Lorenzo Torresani said in a press blurb by the university, AI “cannot tell you why things happen, and it cannot tell you what’s gonna happen next.”
When researchers probed the models’ agency — basically asking them to make complex post-game analysis of stats and trends, like a human broadcaster would — its accuracy fell to just 5 percent.
“A good sportscaster does much more than describe what’s on screen — they explain why a play worked, anticipate what’s next, and… decide which moments matter,” Torresani said. “Our study shows AI is already reasonably good at the descriptive part, but collapses on the rest.”
While sportscasters can definitely breathe a sigh of relief, the study’s findings are also good news for other knowledge workers, at a time when there’s been relentless fear of AI automation turning the job market inside out.
“The same gap shows up in any job whose value lies not in describing what’s visible, but in understanding why events unfold, anticipating what comes next, deciding what matters, and recommending what to do about it,” Torresani concluded.
More on AI in sports: Fans Aghast as New York Jets Say They’re Switching to AI
The post Researchers Put AI Models in Charge of Analyzing Sports, and They Choked Spectacularly appeared first on Futurism.
