What's the impact of team-specific context?
Players' contributions are not only a function of their true ability and luck, but also contingent on team-specific effects. How can we measure how big these team-specific effects are?
On his great blog, Tiotal Football talks about projecting the future contribution of a player toward our team's goal difference. His approach is based on the player's historical contribution which is then adjusted for factors that may have influenced his performance. Combined with some qualitative information, this will result in a team-specific projected marginal goal difference contribution.
One of the factors we need to adjust for is team effects. The basic premise is that players' contributions are not only a function of their true ability and luck, but also contingent on team-specific effects. How can we measure how big these team-specific effects are? What percentage of the observed goal difference contribution comes from the player's actual skill, random chance, and team-specific effects like his teammates, his role, etc.?
Consistency of goal difference contribution from year-to-year
In many ways, this question about team context is a question about the consistency of stats. Let's say the historical contribution of a player is 0.5 GDC/90 (GDC = goal difference contribution). Whether or not we can project a similar number next year depends on the consistency of that stat year to year. A stat that is consistent from year to year (i.e. his numbers next year will be very similar to this year) is likely picking up true, context-independent player skill and vice versa.
There have been a few studies on the consistency of certain stats in football. Devin Pleuler, for example, demonstrated weak year-to-year correlation of superior shot placement. An overperformance of expected goals in one year gives little to no indication of an overperformance in the next year. Since this metric is not consistent from year to year, it most likely doesn't pick up true, context-independent player skill.
One problem with year-to-year correlations is that they would only tell us about the random chance part, not the context in which we are interested. This may be sufficient in other sports such as Baseball because context doesn't play such a big role. However, if we accept that context does matter in football, it doesn't help us to answer our question which is: how team-context-dependent is that stat?
One way to approach this would be to more explicitly tie year-to-year correlations to the context changes that we are interested in (i.e. team-specific effects). Instead of looking at all players, we can split them into two groups — those who played on the same team in year one and year two, and those who played on a different team in year one and year two. We can then use year-to-year correlations for the same-team players as our baseline to evaluate how much dropoff there is for different-team players. This allows us to control for team contexts and evaluate how much stats differ when looking at players who changed teams.
Year-to-year correlation: VAEP
Here's an example of this method. I looked at VAEP from 2012 to 2021 in all leagues with at least two seasons of data. (Thanks to Canzhi Ye for calculating and Tony for providing me with the data).
VAEP stands for Valuing Actions by Estimating Probabilities and was developed by Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. As the name suggests, it's "a framework for valuing any type of player actions based on its impact on the game outcome." It looks at two game states and evaluates how much the player's action in game state 1 impacted the probability of scoring or conceding a goal in game state 2. For example, Busquets has the ball in midfield (game state 1). He then plays a pass to Messi into the final third (game state 2). Let’s say because of Busquets's pass, Barca's chance of scoring increased by 0.03 and its chance of conceding decreased by 0.01. Therefore, Busquets's action is valued at 0.04 VAEP.
One caveat with same-team vs. different-team year-to-year correlations in football comes from league effects. Different leagues have different competitive levels, so there might be a bias when we directly compare MLS numbers to Premier League numbers. To avoid that, I excluded all players who played for teams in different leagues in the different-team group. Therefore, we only compare players who played for the same team in years one and two with players who played for different teams within the same league. For example, Timo Werner's transfer from Leipzig (Bundesliga) to Chelsea (Premier League) would be excluded while Jack Grealish's transfer from Villa (Premier League) to Man City (Premier League) would be included. I also set a minutes threshold (900 minutes) and adjusted VAEP to a per 90 basis.1
The numbers show fairly high correlations for both groups. The correlation for different-team players is lower which makes sense because their contexts have changed a lot more than for players who stayed on the same team. The difference of 0.08 tells us the impact of between-team context, i.e. the changes in context from switching teams.
Differences per position
We know that almost all actions on the pitch are context-dependent — some more than others. While attackers are able to create some actions on their own, defensive performances are much more a result of the team's playing style. Put another way, the effect of team context might be more pronounced for some positions than others. The main problem with this dataset is that it doesn't distinguish between positions per season. For example, Kimmich is recorded as a D(CR), DM(C) because he played both positions throughout his career. However, we don't know in which season he played which position. This is particularly problematic for players who played in defense and midfield. I excluded most of those cases, but there are still caveats. But more on that later, let's look at the results.
The first thing that pops out is that the correlation coefficients per position are lower compared to the correlation coefficient of all players. A scatterplot might help to illustrate why.
If we evaluate all positions together, we would see a clear relationship between VAEP-1 and VAEP-2 that covers a wide range of values. If we evaluate only one position, we would see a smaller portion of the relationship, with a smaller range of values and higher variance which could result in a lower correlation.
Along those lines, we can also see that the correlation coefficients of defenders and forwards are lower. Defensive performances are context-dependent, so you would expect a dropoff when switching teams. However, this doesn't explain the relatively low correlation for players who stay on their team. One part of the explanation might be that within team-context matters a greater deal for defenders (so far we‘ve only talked about between team context). For example, we know that coaches change every 13 months on average in Spain and Italy. If every coach has his own tactical system and if defensive performance depends strongly on the tactical system, this might explain some of it.
For strikers, the explanation might be a little easier as VAEP is biased toward attacking players. "That is because VAEP generally assigns goals high action values such that players can boost their rankings by scoring many goals." Since finishing is not a consistent stat, strikers have a higher variance. Other possession value models like g+ handle this differently, so their correlation might be higher.
When we evaluate the differences for players who changed teams, the dropoff for attacking midfielders is most eye-catching. I’m not sure I have a very good explanation for it. For one, the same-team correlation is much higher, so a higher dropoff isn't that surprising. Beyond that, maybe it comes down to the number of opportunities. If we think about how attacking midfielders create value through on-ball actions, it's passing, dribbling, crossing, and shooting. Perhaps, they just don't get the ball as often, i.e. don't have as many opportunities, to create a similar output when switching teams (think Coutinho going from Liverpool to Barcelona). The same can be true the other way around. If a player goes to a weaker team where he takes on a larger role, he may have more opportunities and therefore boost his VAEP.
Caveats
There are caveats when using year-to-year correlations. For example, it assumes that player abilities are stable from one season to another. That's obviously not true. Players improve and decline throughout the season in addition to other context changes which may underestimate the stats' reliability.
In addition, I already mentioned the position issues. While I tried to exclude players with unclear positions, there are certainly still players whose position isn’t accurate. More generally speaking, positions themselves are rarely ideal. Players can play in the same position but still have very different roles. One way to work around that is to use other data sources, calculate player roles and merge them in, hoping the sample size will remain large enough.
Another point worth highlighting is that we're effectively comparing the latest season of one player at a club with his first season at another club. It's reasonable to argue that these are two different cases. Whereas the player could have played multiple years for his previous club and therefore performs well, he might need time to settle in at his new club and improve over time. One example of how to work around that is to look at the average of two seasons or take the second season at his new club as the reference.
There is also some selection bias due to the introduction of a minute threshold. This will likely exclude some players who changed clubs but didn't see enough minutes to reach the threshold. This isn't ideal, but I honestly don't know a good way to work around that without accepting other biases.
What's next
We have seen how reliable VAEP is year-on-year, how team-context-dependent it is, and how that differs by position. One interesting follow-up study would be to compare the context-dependency of VAEP to other contribution metrics, such as g+, EPV, or Expected Threat. While they largely serve the same purpose, there might be details that differ. This wouldn't tell us if one is better than the other, but it would tell us if one is more context-dependent than the other.
Another thing I'd like to write more about is how we can use this input to decide how much we regress historical contributions to the mean. In the introduction, I referenced Tiotal Football's idea of stripping out team effects to make more accurate player contribution projections. We can use the context-dependency as an input to decide by how much we regress his previous contributions.
There have been a few studies that calculated adjustment rates when moving between leagues (like here and here). We could use those and convert the numbers to the same league level. However, I don't think it's worth doing that given the large enough sample and the potential bias of these adjustment rates