The Difficulty in Predicting Saves

The basic fantasy baseball stats –- wins, ERA, WHIP and strikeouts -– may be ultimately flawed in themselves, but once the entire puzzle is put together the best pitchers find their ways to the top of the list. Roy Halladay, CC Sabathia, Clayton Kershaw, Felix Hernandez: these guys find themselves at the top of fantasy draft boards and the WAR leaderboards every season.

But oh, the save -– fantasy baseball’s great monkey wrench. The save is a monster in and of itself, as evidenced by the furious Closer Run. Players with little other value find themselves highly coveted simply on the basis of being the one tabbed with finishing out the ninth inning on a nightly basis. With just 30 such jobs available, identifying those prepared to move into that role can determine the balance of power in a league.

At the beginning of the season, there is little that will help project saves more than meticulously poring through depth charts and determining who will win each individual job. (Or maybe reading RotoGraphs. Either way.) But in any serious league, this knowledge will be required to simply stay level with the competition. The real gains to be made are found by figuring out which closers are most likely to tank and, in turn, which relievers are most likely to move up and take their spots.

Although much of baseball is a meritocracy, the closers’ position is an oligarchy. We hear about the “closer’s mentality” at least once per year, but it most certainly is not a media creation. Whether it comes from the top of the organization or from managers themselves, save opportunities go to those who have saved games before –- the 107 seasons of 20 or more saves since 2008 have been distributed among only 53 different relievers.

2011, however, was a particularly volatile year for closers, but perhaps we shouldn’t be too surprised. A number of established closers made their exits from the role, whether through retirement (Trevor Hoffman, Billy Wagner), free agency (Bobby Jenks, Rafael Soriano, Octavio Dotel), trade (Francisco Rodriguez, Matt Lindstrom) or injury (Jonathan Broxton, David Aardsma, Brad Lidge). As such, 2011 saw a spike in pitchers who enjoyed an increase of at least 20 saves over the past season.

The 14 pitchers to save 20 more games in 2011 than 2010 fit into a few distinct classes:

The Injury Replacement:
Members: Ryan Madson, Brandon League
Neither the Phillies nor the Mariners found themselves with experienced closers on hand after their first choices, David Aardsma and Brad Lidge respectively, wound up on the DL. Madson and League were clearly the best options for their managers, and each had successful seasons in their first as full-time closer. We see these types yearly.

The Opportunist:
Members: Sergio Santos, Mark Melancon, Fernando Salas, Joel Hanrahan
Melancon and Hanrahan stepped in to become effective pitchers on bad teams. Santos and Salas became the go-to youngsters when the incumbents (Ryan Franklin and Matt Thornton) fizzled early in the season. This is all part of the natural closer cycle –- as bad teams become unwilling to pay for closers, younger relievers get chances to step in, and as good teams are forced to switch closers due to the failures of established closers, talented pitchers can get a chance.

The Resurgent Veteran:
Members: Jose Valverde, Kyle Farnsworth, Francisco Rodriguez, J.J. Putz
Valverde simply had the best (and luckiest) season of his life, recording 49 saves without a failure despite a strikeout-to-walk ratio barely over 2.0. Farnsworth was the beneficiary of Rafael Soriano’s exit in Tampa Bay and Putz stepped in and rectified the disaster in Arizona. Rodriguez returned from injury to pitch well for New York before a trade to Milwaukee.

The Closer of the Future:
Members: Craig Kimbrel, John Axford, Drew Storen, Jordan Walden
This was the wrinkle 2011 threw at us, as four young closers stepped into the role of closer and performed admirably –- arguably at an elite level. This is something that just doesn’t happen very often, and the doors were opened for two of them by the retirements of Billy Wagner and Trevor Hoffman.

If there’s one key thing to note about these players –- particularly those outside that more rare class of inexperienced phenoms –- it’s that they weren’t easily identifiable by their strong performances from last season. Many of these pitchers weren’t in the closer’s role to begin with last year, so as a proxy for save percentage, we can take a look at their shutdown percentage –- calculated as shutdowns divided by (shutdowns plus meltdowns). This eliminates mop-up roles and simulates saves and holds using win probability instead of the notably finicky save statistic.

Last year, the league notched 3,694 shutdowns against 2,008 meltdowns for a shutdown percentage of 64.7%. Six of the 14 pitchers who saw save gains of at least 20 saves were below that mark, and another two were within five percentage points of it.

Conversely, all six pitchers who found themselves falling at least 20 saves –- Bobby Jenks, Brad Lidge, Matt Capps, Matt Lindstrom, Rafael Soriano and Ryan Franklin -– all notched above-average seasons in terms of shutdown percentage before falling off, whether the drop was due to injury, role changes, or ineffectiveness.

As a whole, both save percentage and shutdown percentage correlate negatively with a pitcher’s change in saves the next season!

This is yet another demonstration of the power of regression to the mean and small sample size. With the limited seasons relief pitchers see –- often recording fewer than 200 outs in a season –- relievers rarely see enough hitters to establish their true talent level. Big performances as a closer for a year can mean very little, as can supposed dips from big names.

Unfortunately, this makes it tough for there to be one specific statistical rule for finding those save diamonds in the rough and figuring out who to avoid like the plague after a big year. It requires a mix of the qualitative –- figuring out which roles are vulnerable and who is in place to move up –- and the quantitative -– figuring out who had down years despite good underlying skills and who overachieved. Look for those groups and you can find the saves necessary to win the category and win your league.

We hoped you liked reading The Difficulty in Predicting Saves by Jack Moore!

Please support FanGraphs by becoming a member. We publish thousands of articles a year, host multiple podcasts, and have an ever growing database of baseball stats.

FanGraphs does not have a paywall. With your membership, we can continue to offer the content you've come to rely on and add to our unique baseball coverage.

Support FanGraphs

Jack Moore's work can be seen at VICE Sports and anywhere else you're willing to pay him to write. Buy his e-book.

2 Responses to “The Difficulty in Predicting Saves”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. PoopyMcTurdypants says:

    I would also argue that figuring out which teams have managers that are more or less willing to stick with the ‘established’ guy is also important. In Detroit for example, Valverde is a sure fire bet to close all year even if he does stink up the joint because 1. Leyland is his manager and 2. Leyland will give him credit for last years results this year.

  2. jdrotskoff says:
    FanGraphs Supporting Member

    I watched every appearance Ryan Franklin made in 2011, and it is not possible that he had a shutdown percentage of 0.879… NOT POSSIBLE. If it IS possible, SD and MD are SEVERELY FLAWED.

    In 2011, Franklin appeared in 21 games. He allowed runs in 14 of those games. In 12 of those 14 games, he allowed at least as many earned runs as he had innings pitched in those respective games. So in 12/21 games, he certainly did anything but increase the Cards’ chances of winning, and you could easily make the leap that he actually hurt the chances. How that equates to a SD% of almost 90, I’ll never understand.

    Re-reading, it says “before falling off,” but that furthers my point. He was never “ON” IN 2011. He allowed at least as many ER as IP in six of his first nine games, and eight of his first twelve. He only had nine more appearances the entire year before being released.

    I’m not trying to knock FG, because I’m obsessed with this site, but i just can’t understand this one.