For many years (decades?) a common suggestion was that one should attempt to gain some muscle mass mass (through resistance training and possibly overeating) prior to beginning a diet. Well meaning individuals would suggest you spent 3-4 weeks or more training hard and eating well to gain muscle mass. The goal was to raise metabolism so that the diet would go more effectively.
In that current data indicates that each pound of muscle might burn an additional 6 calories (as opposed to older values of 25-40 cal/lb or even higher) (1), this argument is no longer tenable; to significantly affect metabolic rate would require a monstrous gain of muscle mass, far more than you could gain in 3-4 weeks.
Even if you gained 10 pounds of muscle, that would only add up to an additional 60 calories burned per day, hardly enough to worry about and certainly not enough to affect the following diet. Which isn't to say that diets don't work better after short or even medium periods of overfeeding, mind you, it's simply not because of gains in muscle mass. A more recent idea making the rounds in bodybuilding nutrition is that, prior to trying to gain lean body mass, people should diet down first. This reasoning is based on a variety of data that has examined the changes in body composition that occur when you overfeed either thin or fat individuals (see for example, reference 2 or just about anything Gilbert Forbes has written).
A Primer on the P-ratio
The above recommendation is based on a lot of data on something called the P-ratio (which stands for partitioning ratio) which essentially represents the proportion of protein (LBM) you gain relative to the total weight you gain (this isn't the technical definition of P-ratio, by the way, I'm just trying to simplify it a bit).
Now, a lot of factors control P-ratio including genetics, hormones, diet and training (to a smaller degree than you'd expect) and probably some I'm forgetting (3). But by and large, the primary predictor of P-ratio is starting body fat percentage. Basically, your starting body fat percentage predicts the great majority of what you will lose/gain when you diet/overfeed (4).
So, when you diet, the fatter you are, the less LBM (and more fat) you will lose. Conversely, the leaner you are, the more LBM and less fat you will tend to lose when you diet. This makes sense in evolutionary terms, the more fat you have to lose, the more your body can lose without having to burn off muscle tissue; the leaner you get, the less fat you have and the more muscle you end up losing. Anyone who's dieted naturally to sub 10% body fat levels knows this to be true: the leaner you get, the more muscle mass you tend to lose.
So what about overfeeding and gaining weight? Well, in general, the same holds but in reverse: leaner individuals will tend to gain more LBM and less fat and fatter individuals will tend to gain more fat and less LBM. This actually makes sense when you think about it. The fat individual loses a lot of fat/a little LBM when they diet and gains a lot of fat and little LBM when they overfeed while the leaner individual does the opposite. P-ratio appears to be constant going in both directions. That is, P-ratio appears to be constant for a given individual (5).
So, typically, when overfed, thin/lean individual will gain 60-70% lean body mass (LBM) while fat individuals may gain only 30-40% LBM. Note that these percentage gains are without exercise, simply with overfeeding from a starting body fat level. Although research hasn't examined overfeeding nearly as much as underfeeding, we might expect intensive weight training to skew these numbers to an even better point.
So far, so good right; it sure seems like the leaner you are, the better your body composition changes will be during overfeeding? So get lean and then train and eat and you should gain piles of muscle back, right?
The Problem: Naturally Lean People vs. Dieted Down People
The problem with the above analysis, exciting as it sounds, is that there are significant differences between folks who are naturally lean (on whom the original overfeeding research was done) and subjects who have been dieted to leanness.
Let's consider, for a second the likely physiology of those folks who stay naturally lean. Based on the Geneticcs Hypothesis (3), we'd expect them to have pretty good hormonal status in terms of thyroid levels, low or normal cortisol, maybe decent levels of testosterone, GH and IGF-1. They probably also show a normal nervous system output and an ability to increase fat oxidation when calories are raised as well.
We'd probably expect them to exhibit a spendthrift metabolism (6), one that cranks up in response to overfeeding to burn off excess calories. It wouldn't be surprising if they were the ones who showed a great deal of Non-Exercise Activity Thermogenesis (NEAT, 7) which is what allows them to burn off excess calories without getting fat. All of this, almost certainly with other factors would all contribute to their general lack of fat gain during overfeeding. Of course, if fat gain is limited during overfeeding, that would tend to mean that any weight gain will tend to be LBM, as the P-ratio data described above indicates.
The problem is that the above physiological profile in no way describes individuals who have dieted down to a low body fat percentage. Rather, dieted individuals typically show a biology that is absolutely not geared towards anything except packing the body fat back on. Typically, the metabolic consequences of dieting include a lowered metabolism, decreased fat oxidation, decreased HSL activity, increased LPL activity impaired hormonal status (including lowered testosterone and raised cortisol), decreased thermogenesis from a reduction in both thyroid levels and nervous system output and a host of other metabolic defects. All of these serve to both slow fat loss during the diet and ensure rapid fat regain when food is reintroduced.
For example, in the classic starvation study (the Minnesota Semi-Starvation study) men were dieted for 6 solid months reaching 4-5% body fat at the end of the study. Then they were refed and body composition was tracked. By the theory being advocated, they should have gained lots of LBM and little fat during refeeding, they were clearly super lean to start out with. But this is absolutely not what happened.
As would be expected based on the metabolic adaptations to dieting, their bodies were mainly primed to replenish fat stores. Reductions in metabolic rate, fat oxidation and thermogenesis all contributed to a preferential gain of body fat and these systems didn't reset themselves until all of the body fat lost had been regained (8). Quite in fact, signals from body fat (i.e. leptin and the rest) are the mechanism behind this physiology (9).
The bottom line is that, in dieted down individuals, the body is primed to gain body fat at the expense of LBM to replenish what was lost during the diet. Again, this is fundamentally different than looking at genetically lean individuals (for whom a low body fat percentage is their normal level) in terms of what happens when they are overfed.
And even without this research available, anybody who's dieted to a low body fat percentage can attest to the above. Regardless of the theories being advocated by the individuals looking just at Forbes' data on P-ratio, the end of the diet is a time when you gain body fat the most easily. Even a brief look at the real world should have pointed out why the theory was incorrect in the first place.
Now Watch Me Backpedal
Having hopefully shown you why I think the idea that getting lean first will magically let you pack on the LBM without fat gain, I'm going to backpedal and say that that doesn't mean I think that dieting first is always a bad idea. Quite in fact, there may be very good reasons to diet prior to going on a mass gaining phase. Simply don't expect it to cause the types of magic that some are claiming.
Part of the reason that preceding a mass gaining phase with a diet is one of practicality. If you want to compete in a bodybuilding contest, you need to be sufficiently lean to start with (10-12% body fat for males) to have a chance of coming in on time. That may mean keeping body fat in check by dieting prior to trying to add mass. Similarly, if you simply want to get lean for appearances sake, you need to keep body fat under control.
Meaning this: if you start a mass gaining phase at too high of a body fat percentage (say 12-15%), you're going to gain some fat during that phase and end up in the high teens or worse. This makes dieting back to a non-fat assed body fat percentage a real hassle. Better to keep things in check by alternating periods of cutting and gaining.
As well, it seems empirically that once body fat gets to the 15% range or so for men, fat gains tend to accelerate during mass gaining phases. I suspect this is due to the development of systemic insulin resistance which causes calories to go into fat stores more readily. Keeping body fat levels below that may be helpful.
I should mention that there was always an anecdotal idea that mass gains were best with body fat about 10-12% body fat (for men, add 9-12% for women). While I had always dismissed this as being an excuse to stay fat, I suspect it's probably close to correct. Based on what's going on hormonally and physiologically at both low and higher body fat percentages, this may very well be a sweet spot for mass gaining. You're fed and healthy enough to lift well and make gains but not so fat that other problems arise.
Ok, enough theory crap. Based on the above data, here's what I would generally recommend to bodybuilders or athletes who want to put on muscle mass (i.e. all of them).
1. If you're above 15% body fat (about 24-27% for women), diet first. If you can get to the 10-12% (19-24%) body fat range or so, I think you'll be in an overall better position. Trying to get super lean will probably end up screwing you in the long run because your body will be primed to put back fat on (and most other physiological systems are screwed up as well) when you get super lean.
2. After finishing your diet, regardless of how lean you get, take 2 weeks to eat at roughly maintenance calorie levels before starting your mass gaining phase. The reason has to do with the physiological adaptations to dieting described briefly above. Although you can't reverse all of them short of getting fat again (or fixing the problem pharmaceutically), 2 weeks at maintenance, which by definition should be higher calories than you were eating on your diet, will help to normalize some of them. Leptin, thyroid, SNS output should improve a bit, along with other hormones, putting you in a better place to gain mass without super excessive fat gain. Make sure to get at least 100 grams of carbs/day or more during this phase so that thyroid will come back up.
3. Only try to add mass/bulk until you hit the top end body fat percentage listed in #1 above. So that's about 15% body fat for men and 24-27% body fat for women. What this would mean in practice is that you diet to 10-12% body fat for men (22-24% for women), eat at maintenance for two weeks to try and normalize things, and then add mass until you hit 15% body fat for men (22-24% for women) and then diet back down. Over a number of cycles, you should be able to increase your muscle mass while keeping body fat under control.