Most people know that white bread is a diet don’t: One of the easiest swaps you can make for a major health boost, according to everydayhealth.com is ditching refined flour in favor of whole wheat. But it’s not just that loaf you need to watch out for. The basket of rolls delivered to the table when dining out, the French baguette you grab on your way home to accompany dinner, your Saturday-morning bagel ritual, and Friday pizza night all come with a side of less-than-desirable health risks.

Here’s an unpleasant reason to nix the bread basket:

Little nutritional value. Yes, food is delicious, but at the end of the day we are eating for one reason: to nourish our bodies. And white bread made with refined flour fails to accomplish this goal. “When a grain is refined, such as in the making of flour for white bread, the outermost and innermost layers of the grain are removed. This removes the fiber and some (25 percent) protein, leaving behind the starch,” says Erin Palinski-Wade, RD, author of Belly Fat Diet For Dummies. You may see “enriched flour” on the label. While this flour has had some nutrients like B vitamins and iron added back in after the refining process, it’s still lower in fiber and protein than whole-wheat flour. Opting for whole-grain varieties carries with it a dose of healthy fiber and more protein, adding a nutritional boost to meals.