Actually, do. Do sweatshirt it. I don't end up wearing them nearly enough in outfit posts, but when I'm running out to do errands in the fall I almost always throw on a sweatshirt before I leave the house. It's not the fanciest of outerwear and it's definitely less pretty than a matching cardigan or tailored trench coat but it's warm, cozy, and it has the added bonus of autumn-izing my springy wardrobe. And probably the best part is that they're super cheap. I got this one at Boscovs for like $8 during their labor day sale.
I think the key to incorporating a sweatshirt into an otherwise dressy outfit is to size down. You don't want a giant slouchy sweatshirt, it should be fitted enough that it still looks half decent. And it really should be worn with an outfit that's put-together. My goal is to dress-down a pastel, frilly ensemble with a sweatshirt, not to be sloppy, if that makes sense.