How to get our scaredy cat comfortable around my husband?

We adopted an adult cat from the pound last summer. He had been left to starve in a bank-owned home, so god knows how he was treated before we got him! He will snuggle with me when he's in the mood, but runs from my husband.