Do the small gaps in ductwork really make that big of a difference in energy usage?

Absolutely. As cited above, the Department of Energy estimates that 20% of air in a typical home is wasted because of leaky duct systems. Furthermore, leaky systems are so common that even the majority of brand new homes are in need of air sealing and duct sealing.