Flowering shrubs are invaluable in the garden, and many have pleasant perfumes as well as lovely flowers.
If possible, buy shrubs when they are in bloom to be sure that you like the smell. Because of genetic variability, shrubs that are not named cultivars may not always smell or behave exactly as one hopes. Named cultivars on the other hand, since they are clones of the original plant, ought to be identical and behave predictably.
One shrub that I enjoy in my garden is Vitex agnuscastus. It is commonly known as chaste tree, and it has pretty lavender-blue flowers in midsummer. It grows well in my zone 5 garden and even survived a major relocation to a new bed and a terrible drought a year or so ago. So it seems pretty tough and adaptable. It has a graceful form and fresh green foliage and dies to the ground each winter. It is about 5 feet high in my garden, and I use the blooms in flowers arrangements as their shape reminds me of lilacs.
The bush seems untroubled by pests. The scent when the boughs are cut is astringent and pungent and may not appeal to everyone, though I am untroubled by it.