In addition to the sweet peas, I will be growing a number of other flowers for fragrance. Almost all of them are white. White flowers are known for evening fragrance and that is when most of these at at their best. You can get seds for all these from Select Seeds. Here is what I will be growing:
Evening Stock (Matthiola incana)
These smell out of this world. The flowers are nothing showy. They close during the day and have no noticeable smell. At night they emit a powerful scent a bit like lilies but lacking that heaviness that lilies have if you get too much of it. These like to be direct sown but I had good luck last year starting them ahead and transplanting them when they were only a few weeks old. Evening stock is cold hardy and will self sow after the first year. I like to plant them outside the bedroom window where I can smell them at night.
Petunia (Petunia x hybrida) 'Rainmaster'
Petunias are usually grown as bedding plants but the older varieties smell exceptional at night, especially the white and purple varieties. 'Rainmaster' (it's named for not turning to mush after a rain) is one of these older varieties. Select Seeds says it dates from 1823.
Angel's Trumpet (Datura inoxia) 'Evening Fragrance'
These self seed and you see them all over the place. The flowers smell sweet at night but the leaves smell unpleasant if you brush against them, "similar to rancid peanut butter" says Wikipedia. The seed pods look like prickly grenades. All parts of the plant are toxic.
Tobacco (Nicotiana alata) 'Jasmine'
Nicotiana are known for their sweet evening fragrance. This is less true of the shorter bedding varieties that you now see at the Farmer's Market. If you want them for fragrance, look for seed packets that say they smell good. They are easy to start from seed, and self sow. Most of the fragrant ones partially close during the day.
Fringed Pink (Dianthus hybrida) 'Rainbow Loveliness'
These are highly fringed pinks with a sweet, hard to describe scent. Unlike other dianthus they do not smell like cloves. Around my house we call the fragrance "fabric softener" though I wouldn't say they really smell like that. They are perennial but bloom the first year from seed with an early start.
Last year I saved seeds from some white flowering plants. I expect these seeds to yield at least 50% white plants this year. If I have the discipline to rogue out the non-white flowers, the seeds I collect this year should yield almost 100% white flowers next year.
Heliotrope (Heliotrope arborescens)
Heliotrope has a confectionary smell like almond and vanilla. One common name is Cherry Pie Plant. I have had mixed luck growing heliotrope from seed. It smells nice but after smelling a white one at Hidden Lake Gardens, I decided to spring for 3 white heliotrope plants from Select Seeds. The one at Hidden Lake Gardens smelled far stronger than anything I have ever grown from seed. I later read that the white is sterile and you can't grow it from seed. Will let you know what how they turn out.
Stuff Growing Around the Yard
Because I have been living in the same place for more than 10 years I have a number of other fragrant perennials and shrubs. Most of these make nice cut flowers:
Korean Spice Viburnum (Viburnum carlesii)
These smell intensely of clove. There is a hybrid called 'Mohawk' which also smells great.
Mockorange (Philadelphus coronarius) 'Innocence'
I picked this up as a tiny plant many years ago. I really like the smell of mockorange but have read that some people do not like it. 'Innocence' had variegated leaves and very strong scent. Mockorange is not known for making a pretty shrub. Each year it looks more and more gawky and ungainly. I read somewhere that it should be cut back severely after flowering since it flowers on year old wood. I did this last year and the shrub re-grew a large number of new stems which should flower this year. If it works, than severe pruning is the answer because even if it smells great for a couple weeks, it has to look presentable for the other 50 weeks or what's the point?
Lilac (Syringa vulgaris) 'Mme Lemoine'
This is a white double lilac. Like all lilacs, it smells terrific. The double flowers make it showier for cutting though I can't say it smells any better than any other lilac.
Grenadin Carnations (Dianthus caryophyllus)
Last year I bought 6 Grenadin dianthus from Coleman's. These are biennial or short lived perennials so they need to be replaced each year. I use them mostly for cutting.
I planted a 25 foot double row of peonies at my house when I was growing cut flowers. Some peonies, like 'Festiva Maxima' smell very nice. A number of other ones smell vaguely unpleasant. I have heard them described as smelling like pollen. Some people call this kind of fragrance a "nose twister". Other examples include marigolds, feverfew and Sweet Annie.