Trying to find the source of a fragrance

For a while, I've been obsessed with the fragrance of a particular plant, but because I would smell it out in public, I could never find out which plant it would be coming from. I would catch it in Northern and Southern California as well as Virginia. It's very milky/dewy with grassy notes, very faintly citrus-y, and very sweet. I don't think it's classically "floral."

At first, I thought it to be sweetgrass, but it was too cloying and citrusy (though I could try smelling it again to really see). Then, I thought it to be daylily (some yellow fragrant variety), as I would catch that fragrance walking past daylily grass in Southern California. This one still could be a possibility, as I have also encountered it in Northern California, however, never when they are blooming.

Finally, I now suspect it to be bank catclaw, as I pass by a large bush of it pretty much everyday right now. I smell the scent I like, but there are no daylilies around, and I have heard that this plant is very strongly fragrant. However, people say it's more akin to cinnamon/baked goods than grassy and milky. This plant also is not in Virginia or Northern California, so it may just be a similar scent (or the true plant is still hiding nearby).

Does anyone have any ideas? I know it's a bit hard finding something with a clue that is so subjective and elusive as scent, but I've been trying to find it for years and it's killing me not knowing what it actually is.

The one thing I do know, is that the smell does not come from the flowers of this mysterious plant; I have never smelled it amongst flowering plants. </3