Collecting Ammo

Question for y'all: like many I've seen, I want to start a few displays of every round I can get my hands on. For those of you who have done this, what were some tips/sources to finding individual or small quantities of the rarer rounds?

I've thought about going to my LGS's and asking if they have any loose rounds they'd be willing to sell but wasn't sure if that would be considered annoying (don't want to be 'that guy' lol).

FWIW I've seen pre-made displays with pretty good selections available for purchase but I want the challenge of tracking everything down over the years (just not sure where to start).

Thanks in advance!