Filter media and garden

I've been using my (tropical freshwater) aquarium water to water the plants and flowers in my garden, which good results. I've been throwing out the used media though, and it just occurred to me that perhaps I could shake out the used media in some (non-aquarium) water and use that as well?

I guess I'm not sure exactly what my question here is, perhaps there are some pitfalls, especially if it's not just the coarse filter?