Wasps nest

We have a new, fairly large wasp's nest in our garden. It's a long garden and no houses directly at the back, so probably about 100 feet-ish from a house (ours or our neighbours). We have kids. Should I get it removed or destroyed or can I leave it? I hate the thought of destroying it because I know that wasps are still part of nature and the circle of life. They don't bother me but is the sensible thing to get rid of it? Thanks