Why didn't the Minbari hunted all the Soul Hunters down?

The Soul Hunter race are a real antagonist for the Minbari people. So why didn't they just launch a holy war against them and wipe them out?