Is it possible to have Halsin at the Grove fight?

I can't seem to figure out if it's possible to free Halsin without triggering the entire goblin camp attacking me. I'm hoping to find a way to get him back to the grove and still have Minthara and her horde alive to attack it.