Sail to the isle of the mists quest only works when taking a boat from skellige harbor?

On nintendo switch. If I take any boat from skellige aside from skellige harbor, the directional marker points to the nearest fast travel signpost.

Thanks