mt abang hike

hello! wondering if we can hike up mt abang without a guide, starting at about 8-9am. Heard we would most likely get harassed by locals but is it possible to just reject them ? After paying 100k~ for entrance fees, would we be able to go up peacefully?