Is 6 weeks in a sling after a dislocation a bit too much?

I dislocated my shoulder for the first time on July 18th, and the doctor told me to stay in a sling for 6 weeks and then start physiotherapy.

However, lately i've gotten annoyed with keeping my arm immobile. I still am smart about the use of my arm though. I avoid external movement, I wear a brace for extra support cause im paranoid, and I do wear a sling to bed for extra precaution.

So I was just wondering, should I follow the doctor's advice? Or can I finally take off the sling?