I tried to take off the door handle this morning (3 screws) but it still won't come off, do i have to take the whole door panel off to get at the part where the pwr window and power mirrior buttons are?

i want to try and fix the heated mirrior button.