Be careful about tapping with a hammer since it's bakelite. You might want to put a piece of wood between the two to cushion the blow.
Mine was tight and if you look at the end of the bracket there is a wire acting as a spring clip to press against the controller to hold it snugly. If you use a thin screwdriver to get under the edge of the wire and pull it back it will loosen the tension it is pressing against the controller and allow you to remove it more easily. Slide the controller downward.