It's been very well established that the thumb is a nice sturdy, short, thick, little digit, built for strength, but not for sensitivity or speed.
Although there will always be occasional exceptions to nature's 'rules, the thumb generally cannot do anything as fast or as accurately as any of the fingers. For instance, try tapping your thumbs on a desk or a table - no way can a normal human thumb tap either as fast or as accurately as a finger. Try picking your nose with your thumb, or almost any other pleasurable activity. The fingers have it every time and, again, generally speaking, the most versatile digit is the middle finger and not the index finger, as is often supposed. The thumb is for exerting a good firm grip and using it to operate the plunger on a controller actually deprives you of the stability of that grip. Certainly, you can try to compensate for this and, if you practice enough with any inferior method of carrying out ANY activity at the expense of not practicing the known superior method, it will appear that the inferior method is superior. But this is purely due to neglect of the better method.
For instance, if you spent a year or two with your arms and hands immobilized, and diligently practiced use of the feet, you might well find that you could indeed peel a banana better by the foot than by the hand. A very much more realistic and real-world example is that many people can type on a keyboard, seemingly rather fast, using just one or two fingers of each hand and those same people can hardly type at all using long-established all-digit methods. Yet someone who HAS learned to use the standard method can EASILY double their speed.
There's a thought - just try typing with your thumbs!
Each to their own preferences, but the fastest way to operate your plunger is to use either the index or middle finger - even if you have to hold it a little bit unconventionally.