left ear starts hurting after about 50 minutes of swimming

Everything I have read about 'swimmers ear' points to this not being it.   What I think is going on is the constant changing of pressure with breathing.  I breath in every 3rd stroke so I breath on both sides but when I breath out I wonder if most my 'bubbles' are going into my left ear and causing it to depressure again.   I know if I would stay in the water and not lift my left ear out of the water this will help prolong my swim without discomfort.  I know that ear plugs(one ear plug for me) would help the situation but I'm wondering if there is something with my technique dealing with my left arm going around or breathing out or something that is causing this to happen.

It eventually builds up to where it really is hurting and I then stop my swim.  About 30 seconds after I'm out of the water I don't even remember my left ear hurting anymore.

Anyone have any idea at what is going on?

Thanks,

Ken