I get that. Not being able to pre-select (which is very common in SolidWorks) limits your selection ability significantly and is against your natural instinct. I am still convinced that changing the mouse gesture button would easily solve those issues (or just make it changeable).
I played around more with holding the CTRL button while using mouse gestures and it mostly solves my problems with it. So it’s a kind of okay-ish workaround.