Left click detection (in general)

So I am trying to make a stealth system. Basically I will give a player invisibility but remove it when they do something involving left or right cliclking.
So eating food, or attacking with a weapon (regardless if they hit or not) removes the invisibility.
Any way to make this work? I looked at scoreboard paramters but there is only damage_dealt, which would only work when a player lands the hit.