Your problem is as above, you're using an SDS+ bit in a standard hammer drill.
The hammer drill has a mechanism that makes the whole chuck, together with the bit in it, vibrate "in and out".
An SDS drill has a mechanism whereby the whole bit is a loose sliding fit in the chuck, and a mechanism inside the drill bashes the blunt end of the drill, whacking the business end into the masonry, like someone using a hammer and chisel, so there's less "weight" to move in and out.
SDS drills are very good for making holes in masonry, but **** at precision.
So, anyway, as above. Use your smallest masonry bit first, then enlarge the hole with a bigger bit (don't press too hard when you enlarge the hole or you may shear off the tungsten carbide piece brazed into the end of the drill) You might care to buy a small diameter "normal", non-SDS bit if you haven't got a small SDS one (by the way, SDS bits don't fit well into standard chucks).
Don't rapidly cool the bit in water, it is likely to k-nacker your bit. Tungsten carbide is a sintered material that's susceptible to thermal shock.
Edit:
This is the Argos SDS drill:
https://www.argos.co.uk/product/4533102?clickSR=slp:term:guild drill:6:37:1
This is what I assume you have:
https://www.argos.co.uk/product/4643896?clickSR=slp:term:guild drill:4:37:1