try a bigger fin

yes the side fins would have made a huce difference.
remember to keep the shaft close to the board when paddling, the further out you are; the more the board will want to turn.
but at the end of the day, once u nail your technique, uze will be straight lining like a champ.