There are two theories – a collision theory and a gravitational-tidal theory.
The first theory suggests that when the solar system was being formed Venus (like Earth) may have experienced an encounter with one of the many large bodies still orbiting .
In case of the material formed a separate moon but in case of Venus it was absorbed into its body. This body would have contributed mass kinetic energy as well as angular momentum. This led to a new spin direction and speed for Venus that was seriously altered from its initial state resulting in Venus having a retrograde (backward) rotation.
The second (and newer) theory suggests that Venus’s rotation slowed to a standstill and then reversed direction. The sun’s gravitational pull as well as tidal effects from other planets led to Venus’s axis being shifted to a variety of positions throughout the planet’s evolution.
Venus could just settle into one of four stable rotational states – two in either direction with the planet being more stable in one of the two retrograde states. This led to Venus rotating backwards.