Might help if you mentioned specs for the boat, engine and, possibly, transmission.
FWIW: What you describe sounds a lot like the behaviour of the reversing gear on our A4. In reverse there is no detent. In forward, there is, but it may not be all that pronounced after a dozen feet or better of shift cable and an 8" long shift lever.
Our prop turns gently and intermittently when the reversing gears is in neutral. Forgot to note which direction.