It's not unusual, especially with in-line shrouds, to have a mast "pump" under certain wind conditions as you describe. It may be due to too much tension, or too little. Often Babystays were used to discourage this problem.
Double lower shrouds can be adjusted to stabilize the middle of the mast, but I believe your rig has single in-line lowers so you won't have that option.
It may be something you'll just have to live with unless you're willing to add a babystay.
1984 Fast/Nicholson 345 "FastForward"
".. there is much you could do at sea with common sense.. and very little you could do without it.."
Capt G E Ericson (from "The Cruel Sea" by Nicholas Monsarrat)