Carefully look for a fuse on the solenoid circuit.
Typical small Yanmars QMs, GMs etc. usually had a fuse on the solenoid circuit. Usually this is mounted on the aft portside of the engine ... and is very subject to corrosion in the wiring / connectors because the components were not tinned. What happens is that with a few trial attempts of starting, the corrosion in the fuse body/holder heats up and dries out the corrosion enough lower the resistance to allow enough current to flow to energize the solenoid coils.
Next time the engine doest want to start.... open the engine cover, locate the solenoid and with an insulated screwdriver simply short/jump between the large HOT battery terminal and the smaller wire from the control panel. If the 'shorting' between these two terminals immediately *engages the solenoid and the engine starts* (dont get you clothing, etc. caught in the 'moving parts' when doing this) .... then you have HIGH RESISTANCE in the solenoid control circuit (usually a corroded fuse/holder). If you actually have one of these notorious engine mounted fuse holder ... entirely replace it (dont waste your time trying to clean it) and replace the whole wiring (with 'tinned' wire) all the way back to the panel mounted 'start' button.
I'll offer odds that corrosion & high resistance in the solenoid energizing circuit (usually the fuse holder) is your problem as this is common/notorious on small Yanmars.