I think I might have a solution.
I am planning a 2 battery system that will incorporate 2 x KW 80w panels charging through the Blue Sky SB2512i mppt regulator to a 105a/h deca seamate AGM battery.
The AGM will be the primary battery and there will be 100a/h flooded cell auxillary battery to provide CCA capacity for engine start and anchor winch.
The Blue Sky regulator has a dual battery function that will combine the auxilary battery once the prime battery charge setpoint has been reached.
Noting the absorption voltage for AGM is 14.3v and 14.4v for flooded therefore these batteries will be compatible when combined.
As noted previously on the engine side I have a 60A alternator charging through a Blue Seas automatic charging relay (ACR) this will perform a similar function to the Blue Sky regulator in that it allows one battery to receive charge until a set point is reached and then both batteries are combined.
The 60A alternator will charge the start / anchor winch flooded cell battery as the prime before combining both batteries for charging. (this will ensure the start battery is always ready for an emergency)
Therefore the primary battery when the solar system is charging will be the AGM and the primary battery when the alternator is charging will be the flooded acid battery.
I understand from the Blue Sky distributor that the solar system can remain connected while the alternator is charging.
I would not advise you to use your windlass off the starting battery. Wire the windlass to the house bank and double the size of it if you plan to spend time cruising. A single 100Ah battery will have 50 usable Ah's from 100% state of charge and only 30Ah's when cycling between 50% and 80% when off crusing. AGM batteries like to be recharged to 100% as often as possible so the solar is a good idea.
There is no such thing as "recharging the start battery first" with the ACR. It is a dual sensing device that automatically combines when a voltage of about 13.2V is present at either bank. If you are going to have a "dedicated" start battery I would use it as a dedicated start battery not a windlass and starting battery.
With a solar panel you will spend a lot of time "combined" and I generally don't like mixing battery types in this scenario. AGM and wet cells accept current very differently. As Brian said an Echo Charge would be a better choice if mixing types but then you REALLY DO NOT want to run your alt to the start bank or use a windlass off it.
Keep in mind that your Blue Sky will NOT combine the banks. It will divert a max of up to 2A to a second battery. Also I think that feature is only available in the iX models not the i models IIRC? You don't need that feature as you already have an ACR that will turn on an combine the banks when 13.2V is reached.
Like Brian has said you'll be best served to feed all charge sources to the house bank and also feed your windlass there especially with the Blue Sea DCP switch which does not allow you to choose a second or reserve bank when or if you need it..
Starting an 8HP diesel can be done with even the smallest of personal watercraft battery and AGM's make great starting batteries.
A 100Ah house bank is very small by today's standards but we don't know what you'll be using off it so it may be okay for a day or so....