This is the kind of a question that would require a book to answer properly, but I will take a stab at it. I apologize in advance for the length of my reply. Most of this response was written as a series of articles meant for another venue and so I am not sure that this flows all that well either, and for that I also apologize.
I think that the terms 'offshore' and 'coastal' get bandied about quite freely without any real thought about what the differences are. Even the term ‘race boat’ is a bit vague since all kinds of boats are raced in all kinds of differing types of competition. Race boats can therefore vary quite widely depending on the type of racing that they are intended for. I am assuming that you are not asking about small one design race boats as much as boats that at least to one extent or another can be raced or cruised in a pinch.
For the most part, race boats are optimized to perform better than the racing rating rule under which it is intended to race. This has a lot of implications. Under some rules (IMS and IRC for example) race boats are optimized to be fast and easy to handle across a wide range of conditions, producing great all around boats, but in the worst cases (International, Universal, CCA and IOR rules for example), the shape of the hulls, and design of the rig are greatly distorted to beat the shortcomings and loopholes in the rule, producing boats that become obsolete as race boats, and to a great extent as cruising boats as well, once the rule becomes history.
While the EU does have a system that certifies boats into one of 4 categories, this rating system was intended to remove trade barriers between the various EU countries. It represents the lowest common denominator between all of the regulations that pre-existed the formation of the EU. A boat that is certified as meeting the CE Small Craft Directive, in the offshore category, has met this minimum standard but it does not certify that the vessel is actually suitable for offshore use. For example the EU standards do not look at motion comfort, or the suitability of the interior layout for offshore use. Stripped out racers with minimal tankage and fragile rigs can and do obtain offshore certification. The U.S. does have the ORC, ABS, and ABYC standards which are somewhat helpful, but again does not certify that the vessel is actually suitable for offshore use
In a broad terms, a well made coastal cruiser should be more expensive than a dedicated offshore distance cruising boat, because it needs to be more complex and actually needs more sophisticated engineering and construction than most people will accept in a dedicated offshore boat. When new, the high tech materials used, and the first class hardware generally employed, make racers comparatively expensive as well. They are also expensive to maintain in full race condition since maintaining a smooth, fair bottom, good sails, running rigging. and sophisticated electronics does not come cheaply. But as they grow older and less competitive, they often become real bargains.
In a general sense, all boats are a compromise and with experience you learn which compromises make sense for your own needs and budget. Most times the difference between an optimized race boat, coastal cruiser and a dedicated offshore cruising boat is found in the collection of subtle choices that make a boat biased toward one use or the other. A well designed and constructed coastal cruiser will often make a reasonable offshore cruising boat and club level racer, while traditional dedicated offshore cruising boats usually make very poor racers or coastal cruisers.
Which brings up another key point. I would think that most knowledgeable sailors use the term ‘offshore cruiser’, they generally think of traditional, long waterline, full keeled or long fin keeled, heavy displacement, cutters or ketches. But in recent years there has been a whole series of ‘modern offshore cruisers’, which have been designed to take advantage of the research into stability, motion comfort, performance, and heavy weather sail handling that emerged as the result of the Fastnet and subsequent disasters. These boats tend to be longer for their displacement, often have fin or bulb keels, and carry a variety of contemporary rigs such as fractionally rigged sloop rigs. Depending on the specifics of the boat in question, a race boat may also make a reasonable coastal cruiser or offshore cruiser but will rarely be ideal as either and will generally take some adaptation to reach a reasonable standard for these applications.
Looking further, when I think of the distinctions between a raceboat, vs. coastal cruiser vs. a dedicated offshore boat, there are specific attributes that I would look for:
A typical well-used coastal cruiser might only sail five hundred to a thousand miles a year. Most do less. A well-used offshore cruiser may do as much as 20,000 to 30,000 miles in a single year (10,000 15,000 being more typical). Whether traditional or modern, offshore cruising boats need to be designed to stand up to the long haul. A single year of offshore cruising can literally be the equivalent the abuse encountered in 20 or 30 years of coastal cruising.
Traditional offshore cruisers come in a range of flavors. Whether fiberglass, steel, alloy or timber, traditional offshore cruisers tend to have robust hulls that are simply constructed. Hull panels tend to be very heavy, accessible and maintainable. Internal framing tends to be widely spaced or almost non-existent. Engineering tends to be simple and reliable. Materials tend to be low tech, which is not necessarily a bad thing. The down side is that a weight goes into these structures using up valuable displacement that could be used for additional carrying capacity or ballast. Some of his weight is carried high in the hull and deck structure reducing stability and increasing roll and pitch.
Modern offshore cruisers tend to use higher tech materials and structural design. Some robustness and redundancy may be given up, but often the better of these newer designs have greater strength despite their lighter weight. These newer designs often take advantage of sophisticated framing systems and purposefully selected alloys or laminates. They often benefit from careful engineering intended to improve impact resistance and longevity.
Whether traditional or modern, offshore cruisers need to be able the cyclical loadings that insidiously wear out a boat over long passages. Larger margins of safety are required. In offshore cruising boats more than the other types, a little weight added, an often breed a whole lot more weight. A little added weight has a way of ricocheting through the whole design cycle. A little weight added means that perhaps the sail area needs to be increased. The increased sail area means a little more ballast. The added ballast perhaps means larger keel bolts and more robust transverse frames. This additional weight and sail area means higher stress on the rigging and so perhaps heavier rigging and attachment points get added, and that means perhaps a decrease in stability or perhaps a bit more ballast. The added weight means more drag and so fuel consumption increases and perhaps so does the size of the fuel tanks. And with all that added weight the designer is then faced with an under-canvassed design or else adding a sail area and risking going though another round of weight addition. Which is why, when all is said and done, traditional offshore cruising boats tend to be so much heavier than race boats, coastal cruisers or even more modern offshore designs.
Coastal cruisers generally benefit from better performance than offshore boats and do not have as stringent a requirement for a robust structure as and offshore boat. As a result coastal cruisers greatly benefit from lighter construction using modern materials and methods. Redundancy and self-sufficiency is less of a requirement. Fully lined interiors and other conveniences are often the norm on cruisers. Even quality coastal cruisers use molded force grids or pans that are glued in rather than laid up in place. Framing is often wider spaced and less robust. Hull panels are often cored and thinner than on an offshore boat. Rarely do they receive the careful workmanship that is required for a quality race boat, or the high safety factors ideally applied to a dedicated offshore cruiser. Then again they don’t need either as their use and abuse is generally much less harsh then encountered in the life cycles of either racing or offshore cruising boats.
Race boats generally benefit from the most sophisticated engineering of the three. Weight is the enemy of speed and motion comfort and so great attention is paid to reducing weight where weight can be reduced. But since breaking a boat is a very slow way around a racecourse, with some notable exceptions for specific racing classes, and racing periods, race boats are surprisingly tough. They are designed for very heavy loadings compared to coastal cruisers since racing crews will often carry on in no matter what nature throws at them, carrying far more sail than one might normally consider prudent. Their larger sail area to weight ratios, proportionately higher ballast ratios, their use of low stretch line and sails, the willingness to carry a lot of sail into higher wind ranges, and placement of the crew weight (often as much as 15% or more of the displacement of the boat) out on the rail as moveable ballst results in enormous strains compared to similar displacement offshore or coastal cruisers.
By the same token, race boats are designed with smaller safety margins so the engineering better be right. The problem with smaller safety margins is that over time race boats wear out quicker than other types of boats. Designers and owners somewhat see that as acceptable since rules also change over time making race boats more likely to become obsolete. Historically there is nothing man made (except perhaps a 15 year old computer) that is quite as obsolete as an obsolete rule beating race boat, and so historically designers are more willing to view them as disposable. At least with some of the newer rules, the boats being produced are good all around boats and quite a bit more robust and so may find a long useful life cycle.
Race boats generally use higher grade and higher tech materials. They are often the first to benefit from advances in structural design. They often have fewer openings in the hull and deck, which results in much greater stiffness and potentially less fatigue issues. Structural workmanship is often as good as it gets in the world of building yachts even if the interior finishes often seem a little crude. Race boats often gave very sophisticated internal framing systems, which take up room within the interior but make them far sturdier than their light weight would seem to imply.
All of that said, this has not always been the case, CCA era race boats often suffered from the mediocre engineering and poor laminating practices of the day, and IOR era and early IMS era boats often had fragile rigs.
On a coastal cruiser there should be good wide berths, with enough sea berths for at least half of the crew for that night run back to make work the next day. An offshore cruiser is often handled by a smaller crew and so fewer berths and fewer sea berths are necessary. The berths on an offshore boat should be narrower and have leeboards or lee cloths. On both I am looking for a well-equipped galley but the galley needs to be larger on a coastal cruiser so that there is adequate space to prepare meals for the typically larger crew or a raft-up. Refrigeration is less important on a coastal cruiser, where ice is typically readily available at the next port of call, although the case can be made for no refrigeration or icebox if you are going offshore. Race boats tend to have Spartan accommodations. Offshore oriented race boats will often have enough seaberths to sleep half the crew on either side of the boat so that the off watch crew can be tacked along with the boat. Water tanks are often reasonably sized to take care of a race boat’s large crews, but fuel tankage is often a bad joke. Storage is generally huge to carry a race boat’s large sail inventory, but it does not work well for carrying groceries, spares and supplies.
A comfortable cockpit for lounging is very important on a coastal cruiser. It should be larger than an offshore boat to accommodate a larger number of people which is OK since pooping is less likely to occur doing coastal work. Dedicated offshore boats generally have smaller cockpits with very large drains. The cockpits are deeper and have taller coamings to protect the crew. This makes them less comfortable for lounging and less easy to move in and out of. Ideally offshore boats have bridge decks that are higher than the lowest point of the coaming and companionway slides that can be locked in place to avoid down flooding. All of which makes moving about a bit less convenient. Race boat cockpits tend to need the wide open spaces to house the vast crowds that inhabit them on the race course. If you think of a race boat cockpit at a mark rounding, you have a helmsman, mainsail trimmer, guy trimmer, sheet trimmer, pit person, and perhaps a grinder or two. That’s a whole lot of people and each need their own space to that voodoo that they do so well. Coamings and seats just get in the way. Modern racer-cruisers often have removable seats that double as cockpit lockers and which are removed for racing (along with a few hundred pounds of the ‘unnecessary gear’ used to deliver the boat to the race course).
While gear for offshore boats need to be simple and very robust, coastal cruisers need to be able to quickly adapt to changing conditions. For the coastal cruiser greater purchase, lower friction hardware, easy to reach cockpit-lead control lines, all make for quicker and easier adjustments to the changes in wind speed and angle that occur with greater frequency. There is a big difference in the gear needed when, ‘we’ll tack tomorrow or the next day’ vs. auto-tacking or short tacking up a creek.
Offshore boats need to be heavier. They carry more stuff, period. The traditional rule of thumb was that an offshore boat needs to weigh somewhere between 2 1/2 and 5 long tons per person. A coastal cruiser can get by with less weight per crew person but generally is cruised by a larger crew. The problem that I have is that most offshore sailors and many coastal cruisers seem to start out looking for a certain length boat and then screen out the boats that are lighter than the displacement that they think that they need. This results in offshore boats and some coastal cruisers that are generally comparatively heavy for their length. There is a big price paid in motion comfort, difficulty of handling, performance and seaworthiness when too much weight is crammed into a short sailing length.
I suggest that a better way to go is to start with the displacement that makes sense for your needs and then look for a longer boat with that displacement. That will generally result in a boat that is more seaworthy, easier on the crew to sail, have a more comfortable motion, have a greater carrying capacity, have more room on board, and be faster as well. Since purchase, and maintenance costs are generally proportional to the displacement of the boat the longer boat of the same displacement will often have similar maintenance costs. Since sail area is displacement and drag dependent, the longer boat of an equal displacement will often have an easier to handle sail plan as well.
-Keel and Rudder types:
I would say unequivocally that for coastal cruising a fin keel is the right way to go here. The greater speed, lesser leeway, higher stability and ability to stand to an efficient sail plan, greater maneuverability and superior windward performance of a fin keel with spade rudder (either skeg or post hung) are invaluable for coastal work. Besides fin keels/bulb keels are much easier to un-stick in a grounding. In shallower venues, a dagger board with a bulb or a keel/centerboard is also a good way to go.
There is a less obvious choice when it comes to the keel and rudder type for offshore cruising. Many people prefer long or full keels for offshore work but to a great extent this is an anachronistic thinking that emerges from recollections of early fin-keelers. Properly engineered and designed, a fin keels with a skeg hung rudder can be a much better choice for offshore work. There is the rub. Few fin keelers in the size and price range that most people purchase for offshore cruising are engineered and designed for abuses of dedicated offshore cruising. That said, popular offshore cruisers like the Pacific Seacraft, and Valiants have adapted skeg-hung spade rudders while the Island Packets have chosen to use a rudder post hung spade rudder.
Of course race boats thrive on stability and low drag. For them, under most racing rules, there is only one choice, fin keels and spade rudders. Under some of the older rules, (International, Universal and CCA) race boats often had fin keels with attached rudders. This was the worst of all worlds. The boats do not track as well as a spade rudder but have all of the negatives of a keel hung rudder (greater exposure to damage being close to the depth of the keel, higher loading, less maneuverable). Newer race boats have minimal foil area and large bulb keels. This combo was chosen for greater stability and therefore sail carrying capacity as well as minimal drag. As a side benefit this keel type has been demonstrated to offer increased seaworthiness and motion comfort that results from a significantly lower center of gravity relative to the vertical center of buoyancy, better dampening, and the ability to stall at high side loadings which reduces the likelihood of being rolled in a large breaking wave. In US Naval Academy studies of groundings, bulb keels were also shown to be the easiest to extract from a grounding, which is a very good thing considering that more modern race boats generally have significantly deeper drafts.
Good ground tackle and rode-handling gear is important for both cruising types but all-chain rodes and massive hurricane proof anchors are not generally required for coastal cruising. Race boats often lack proper anchor handling gear or in the most extreme cases, they may even lack fixed cleats to tie up with. Frankly from the racers point of view these are simply things that get in the way of that perfect hoist, douse , tack, or jibe.
At least on the US East Coast, (where I sail and so am most familiar with) light air performance and the ability to change gears is important for a coastal cruiser. It means more sailing time vs. motoring time and the ability to adjust to the 'if you don't like the weather, wait a minute' which is typical of East Coast or Great Lakes sailing. If you are going to gunkhole under sail, maneuverability is important. Windward and off wind performance is also important.
With all of that in mind, I would suggest that a fractional sloop rig with a generous standing sail plan, non- or minimally overlapping jibs, and an easy to use backstay adjuster is ideal. This combination is easy to tack and trim or change gears on. I would want two-line slab reefing for quick, reliable on the fly, reefing. I would want an easy to deploy spinnaker as well.
For offshore use, traditional cruising boats tend to have a very high drag relative to their stability and so low aspect ratio rigs are important. Depending on the size of the boat, cutter and ketch rigs are the time-tested solution. They work reasonably well as long as simplicity and performance are not important.
More modern, lower drag offshore cruisers seem to be using fractional sloop rigs with a great deal more frequency for many of the same reasons as coastal cruisers. But modern offshore cruisers can be found with the full range of rigs; masthead sloops (with and without removable jib stays), cutters, ketches, even schooners you name it.
Racers are only concerned with efficiency, the most drive for the least sail area as measured by the rule. It is this last phrase that has lead to terrible distortions to rig proportions relative to what is easy to handle or actually efficient in an absolute sense. For example, the CCA under penalized genoas and mizzens, and so yawls and huge overlapping headsails appeared. The IOR fairly measured mizzen sail area and so yawls disappeared again, but the IOR over penalized mainsails and under penalized headsails and so masthead sloops with tiny high aspect ratio mainsails and huge genoas and spinnakers became the rage. The IMS measures the impact of sails more fairly and so fractional rigs with their ease of shifting gears has become the rage.
The bad news for coastal cruisers is that the racing rig fad dujour often shows up on next year’s coastal cruiser. The really bad news is that since coastal cruisers often stay in production for many years these bad fad ideas often stay in the coastal cruiser marketplace for a very long time. If you doubt that look at the IOR proportioned rig on most Catalinas.
I think that speed is especially important to coastal cruising. To me speed relates to range and range relates to more diverse opportunities. To explain, with speed comes a greater range that is comfortable to sail in a given day. In the sailing venues that I have typically sailed in, being able to sail farther in a day means a lot more places that can be reached under sail without flogging the crew or running the engine. When coastal cruising speed also relates to being able to duck in somewhere when things get dicey.
It can be argued that speed is less important to the offshore cruiser. What’s an extra knot when you have an ocean to cross. On the other hand, a little more speed can allow a crafty distance cruiser to pick the wind system that he is sailing in or keep moving in a doldrum. It can mean more sailing relative to motoring and so a lower requirement for fuel, stores and water capacity. It can mean somewhat less expense for a given passage. It can mean more time in an interesting port relative to time at sea, which is an advantage to those who prefer portside to offshore, but which is a disadvantage to those for whom cruising is all about the passage making.
Of course, race boats are all about speed, speed in all conditions. The best race boats are fast in all conditions and are quick to shift gears on the fly dealing with whatever Mother Nature throws at them. I often hear how cruising boats are faster than race boats in a breeze. I just have not seen that at all. Big wind or small, most modern race boats are radically faster than their non- racing sisters from the same era. The possible exception to this would be the CCA era boats with their short waterlines, and the pre-Fastnet IOR boats with their distorted hullforms and rigs.
Good ventilation is very critical to both cruising types. Operable ports, hatches, dorades are very important. While offshore, small openings are structurally a good idea, for coastal work this is far less of an issue. Ventilation is irrelevant to most dedicated race boats, but boats intended as racer cruisers need all the ventilation they can get, if nothing else to dry out after a spinnaker that has been shrimping gets dumped down below to be packed.
-Visibility and a comfortable helm station:
Coastal boats are more likely to be hand steered in the more frequently changing conditions, and higher traffic found in coastal cruising and are more likely to have greater traffic to deal with as well. A comfortable helm position and good visibility is critical. Offshore, protection of the crew becomes more important. Racers need to see their sails, the waves, and their competition. Visibility is important, but often gets compromised by the use of deck sweeping jibs and low booms that wipe out visibility from ‘the high side’.
Storage and Tankage:
There is a perception that coastal cruisers so not need storage. I disagree with that. Coastal cruisers need different kinds of storage than an offshore boat but not necessarily less storage. Good storage is needed to accommodate the larger crowds that are more likely to cruise on a short trip. Good water and holding tankage is important because people use water more liberally inshore assuming a nearby fill up, but with a larger crew this takes a toll quickly. Holding tanks are not needed offshore but they are being inspected with greater frequency in crowded harbors and there are few things worse than cruising with a full holding tank and no way to empty it. Offshore boats generally need larger fuel tanks.
Motion Comfort and Seakindliness
This is probably the most controversial of the topics. It is important to understand that when it comes to motion comfort, there is no universally right, one size fits all. Relative motion comfort derives from the amount of motion and rate of acceleration that is inherent in the design of a boat as it is actually loaded. How comfortable the motion feels results from the combination of those two factors; amount of motion and the rate of change in motion. It is very hard to design a boat that inherently has both a minimal range of motion, and which also has slow acceleration rates. It can be done, but historically it wasn’t.
Complicating this discussion further, is that fact that people are affected by motion differently. In a U.S. Navy study of motion sickness, it was found that of the people who are prone to motion discomfort, roughly one third were predominantly affected by the amount of movement, but were minimally affected by the rate of change. Another third were predominantly affected by the rate of change, but were minimally affected by the amount of movement. And the remaining third were affected by both the amount and the rate of change. The ideal solution then is to find out how you personally are affected by motion and then to seek a boat with a motion consistent with your own natural preference.
There are a lot of factors that affect a boat’s natural motion but the biggies are inertia, stability, buoyancy distribution, and dampening.