Tadd...The cheap GPS's are just as accurate as the expensive ones so if all you need to know is your lat/long then they will be fine. Most of the small handhelds are designed for hikers etc. rather than mariners so lack the larger multi-data readouts (course, velocity, SOG, etc.) that you get on the slightly larger units (Garmin GPS72,76 etc.) and of course, color screens and charts won't be found on the sub-$100 stuff. Some allow you to store waypoints...others don't permit direct entry. Many times the # of storable routes is not very high. You also generally get a small, less readable screen and may not be able to output NEMA data to other electronics. Some of the above may be important to you...or not. Even the $99 stuff outperforms and out-features units from just a few years ago and can take you anywhere in the world if you have enough batteries! (Get one that can take an adapter!)
Once you get to know your cheap GPS and lust for something more powerful...you can always stash the cheap one in a waterproof box with a bunch of batteries as your backup...I have two like that now!