How old does a horse have to be to start training? Is there a minimum age before harness training or even actual riding? I know it can depend a lot on the horse, but what about the actual physical limitations of the horse?

Training from the ground should start at birth with imprinting and move to ground manners and conditioning to new situations and things. As for when a horse should be ridden 3/4yrs is pretty good to start a horse as earlier can ruin their legs and backs and lead to long term problems. Race horses are often started earlier but have shorter careers as the younger the better for weight restrictions. Harness training other than just putting it on the horse would fall under the same thing. You can harness and ground drive a horse without them pulling anything, lunge, ground drive and do plenty of things that will train a horse to deal once a rider is on their back.

Good bit if information there, I wasn't sure the difference between a race and a non race horse if it might differ or not for training times either.

The only difference I've ever really seen between training to race and not is the regard to the longevity of the horse's soundness. Racing is pushed as fast as possible and if the horse only gets a couple good years, thats fine: they can sell it on to a regular riding home for retraining. With regular training your goal is a sound horse for the next 20 or more years. You wouldn't strap 20lbs on your toddler and its the same with a foal.