Please be careful when it comes to horses. And please don't fall into the gottta have this or that just because some horse person tells you you need it. Stick to the basics. saddle, bridle and such no need for fancy signature stuff unless you get into showing. When you keep it simple and inexpensive then you can have fun without worrying too much about the cost. Look for stuff at auctions or your local feed store that is used to start out. Only possible exception might be in a saddle. Take the lessons and if need be when you go to buy your first horse pay someone you trust to go with you and vet check any horse you buy. As with anything there are those out there that would take advantage of someone new to the activity.
There is an annual "Small Farm Expo" that is put on in Pennsylvania, New York and New Jersey. I don't know where you are in Ontario or if anything like this is available to you. The last New York state show was held in New Paltz. 2005 was in PA and the 2006 show will be in Augusta, NJ. I had attended the first show in 2001 which was held in PA. Various avenues for small farm income were discussed via workshops and demos. Goats, sheep, cattle, horses, alpacas, etc., they had it all covered. The show should go back to New York for 2007 but the location does change.
Maybe with a little searching or some member feedback, you can find a similar venue convenient to you.
Some folks raise them to sell, some to milk, some to eat, you name it.
We bought two full blooded ( non-registered ) bred Dexter cows for $900 in 2004. Registered cows or heifers run around 1000 - 1500 normally.
We purchased a registered bull in late 2004 after the cows had produced a couple heifer calves, the bull was $450. The bulls generally are cheaper than the cows as more folks want the heifers and cows for breeding stock.
Last year the cows produced a heifer calf and bull calf from the original cows and the bull we purchased, they ended up being red ( Dexters can be black, dun or red, the red is kind of a brownish red. ) Red is the rarest, to bad our cows aren't registered so I could register these new calves and sell them that way. I haven't decided what to do with these new calves, most likely I will put them up for sale once I wean them in the spring.
There is a good website with message boards about Dexters at: