Sometimes the poorly drained organic material can heave and give troubles. I would recomend you look into removing the topsoil one way or another and get some coarse material in below your gravel. The way it has been explained to me is, look at how railway beds are built. They can't have any heaving.