6. James Madison
Net worth:
$101 million
Our fourth president, James Madison is known as the "Father of the Constitution". He was one of our richest presidents, but also one of the smallest: he stood at 5'4" and never weighed more than 100 lbs.
How he made his riches
His father James Madison Sr. had a giant plantation named Montpelier and hundreds of slaves. This platnation made him the largest land owner in Orange County, Virginia - an area where slaveholders were the political elite. In 1801, while serving as Secretary of State, the 50-year old Madison inherited all of his father's land and holdings.