Serving under President George W Bush, he was a leading proponent of the administration's "war on terror".
US forces launched a campaign in Afghanistan after the 9/11 attacks and invaded Iraq in 2003.
Mr Rumsfeld resigned in 2006 amid the fallout of the conflict, but always defended his record.
Reacting to the news, President Bush described him as "a very good man" and an "exemplary public servant" who "never flinched from responsibility".
Mr Rumsfeld's family said he died at home in the town of Taos, New Mexico, on Tuesday.
"It is with deep sadness that we share the news of the passing of Donald Rumsfeld, an American statesman and devoted husband, father, grandfather and great grandfather," they said in a statement.
Born in Chicago in 1932, Mr Rumsfeld's political and private career spanned decades.
He first came to Washington DC in the 1960s and served in multiple positions under Richard Nixon and Gerald Ford.
In 1975, he became the youngest person to be named defence secretary and later became the oldest to serve in the role during his second stint under President Bush.
Ahead of the invasion of Iraq in March 2003, Mr Rumsfeld made the case for the invasion. He argued that Iraqi weapons of mass destruction posed a danger to the world, but no such weapons were ever found.
One of his most memorable moments came in 2002 when he was asked about the lack of evidence for Iraqi weapons of mass destruction. He gave a much-mocked answer about "known knowns" and "known unknowns".
Mr Rumsfeld, however, largely remained defiant over his record after leaving office. In a 2011 memoir, he defended his decisions around the Iraq war but did express regret over some of his comments.