The vote was held past midnight after opposition parties brought a motion against him, which was upheld by the Supreme Court.
Mr Khan had said he would not recognise an opposition government, claiming - without evidence - that there was a US-led conspiracy to remove him.
The assembly will now appoint a new prime minister.
Pakistan's parliament will meet to vote for the country's new leader on Monday.
That person will be able to hold power until October 2023, when the next election is due to be held.
Ayaz Sadiq, who is in charge of the national assembly while there are no ruling party members or designated speakers, said nomination papers for candidates should be filed by 11:00 local time (06:00 GMT) on Sunday.
Mr Khan becomes the first Pakistani prime minister to be ousted by a no-confidence vote.
The Supreme Court ruled on Thursday that Mr Khan, 69, had acted unconstitutionally when he blocked the no-confidence vote and dissolved parliament.
This made several opposition members furious, with some accusing the prime minister of treason.
Minutes before the vote started, the speaker of Pakistan's lower house of parliament - an ally of Mr Khan - announced his resignation. Members of Mr Khan's party (PTI) left the building, insisting he was the victim of an international conspiracy.
Opposition parties were able to secure 174 votes in the 342-member house in support of the no-confidence motion, the house speaker said, making it a majority vote.
In a tweet, opposition leader Shehbaz Sharif said Pakistan and its parliament were "finally freed from a serious crisis".
Mr Sharif added: "Congratulations to the Pakistani nation on a new dawn."
Meanwhile, PTI senator Faisal Javed Khan said the 69-year-old walked out of his prime ministerial residence "gracefully and he didn't bow down".
The senator went on to say that Mr Khan had "lifted the entire nation".
The former captain of Pakistan's national cricket team was elected prime minister in 2018, and promised to fight corruption and fix the economy.
But those pledges have gone unmet with the country gripped by a financial crisis.
In late March a series of defections deprived him of his majority and left him fighting for his political career.
The BBC's Secunder Kermani says Mr Khan is widely regarded as having come to power with the help of Pakistan's army, but now observers say they have fallen out.
Mr Khan has repeatedly said that Pakistan's opposition parties are working with foreign powers. He also claims that he is the target of a US-led conspiracy to remove him because of his refusal to stand with Washington on issues against Russia and China.
The US has said there was "no truth" in these allegations, and Mr Khan has never provided any evidence.
He visited Moscow to meet President Vladimir Putin as Russia was launching the invasion of Ukraine and has previously criticised what the Bush administration called the war on terror.
Saturday's vote came after opposition lawmakers put forward a no-confidence motion to parliament last Sunday, in a bid to oust Mr Khan from power.
But parliament's deputy speaker Qasim Suri - a member of Mr Khan's political party - swiftly blocked the vote, saying it showed "foreign interference". Mr Suri also said that it went against the constitution, which calls for loyalty to the state.
Mr Khan's government went on to dissolve parliament and called for a snap election to be held. This angered several opposition members, with some accusing the prime minister of treason for blocking the vote.
Opposition figures submitted a petition to the Supreme Court to assess the situation.
On Thursday, Pakistan's top court ruled that Mr Khan's decision to stop the vote from going ahead was unconstitutional. It ordered that the no-confidence vote should go ahead again.
However an impasse over the vote continued well into Saturday evening, prompting the speaker of the lower house of parliament - Asad Qaiser, an ally of Mr Khan - to resign.