After a nine-hour session, the chamber voted 45-55 against appointing Nikol Pashinyan, who was the only candidate nominated to replace the ex-president who resigned as prime minister due to the protests.
Tens of thousands of opposition supporters joined a protest rally in Yerevan on Sunday evening, hoping that a massive show of strength would thrust their leader to power.
But he was still six votes short of the 53 he needs from the 105-seat legislature, where the Republican Party has a majority. Sargsyan was appointed PM by the parliament on April 17 but was forced to resign under public pressure less than a week later.
Republican deputies accused Pashinyan of exploiting the young people who have been his core supporters.
Pashinyan has pledged to keep Armenia close to Moscow, saying the changes he wants to make would instead focus on rooting out graft.More news: Encounter breaks out in Jammu and Kashmir's Pulwama
Parliament said in a statement a vote will be held on May 1 to elect a new leader, potentially spelling a way out of the post-Soviet nation's biggest political crisis in years.
However, as a debate unfolded in parliament to discuss Pashinyan's candidacy for the vacant prime minister's post, it became clear the ruling elite was not willing to surrender power.
Speaking to AFP in an interview Monday, Pashinyan said Armenians themselves wanted to determine the future of their country and an explosion of tensions was just a matter of time.
The opposition in this Caucasus Mountains country saw Sargsyan's move into the prime minister's chair as an attempt by him to stay in power indefinitely. "A total strike is announced", Pashinyan said in comments carried by the Interfax news agency.
"I was hoping to hear a conceptual speech of a candidate for the prime minister, but instead I was witness to a non-program speech which sounds pleasant to people's ears", senior Republican party member Eduard Sharmazanov told the chamber. The Republican Party of Armenia earlier had announced that they would not hamper the "election of the people's candidate", and also said that they will not put forward their own candidate.