"I think there will be quite a few people who will now be pretty firmly of the view that she should resign", Vaizey told BBC Oxford in an interview on Thursday, when asked whether May should step aside.
A spokesman for the Prime Minister confirmed there will be a "thorough investigation into what's happened [with regards to security], and that despite the clear issues with her delivery, the Prime Minister is in good health and was pleased with the way the speech went".
Mr Shapps, who has claimed to have the backing of around 30 MPs - with some Cabinet members also privately offering support - said the demands for an election were growing.
She told Sky News: "It is a fact that since the moment of the election several Remain MPs like Nicky Morgan and Anna Soubry have called for her to go because they want a pro-Remain leader". Do some support it?
It is surprising that the PM's tenure in Downing Street has lasted so long after her disastrous decision to call a snap election after Easter.More news: Black Stars hit with injury woes ahead of Uganda clash
The latest round of political uncertainty surrounding May pushed down the value of the pound against the euro and the dollar on currency markets. "MPs' confidence in Theresa May is at a low point", said Jasper Lawler, head of research at London Capital Group, an online trading platform.
A prankster handed Theresa May a P45, ostensibly from Boris Johnson, during her speech at the Conservative Party conference in Manchester.
The Conservatives have 316 MPs in parliament.
The Tunbridge Wells MP said the party now needed to concentrate on being "effective and cool-headed" in government and that there would be no patience with any "distractions". "Grant has many talents but one thing he doesn't have is a following in the party, so really I think this is going to fizzle out to be perfectly honest".
"There's nothing wrong with thinking we can do better than this and I'm afraid to say that does involve, though Theresa May is a very fundamentally decent person, having a new leader". Grant has many talents but one thing he doesn't have is following in the party.
Other MPs expressed concern that any plot to remove the Prime Minister would seriously damage Brexit talks with the EU.