Under the new measures, banks with liabilities of more than $100 billion will be taxed 0.06% on those "liabilities".
The measure has also been criticized by the individual banks affected.
The Treasurer has jettisoned decades of conservative fiscal orthodoxy around deficits and, particularly, debt.
It sees the unemployment rate at 5.75 percent in 2017/18, easing from a 13-month high of 5.9 percent now while it pegged the consumer price index (CPI) at 2 percent, climbing to 2.5 percent by 2020/21.
"This budget is about making the right choices to secure the better days that are ahead", Morrison told reporters when he arrived at Parliament House. "It does not pretend to do things with money we do not have".
Labor has yet to confirm whether it will support the hike included in the 2017 budget which will cover a $3.8 billion gap in funding for the National Disability Insurance Scheme.
Defence spending will hit two per cent of GDP by 2020/21. Other measures outlined in the budget include an additional A$321 million for the national police force for counter terrorism purposes; and a boost of A$428 million to be spent on early childhood education.
Among revenue boosting measures was a six-basis point levy on the liabilities of banks with liabilities of more than A$100 billion from July 1, a move that will help the government raise A$6.2 billion through 2020/21 to aid budget fix.
For a worker earning $50,000 annually, it'll be an extra $250 per year.More news: Tripadvisor Inc (TRIP) Receives "Hold" Rating from Stifel Nicolaus
In Opposition, the Liberal Party's shadow treasurer Joe Hockey in 2010 described then Treasurer Wayne Swan's mining tax as "a bad tax" that would be "a disincentive for further investment". The inland rail line, to receive A$8.4 billion in funding, will run from Melbourne to Brisbane, spanning 1,700km, and will help farmers to move produce and exports.
"Give me a break - that's what Australians will be saying", he said.
The levy will raise $1.2 billion over the next four years. Currently, the withholding tax is only taken when a foreigner sells a property worth $2m or more.
Foreign property owners will also face fines of at least A$5,000 if they fail to either occupy or lease the home for more than six months a year.
The Government has also heard the anger over housing prices.
"This measure is meant to encourage foreign owners of residential property to make their properties available for rent where they are not used as a residence and so increase the number of dwellings available for Australians to live in", Mr Morrison said.
Despite the Treasurer's promise to ease cost-of-living pressures, it is becoming apparent that some Aussies will be slugged with a "university tax", a "health tax" and a "banking tax", while high-income earners get tax relief.
Prime Minister Malcolm Turnbull dismissed claims that absorbing the cost would reduce profits to the extent it would have a negative effect on bank shares and, hence, superannuation.