The former COAS has been given permission to head the military alliance after General Headquarters approval, according to Defence Minister Khawaja Asif.
On January 11, Khwaja Asif had said that former army chief General Sharif didn't seek "no objection certificate" (NOC) for any foreign vocation.
Three government officials said a flight took Raheel Sharif to Saudi Arabia on Friday.
Opposition parties have been criticising the government over the controversial appointment, claiming that the alliance is formed against Shia-majority Iran and will harm Pakistan's relations with its neighbour.
The Prime Minister said the two countries have maintained cordial and brotherly relations marked by commonalities on all issues of mutual interest at regional and global level.More news: US Treasury Department Stops Short of Calling China a Currency Manipulator
"Iran and other countries opposed of the alliance will also benefit from this".
Since the news of the alliance first surfaced, there have been concerns about its nature and how it may affect a pre-existing parliamentary resolution on Yemen passed unanimously by lawmakers calling for "neutrality in the conflict" in 2015.
Earlier this week, Iran's ambassador to Pakistan Mehdi Honardoost expressed reservations over Sharif's role as head of the Saudi-led military alliance.
He said that special attention is given to Pakistani community residing in Saudi Arabia with a particular focus on ease of access to the community to consular services.