Hogan Gidley, White House deputy press secretary, said the president would host a "working lunch in the private dining room off the Oval Office" with West and Brown.
West's first visit to Mr Trump was at his eponymous Manhattan skyscraper in 2016, a month after he won the election, but this was the rapper's first trip to the White House. Instead, the company has suggested in the past that users select passcodes with unique significance only they may know.
During one pause, Trump seemed to acknowledge the oddness of the moment, saying, "That was quite something".
West, who has broken ranks with much of the left-leaning entertainment industry to support Trump, recently took fire for wearing one of the red baseball caps handed out at the president's raucous "Make America Great Again" rallies.
West's wife, the reality TV star Kim Kardashian West, sat down with the president earlier this year to discuss criminal justice reform.
West has made headlines today about his behavior during the sit-down. which included a couple classic Kanye rants, a heap of Trump praising and even a hug between the two men.
On Thursday, West attributed his support for the president to not having "a lot of male energy" in his home growing up. West said they need to create factories.More news: Banksy shocks art world by shredding million-dollar work at auction
In an impassioned but sometimes hard to follow 10-minute speech, West talked about the "male energy" of Trump's 2016 campaign, the USA relationship with North Korea and his belief that he was misdiagnosed with bipolar disorder.
"Illegal guns is the problem, not legal guns", West said.
After he finished, The president concluded, "He can speak for me any time he wants. he's a smart cookie".
The club shared an image of the typically straight faced Kanye West, with the caption: 'The global break has got us like'.
"I tell you what, that was pretty impressive", Trump said of West's speech. "He might not have thought he'd have a insane mother-- like (me)".
He said: "What I think is we don't need sentences, we need pardons".
"It was from the soul, I just channeled it", West said.
Ike Barinholtz, who directed the upcoming Trump-inspired political satire "The Oath", said the meeting was inappropriate given the hurricane that is wrecking havoc on the east coast.
"If he don't look good, we don't look good", said West.