Just a week after David Letterman announced his retirement from Late Show, CBS has found his replacement–Stephen Colbert. The quick-witted South Carolinian will bring his infectious brand of humor to the late night show, something that CBS lost over the years as Letterman aged and Craig Ferguson did whatever it is Craig Ferguson does.

While most know Colbert for his political humor and outrageous stunts, he also happens to be a music aficionado (I mean, the guy hosted COLBCHELLA!) with a serious musical pallet that opens himself up to some of the hardest-to-reach artists. If there is anyone to rival Fallon’s musical throne, it’s this guy. From Radiohead to Jack White to Arcade Fire, here are five of Colbert’s best musical guests on his show.

Jack White

Colbert and White have a long history. In fact, it wouldn’t surprise me if White appeared on his show during opening week. Here he is sitting down for a backstage interview and a performance of “Freedom at 21” in support of Blunderbuss.

Interview:

“Freedom at 21:”

Radiohead

One of Colbert’s biggest gets in his time on Comedy Central had to be Radiohead’s appearance during their King of Limbs promotional tour. While the band didn’t do much press, they certainly took to Colbert’s show (Yorke recently returned with Atoms for Peace) despite the interview garnering some reaction for Radiohead’s disinterest throughout. One thing to remember: With Colbert, nearly everything is planned. The band also performed twice, so you can check out the interview portion and “The National Anthem.”

Interview:

“The National Anthem”

Gregg Allman with The National

An accomplished musician in his own right (he has two Grammys!), Colbert joined The National and Gregg Allman for a Christmas performance of “Silver Bells” as the show came to a close. One of his biggest similarities to friend (and now ratings rival) Jimmy Fallon remains his love affair with music and his affinity for jumping on stage with big time artists to create a once-in-a-lifetime performance.

Arcade Fire

Billed as “The Reflektors,” the Montreal rockers made one of their first public appearances in support of their latest album on The Colbert Report to perform “Normal Person” and “Afterlife.” Win and Will Butler also sat down for a hilarious interview where the host not only takes a shot at Arcade Fire (“those guys are pretentions, huh?”) but also gets a reconciliatory hug from Win following a joke about Colbert being a terrible person.

Interview:

“Normal Person”

LCD Soundsystem

Along with Radiohead, this may be Colbert’s biggest get ever. LCD Soundsystem shut down operations in 2011 but not before they graced the late night TV stage one more time. It wasn’t Letterman or Leno or Fallon who got Murphy and company to perform, it was Colbeagle The Eagle himself. Watch Murphy and Colbert talk (where Colbert pleads him to reconsider retirement) and then check out the full band signing off with “I Can Change.” If only you could change your mind, James.

Interview:

“I Can Change”