“It was crazy,” the Tony Award winner — nominated again this year for his role in ‘Back to the Future: The Musical’ — tells PEOPLE of his time on the hit ABC primetime soap

Roger Bart will always have love for Desperate Housewives — even if he once thought he might never get another role after it.

In an exclusive conversation with PEOPLE at the 2024 Tony Awards Meet the Nominees junket earlier this month, the actor, 61, opened up about starring on the popular ABC series and how he thought his career would be over after he played George Williams, a pharmacist who became a love interest of Marcia Cross’ Bree Van de Kamp after poisoning her husband to death.

According to Bart, his “favorite and most terrifying” moment on Desperate Housewives was when they filmed his character’s death scene. “I asked the director, ‘You think I’m ever going to work again after this?’ Because I was realizing, just by walking through airports, this is impactful,” he said.

Noting that the director referred to his role as his “Gilligan” — a nod to Bob Denver’s role on Gilligan’s Island — Bart, who is nominated for his featured actor performance as Doc Brown in Back to the Future: The Musical, recalled the reaction he got even after his character’s demise.

“I was walking through an airport and the amount of attention I got for it, because back then everyone tuned in at 9:00 on a Sunday, it was like 23 million people, it was crazy,” he explained. “… It was really something.”

Bart also recalled going on The View at the time, stating, “I remember the ladies going, ‘So you are dead?’ And I said, ‘I am.’ That was literally the only reason I was on The View. It was so funny.”

‘Desperate Housewives’ Cast: Where Are They Now?

Reflecting on when he first got the role of Williams, Bart said, “I had kind of no idea what it was going to be.”

“Matter of fact, I even shot my first scenes just thinking I was sort of like this druggist, and then, Mark Cherry, who created the show, came to our little trailer area and goes, ‘Do you want to know what we have planned for the character?’ ” he continued.

“I said, ‘Well, yeah. Unless it hurts me,’ and he said, ‘Well, you’re going to be a sociopath stalker.’ So I thought, ‘Oh, this is going to be really fun,” Bart added.

The Tony Award winner said his scandalous role “did not disappoint,” noting, “It put me in such incredible situations.”

“With that show, I can go anywhere in the world [and be recognized]. I’ve gone to some crazy places, but that show will be the one where someone walking across the bridge will say, ‘Are you?’ ” Bart explained. “It is great to have been a part of that.”

Eva Longoria Says She’d Be the ‘Only’ Star of ‘Desperate Housewives’ to Do a Reboot: ‘I Miss Gabby’

Looking ahead to the idea of potentially reviving Desperate Housewives in the future, Bart said, “I feel like the ladies might be up for it.”

“And God forbid that they ever say The Men of Desperate Housewives: The Series, because that would be a travesty,” he added. “I don’t really know. It’s kind of a funny idea. I don’t know.”

The 2024 Tony Awards will take place on Sunday, June 16, at Lincoln Center’s David H. Koch Theater in New York City. Viewers can watch the show on CBS and Paramount+ beginning at 8 p.m. EST. Select awards will also be handed out on a preshow that will stream on Pluto TV.