But the late-night host insisted that what's got people really riled is the misconception over which Palin daughter the joke was about.
On Monday's edition of "Late Show," Letterman explained that the risque joke thought by some to have targeted Palin's underage daughter, Willow, was actually referring to 18-year-old daughter Bristol. The name of the daughter wasn't mentioned in the joke, which was part of Letterman's monologue on last Monday's show.
It was "a coarse joke," ''a bad joke," Letterman told viewers. "But I never thought it was (about) anybody other than the older daughter, and before the show, I checked to make sure, in fact, that she is of legal age, 18."
"The joke, really, in and of itself, can't be defended," he declared.
Even so, the ongoing outcry, led by Palin and her husband, Todd, has centered on Letterman intending to make a joke about the Palins' 14-year-old daughter having sex with a Yankees baseball player.
Todd Palin issued a statement last week that said "any 'jokes' about raping my 14-year-old are despicable."
And Sarah Palin charged Letterman with "sexually perverted comments made by a 62-year-old male celebrity."
On Monday's show, Letterman said, "I'm wondering, 'Well, what can I do to help people understand that I would never make a joke like this?' I've never made jokes like this, as long as we've been on the air, 30 long years."
If that's the misconception — that the joke was aimed at a 14-year-old — he understands why people are upset, he said. "I would be upset myself." Letterman has said from the beginning that he thought the Palin daughter who attended the Yankee game was Bristol.
"I feel that I need to do the right thing here and apologize for having told that joke," he said. "It's not your fault that it was misunderstood, it's my fault that it was misunderstood."
He apologized to both daughters, "and also to the Governor and her family and everybody else who was outraged by the joke," he said as the studio audience applauded. "I'm sorry about it, and I'll try to do better in the future."
Related Content from AccessHollywood.com:
VIEW THE PHOTOS: Photo Fun With Hollywood's Hottest Comedians!
PLAY IT NOW: Sarah Palin & David Letterman Throw Down Over 'Late Night' Joke
PLAY IT NOW: Letterman Makes Late Night Comeback