"No, Alex, I think it's 'baker-man.' Like Pattycake is the baker's nickname or that's just what he does, he pats cakes."
"So there's not a 'baker's man'?"
"No."
I've been singing it wrong. He sounds like he knows, but I am still not sure.
"So if it's a name, is it Patty Cake or Paddy Cake, like he's Irish? And why are we in such a hurry for the cake?"
"Oh Christ, Alex. I don't know."
*********UPDATE**********
I found this from 1901, perhaps confirming that the origins of "pat-a-cake" are racist: