Australian Prime Minister Tony Abbott Thursday expressed regret at winking during a testy radio chat with a pensioner who works on a sex phone line, as he battled a fierce backlash. A video feed from the studio showed the conservative leader winking and grinning to the programme's host when the female caller revealed what she did to make ends meet, as she complained about swingeing budget cuts imposed by Abbott's government. Abbott, who was the target of a fiery misogyny speech by former leader Julia Gillard that went viral in 2012, said the wink was to signal he was happy to proceed with the call. But as a political and social media furore grew over the incident, he admitted it was wrong. "I shouldn't have done it... I should have been more focused on the caller and less focused on the interviewer," he told the Nine Network. Asked if he was being disparaging towards the woman, he said no. "Mistakes are always regrettable ... and I will do my best having made a mistake yesterday to make none today," he said. Labor opposition leader Bill Shorten said Abbott had lost touch with the electorate. "I'm sure the prime minister's minders would've had a collective forehead-slapping moment, and said, 'oh my goodness, Tony's at it again'," Shorten said. "I don't know what was going through his mind then. It just shows how out of touch the prime minister is. " The Greens called on Abbott to step down from his dual role as the government's minister for women. "We need a minister for women who is actually a woman and who is prepared to stand up for women -- not simply dismiss their problems with a wink and a smirk," Greens Senator Larissa Waters said. The Abbott government's first budget, handed down last week, axed health and education spending while tightening welfare benefits to help bring the deficit under control. The caller, Gloria, told him she was a struggling 67-year-old pensioner with three chronic, incurable medical conditions, two life-threatening. "I just survive on around Aus$400 ($369) a fortnight after I pay my rent and I work on an adult sex line to make ends meet. That's the only way I can do it," she said. She called back to ABC radio on Thursday to say she was glad the wink had got so much coverage. When the presenter said the incident had been curious, she replied:  "Curious is not the word ... sleazy, slimy." She later labelled Abbott "a stupid man". Foreign Minister Julie Bishop, the only woman in Abbott's cabinet, came to his defence, denying his wink was sexist. "I think people are far too quick to judge others through this prism of everything being about misogyny or sexism," she said, adding that he always treated her with respect. It has been a bad week for Abbott with opinion polls showing his popularity has plunged since the budget was released.