Simon Cowell is not happy with the way he began his relationship with Sarah Silverman -- by having an affair with her while she was married to his best friend Andrew Silverman. "I regret that part," Cowell told British newspaper Mirror about the affair. "But then of course you have a baby and you look at the baby and you kind of go, 'This is what happened from it. ... It is not something I am proud of or wanted to happen in terms of hurting anyone. It just happened." The 54-year-old music mogul went on to say that while the situation wasn't ideal, sometimes, "you have to deal with it and man up to it." "You have to accept the responsibility and the criticism," Cowell said. "My life has changed, but I am probably more focused on what I am doing now than I was a year ago. Maybe because it puts things in perspective having two people in my life who I am responsible for. ... Now the house is full of people, it has become a home rather than a house." Silverman divorced her husband in August 2013 a month after he filed the papers in NYC citing adultery. Silverman and Cowell welcomed their first son, Eric, on February 14, 2014.