Not just aristocracy, but anyone who owned them, which would include middle class people. Here's a famous example for you: Cato the Elder married a freedwoman, Salonia who was a slave at the time the relationship began. read more
A slave was not considered human: A man had the right to do anything he pleased with his slaves but a woman who had sexual intercourse by a slave was killed (the men made the rules) Roman men who did not have slaves on his own were supposed to visit the many brothels. It was not a social taboo, just something normal. read more
While I'm sure that some Roman women had sex with their slaves, the culture was also incredibly male-dominated. It was incredibly normal for a slave owner to sleep with his slaves - especially in the aristocracy, a young aristocrat's first sexual experiences were probably with his slaves. read more