Is abstract algebra used in machine learning?

Yes. One can consider group transformations on images to get a representation of the image that is invariant to those transformations. One can learn more about i-theory (invariant theory) in the course website for 9.520 statistical learning theory and applications at MIT: 9. read more

I also learned in a theoretical computer science course some algebraic geometry applied to machine learning. read more

In my experience (YMMV) linear algebra is very much used as a tool in Machine Learning and Computer Vision, with little interest in the underlying mathematics. That said, you'll need a more mathematical understanding than, say, a games programmer or a basic researcher in the natural sciences. read more

Secondly, abstract algebra and machine learning are just some small subjects. You can get very good at them, almost master them if you try really hard. But they are always being improved by other masters, new things are being invented each day and you should never stop learning if you truly want to call yourself a master. read more