The most effective way to scale, rotate, shear and reflect is to use the power of your graphics card - for example through OpenGL.
If you still want to do bitmap pixel operations yourself, typically you do this using linear algebra. This is not super easy to figure out, so I recommend finding some good study material, for example this book.
The scope of this is way too large to get a good answer here: I'd recommend reading a good reference on the topic. I've always liked the Foley and VanDam...
The book "Computer Graphics: Principles and Practice" (known in the Computer Graphics circles as the "Foley-VanDam") is the basic for most computer graphics courses, and it covers the topic of implementing a ray-tracer in much detail. It is quite dated, but it's still the best, afaik, and the basic principles remain the same.
I also second the recommendation for Eric Lengyel's Mathematics for 3D Game Programming and Computer Graphics. It's not as thorough, but it's a wonderful review of the math basics you need for 3D programming, it has very useful summaries at the end of each chapter, and it's written in an approachable, not too scary way.
In addition, you'll probably want some OpenGL or DirectX basics. It's easier to start working with a 3D API, then learn the underlying maths than the opposite (in my opinion), but both options are possible. Just look for OpenGL on SO and you should find a couple of good references as well.
The classic textbook here is Foley and vam Dam. Hearn and Baker is quite good, as is the Watt book.