My two biggest shelf pulls are Real-Time Rendering [0] by Akeine-Moller, Haines and Hoffman. 3D Math Primer for Graphics and Games [1] by Dunn & Parberry
Part of why it's difficult is that there are a tons of concepts belonging to 'real-time rendering' which three.js is built on top of. These are things like how a 3D scene is projected from a particular viewpoint using a 'camera', how lighting works, the split in vector/raster techniques for representing 3D shapes vs their surfaces, the role of shaders in modern GPU architectures etc. I'd recommend the book 'Real-time Rendering': https://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akeni...
Don't think there is any such book specifically for UE and its source. But there's a lot of good books on realtime rendering and graphics programming in general.
GPU Gems, Shader X and GPU Pro are good series for learning specific graphics programming techniques.
[1a, 1b, 1c] Computer Graphics, Principles and Practice Series [2] Physically Based Rendering [3] Real Time Rendering
---
[1a] https://www.amazon.com/Computer-Graphics-Principles-James-Fo... [1b] https://www.amazon.com/Computer-Graphics-Principles-Practice... [1c] https://www.amazon.com/Computer-Graphics-Principles-Practice... [2] https://www.amazon.com/Physically-Based-Rendering-Theory-Imp... [3] https://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akeni...
[0] https://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akeni... [1] https://www.amazon.com/Math-Primer-Graphics-Game-Development...
I also read a ton of presentations and papers. Highly recommend the famous PBR SIGGRAPH course notes [0], especially the intro to light & physics by Naty Hoffman. GPU-Driven Rendering Pipelines [1] is another recent goodie.
[0] http://blog.selfshadow.com/publications/s2013-shading-course... [1] http://advances.realtimerendering.com/s2015/aaltonenhaar_sig...
Edit: I'd also add: I wouldn't bother learning much OpenGL/WebGL to begin with (except shader programming in GLSL, since there's no good alternative abstraction for that). If you end up liking working with 3D graphics, go back and learn some about it since it'll help you understand performance concerns better—but meanwhile, knowing it is just an optimization you don't need yet. It's true three.js is built on top of it, but the significant principles you need to use three.js effectively fall under real-time rendering, not OpenGL.
GPU Gems, Shader X and GPU Pro are good series for learning specific graphics programming techniques.
https://www.amazon.co.uk/Game-Programming-Patterns-Robert-Ny...
Realtime rendering overview: https://www.amazon.co.uk/Real-Time-Rendering-Third-Tomas-Ake...
Related math: https://www.amazon.co.uk/Math-Primer-Graphics-Game-Developme...
Other recommendations:
http://mrelusive.com/books/books.html
http://fabiensanglard.net/Computer_Graphics_Principles_and_P...
It's fun to explore the source though, and NVIDIA has some cool experimental branches of the engine with their stuff integrated. https://github.com/NvPhysX/UnrealEngine