It's Problem 7.29 in the third edition of Michael Sipser's Introduction to the Theory of Computation (you can find it by searching for "coloring" when using the preview function on Amazon). It's also in The design and analysis of computer algorithms by Aho, Hopcroft and Ullman as Theorem 10.12, though the proof there seems a little different from what you've sketched. (I hope that the Google Books link works. Sometimes it won't show the right preview.)
Sipser's Introduction to the Theory of Computation is the standard textbook. The book is fairly small and quite well written, though it can be pretty dense at times. (Sipser is Dean of Science at MIT.)
You may need an introduction to discrete math before you get started. In my udergrad, I used Rosen's Discrete Mathematics and Its Applications. That book is very comprehensive, but that also means it's quite big.
Rosen is a great reference, while Sipser is more focused.
Google hasn't been helpful because no such algorithm exists. Check out Rice's Theorem for the impossibility.
Let S be the set of languages that you can reduce SAT to in polynomial time.
SAT is clearly in S, and we know some machine recognizes it.
The empty language is not in S (even if P=NP, so that SAT is P-complete), and we know some machine recognizes it.
By Rice's Theorem, no machine decides, when given a machine as input, whether that machine recognizes a language in S.
(we assume that the "any custom problem" input is as a machine encoding)
I see that you make a lot of questions about computational complexity, but do not have a good foundation. Many things you propose or ask already have known impossibility results. May I suggest you have a look at Sipser (https://www.amazon.com/Introduction-Theory-Computation-Michael-Sipser/dp/113318779X)? That will give you a better understanding of computability and complexity to understand the feedback you're getting.
this is the canonical undergrad book
it's colloquial and has a lot of diagrams and "intuition". people really like it but i actually think hopcruft ullman is better because it's more structured
(Edit: added the title and author name in my post)