Learn the mechanics of compiler construction first - that'll inform everything about your language, and is a prerequisite to actually building something other people can use. I'd recommend The Dragon Book [1], SICP [2], and Modern Compiler Implementation in ML [3] for this.
After that, it's a question of where you want to go with your language, and what distinguishing features you'd like it to have relative to other languages. Languages which just change some syntax and keywords but otherwise look like C are common, but they tend not to get usage beyond their creators. Much of the academic research lately is about type systems (TAPL has already been mentioned) - there's research into dependent types, effect typing, total functional programming, linear types already made it into Rust's borrow checker, etc.
However, many of the worst problems in industry - dependencies, versioning, smoothly migrating from a prototype to robust production code, code navigation, security, parallelization & distribution, the memory hierarchy - have gotten relatively little attention in academic research, and are desperately awaiting solutions. If you have some industry experience, it may be worth trying to imagine how you might fix these problems if you were allowed to change the programming language at will.
After that, it's a question of where you want to go with your language, and what distinguishing features you'd like it to have relative to other languages. Languages which just change some syntax and keywords but otherwise look like C are common, but they tend not to get usage beyond their creators. Much of the academic research lately is about type systems (TAPL has already been mentioned) - there's research into dependent types, effect typing, total functional programming, linear types already made it into Rust's borrow checker, etc.
However, many of the worst problems in industry - dependencies, versioning, smoothly migrating from a prototype to robust production code, code navigation, security, parallelization & distribution, the memory hierarchy - have gotten relatively little attention in academic research, and are desperately awaiting solutions. If you have some industry experience, it may be worth trying to imagine how you might fix these problems if you were allowed to change the programming language at will.
[1] https://www.amazon.com/Modern-Compiler-Implementation-Andrew...
Big List of Resources:
Legend:
Modern Compiler Implementation in ML (I'm not a fan of the C/Java versions)
https://www.amazon.com/Modern-Compiler-Implementation-Andrew...
If you are fluent in a mainstream OO language - Java, C#, Ruby etc
Programming Language Processors in Java: Compilers and Interpreters
( the code is in Java but can be trivially ported into any OO language )
https://www.amazon.com/Programming-Language-Processors-Java-...
Most compiler programming books use lex/yacc versions for lexing and parsing. Imo, this isn't a good way to learn lexing/parsing, and using recursive descent or combinator parsing approaches is (imho) the right way to begin.
If you want to know how tools like lex and yacc are built, then Holub's "Compiler Construction in C" is very comprehensive and goes into great detail about the required CS theory- (automata DFA, NFA etc).
The book seems to be out of print, but used copies are worth buying (imho)
https://www.amazon.com/Compiler-Design-C-Prentice-Hall-softw...