Certainly all systems have undocumented stuff that are left undocumented because the users are not supposed to use it. If they would, they couldn't change it.
If you really want to know about these things you should get a copy of [Windows Internals](https://www.amazon.com/Windows-Internals-Part-architecture-management/dp/0735684189/ref=dp_ob_title_bk) - doesn't have to be the latest edition. This is for Windows, obviously - I assume there are similar references for Linux (though I don't have any specifics).
https://www.amazon.co.uk/Windows-Internals-Part-architecture...
Certainly all systems have undocumented stuff that are left undocumented because the users are not supposed to use it. If they would, they couldn't change it.
- MacOS (pre-OS X)
"Revolution in the Valley"
https://www.amazon.com/Revolution-Valley-Insanely-Great-Stor...
https://www.amazon.com/Mac-OS-Internals-Approach-paperback/d...
https://www.amazon.com/Symbian-OS-Internals-Real-time-Progra...
"Symbian OS Platform Security: Software Development Using the Symbian OS Security Architecture"
https://www.amazon.com/Symbian-Platform-Security-Development...
"The Symbian OS Architecture Sourcebook: Design and Evolution of a Mobile Phone OS"
https://www.amazon.com/Symbian-OS-Architecture-Sourcebook-Ev...
- Mesa and Mesa/Cedar
https://www.amazon.com/Be-Developers-Guide-Development-Team/...
"Be Advanced Topics"
https://www.amazon.com/Be-Advanced-Topics-Development-Team/d...
- Windows
https://www.amazon.de/Windows-Internals-Part-Pavel-Yosifovic...
https://www.amazon.de/Windows-Internals-Part-Mark-Russinovic...
https://channel9.msdn.com/Shows/Going+Deep/Rob-Short-and-ker...
https://channel9.msdn.com/Shows/Going+Deep/Doron-Holan-Kerne...
https://channel9.msdn.com/Events/Build/2012/3-005
Not everything by a long shot, plenty more to re-discover like VMS,IBM i and Z, Unisys ClearPath, mbed,...
Just keep an open mind and don't idolatrize UNIX, yes it has a couple of good ideas, but they don't make it the be all end all of OS design.
Raymond Chen of "Old New Thing" fame and all his friends[1]
The creator of process monitor occasionally writes nice blog posts about troubleshooting[2]
A lot of historical NT stuff is still valid, or at least mildly relevant today.
[0]https://www.amazon.com/Windows-Internals-Part-architecture-m...
[1]https://blogs.msdn.microsoft.com/oldnewthing/
[2]https://blogs.technet.microsoft.com/markrussinovich/2005/10/...