There really isn't anything TornadoFX specific to this process, but I have two recommendations:
Don't create a custom control unless you really need to. To add custom functionality to a Button or a TextField, install a custom skin instead.
If you really need a custom control, consider extending the Control class. For more layout centric components you might extend a layout instead, or simply Pane.
To learn more about custom components, consider reading this brilliant book:
Also check out these videos: