How to collect your training depends on your domain. What kind of natural language text do you want to process? There is [this book](https://www.amazon.com/Natural-Language-Annotation-Machine-Learning/dp/1449306667) which helps you to understand how the collection and annotation process should work. If you need to create custom NER models, [this project](https://git.noc.fh-aachen.de/marc.schreiber/Towards-Effective-NLP-Application-Development) can help you to create your models. Currently, we are working on annotating NEs and it should be done in the next couple of weeks.