A counter-melody is a 'less important' melody played against the primary melody. Historically there was a period when playing multiple melodies against each other was the absolute essence of music, and folks like Bach dominated (called 'counterpoint' music). A very famous example, that is 'simple' so easy to follow is Bach's Invention #1: https://www.youtube.com/watch?v=xzU7xQmmXGE
His 'inventions' were for two melodies. He also wrote 'sinfonias' which were for three voices, and are a bit more complex, for example his Sinfonia No.2: https://www.youtube.com/watch?v=UoZwnXDjbV0
Listening to this with headphones while trying to consciously follow all melodies is quite a peculiar experience. :)
And if you want to feel depressed and talentless, check from 10:54: https://www.youtube.com/watch?v=9XCUcZ5KK7Q. Here he starts with the primary melody and then has the second melody join. This is exhausting to *really* listen to and my peanut brain is too small to ever hope to play something like this. Shredding? Sure, I can start slow, practice a ton and end up fast. But having both hands play *this* independently? Awe-inspiring. Sitting down and composing this on a piece of paper almost 400 years ago? Madness. :D
In baroque counterpoint the goal is to make every melody interesting in and of themselves, and make it sound 'effortless harmoniously together'. It should sound like these melodies were born together. In practice this aint easy at all and you'll be tempted to see one as the 'primary melody' and the others as 'subservient' where you can take shortcuts to make them fit the main melody. At that point it is no longer true counterpoint but you can still call it a counter-melody. Taken further you'll have things like arpeggios; parts that obviously fit the primary melody but are themselves so bland they are clearly accompaniment instead of a melody in their own right.
If you're interested, a very well-regarded (though rather pedantic) book that starts at the basic and offers exercises is the many century old https://www.amazon.com/Study-Counterpoint-Johann-Joseph-Parnassum/dp/0393002772. If you want to go *really* old-school you can go as close to the original here: http://hz.imslp.info/files/imglnks/usimg/3/31/IMSLP370587-PMLP187246-practicalrulesfo00fuxj.pdf
I think it'll help most songwriters/composers to know the basics, even if you dont care about classical counterpoint at all. :)
Without question, the best way to learn how to arrange is to put in the work transcribing some of your favorite arrangers and dissecting the way they approach things.
Study the chord progressions they use and analyze their voicing. Break down how they use counterpoint vs. countermelody. Pay attention to how they use every single voice, common articulations, and where in the range do they have each part "live" (1st vs. 2nd vs 3rd).
Write down what you observe about how they do things, try to put it into words. Compare/contrast between arrangers. This will help you better internalize what they are doing and help you to find your own style.
Before doing any of this, however, I suggest to read, read, read. Here are a few books to get your started:
The Study of Counterpoint - Johann Joseph Fux
Contemporary Counterpoint: Theory & Application - Beth Denisch
Guide to the Practical Study of Harmony - Peter Ilyitch Tchaikovsky
Principles of Orchestration - Nikolay Rimsky-Korsakov
Essential Dictionary of Orchestration - Dave Black & Tom Gerou
Treatise on Instrumentation - Hector Berlioz & Richard Strauss
Arranging for Horns - Jerry Gates
Another excellent resource is Bandestration - https://bandestration.com/
Another great read that is HIGHLY applicable to writing for marching music is:
Fundamentals of Musical Acoustics
If you are interested to explore interplay between wind/percussion arranging and electronics:
Acoustic and MIDI Orchestration for the Contemporary Composer - Andrea Pejrolo