The Audio Programming Book (MIT Press)

Though your question is quite broad I can give some insight into where to start.

First you will need a copy of the VST3 SDK from Steinberg

From my experience there is quite a large learning curve for getting started with the VST SDK. So I would suggest finding a library that provides a wrapper on top of it. For example: JUCE

Using a wrapper will help you get past a lot of the boiler plate programming that needs to be done to make a VST work.

As far as waveform generation, filtering and other DSP related concepts...

There is so much to the subject that I couldn't even begin to describe it.

Take a look at musicdsp and The STK for some basic concepts.

Get a book on the subject. A classic would be The Audio Programming Book

Additionally you need to make sure that you have a strong grasp of audio theory. Take A look at this.

And of course Google is your friend.

EDIT:

To answer your question more completely. Yes C++ (or C) would be the language of choice for this kind of application (though not the only possible choice)

I would look into using an Audio API before delving into VST development, this will allow you to work on your skills without the trouble of VST development.

Some Suggestions would be:

• PortAudio (written in C)
• RtAudio (written in C++, but not c++11 or higher)

Which are two of the big name Audio Libraries.

Or you could take a look at libzaudio Which is an Audio API that I am currently working on. (It currently depends on PortAudio, but provides a more modern C++11 style way of working with audio)

Assuming you have installed PortAudio & libzaudio the following will generate a sin 440hz wave for one second:

#include <iostream>
#include <cmath>
#include <libzaudio/zaudio.hpp>

int main(int argc, char** argv)
{
    try
    {
        //bring the needed zaudio components into scope
        using zaudio::no_error;
        using zaudio::sample;
        using zaudio::sample_format;
        using zaudio::stream_params;
        using zaudio::time_point;
        using zaudio::make_stream_context;
        using zaudio::make_stream_params;
        using zaudio::make_audio_stream;
        using zaudio::start_stream;
        using zaudio::stop_stream;
        using zaudio::thread_sleep;

        //create an alias for a 32 bit float sample
        using sample_type = sample<sample_format::f32>;

        //create a stream context with the default api (portaudio currently)
        auto&& context = make_stream_context<sample_type>();

        //create a stream params object
        auto&& params = make_stream_params<sample_type>(44100,512,0,2);

        //setup to generate a sine wave
        constexpr sample_type _2pi = M_PI * 2.0;
        float hz = 440.0;
        sample_type phs = 0;
        sample_type stp = hz / params.sample_rate() * _2pi;

        //create a zaudio::stream_callback compliant lambda that generates a sine wave
        auto&& callback = [&](const sample_type* input,
                              sample_type* output,
                              time_point stream_time,
                              stream_params<sample_type>& params) noexcept
        {
            for(std::size_t i = 0; i < params.frame_count(); ++i)
            {
                for(std::size_t j = 0; j < params.output_frame_width(); ++j)
                {
                    *(output++) = std::sin(phs);
                }
                phs += stp;
                if(phs > _2pi)
                {
                    phs -= _2pi;
                }
            }
            return no_error;
        };

        //create an audio stream using the params, context, and callback created above. Uses the default error callback
        auto&& stream = make_audio_stream<sample_type>(params,context,callback);

        //start the stream
        start_stream(stream);
        //run for 1 second
        thread_sleep(std::chrono::seconds(1));
        //stop the stream
        stop_stream(stream);
    }
    catch (std::exception& e)
    {
        std::cout<<e.what()<<std::endl;
    }
    return 0;
}

(Pulled from one of my example files)

I would be happy to explain more in detail if you would like to contact me elsewhere. (A long discussion on the topic would not be permitted on Stack Overflow)