Blender's comment is important: different types of processing are required depending on the type of "cartoon" image that you want. It may not be feasible to automatically determine what type of image processing is required.
You might save yourself a lot of hassle by implementing 3 or 4 different image processing techniques, run all the techniques on each image, and then allow a user to choose the processed image that looks best.
Some processing techniques include the following:
If your image looks most like the Christopher Walken sample, then some form of binarization should work. To tinker with this, use the "Posterize" filter in Photoshop or GIMP. Binarization is an extreme case of posterization, with all colors (or in "grayscale" images, all brightnesses) squeezed down to just two: pure black and pure white. Choosing the right threshold automatically can be tricky, so it could help to offer the user a means to adjust the threshold.
You'll probably find that a simple binarization won't work as well as you'd like, but I'd suggest getting that code working first, identify its weaknesses, and then tinker further.
To learn more about thresholding, binarization, region labeling, etc., check out one of the most widely used textbooks on image processing:
Digital Image Processing by Gonzalez and Woods (3rd edition)
(The retail price is high. Check addall.com to find a cheaper international edition, which could cost as little as $35.)
As long as you can access individual pixels, write individual pixels, work with matrices, etc., you can implement image processing. In GD you'll want to look into the built-in function to perform a convolution:
To use Sobel to find vertical lines, the 3x3 kernel would look like this:
|-1 0 1|
|-2 0 2|
|-1 0 1|
Rotate that kernel 90 degrees and you have a kernel to find horizontal lines. You can read more about the Sobel kernel at Wikipedia:
As you say, "bwconnhull" and "bwareaopen" aren't directly supported, but there are a number of foundational morphological functions that are available. Unfortunately, it's a bit of work to recreate those two routines using the smaller subset of functions (such as bwperim, bwselect, bwtraceboundary, and bwmorph). I believe it is actually possible, but the implementation will depend a bit on how exactly the routines are being used in your code.
A good guide to morphological operations and their relationship to each other is given in Gonzales and Woods (http://www.amazon.com/Digital-Image-Processing-3rd-Edition/dp/013168728X). In my old copy, it's in Chapter 8 under Morphology, but I think it's in Chapter 9 in newer editions.
Sadly, I know of no drop-in replacement, and you'll end up writing new ones and testing them in your application. On the plus side, morphological operations are very well explained and defined, and they relate to each other in an elegant way, so you should have all the tools you need in those other functions.
Seems you lack basic knowledge of Digital Image Processing, I recommand to you this book.
Digital Image Processing (3rd Edition) Rafael C.Gonzalez / Richard E.Woods http://www.amazon.com/dp/013168728X
For basic operation using OpenCV(which I am familiar with), here is an example:
int main(int argc, char *argv)
IplImage* img = 0;
printf("Usage: main <image-file-name>/n/7");
// Load image
printf("Could not load image file: %s\n",argv);
// acquire image info
height = img->height;
width = img->width;
step = img->widthStep;
channels = img->nChannels;
data = (uchar *)img->imageData;
printf("Processing a %dx%d image with %d channels/n",height,width,channels);
// create display window
cvMoveWindow("mainWin", 100, 100);
// reverse image
// display reversed image
cvShowImage("mainWin", img );
printf("height=%d width=%d step=%d channels=%d",height,width,step,channels);