On many platforms (such as any modern x86-compatible), many of the maths functions are implemented directly in the floating-point hardware (see for example http://en.wikipedia.org/wiki/X86_instruction_listings#x87_floating-point_instructions). Not all of them are used, though (as I learnt through comments to other answers here). But, for instance, the sqrt library function is often implemented directly in terms of the SSE hardware instruction.
For some ideas on how the underlying algorithms work, you might try reading Numerical Recipes, which is available as PDFs somewhere.
I think it's a very bad idea to calculate two very large numbers and hope that the quotient comes out as something sensible.
Start by taking the natural log:
ln(n!/(q!)^r) = ln(n!) - r*ln(q!)
You can use gammaln() for the two function values, simplify, then take exp() to get the result you want:
value = exp(gammln(n+1) -r*gammln(q+1))
Numerical Recipes has a nice chapter on how to implement functions like gammln().
Method of Least Squares
This book Numerical Recipes 3rd Edition: The Art of Scientific Computing (Hardcover)
has all you need for algorithms to implement Least Squares and other techniques.