There are so many variables that could be involved that I find this question too general to answer. Some things you might want to check out are:
But since security is a pretty complex topic and important to get right (it is only as strong as the weakest link) and WCF has so many different options, configuration properties etc... it would be smart to get some books and really get aquinted with the intracies.
I can personally recommend Programming WCF 2nd edition and Learning WCF: A hands-on guide
After reading your previous question I also suggest you give tracing a try. WCF logs quite a bit about whats going on internally and when certain limits are hit. See http://msdn.microsoft.com/en-us/library/ms733025.aspx for more information about how to configure it. Pay special attention to the part about how XmlWriterTraceListener isn't thread-safe and might block execution flow and this severly impact performance (not something you want when debugging performance) and how to overcome this.
Lastly I think John Saunders his suggestion is very valid and if you do start with a trivial example and do not encounter this problem then I suggest you re-add your current functionality in small increments (or in a binary sort type of fashion) to see what it is that is causing the issue.
Not only can WCF be used I would highly recommend it instead of the more low level suggestions unless there is a very specific reason you can't use WCF (which I can't imagine with the gazillion config options). You have different layers you can apply security at in WCF the message level meaning the contents is encrypted and checksummed and transport meaning the underlying connection is encrypted with SSL on HTTPS for instance. This post by Aaron Skonnard nicely lists the different WCF bindings and their supported modes of transport, transaction handling, layers of security etc...
WCF can be a bit daunting at first because it supports so many things but Programming WCF Services 2nd edition should get you on your way very fast and it's an excellent book!
If you're only just starting to do web services, I would strongly recommend you go and check out WCF. It's the current and future standard for communications between machines - web services and a lot more.
The book I always recommend to get up and running in WCF quickly is Learning WCF by Michele Leroux Bustamante. She covers all the necessary topics, and in a very understandable and approachable way. This will teach you everything - basics, intermediate topics, security, transaction control and so forth - that you need to know to write high quality, useful WCF services.
Learning WCF http://ecx.images-amazon.com/images/I/51rz7yhbyxL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg
The more advanced topics and more in-depth look at WCF will be covered by Programming WCF Services by Juval Lowy. He really dives into all technical details and topics and presents "the bible" for WCF programming. He just recently completed a third edition, which covers WCF in .NET 4 and AppFabric and the Azure Service Bus, too.
Programming WCF Services by Juval Lowy
IDesign.net - Juval Lowy's website
Keith Elder Demystifies WCF @ dnrTV
Miguel Castro: Extreme WCF @ dnrTV
In your binding configuration, there are four timeout values you can tweak:
The most important is the sendTimeout, which says how long the client will wait for a response from your WCF service. You can specify hours:minutes:seconds in your settings - in my sample, I set the timeout to 25 minutes.
The openTimeout as the name implies is the amount of time you're willing to wait when you open the connection to your WCF service. Similarly, the closeTimeout is the amount of time when you close the connection (dispose the client proxy) that you'll wait before an exception is thrown.
The receiveTimeout is a bit like a mirror for the sendTimeout - while the send timeout is the amount of time you'll wait for a response from the server, the receiveTimeout is the amount of time you'll give you client to receive and process the response from the server.
In case you're send back and forth "normal" messages, both can be pretty short - especially the receiveTimeout, since receiving a SOAP message, decrypting, checking and deserializing it should take almost no time. The story is different with streaming - in that case, you might need more time on the client to actually complete the "download" of the stream you get back from the server.
There's also openTimeout, receiveTimeout, and closeTimeout. The MSDN docs on binding gives you more information on what these are for.
To get a serious grip on all the intricasies of WCF, I would strongly recommend you purchase the "Learning WCF" book by Michele Leroux Bustamante:
Learning WCF http://ecx.images-amazon.com/images/I/51GNuqUJq%2BL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA240_SH20_OU01_.jpg
and you also spend some time watching her 15-part "WCF Top to Bottom" screencast series - highly recommended!
For more advanced topics, you should definitely check out Juwal Lovy's Programming WCF Services book.
Programming WCF http://ecx.images-amazon.com/images/I/41odWcLoGAL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA240_SH20_OU01_.jpg