WinHttpOpen, WinHttpConnect and multithreaded requests

[John M]

Hi,

I'm porting a WinInet project to use WinHttp and am running into some hidden
allocation issues.

What I've noticed is that if I have 512 threads running simultaneously and
each of them calls WinHttpOpen, I get about 250MB allocated that never get
freed up. I've tried this with smaller numbers of threads and it's almost
exactly 428KB (438,272 bytes) that gets lost for each WinHttpOpen. Has
anybody else ever seen this?

Reading some of the older posts in this group, I'm sensing that maybe I only
need to make a single WinHttpOpen call and then let each thread use that
handle for it's subsequent Connects. Am I right about using just the one
WinHttpOpen call?

A little more reading (http://tinyurl.com/e34kl) makes me think that I might
only need to make a single call to WinHttpConnect as well. For the project
that I'm currently working on, all of the requests will be going to the same
server on the same port. Would I be safe to just make one single
WinHttpConnect call in this case?

I think that the WinHttpConnect calls don't hold any allocated memory after
their internal threads die off. Has anybody else seen different?

Thanks,
John

[Stephen Sulzer]

Hi John,

Concerning the large memory footprint: WinHttp's memory manager uses
per-thread heaps to improve scalability. So each thread that calls into the
WinHttp API will incur the overhead of a separate "heap manager/cache"
object. I doubt that sharing one set of WinHttpOpen & WinHttpConnect handles
across the threads--which may be a perfectly valid thing to do--will reduce
the memory usage by much.

When a thread terminates, the associated per-thread WinHttp heap object may
be kept alive in order to be reused by a new thread--so that would appear as
a memory leak. But the memory for these objects should be released if
WinHttp.dll is dynamically unloaded from the process.

In any case, having hundreds of threads within an application is not a
scalable design in Windows. Each thread will allocate its own stack, which
alone could add up to several megabytes of memory for 500+ threads. (Does
your 250MB measurement include the virtual memory reserved for each thread's
stack?)

The recommended design for a scalable Windows application is to break the
workload into individual "workitems" which are queued to be processed by a
worker thread pool--with the number of worker threads in the pool
proportional to the number of CPUs on the system--and to use asynchronous,
non-blocking I/O calls. WinHttp in asynchronous mode will, by default,
create 1 worker thread per CPU to process asynchronous HTTP requests.
Systems like IIS and ASP may spin up to 25 or more worker threads per CPU.
But 500+ threads is overkill--unless perhaps you are running on a 16- or
32-processor server box. :)

Regarding how many WinHttpOpen handles you need within your application: a
separate WinHttpOpen handle should be used for each _authenticated_ "user"
within the process; or, the WinHttpOpen handle can be shared among a
collection of anonymous clients. WinHttp was designed for multi-user
applications, and the WinHttpOpen handle is the key to partitioning the
users that exist within the process. If your application is multi-user, and
if the HTTP requests the clients send require authentication (or involve the
use of HTTP cookies that contain private data), then each client should be
assigned its own WinHttpOpen handle (and, therefore, its own WinHttpConnect
handle). If the users within the process are anonymous (the HTTP requests
they make using WinHttp do not require any kind of authentication), then it
is fine to use just one WinHttpOpen handle for them. WinHttp handles are
thread safe, so it is fine to share a WinHttpOpen handle across multiple
threads.

Hope that helps.

Regards,
Stephen

[John M]

Hi Stephen,

Yes, that helps tremendously. I'm sorry to have taken so long to thank
you, but I needed to absorb the information you sent... there is plenty
in there to think about.

1) "I doubt that sharing one set of WinHttpOpen &"... you're right... I
tried it and there was some decrease in memory use but not enough for
it to be significant.

2) "the associated per-thread WinHttp heap object may be kept alive"...
In our circumstances, it is _always_ kept alive. Other than
dynamically unloading teh WinHttp DLL, is there any other step I can
take to ask WinHttp to release those heaps?

3) "having hundreds of threads within an application is not a scalable
design in Windows"... OK... That's a very, very important thing for me
to learn. Thank you. Do you know of any good books that have been
published that can illustrate this type of design?

4) "(Does your 250MB measurement include the virtual memory reserved
for each thread's stack?)"... I took those measurements _after_ the
threads had finished and exited so I'd have to say "no" to that.

5) "Systems like IIS and ASP may spin up to 25 or more worker
threads"... oddly enough, one of the reasons they designed our code to
use the 500+ threads was because someone discovered something that
sounds very much like what you are describing. Do you know enough
about how .NET and ASP handle Web Service requests to answer this next
question... If a .NET Web Service receives 500 simultaneous requests,
will it answer each and every one of them? I hope that what you
mentioned about the worker threads means that it would funnel them down
internally and process them all.

6) "use asynchronous, non-blocking I/O calls"... I've found this topic
(http://tinyurl.com/7hw36) in the WinHttp MSDN documents and am going
to start working on the sample application to understand how to build
the asynchronous calls. Thank you. I hope I can find the example code
on the Platform SDK CD.

7) "Regarding how many WinHttpOpen handles you need"... thank you,
thank you. I've read the WinHttp documentation but this was never
clear to me. Now I understand when each one of these objects needs to
be created.

Best regards and thank you for your time,
John

[Stephen Sulzer]

Off-hand, I don't have any book recommendations. Using a small thread pool
(scaled to the number of CPUs on the machine) to process a queue of incoming
requests is a fairly common server architecture. There are a number of
articles on MSDN and elsewhere on the web that talk about this kind of
design. Here are a couple random links:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/progthrepool.asp

http://msdn.microsoft.com/msdnmag/issues/03/06/Threading/default.aspx

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndllpro/html/msdn_scalabil.asp

In a server application like IIS/ASP.NET, as requests come in they are
queued up and then processed by a worker thread pool. (The incoming HTTP
requests to IIS are initially processed by IIS's I/O worker thread. If the
request then requires the execution of ASP.NET code, a request is split off
and put into the ASP.NET worker thread pool.) So if 500 requests come in,
then yes, the server should be able to process them all, although it will
only work on maybe a couple dozen of them concurrently. But if the number of
requests waiting in the queue exceeds a certain threshold (the backlog of
waiting requests gets too large), then the server will start rejecting any
new requests that come in, until enough pending requests are processed and
the size of the queue decreases below the cut-off threshold.

Another issue to consider: are you building your server application using
.NET/C# or using "unmanaged" C++ with the Win32 API? If you are using
.NET/C#, then consider using the System.Net webclasses for sending HTTP
requests, not WinHTTP. Use WinHTTP if you are writing a native Win32/C++
application.

Regards,
Stephen

[John M]

Hi Stephen,

Thank you for the links. The part 3 information (I/O Completion Ports)
in the third link is very helpful for figuring out the
WorkItem/WorkerThreadPool architecture. I'm giving that a try based on
a couple of examples (http://tinyurl.com/dprra ,
http://support.microsoft.com/kb/q197728/) I found. As a first pass, it
actually made things worse for me, but that might be my early, limited
understanding.

Unfortunately, the design of this application doesn't use the .NET
Framework. I think that the information about the
500request/25threadPool IIS/ASP might have been enough to shift this if
we had known. Oh well.

Regards,
John

Post by Stephen Sulzer
Off-hand, I don't have any book recommendations. Using a small thread pool
(scaled to the number of CPUs on the machine) to process a queue of incoming
requests is a fairly common server architecture. There are a number of
articles on MSDN and elsewhere on the web that talk about this kind of
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/progthrepool.asp
http://msdn.microsoft.com/msdnmag/issues/03/06/Threading/default.aspx
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndllpro/html/msdn_scalabil.asp
In a server application like IIS/ASP.NET, as requests come in they are
queued up and then processed by a worker thread pool. (The incoming HTTP
requests to IIS are initially processed by IIS's I/O worker thread. If the
request then requires the execution of ASP.NET code, a request is split off
and put into the ASP.NET worker thread pool.) So if 500 requests come in,
then yes, the server should be able to process them all, although it will
only work on maybe a couple dozen of them concurrently. But if the number of
requests waiting in the queue exceeds a certain threshold (the backlog of
waiting requests gets too large), then the server will start rejecting any
new requests that come in, until enough pending requests are processed and
the size of the queue decreases below the cut-off threshold.
Another issue to consider: are you building your server application using
.NET/C# or using "unmanaged" C++ with the Win32 API? If you are using
.NET/C#, then consider using the System.Net webclasses for sending HTTP
requests, not WinHTTP. Use WinHTTP if you are writing a native Win32/C++
application.
Regards,
Stephen

[Stephen Sulzer]

Hi John,

I should have mentioned this in my previous post: Note that you cannot use
the I/O Completion Port programming model directly with WinHttp
requests--you cannot associate a WinHttp handle with an IO completion port.
When using WinHttp in async mode, your application must use WinHttp's
callback mechanism. Internally, WinHttp will use non-blocking, "overlapped
I/O" Winsock sockets with an IO completion port to process the request. The
calls from WinHttp to your status callback function occur on WinHttp's
worker thread that services its IO completion port.

In your WinHttp status callback function, your code could of course post a
notification to your own IO completion port, to process the callback
notification asynchronously on your application's own worker thread. That
may be a good design (despite the extra thread context switching), as it is
recommended to not block too long within the callback so that WinHttp can
get back quickly to processing more I/O. The WinHttp API also provides an
option (WINHTTP_OPTION_WORKER_THREAD_COUNT) that can be set on startup
(before calling WinHttpOpen) to specify how many worker threads it should
create to service its internal IO completion port.

- Stephen

Post by John M
Hi Stephen,
Thank you for the links. The part 3 information (I/O Completion Ports)
in the third link is very helpful for figuring out the
WorkItem/WorkerThreadPool architecture. I'm giving that a try based on
a couple of examples (http://tinyurl.com/dprra ,
http://support.microsoft.com/kb/q197728/) I found. As a first pass, it
actually made things worse for me, but that might be my early, limited
understanding.
Unfortunately, the design of this application doesn't use the .NET
Framework. I think that the information about the
500request/25threadPool IIS/ASP might have been enough to shift this if
we had known. Oh well.
Regards,
John

[Stephen Sulzer]

Post by John M
2) "the associated per-thread WinHttp heap object may be kept alive"...
In our circumstances, it is _always_ kept alive. Other than
dynamically unloading teh WinHttp DLL, is there any other step I can
take to ask WinHttp to release those heaps?

No, I don't recall there being any way to control that behavior of WinHttp's
memory manager. Constraining the number of threads that call into WinHttp is
the only thing that will help. A couple threads using WinHttp in
asynchronous mode can manage a large number of concurrent HTTP requests.

- Stephen