WinHttpSetOption and WinHttpQueryOption

[DerekG]

I am writing a Windows service for Win2kSP3 which accesses the
internet. I have been trying to find a method which acts more like
CInternetSession::OpenUrl because all I want to do is read a web page,
but I have not been able to find anything like that in this API. The
whole reason why I am switching to this API is because I read that
WinInet does not function properly in Windows services, but this API
supposedly does.

I have the latest Core and Internet SDK. I finally got my code to
compile and have been trying without any success to get this API to
work for me. The trouble that I am experiencing is with
WinHttpSetOption and WinHttpQueryOption; WinHttpOpen appears to work.

My questions are thus:
1) is there a method like CInternetSession::OpenUrl available (ie.
WinHttpOpenUrl) so that I can just read a web page?
2) what am I doing wrong in the code snippet below which causes me to
receive error 87 for WinHttpQueryOption and error 12018 for
WinHttpSetOption?

My code snippet is as follows:

m_hInternetSession = WinHttpOpen(L"MyApplication",
WINHTTP_ACCESS_TYPE_NAMED_PROXY, L"http://my-proxy.com:8080",
WINHTTP_NO_PROXY_BYPASS, NULL);
if (m_hInternetSession)
{
DWORD dwLogonPolicy = 0;
TCHAR szUsername[128] = "username";
TCHAR szPassword[128] = "password";

dwCount = sizeof(DWORD);
if (WinHttpQueryOption(m_hInternetSession,
WINHTTP_OPTION_AUTOLOGON_POLICY, (LPVOID)&dwLogonPolicy, &dwCount) ==
FALSE)
{
dwCode = GetLastError();// Error 87
}
dwCount = lstrlen(szUsername);
if (WinHttpSetOption(m_hInternetSession,
WINHTTP_OPTION_PROXY_USERNAME, szUsername, dwCount) == FALSE)
{
dwCode = GetLastError();// Error 12018
}
dwCount = lstrlen(szPassword);
if (WinHttpSetOption(m_hInternetSession,
WINHTTP_OPTION_PROXY_PASSWORD, szPassword, dwCount) == FALSE)
{
dwCode = GetLastError();// Error 12018
}

Thanks in advance,
dg

[Stephen Sulzer]

Yes, WinHTTP should be used instead of WinInet for Windows services.

Are you using WinHTTP 5.0 (winhttp5.lib/winhttp5.dll) or WinHTTP 5.1
(winhttp.lib/winhttp.dll) ? I'm assuming that you are using version 5.1 on
Windows 2000 SP3.

Post by DerekG
1) is there a method like CInternetSession::OpenUrl available (ie.
WinHttpOpenUrl) so that I can just read a web page?

No, there is no high-level function in the WinHTTP Win32 API equivalent to
WinInet's InternetOpenUrl function, so you will have to write more code. If
you look in the WinHTTP sample code in the SDK (under \samples\web\winhttp),
you might find some code there that you can reuse.

Post by DerekG
2) what am I doing wrong in the code snippet below which causes me to
receive error 87 for WinHttpQueryOption and error 12018 for
WinHttpSetOption?

The WinHTTP option flags are documented on-line at:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winhttp/http/option_flags.asp

which also includes a table listing which types of handles (Internet Session
or Request) support which options.

WinHTTP error codes are in the 12000 range and are listed in the winhttp.h
SDK file. (And are documented on-line at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winhttp/http/error_messages.asp.)
Other Win32 error codes can (usually) be found in the winerror.h SDK file.

I am not sure what is causing the error 87 (incorrect parameter) with the
WinHttpQueryOption(WINHTTP_OPTION_AUTOLOGON_POLICY) call. Although the
documentation says that the AUTOLOGON_POLICY option is not supported on a
Session handle, I believe that is a mistake in the docs, and that it should
be OK to query this option on the Session handle (at least for WinHTTP 5.1).

You should try querying the option on the Request handle and see if that
fixes the problem. The default setting for the AUTOLOGON_POLICY is
WINHTTP_AUTOLOGON_SECURITY_LEVEL_MEDIUM. I don't think you will need to
change this setting in order to authenticate with a proxy server.

The error 12018 (incorrect handle type) for the WinHttpSetOption calls means
that you cannot set the PROXY_USERNAME and PROXY_PASSWORD options on the
Session handle. These options are supported on the Request handle only.

Another concern in your code is the declarations of the szUserName and
szPassword string variables. The WinHTTP Win32 API is Unicode-only, so the
code must ensure any strings parameters are Unicode (this requirement does
not apply to POST data, however, which should be given in bytes). The
"username" and "password" string literals should be prefaced with an L to
make them Unicode.

Stephen

[DerekG]

First of all, I would like to thank you for getting back to my initial
questions so quickly. You have been a tremendous help. And to answer
your question about which version I am using, yes, I am using 5.1.

I changed the WinHttpSetOption to use the request handle and it does
appear to work; I did view the help page on the options, but I did not
see any reference to any handles (ie session nor request) that we are
supposed to use for WINHTTP_OPTION_PROXY_USERNAME nor for
WINHTTP_OPTION_PROXY_PASSWORD. However, WINHTTP_OPTION_PROXY does say
that we can use it for either the session or the request.

Now I have a couple more questions. I am trying to send a URL much
like the one used in the WinHttpCrackUrl example (ie
http://search.msn.com/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=wininet).
The code below works when I am authenticated prior to running my
code, but when I am not authenticated prior to running this function I
get the company proxy authentication html for a response which I am
trying to avoid. As you can see in the code below, I attempt to set
the proxy username and password. Why isn't this working (ie avoiding
the company proxy authentication page)? None of the code examples
appear to talk about this or did I do miss something?

tia,
dg

=====================================================

void MakeInternetRequest()
{
BSTR bstrObjectName = NULL;
BSTR bstrUsername = NULL;
BSTR bstrPassword = NULL;
BOOL bRequests = FALSE;
DWORD dwCode = 0;
DWORD dwSize = 0;
DWORD dwDownloaded = 0;
CString sBuffer = _T("");
HINTERNET hInternetSession = NULL;
HINTERNET hConnection = NULL;
HINTERNET hRequest = NULL;

hInternetSession = WinHttpOpen(
L"TestApplication",
WINHTTP_ACCESS_TYPE_NAMED_PROXY,
L"http://my-proxy.com:8080",
WINHTTP_NO_PROXY_BYPASS,
0);

if (hInternetSession)
{
hConnection = WinHttpConnect(
hInternetSession,
L"search.msn.com",
INTERNET_DEFAULT_HTTP_PORT,
0);
}

if (hConnection)
{
sBuffer.Format("/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=wininet");
bstrObjectName = sBuffer.AllocSysString();
hRequest = WinHttpOpenRequest(
hConnection,
L"GET",
bstrObjectName,
NULL,
WINHTTP_NO_REFERER,
WINHTTP_DEFAULT_ACCEPT_TYPES,
0);
}

if (hRequest)
{
sBuffer.Format("username");
bstrUsername = sBuffer.AllocSysString();
if (WinHttpSetOption(hRequest, WINHTTP_OPTION_PROXY_USERNAME,
bstrUsername, SysStringLen(bstrUsername)) == FALSE)
{
dwCode = GetLastError();
}
sBuffer.Format("password");
bstrPassword = sBuffer.AllocSysString();
if (WinHttpSetOption(hRequest, WINHTTP_OPTION_PROXY_PASSWORD,
bstrPassword, SysStringLen(bstrPassword)) == FALSE)
{
dwCode = GetLastError();
}
bRequests = WinHttpSendRequest(
hRequest,
WINHTTP_NO_ADDITIONAL_HEADERS,
0,
WINHTTP_NO_REQUEST_DATA,
0,
0,
0);

if (bRequests)
bRequests = WinHttpReceiveResponse(hRequest, NULL);

if (bRequests)
{
do
{
// Check for available data.
dwSize = 0;
if (!WinHttpQueryDataAvailable( hRequest, &dwSize))
printf("Error %u in
WinHttpQueryDataAvailable.\n",GetLastError());

// Allocate space for the buffer.
char* pszOutBuffer = new char[dwSize+1];
if (!pszOutBuffer)
{
printf("Out of memory\n");
dwSize=0;
}
else
{
// Read the Data.
ZeroMemory(pszOutBuffer, dwSize+1);
if (!WinHttpReadData( hRequest, (LPVOID) pszOutBuffer,
dwSize,
&dwDownloaded))
printf("Error %u in WinHttpReadData.\n", GetLastError());
else
{
if (strlen(pszOutBuffer) > 0)
cout << pszOutBuffer << endl;
}

// Free the memory allocated to the buffer.
delete [] pszOutBuffer;
}
} while (dwSize>0);
}
}

if (bstrObjectName)
SysFreeString(bstrObjectName);
if (bstrUsername)
SysFreeString(bstrUsername);
if (bstrPassword)
SysFreeString(bstrPassword);

if (hRequest)
WinHttpCloseHandle(hRequest);
if (hConnection)
WinHttpCloseHandle(hConnection);
if (hInternetSession)
WinHttpCloseHandle(hInternetSession);
}

[Stephen Sulzer]

I really don't see anything wrong with your code.

Double check that the values of the bstrUserName and bstrPassword strings
are what you expect them to be.

If the password is actually an empty string, then you must specify
SysStringLen(bstrPassword)+1 as the length (you can't specify zero for the
length of the buffer.)

In the case that it works--when you preauthenticate with the proxy before
sending the HTTP request--how do you authenticate with the proxy? Is there
any additional information you use when preauthenticating with the proxy
(such as a 'Realm') that is not being specified in the code sample you gave?
Also, if the username is of the form "domain\username", be sure that the C
string in the code is "domain\\username".

It may also help to get a capture of the network traffic between the WinHTTP
client and the proxy server. Check the network traffic data to ensure that
WinHTTP is trying to authenticate with your server. (If WinHTTP is trying to
authenticate, then it suggests there is something wrong with the
credentials.) You can use a tool like NETMON or the WinHTTP Trace utility to
capture the network trace.
The WinHTTP Trace utility (winhttptracecfg.exe) is available as part of the
Windows Resource Kit. Documentation for the tool is at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winhttp/http/winhttptracecfg_exe__a_trace_configuration_tool.asp
and the resource kit can be downloaded from:
http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&DisplayLang=en

WinHTTP authentication support is documented on-line at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winhttp/http/authentication_in_winhttp.asp

Note that the sample code given in the documentation uses a different
WinHTTP API--WinHttpSetCredentials--for supplying the username and password.
However, using WinHttpSetOption to supply credentials is supported, so you
should not need to change your code.

As for the information about which handles support which options, the list
is given at the very end of the help page I pointed to.
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winhttp/ht
tp/option_flags.asp; scroll to the very bottom to see the table)


Stephen

[DerekG]

Post by Stephen Sulzer
Double check that the values of the bstrUserName and bstrPassword strings
are what you expect them to be.

I have now done that and everything seems to be fine.

Post by Stephen Sulzer
If the password is actually an empty string, then you must specify
SysStringLen(bstrPassword)+1 as the length (you can't specify zero for the
length of the buffer.)

Its not empty.

Post by Stephen Sulzer
In the case that it works--when you preauthenticate with the proxy before
sending the HTTP request--how do you authenticate with the proxy?

The way I authenticate is to attempt to access a website outside of
the company LAN (ie like this news group). However, my company does
use a PAC file. Would that somehow get in the way even though for my
test I am hard coding the same proxy server that it returns?

Post by Stephen Sulzer
Is there
any additional information you use when preauthenticating with the proxy
(such as a 'Realm') that is not being specified in the code sample you gave?

No, all I have to do when I authenticate is type in my username and
password.

Post by Stephen Sulzer
Also, if the username is of the form "domain\username", be sure that the C
string in the code is "domain\\username".

No, all I have to do when I authenticate is type in my username and
password.

Post by Stephen Sulzer
It may also help to get a capture of the network traffic between the WinHTTP
client and the proxy server. Check the network traffic data to ensure that
WinHTTP is trying to authenticate with your server. (If WinHTTP is trying to
authenticate, then it suggests there is something wrong with the
credentials.) You can use a tool like NETMON or the WinHTTP Trace utility to
capture the network trace.

Unfortunately, I can't use either one of the tools mentioned because I
am running Win2k (ie not the server edition). WinHTTP Trace does not
appear to support Win2k and NETMON appears to only be a Win2k Server
utility so when I go to install it as a Windows Component it is not in
the list of available utilities to install. I will have to look
around for utilities that I can use in that regard.

Post by Stephen Sulzer
As for the information about which handles support which options, the list
is given at the very end of the help page I pointed to.
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winhttp/ht
tp/option_flags.asp; scroll to the very bottom to see the table)

I completely missed that -- thanks for pointing that out.

Thanks,
dg

[Stephen Sulzer]

I think we are stuck until you can get a network capture.

If you have trouble getting a tracing tool to work, then can you just print
out the response headers please? You can query for all the response headers
(after WinHttpReceiveResponse) using:

WinHttpQueryHeaders(hRequest,
WINHTTP_QUERY_RAW_HEADERS_CRLF,
WINHTTP_HEADER_NAME_BY_INDEX,
ResponseHeaders, &dwSize, 0);

where ResponseHeaders is a large (say 64K) Unicode string buffer allocated
in your code (and dwSize is the size of this buffer in bytes).

Try querying for the response headers in two cases: 1) a request with the
WinHttpSetOption calls that supply the proxy name and password, and 2) a
request with no WinHttpSetOption calls.

The WinHTTP Trace tool should work on Windows 2000 Pro. The tracing
capability is already built into WinHTTP; all the winhttptracecfg.exe tool
does is set a registry key to enable/disable tracing. Running the tool does
require administrator privileges, however. (Also, even though the resource
kit is titled "Windows Server 2003 Resource Kit Tools", the
winhttptracecfg.exe can be run on Windows 2000.)

There is also a much smaller download for just the WinHTTP Trace Cfg tool
available at:
http://download.microsoft.com/download/2/8/9/289387e3-55fe-45b3-b532-87c6d21e54ec/Winhttptracecfqui.exe

I don't think the use of autoproxy scripts (the PAC file) in you network
environment is related to the problem, if you are hard-coding a proxy server
name in the WinHttpOpen call. (But it might be next on the list to
investigate if looking at the network trace/response headers does not help.)


Stephen

Post by DerekG

Post by Stephen Sulzer
Double check that the values of the bstrUserName and bstrPassword strings
are what you expect them to be.

I have now done that and everything seems to be fine.

Post by Stephen Sulzer
If the password is actually an empty string, then you must specify
SysStringLen(bstrPassword)+1 as the length (you can't specify zero for the
length of the buffer.)

Its not empty.

Post by Stephen Sulzer
In the case that it works--when you preauthenticate with the proxy before
sending the HTTP request--how do you authenticate with the proxy?

The way I authenticate is to attempt to access a website outside of
the company LAN (ie like this news group). However, my company does
use a PAC file. Would that somehow get in the way even though for my
test I am hard coding the same proxy server that it returns?

Post by Stephen Sulzer
Is there
any additional information you use when preauthenticating with the proxy
(such as a 'Realm') that is not being specified in the code sample you gave?

No, all I have to do when I authenticate is type in my username and
password.

Post by Stephen Sulzer
Also, if the username is of the form "domain\username", be sure that the C
string in the code is "domain\\username".

No, all I have to do when I authenticate is type in my username and
password.

Post by Stephen Sulzer
It may also help to get a capture of the network traffic between the WinHTTP
client and the proxy server. Check the network traffic data to ensure that
WinHTTP is trying to authenticate with your server. (If WinHTTP is trying to
authenticate, then it suggests there is something wrong with the
credentials.) You can use a tool like NETMON or the WinHTTP Trace utility to
capture the network trace.

Unfortunately, I can't use either one of the tools mentioned because I
am running Win2k (ie not the server edition). WinHTTP Trace does not
appear to support Win2k and NETMON appears to only be a Win2k Server
utility so when I go to install it as a Windows Component it is not in
the list of available utilities to install. I will have to look
around for utilities that I can use in that regard.

Post by Stephen Sulzer
As for the information about which handles support which options, the list
is given at the very end of the help page I pointed to.

(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winhttp/ht

Post by DerekG

Post by Stephen Sulzer
tp/option_flags.asp; scroll to the very bottom to see the table)

I completely missed that -- thanks for pointing that out.
Thanks,
dg

[DerekG]

Post by Stephen Sulzer
I think we are stuck until you can get a network capture.
If you have trouble getting a tracing tool to work, then can you just print
out the response headers please? You can query for all the response headers
WinHttpQueryHeaders(hRequest,
WINHTTP_QUERY_RAW_HEADERS_CRLF,
WINHTTP_HEADER_NAME_BY_INDEX,
ResponseHeaders, &dwSize, 0);
where ResponseHeaders is a large (say 64K) Unicode string buffer allocated
in your code (and dwSize is the size of this buffer in bytes).

Does WinHttpTrace require proxycfg and isn't proxycfg supposed to be
installed with Win2k SP3? I have Win2k SP3, but I don't have proxycfg
on my system; my attempts to use WinHttpTrace have not produced the
log file as I had hoped (I registered the ocx and used the web page
interface to control the functionality). I went to look at the list
of tools that are installed in the Windows Server 2003 Resource Kit
Tools, but I didn't see proxycfg listed there. I tried to download
the kit anyway, but was unable to do so (I don't know why), but I was
able to get WinHttpTrace from your above link. My attempts to find a
suitable trial version of a packet sniffer have been disasterous
because each one of the trial versions is handicapped in some crucial
way... I am still trying to find a packet sniffer that will allow me
to get all of the information I need for this short duration.

Post by Stephen Sulzer
Try querying for the response headers in two cases: 1) a request with the
WinHttpSetOption calls that supply the proxy name and password, and 2) a
request with no WinHttpSetOption calls.

After some more testing I have found that
WinHttpSetOption/WinHttpSetCredentials does nothing for me. It does
not matter if I call those functions because they don't appear to do
anything; the only thing that appears to matter is if I
pre-authenticate (ie. open a web page, authenticate on the proxy, then
run my test application).

Post by Stephen Sulzer
The WinHTTP Trace tool should work on Windows 2000 Pro. The tracing
capability is already built into WinHTTP; all the winhttptracecfg.exe tool
does is set a registry key to enable/disable tracing. Running the tool does
require administrator privileges, however. (Also, even though the resource
kit is titled "Windows Server 2003 Resource Kit Tools", the
winhttptracecfg.exe can be run on Windows 2000.)

I do have administrator privileges, but I am unable to use
WinHttpTrace because of above stated problems.

I will let you know more when I can actually sniff the packets.

Thanks,
dg

[Stephen Sulzer]

WinHTTP tracing does not require the proxycfg.exe tool. The proxycfg.exe
tool sets a registry key to tell WinHTTP what proxy to use, so that the name
of the proxy does not need to be hard-coded in the WinHttpOpen call.

What output is displayed when you run winhttptracecfg.exe ? A couple
important things to note about WinHTTP tracing:
1. the trace file may not be available on disk until your program or
service that uses WinHTTP shuts down. WinHTTP flushes the trace file to disk
when winhttp.dll is unloaded from memory.
2. the 'FileNamePrefix' option does not specify the full trace file
name. If the FileNamePrefix is set to "c:\test\winhttplog", for example,
then the trace file will be located in the c:\test directory and the file
name will be "winhttplog-<process-name>.<time>.LOG".
3. WinHTTP picks up the trace settings when winhttp.dll is first loaded
into the process. So ensure that the trace configuration is set before you
start your service or program that uses WinHTTP.

Is the proxy server that you specify in the WinHttpOpen call the same proxy
that you preauthenticate with?

I don't understand how preauthenticating with the proxy server works. I'm
not sure how your proxy would remember that your client is authenticated.
What kind of proxy server are you using and do you know what kind of
authentication protocol it uses (e.g., Digest, Basic, NTLM)? WinHTTP
supports only HTTP web proxies, and only Basic, Digest, NTLM/Kerberos
(Integrated Windows Authentication) and Passport authentication schemes.
WinHTTP does not support SOCKS proxies.

What are the response headers in the case where you haven't authenticated?

Regarding proxycfg.exe, it's unfortunate that it is not in Windows 2000 SP3
or SP4, nor is it part of the Resource Kit. You can obtain proxycfg.exe for
Windows 2000 SP3 via Microsoft PSS; see KB article 830605:
http://support.microsoft.com/default.aspx?scid=kb;en-us;830605. However, you
do not need proxycfg.exe in order to get tracing to work.

Stephen

[DerekG]

Post by Stephen Sulzer
What output is displayed when you run winhttptracecfg.exe ?

The following log is before I authenticated:

12:01:03.860 ::*Session* :: >>>> WinHttp Version 5.1 Build 5.1.2600
Dec 9 2003 01:37:31>>>>Process TestApplication.exe [1860 (0x744)]
started at 12:01:03.860 05/05/2004
12:01:05.152 ::*Session* :: WinHttpOpen("TestApplication", (3),
"http://my-proxy.com:8080", "", 0x0)
12:01:05.152 ::*Session* :: WinHttpOpen() returning handle 0x9b4000
12:01:05.152 ::*Session* :: WinHttpConnect(0x9b4000, "search.msn.com",
80, 0x0)
12:01:05.152 ::*Session* :: WinHttpConnect() returning handle 0x9b9000
12:01:05.152 ::*Session* :: WinHttpOpenRequest(0x9b9000, "GET",
"/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=wininet", "", "", 0x0,
0x00000000)
12:01:05.152 ::*Session* :: WinHttpCreateUrlA(0x12fd78, 0x0,
0x10c0000, 0x12fdb4)
12:01:05.152 ::*Session* :: WinHttpCreateUrlA() returning TRUE
12:01:05.152 ::*0000001* :: WinHttpOpenRequest() returning handle
0x9ba000
12:01:05.152 ::*0000001* :: WinHttpSendRequest(0x9ba000, "", 0, 0x0,
0, 0, 0)
12:01:05.152 ::*0000001* :: WinHttpCreateUrlA(0x12fd38, 0x0,
0x10c0000, 0x12fd1c)
12:01:05.152 ::*0000001* :: WinHttpCreateUrlA() returning TRUE
12:01:05.152 ::*0000001* :: Using proxy server: my-proxy.com:8080
12:01:05.342 ::*0000001* :: "my-proxy.com" resolved
12:01:05.412 ::*0000001* :: sending data:
12:01:05.412 ::*0000001* :: 166 (0xa6) bytes
12:01:05.412 ::*0000001* :: <<<<-------- HTTP headers follow below
----------------------------------------------->>>>
12:01:05.412 ::*0000001* :: GET
http://search.msn.com/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=wininet
HTTP/1.1
12:01:05.412 ::*0000001* :: User-Agent: TestApplication
12:01:05.412 ::*0000001* :: Host: search.msn.com
12:01:05.412 ::*0000001* :: Proxy-Connection: Keep-Alive
12:01:05.412 ::*0000001* :: <<<<-------- End
----------------------------------------------->>>>
12:01:05.412 ::*0000001* :: WinHttpSendRequest() returning TRUE
12:01:05.412 ::*0000001* :: WinHttpReceiveResponse(0x9ba000, 0x0)
12:01:05.412 ::*0000001* :: received data:
12:01:05.412 ::*0000001* :: 32 (0x20) bytes
12:01:05.412 ::*0000001* :: <<<<-------- HTTP headers follow below
----------------------------------------------->>>>
12:01:05.412 ::*0000001* :: HTTP/1.0 302 Moved Temporarily
12:01:05.412 ::*0000001* ::
12:01:05.412 ::*0000001* :: <<<<-------- End
----------------------------------------------->>>>
12:01:05.552 ::*0000001* :: sending data:
12:01:05.552 ::*0000001* :: 225 (0xe1) bytes
12:01:05.552 ::*0000001* :: <<<<-------- HTTP headers follow below
----------------------------------------------->>>>
12:01:05.552 ::*0000001* :: GET
/IssueLogin?redirect=http%3A%2F%2Fsearch.msn.com%2Fresults.asp%3FRS%3DCHECKED%26FORM%3DMSNH%26v%3D1%26q%3Dwininet
HTTP/1.1
12:01:05.552 ::*0000001* :: User-Agent: TestApplication
12:01:05.552 ::*0000001* :: Connection: Keep-Alive
12:01:05.552 ::*0000001* :: Host: my-proxy.com:8080
12:01:05.552 ::*0000001* :: <<<<-------- End
----------------------------------------------->>>>
12:01:05.572 ::*0000001* :: received data:
12:01:05.572 ::*0000001* :: 17 (0x11) bytes
12:01:05.572 ::*0000001* :: <<<<-------- HTTP headers follow below
----------------------------------------------->>>>
12:01:05.572 ::*0000001* :: HTTP/1.0 200 OK
12:01:05.572 ::*0000001* ::
12:01:05.572 ::*0000001* :: <<<<-------- End
----------------------------------------------->>>>
12:01:05.572 ::*0000001* :: WinHttpReceiveResponse() returning TRUE
12:01:06.073 ::*0000001* :: WinHttpQueryHeaders(0x9ba000, (0x16),
"<null>", 0x0, 0x12ff08 [0], 0x0 [0])
12:01:06.073 ::*0000001* :: WinHttpQueryHeaders() returning FALSE
12:01:06.073 ::*0000001* :: WinHttpQueryHeaders(0x9ba000, (0x16),
"<null>", 0xdd5b20, 0x12ff08 [358], 0x0 [0])
12:01:06.073 ::*0000001* :: WinHttpQueryHeaders() returning TRUE
12:01:06.073 ::*0000001* :: WinHttpQueryHeaders(0x9ba000,
(0x20000013), "<null>", 0x12ff4c, 0x12ff08 [4], 0x0 [0])
12:01:06.073 ::*0000001* :: WinHttpQueryHeaders() returning TRUE
12:01:06.073 ::*0000001* :: WinHttpQueryAuthSchemes(0x9ba000,
0x12ff0c, 0x12fefc)
12:01:06.073 ::*0000001* :: WinHttpQueryAuthSchemes: error 4317
[0x10dd]
12:01:06.073 ::*0000001* :: WinHttpQueryAuthSchemes() returning FALSE
12:01:06.073 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12ff68)
12:01:06.073 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:01:06.073 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd6d68, 846,
0x12ff0c)
12:01:06.073 ::*0000001* :: WinHttpReadData() returning TRUE
12:01:06.073 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12fef4)
12:01:06.073 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:01:06.073 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd7348, 3901,
0x12ff0c)
12:01:06.073 ::*0000001* :: WinHttpReadData() returning TRUE
12:01:06.093 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12fef4)
12:01:06.093 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:01:06.093 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd5b20, 0,
0x12ff0c)
12:01:06.093 ::*0000001* :: WinHttpReadData() returning TRUE
12:01:06.093 ::*0000001* :: WinHttpCloseHandle(0x9ba000)
12:01:06.093 ::*0000001* :: WinHttpCloseHandle() returning TRUE
12:01:06.093 ::*Session* :: WinHttpCloseHandle(0x9b9000)
12:01:06.093 ::*Session* :: WinHttpCloseHandle() returning TRUE
12:01:06.093 ::*Session* :: WinHttpCloseHandle(0x9b4000)
12:01:06.093 ::*Session* :: WinHttpCloseHandle() returning TRUE

The following is after I authenticated:

12:43:52.996 ::*Session* :: >>>> WinHttp Version 5.1 Build 5.1.2600
Dec 9 2003 01:37:31>>>>Process TestApplication.exe [1884 (0x75c)]
started at 12:43:52.996 05/05/2004
12:43:53.937 ::*Session* :: WinHttpOpen("TestApplication", (3),
"http://my-proxy.com:8080", "", 0x0)
12:43:53.947 ::*Session* :: WinHttpOpen() returning handle 0x9b4000
12:43:53.947 ::*Session* :: WinHttpConnect(0x9b4000, "search.msn.com",
80, 0x0)
12:43:53.947 ::*Session* :: WinHttpConnect() returning handle 0x9b9000
12:43:53.947 ::*Session* :: WinHttpOpenRequest(0x9b9000, "GET",
"/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=wininet", "", "", 0x0,
0x00000000)
12:43:53.947 ::*Session* :: WinHttpCreateUrlA(0x12fd78, 0x0,
0x10c0000, 0x12fdb4)
12:43:53.947 ::*Session* :: WinHttpCreateUrlA() returning TRUE
12:43:53.947 ::*0000001* :: WinHttpOpenRequest() returning handle
0x9ba000
12:43:53.947 ::*0000001* :: WinHttpSendRequest(0x9ba000, "", 0, 0x0,
0, 0, 0)
12:43:53.947 ::*0000001* :: WinHttpCreateUrlA(0x12fd38, 0x0,
0x10c0000, 0x12fd1c)
12:43:53.947 ::*0000001* :: WinHttpCreateUrlA() returning TRUE
12:43:53.947 ::*0000001* :: Using proxy server: my-proxy.com:8080
12:43:54.127 ::*0000001* :: "my-proxy.com" resolved
12:43:54.197 ::*0000001* :: sending data:
12:43:54.197 ::*0000001* :: 166 (0xa6) bytes
12:43:54.197 ::*0000001* :: <<<<-------- HTTP headers follow below
----------------------------------------------->>>>
12:43:54.197 ::*0000001* :: GET
http://search.msn.com/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=wininet
HTTP/1.1
12:43:54.197 ::*0000001* :: User-Agent: TestApplication
12:43:54.197 ::*0000001* :: Host: search.msn.com
12:43:54.197 ::*0000001* :: Proxy-Connection: Keep-Alive
12:43:54.197 ::*0000001* :: <<<<-------- End
----------------------------------------------->>>>
12:43:54.197 ::*0000001* :: WinHttpSendRequest() returning TRUE
12:43:54.197 ::*0000001* :: WinHttpReceiveResponse(0x9ba000, 0x0)
12:43:54.688 ::*0000001* :: received data:
12:43:54.688 ::*0000001* :: 17 (0x11) bytes
12:43:54.688 ::*0000001* :: <<<<-------- HTTP headers follow below
----------------------------------------------->>>>
12:43:54.688 ::*0000001* :: HTTP/1.0 200 OK
12:43:54.688 ::*0000001* ::
12:43:54.688 ::*0000001* :: <<<<-------- End
----------------------------------------------->>>>
12:43:54.698 ::*0000001* :: WinHttpReceiveResponse() returning TRUE
12:43:54.698 ::*0000001* :: WinHttpQueryHeaders(0x9ba000, (0x16),
"<null>", 0x0, 0x12ff04 [0], 0x0 [0])
12:43:54.698 ::*0000001* :: WinHttpQueryHeaders() returning FALSE
12:43:54.698 ::*0000001* :: WinHttpQueryHeaders(0x9ba000, (0x16),
"<null>", 0xdd5b20, 0x12ff04 [1662], 0x0 [0])
12:43:54.698 ::*0000001* :: WinHttpQueryHeaders() returning TRUE
12:43:55.349 ::*0000001* :: WinHttpQueryHeaders(0x9ba000,
(0x20000013), "<null>", 0x12ff4c, 0x12ff04 [4], 0x0 [0])
12:43:55.349 ::*0000001* :: WinHttpQueryHeaders() returning TRUE
12:43:55.349 ::*0000001* :: WinHttpQueryAuthSchemes(0x9ba000,
0x12ff0c, 0x12fefc)
12:43:55.349 ::*0000001* :: WinHttpQueryAuthSchemes: error 4317
[0x10dd]
12:43:55.349 ::*0000001* :: WinHttpQueryAuthSchemes() returning FALSE
12:43:55.349 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12ff68)
12:43:55.349 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:43:55.349 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd5b20, 194,
0x12ff0c)
12:43:55.349 ::*0000001* :: WinHttpReadData() returning TRUE
12:43:55.349 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12fef4)
12:43:55.349 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:43:55.349 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd7288, 7998,
0x12ff0c)
12:43:55.349 ::*0000001* :: WinHttpReadData() returning TRUE
12:43:55.600 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12fef4)
12:43:55.610 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:43:55.610 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd7288, 5196,
0x12ff0c)
12:43:55.610 ::*0000001* :: WinHttpReadData() returning TRUE
12:43:55.770 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12fef4)
12:43:55.770 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:43:55.770 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd7288, 3032,
0x12ff0c)
12:43:55.770 ::*0000001* :: WinHttpReadData() returning TRUE
12:43:55.870 ::*0000001* :: WinHttpQueryDataAvailable(0x9ba000,
0x12ff10, 0x12fef4)
12:43:55.870 ::*0000001* :: WinHttpQueryDataAvailable() returning TRUE
12:43:55.870 ::*0000001* :: WinHttpReadData(0x9ba000, 0xdd5b20, 0,
0x12ff0c)
12:43:55.870 ::*0000001* :: WinHttpReadData() returning TRUE
12:43:55.870 ::*0000001* :: WinHttpCloseHandle(0x9ba000)
12:43:55.870 ::*0000001* :: WinHttpCloseHandle() returning TRUE
12:43:55.870 ::*Session* :: WinHttpCloseHandle(0x9b9000)
12:43:55.870 ::*Session* :: WinHttpCloseHandle() returning TRUE
12:43:56.972 ::*Session* :: WinHttpCloseHandle(0x9b4000)
12:43:56.972 ::*Session* :: WinHttpCloseHandle() returning TRUE

Post by Stephen Sulzer
A couple
1. the trace file may not be available on disk until your program or
service that uses WinHTTP shuts down. WinHTTP flushes the trace file to disk
when winhttp.dll is unloaded from memory.
2. the 'FileNamePrefix' option does not specify the full trace file
name. If the FileNamePrefix is set to "c:\test\winhttplog", for example,
then the trace file will be located in the c:\test directory and the file
name will be "winhttplog-<process-name>.<time>.LOG".
3. WinHTTP picks up the trace settings when winhttp.dll is first loaded
into the process. So ensure that the trace configuration is set before you
start your service or program that uses WinHTTP.

This didn't start working for me until I allowed the program to run to
completion once. I was running my TestApplication in the debugger
expecting a log file to appear and it never did so I would stop the
debugger and it would still not appear. Then, when I allowed the
program to run to completion once, all of my other loggings have
occurred (even when I stop the application in the debugger prior to
program completion).

Post by Stephen Sulzer
Is the proxy server that you specify in the WinHttpOpen call the same proxy
that you preauthenticate with?

Yes, it is.

Post by Stephen Sulzer
I don't understand how preauthenticating with the proxy server works. I'm
not sure how your proxy would remember that your client is authenticated.
What kind of proxy server are you using and do you know what kind of
authentication protocol it uses (e.g., Digest, Basic, NTLM)? WinHTTP
supports only HTTP web proxies, and only Basic, Digest, NTLM/Kerberos
(Integrated Windows Authentication) and Passport authentication schemes.
WinHTTP does not support SOCKS proxies.

I tried doing a WinHttpQueryAuthSchemes in an attempt to retrieve the
authentication means without any success. I have also tried using
WinHttpSetCredentials with every type of authentication scheme listed
in the documentation without any success.

Post by Stephen Sulzer
What are the response headers in the case where you haven't authenticated?

See logs above.


Thanks,
dg

[Stephen Sulzer]

It doesn't look like your proxy server uses a standard HTTP authentication
protocol. I would have expected to see an "HTTP/1.0 407 Proxy Authentication
Required" response in the first trace. Also, the fact that
WinHttpQueryAuthSchemes returned error code 4317 (ERROR_INVALID_OPERATION)
means that WinHTTP did not recognize any authentication challenge. From the
trace, it looks like the proxy server uses some kind of HTML page-based
authentication, and not one of the standard HTTP authentication protocols
such as Basic or Digest.

When you preauthenticate with the proxy server on its log-in web page, do
you fill in your username and password directly into the web page and click
some kind of "Log-in" button? If so, then your proxy is using a custom
authentication scheme that WinHTTP does not support.

In the first trace, the request got redirected to another web page (via the
302 response). WinHTTP followed the redirect to the "IssueLogin" web page
and received a "200 OK" response. From WinHTTP's point-of-view, your
original GET request had succeeded at that point. If the proxy returns an
HTML log-in web page, WinHTTP will not parse the HTML to figure out how to
respond to the authentication. So you cannot use WinHTTP with this proxy
server.

Stephen

[DerekG]

Post by Stephen Sulzer
When you preauthenticate with the proxy server on its log-in web page, do
you fill in your username and password directly into the web page and click
some kind of "Log-in" button? If so, then your proxy is using a custom
authentication scheme that WinHTTP does not support.

Yes, there is a log-in button on a web page.

Post by Stephen Sulzer
In the first trace, the request got redirected to another web page (via the
302 response). WinHTTP followed the redirect to the "IssueLogin" web page
and received a "200 OK" response. From WinHTTP's point-of-view, your
original GET request had succeeded at that point. If the proxy returns an
HTML log-in web page, WinHTTP will not parse the HTML to figure out how to
respond to the authentication. So you cannot use WinHTTP with this proxy
server.

Figures... my lady luck could use an extreme makeover.

Thanks for all of your help; you have been fantastic with helping me
get through this.

Thanks,
dg