Bug 20321 - HttpWebRequest on https with preAuthentication and a given http proxy removes auth header
Summary: HttpWebRequest on https with preAuthentication and a given http proxy removes...
Alias: None
Product: Class Libraries
Classification: Mono
Component: System ()
Version: 3.4.0
Hardware: All All
: --- normal
Target Milestone: Untriaged
Assignee: Martin Baulig
Depends on:
Reported: 2014-06-03 17:50 UTC by Till Lorentzen
Modified: 2016-11-11 09:47 UTC (History)
2 users (show)

Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Till Lorentzen 2014-06-03 17:50:50 UTC
Requesting a Web Resource with https over a http proxy server and enabled preAuthentication failed:

using System;
using System.Net;
using System.Text;


IWebProxy proxy = new WebProxy("http://proxy.example.com:8080/");
proxy.Credentials = new NetworkCredential("user", "password");
WebRequest.DefaultWebProxy = proxy;

HttpWebRequest conn = (HttpWebRequest)WebRequest.Create("https://example.com");
conn.Method = "GET";
conn.PreAuthenticate = true;
conn.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.GetEncoding("utf-8").GetBytes("user:password"));
HttpWebResponse response = (HttpWebResponse)conn.GetResponse();

=> Unauthorized Exception is thrown if the https server needs basic auth

If PreAuthentication is false it works but produces more traffic and delay if I am understanding the PreAuthentication mechanism correctly.

Reproduced on Platforms: Ubuntu (mono version 3.2.8) and MacOS (mono version 3.4.0)

It works fine on Windows without mono.
Comment 1 Martin Baulig 2014-06-04 11:50:22 UTC
Well, you did misunderstand what PreAuthentication does - which is really not surprising, considering how misleading and confusing the name of that property is.  There is also only one slight hint on what it does in the MSDN docs, you need to google for it (especially in combination with "first request") to find out the full story.

What PreAuthenticate actually does is caching and reusing a successful auth token on _subsequent_ requests (to the same server using the same url or a subpath of it), but _not_ on the first request.  However, this can only work if you actually let the http stack do the authentication and - from the http stack's perspective - a successful authentication can only happen after the usual multi-way handshake.

Manually setting the "Authorization" header should only be done to work around broken servers which somehow expect that header on the first request instead of doing the usual handshake.  Of course, this can only work with Basic authentication.  If you do this, then from the http stack's point of view, the connection is unauthenticated (no handshake has happened), so it will not add the header to the next request.

In theory, sending the authorization header on the first request would work when using Basic authentication, but after reading your bug report, I finally understand why MS may have decided against that: there must be some kind of a penalty for those who attempt to abuse Basic authentication as a means to gain a tiny bit of performance.

If you attempt to use PreAuthenticate (without manually adding that header) on Windows, you'll see that it will actually do the normal auth challenge on the first request and then reuse it on all subsequent ones.  This is currently broken on Mono, which I just discovered yesterday by accident while working on another bug.

To solve your problem, first you need to decide whether you want to use PreAuthenticate or manually add the "Authorization" header on each request - but don't do both.

Manually set that header if you either have a broken server that insists on seeing it on the first request or if you really want to trade security for a tiny speed improvement.

Otherwise, if you need to make multiple requests to the same server, use PreAuthenticate and let the http stack take care of the authentication for you.

Note that PreAuthenticate (which should really be called CacheAuthentication or CacheAndReuseAuthentication) also works with the more secure authentication mechanisms such as Digest or NTLM when using HTTP 1.1 and persistent connections.  This will do the usual multi-way handshake on the first request, then keep the connection open and reuse the session's auth header for all subsequent requests.

This may not currently work as expected on Mono, but it's on my todo list.

As a general principle, Basic authentication should be avoided whenever possible - even when using SSL.

While the SSL connection itself is encrypted, Basic authentication needs to add your password in plain text to the http request - which means that your password must be in memory.

On Windows, you can use the NetworkCredential constructor that takes a SecureString to encrypt your password in memory.  When using Digest or NTLM authentication, the networking stack will then only decrypt your password while it needs it to compute the challenge response - so your unencrypted password will only be in your application's memory for a very short period of time.  When using Basic authentication, on the other hand, your clear-text password goes into the HTTP headers, which are normal strings - and normal strings are immutable after they're created and also can't be explicitly garbage collected - thus making it much more easy for an attacker to retrieve your plain-text password from your application's memory.

Unfortunately, we don't support encrypted SecureString in Mono yet, but we might do in future.

The networking stack will also default to using the most secure authentication method that's available - so if the server eventually gets updated to support Digest / NTLM, the networking stack will automatically use it.
Comment 2 Till Lorentzen 2014-06-05 11:46:35 UTC
Thanks a lot for this detailed and fast answer!

I think you could and should downgrade the importance to low.

I fixed the problem by changing the library I used to use a solution like this:

IWebProxy proxy = new WebProxy("http://proxy.example.com:8080/");
proxy.Credentials = new NetworkCredential("user", "password");
WebRequest.DefaultWebProxy = proxy;

HttpWebRequest conn = (HttpWebRequest)WebRequest.Create("https://example.com");
conn.Method = "GET";
conn.PreAuthenticate = true;
conn.Credentials = new NetworkCredential("user2", "password2");
HttpWebResponse response = (HttpWebResponse)conn.GetResponse();
Comment 3 Martin Baulig 2014-06-05 13:28:59 UTC
Cool!  Your new code will now also automatically use a more secure authentication method such as NTLM or Digest if the server supports it.

I'll keep the bug open till I'm done with my new authentication-related web tests.
Comment 4 Martin Baulig 2016-11-11 09:47:23 UTC
I believe this one has been fixed a long time ago.