Bug 52112 - The user defined web proxy is not used by the HttpClientHandler
Summary: The user defined web proxy is not used by the HttpClientHandler
Status: RESOLVED NOT_ON_ROADMAP
Alias: None
Product: Xamarin.Mac
Classification: Desktop
Component: Library (Xamarin.Mac.dll) (show other bugs)
Version: Master
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: Future Cycle (TBD)
Assignee: Marek Safar
URL:
Depends on:
Blocks:
 
Reported: 2017-01-31 18:32 UTC by Xavier Rigau
Modified: 2018-03-19 19:53 UTC (History)
13 users (show)

Tags: xamexttriage xamextfollowed
Is this bug a regression?: ---
Last known good build:


Attachments
Actual HTTP responses (105.13 KB, image/png)
2017-01-31 18:32 UTC, Xavier Rigau
Details
Xamarin Studio errors (116.78 KB, image/png)
2017-02-02 21:30 UTC, Xavier Rigau
Details
Log files for the 3 different handlers. (94.23 KB, application/x-zip-compressed)
2018-03-19 16:54 UTC, Xavier Rigau
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NOT_ON_ROADMAP

Description Xavier Rigau 2017-01-31 18:32:39 UTC
Created attachment 19668 [details]
Actual HTTP responses

In Mac OS X when a user is behind a web proxy the HTTP requests issued by HttpClientHandler don't work. They don't use the proxy settings and either timeout or if there is any server intercepting them you get the reply from the wrong host. For instance, in HP we use an automatic configuration script.

xMacMini:~ xavierr$ scutil --proxy
<dictionary> {
  ExceptionsList : <array> {
    0 : *.local
    1 : 169.254/16
  }
  FTPPassive : 1
  HTTPEnable : 0
  ProxyAutoConfigEnable : 1
  ProxyAutoConfigURLString : http://autocache.hp.com
}

I reported it https://forums.xamarin.com/discussion/comment/249460 but I only got suggestions for workarounds.

-There is plenty documentation about the limitations of the Mono HTTP client.
-Xamarin Studio installer suffers the same problem and doesn't install behind my firewall
-The code we use in our app works untouched in .NET framework and WinRT in Windows.

The post done from inside the corporate network return a HTML response as in the attached picture.
Comment 2 Xavier Rigau 2017-02-02 21:30:43 UTC
Created attachment 19703 [details]
Xamarin Studio errors
Comment 3 Xavier Rigau 2017-02-02 21:31:03 UTC
Comment on attachment 19703 [details]
Xamarin Studio errors

-Xamarin Studio shows "Untrusted HTTP certificate detected" when it starts up behind a web proxy (wired)
-It doesn't if there is no proxy (wireless)

See 2nd attachment
Comment 4 Xavier Rigau 2017-02-02 21:31:10 UTC
-Xamarin Studio shows "Untrusted HTTP certificate detected" when it starts up behind a web proxy (wired)
-It doesn't if there is no proxy (wireless)

See 2nd attachment
Comment 5 Marek Safar 2017-02-06 14:46:49 UTC
Did you try to use CFNetworkHandler with .UseSystemProxy set to true ?

@chamons, I think the XS settings is quite misleading as it mentions iOS 6+ but no Mac.
Comment 6 Chris Hamons 2017-02-06 14:50:59 UTC
Good catch on IDE text, adding Vincent to let him know about that.
Comment 7 Vincent Dondain [MSFT] 2017-03-14 22:17:27 UTC
Not sure what's misleading in the XS setting, what are we talking about? Screenshot maybe?
Comment 8 Marek Safar 2017-03-23 09:58:44 UTC
Vincent, the handler settings indicates the handler can be used with iOS only which is not true
Comment 10 Marek Safar 2017-06-01 21:18:21 UTC
Still waiting on response whether https://bugzilla.xamarin.com/show_bug.cgi?id=52112#c5 fixes the issue for customer
Comment 11 Vincent Dondain [MSFT] 2017-06-07 20:47:19 UTC
@xavier can you please try to use CFNetworkHandler with .UseSystemProxy set to true?

Like Marek recommended, otherwise if you're not experiencing this issue anymore please close the bug.
Comment 12 Xavier Rigau 2017-06-07 21:04:06 UTC
No. Because: 

1) This won't fix the errors in Xamarin Studio/installer/updater.
2) The same code works with Windows HTTPS stack
3) This bug is not unique to HP or custom proxies; it is a Xamarin shortcoming. Xamarin's customers should not have to deal with this issue; fixing it is for the benefit of MS, HP and the Xamarin product.

The mono HTTP stack is just partially implemented see the code here https://github.com/mono/mono/blob/effa4c07ba850bedbe1ff54b2a5df281c058ebcb/mcs/class/System/System.Net/MacProxy.cs#L910 


switch (proxies[i].ProxyType) {
CFProxyType.AutoConfigurationUrl:
// unsupported proxy type (requires fetching script from remote url)
break;

case CFProxyType.SOCKS:
// unsupported proxy type, try the next one
break;


For more context speak with Rob DeRosa.
Comment 13 Miguel de Icaza [MSFT] 2017-06-09 20:13:34 UTC
Question, what Http client handler are you using?

The one in the link above for Mono's HTTP stack only comes into play when using the Mono HttpWebRequest.

On iOS/Mac, we support both a CoreFoundat-based handler for older systems, and an NSUrlSession based one, which is Apple's recommended API for HTTP downloads.

Does using the System.Net.Http.NSUrlSessionHandler solve this problem?
Comment 14 Xavier Rigau 2017-06-22 19:58:14 UTC
Happens with CFNetwork, NSUrlSession and default build settings. Please note that as a library developer I don't have control of this build setting since it is only for apps.
Comment 15 Chris Hamons 2017-06-23 15:25:00 UTC
Xavier - Can you post a snipped of code showing how you are invoking the networking code in question. There are a number of different classes / ways of doing it, and they may trigger different behaviors.
Comment 17 Alex Soto [MSFT] 2018-03-16 18:46:52 UTC
Unfortunately by doing this as a library creator will not honor the user's settings for CFNetworkHandler or NSUrlSessionHandler because you are forcing it into mono's HttpClientHandler.

> var handler = new HttpClientHandler();
> //...
> var request = new System.Net.Http.HttpClient(handler);

As a library creator the recommended ways to deal with this is to either use the System.Net.HttpClient's empty constructor throughout your library or if you really need to use a `ClientHandler` you must explicitly use the CFNetworkHandler or NSUrlSessionHandler for iOS and some code sharing technique [1] or the new multi target approach [2].

By using the CFNetworkHandler or the NSUrlSessionHandler your library will gain great benefits that the current mono does not provide like transparent connection switching from Wi-Fi to cellular, Apple’s native TLS implementation which provides full TLS 1.2+ support, a greater network reliability since you are using Apple's native stack and of course the Proxy system settings.

[1]: https://docs.microsoft.com/en-us/xamarin/cross-platform/app-fundamentals/code-sharing
[2]: https://montemagno.com/new-plugin-for-xamarin-multi-target-templates-for-visual-studio-2017/
Comment 18 Xavier Rigau 2018-03-19 16:51:51 UTC
The need to use CFNetworkHandler or the NSUrlSessionHandler in PCL code is absurd since it brakes the multi-platform premise. Alex, are you saying that Xamarin needs a platform specific HTTP client to work behind proxies? I think is way more reasonable to fill the HTTP client Mac OS X implementation than to ask Xamarin client to write platform specific code. 

This bug bottom line is the unwillingness to fix the HTTP mono stack for Mac OS X; nothing else ;) 

The code works well .NET and UWP as is. 

Anyhow, I removed the handler from the constructor and now the code works with NSUrlSession but not with CFNetworkHandler nor Managed.

I don't feel comfortable forcing all the client from a library to use the NSUrlSession handler.
Comment 19 Xavier Rigau 2018-03-19 16:54:03 UTC
Created attachment 26220 [details]
Log files for the 3 different handlers.
Comment 20 Manuel de la Peña [MSFT] 2018-03-19 17:58:25 UTC
For what I can understand:

1. @Xavier is developing a library to be used by clients.
2. The library has to be able to perform Http request behind proxies if those exist. 
3. The sample code does the following:

    var handler = new HttpClientHandler();
    new System.Net.Http.HttpClient(handler);

As I see it, the problem, is that the assumption that every platform has all the capabilities as others is a misconception, for example, if we were writing a library that does watch for file system events, the approaches between the linux/mac/windows implementation would be completely different, and something similar happens in the network stack.

From the Xamarin perspective, the only way we can ensure that the HttpClient request perform correctly is via the Handlers that Alex mentioned. The HttpClient library is designed to work correctly via following the IoC or dependency injection allowing to pass the correct handler per platform.

Writing code that will choose the handler for the client would probably mean that we would confront problems in the future and since we support several platforms, probably a maintenance nightmare. 

I would follow @Alex recommendations, that is, pass the correct handler to use in the platform BUT to avoid imposing the handler to the library user, as @Xavier mentioned, I would refactor the code not to create the HttpClient for the user library, but to do a dependency injection of the HttpClient to use. It is a simple change from

private System.Net.Http.HttpClient MakeRequestObj([CanBeNull] IEnumerable<KeyValuePair<string, string>> headers)

to 

private void MakeRequestObj(System.Net.Http.HttpClient request, [CanBeNull] IEnumerable<KeyValuePair<string, string>> headers)

This solves some of the issue we face:

1. Does not impose the HttpClient to the user. Is not only that we are requesting you to change the handler per platform, but you are IMPOSING an implementation
to your user. That is, you have decided in advance to use the default handler, in this case, the user won't be able to choose a different handler of their choice, could be one of the handlers in Xamarin/Mono but even a handler of their own.
2. Creating the object in methods makes testing hard.
3. For what you are trying to achieve, it might be better to use a pattern similar to the one used in the HttpClientFactory or DelegatingHandler.

@Xavier I find the following read very useful to understand the HttpClient pipeline and how to take advantage of it in this kind of situations:

https://www.thomaslevesque.com/2016/12/08/fun-with-the-httpclient-pipeline/

You can use either the HttpClientFactory to create a httpclient with the headers etc.. or think of a nice way to use the DelegatingHandler. That will make your library more testable, more user friendly and no one, nor you, not Xamarin/Mono will impose a Handler to you library user, meaning that the will be able to write unit testable code with your library, otherwise, the way you have engineered the library, it looks like they will not be able to pass a mock handler for testing.

Regarding the issue with the handler CFNetworkHandler not picking the proxy settings, can you please show me how are you constructing the CFNetworkHandler and Managed ones.
 I have seen bug request similar and AFAIK something like the following has to be done:

https://stackoverflow.com/questions/23492126/http-traffic-monitoring-issue-when-using-monotouch-httpclient-and-charles-proxy

If you wanted to use the managed one, you would have to do something like:

var handler = new HttpClientHandler {
    Proxy = CFNetwork.GetDefaultProxy (),
    UseProxy = true,
};
var client = new HttpClient(handler);

Or even worse, you can have a proxy per URI, which means that you would have to do some logic like the following:

HttpClient client;
var proxies = CFNetwork.GetProxiesForURL(url, null); // we can have a proxy per url
if (proxies.Length > 0)
{
    var httpClientHandler = new HttpClientHandler{
	Proxy = new WebProxy($"{proxies[0].HostName}:{proxies[0].Port}", false),
	UseProxy = true
    };
    client = new HttpClient(httpClientHandler);
} else {
    client = new HttpClient();
}



Add Comment
Comment 21 Xavier Rigau 2018-03-19 18:44:46 UTC
This is a lot of academical and philosophical reasons to not admit that the issue is lack of support in the mono stack for Mac OSX:

The mono HTTP stack is just partially implemented see the code here https://github.com/mono/mono/blob/effa4c07ba850bedbe1ff54b2a5df281c058ebcb/mcs/class/System/System.Net/MacProxy.cs#L910 


switch (proxies[i].ProxyType) {
CFProxyType.AutoConfigurationUrl:
// unsupported proxy type (requires fetching script from remote url)
break;

case CFProxyType.SOCKS:
// unsupported proxy type, try the next one
break;

The philosophical argument is a slippery slope. By that reasoning anything that is hard to abstract in Xamarin can be punted back to 'this is platform specific'. 

The following arguments are wrong:

1) 'that the assumption that every platform has all the capabilities as others is a misconception' --> In this case they DO have the same capabilities of using HTTP behind a proxy.

2) 'watch for file system events, the approaches between the linux/mac/windows implementation would be completely different' --> Correct, that's is why you would expose a common API in Xamarin and hide the different implementation from the clients. 

3) If passing a handler to HttpClient was so bad why is it even possible? Leave only the default constructor in PCL!!! 

I will follow Alex suggestion and tell my clients to use NSUrlSession handler in their settings. 

It is disheartening that such obvious points are being missed by MSFT:
1) The code works as is on other pltaforms
2) There is no technical reason to not fix mono.
3) The sanctimonious reasons given are not pertinent and just diffuse the issue.
Comment 22 Manuel de la Peña [MSFT] 2018-03-19 18:56:24 UTC
> 1) 'that the assumption that every platform has all the capabilities as others is a 
> misconception' --> In this case they DO have the same capabilities of using HTTP 
> behind a proxy.

In this case, we are not talking about performing a request behind a proxy, we are talking about reading the default proxy from the platform. IT IS platform specific.

> 2) 'watch for file system events, the approaches between the linux/mac/windows 
> implementation would be completely different' --> Correct, that's is why you would
> expose a common API in Xamarin and hide the different implementation from the 
> clients.

Which is what we have, a common interface in the form of HttpMessageHandler implementations. All following the same interface.

> 3) If passing a handler to HttpClient was so bad why is it even possible? Leave only > the default constructor in PCL!!! 

I am missing a point, with this sentence. I am explaining that the handler should be passed.

Anyway, here is no need to discuss about a bug that is solvable passing the handler, Can you please provide the way in which you create the CFNetworkHandler and the ManageHandler to understand why they are not using the proxy? Or should we completely disregard the issue?
Comment 23 Alex Soto [MSFT] 2018-03-19 19:01:55 UTC
Hello Xavier

We really appreciate your comments, we have provided the solution that we think is best for our customers, this is not just an iOS/Mac issue, we also took the same approach in android with AndroidClientHandler, we think that each platform's Http stack will be the best option because of all the features it provides are already there and no need to be reimplemented.

That said we are also and Open Source project and if you think that the mono HTTP stack is just partially implemented please feel free to contribute, we will happily accept any contributions from anyone.
Comment 24 Xavier Rigau 2018-03-19 19:53:39 UTC
1) We are talking about not having to worry about proxies when doing a REST client using the HTTP client. As a REST client my code has no business dealing with proxies, certificates, etc as long as it sticks to the standard. We needed the handler to specify that we accepted GZIP compressed responses.

2) I think you are missing the point that the whole REST client is written in PCL code and has no idea of what platform is running on. Why should it?

3) The handler we were passing deal with the custom fact that we accept compressed responses. This is a PCL/netstandard code which should be able to work independent of the platform.

Passing a handler is NOT the solution for PCL code. Using the default constructor and changing the project setting did the trick. At the expense of using a custom handler for decompression.

var handler = new HttpClientHandler();
if (handler.SupportsAutomaticDecompression)
 handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

@Alex,
One of the reason why Xamarin is frown upon in my company is because HTTP client in iOS and Android. I am not on payroll to contribute to open source ;) I have been forced to go native with the library in all other OSes :(

Thanks for pointing out where the problem was.