Bug 52672 - WebHeaderCollection throws when NetStandard 1.3 library sets User-Agent
Summary: WebHeaderCollection throws when NetStandard 1.3 library sets User-Agent
Alias: None
Product: Class Libraries
Classification: Mono
Component: System ()
Version: 4.6.0 (C8)
Hardware: PC Windows
: --- normal
Target Milestone: Untriaged
Assignee: Marek Safar
Depends on:
Reported: 2017-02-22 17:02 UTC by David A. Sjøen
Modified: 2017-03-07 14:03 UTC (History)
4 users (show)

Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description David A. Sjøen 2017-02-22 17:02:40 UTC
ServiceStack's clients use HttpWebRequest. When using the .NET core version of their library in Xamarin Android, an exception is thrown:

System.ArgumentException: The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name
  at System.Net.WebHeaderCollection.ThrowOnRestrictedHeader (System.String headerName) [0x00021] in /Users/builder/data/lanes/4009/3d959b66/source/mono/mcs/class/referencesource/System/net/System/Net/WebHeaderCollection.cs:574 
  at System.Net.WebHeaderCollection.Set (System.String name, System.String value) [0x0001f] in /Users/builder/data/lanes/4009/3d959b66/source/mono/mcs/class/referencesource/System/net/System/Net/WebHeaderCollection.cs:705 
  at System.Collections.Specialized.NameValueCollection.set_Item (System.String name, System.String value) [0x00000] in /Users/builder/data/lanes/4009/3d959b66/source/mono/mcs/class/referencesource/System/compmod/system/collections/specialized/namevaluecollection.cs:337 
  at System.Net.WebHeaderCollection.set_Item (System.Net.HttpRequestHeader header, System.String value) [0x0001b] in /Users/builder/data/lanes/4009/3d959b66/source/mono/mcs/class/referencesource/System/net/System/Net/WebHeaderCollection.cs:288 
  at ServiceStack.NetStandardPclExport.Config (System.Net.HttpWebRequest req, System.Nullable`1[T] allowAutoRedirect, System.Nullable`1[T] timeout, System.Nullable`1[T] readWriteTimeout, System.String userAgent, System.Nullable`1[T] preAuthenticate) [0x0000a] in <3a9115e15bc9432a81ee49f65fe0d753>:0 

We first registered the issue with ServiceStack (https://github.com/ServiceStack/Issues/issues/520), but it turns out that User-Agent shouldn't be restricted in NetStandard 1.3. Thus, it seems the problem lies with Xamarin.
Comment 1 Marek Safar 2017-02-28 15:04:54 UTC
This is not Mono/Xamarin bug. NetStandard does not specified the implementation only API contract and in this case Xamarin behaves same as .net (if you run the code on .net you get same exception).

The best fix is to update to netstandard API which supports UserAgent property
Comment 2 David A. Sjøen 2017-02-28 15:53:25 UTC
Marek, what do you mean by the last sentence?
Comment 3 Marek Safar 2017-02-28 15:58:54 UTC
I don't know what's blocking ServiceStack to update to the latest netstandard (1.6 at the moment) and set the property directly.
Comment 4 David A. Sjøen 2017-02-28 16:05:38 UTC
I don't know either, but the way ServiceStack does it works in .NET Core, but not in Xamarin. I'm aware that the .NET version of WebHeaderCollection behaves the same way as Xamarin, but shouldn't it follow the behavior of .NET Core when called from a NetStandard DLL?
Comment 5 Marek Safar 2017-02-28 16:09:58 UTC
NetStandard is API contract the method can even throw NotSupportedException and it'll still comply be NetStandard compliant
Comment 6 David A. Sjøen 2017-02-28 17:08:33 UTC
Yes, but that wouldn't be very nice to the developers trying to use that method. :-)
Comment 7 Sergey Zhukov 2017-02-28 17:49:51 UTC
Marek, couple of questions:

1. Does Xamarin support NetStandard 1.3? If yes, should not Xamarin support setting up `User-Agent` header in `HttpWebRequest`?
2. Maybe I missed something, but where in NetStandard 1.6 `UserAgent` property is supported in `HttpWebRequest`? This sample code does not compile on netcoreapp1.1 platform (which is superset of NetStandard1.6):

            System.Net.HttpWebRequest req = null;
            req.UserAgent = "123";

error CS1061: 'HttpWebRequest' does not contain a definition for 'UserAgent' and no extension method 'UserAgent' accepting a first argument of type 'HttpWebRequest' could be found (are you missing a using directive or an assembly reference?)

Also, https://packagesearch.azurewebsites.net/ does not say anything about `UserAgent` property in `HttpWebRequest`

3. Does this statement true about Xamarin platform: "If you want to support iOS, Android and UWP, then NET Standard 1.4 is the highest you can use.". https://oren.codes/2016/07/09/using-xamarin-forms-with-net-standard/

If yes, that means that NetStandard 1.6 cannot be used to support all of these platforms anyway.

We try to keep lowest possible version of NetStandard to support as much platforms as possible, and we can create additional version of library, which is targeted to NetStandard 1.6, but currently I don't see how targeting library to .NetStandard 1.6 can help to resolve this particular issue.
Comment 8 Marek Safar 2017-03-02 15:47:57 UTC
Ad 1, Yes Xamarin supports all NetStandard versions.
Ad 2, My mistake I was checking NetStandard 2 instead of 1.6 where this property is not present.
Ad 3, I cannot speak of UWP and what happens when you try to run NetStandard 1.5+ or newer on it.

I checked netcore sources and it looks like they will match Xamarin and .NET behaviour with NetStandard 2.0. They will throw when you call Set with restricted members as done .NET and Xamarin with NetStandard 1.x.

Relevant ticket is https://github.com/dotnet/corefx/issues/12139
Comment 9 Sergey Zhukov 2017-03-02 16:23:57 UTC
We made a workaround to avoid this issue. At the first we trying to find `UserAgent` property by reflection, if not found only then trying to set up it on `User-Agent` header in collection. This should help in such situation, when `UserAgent` property is present in platform, but not available to compiler.
Comment 10 Marek Safar 2017-03-06 15:37:23 UTC
Hmm, that won't easily work as Xamarin linker can remove the property if not used directly anywhere else
Comment 11 Sergey Zhukov 2017-03-06 16:59:10 UTC
How can this occur? Property will be available via type metadata browsing, but when calling setter on it will not exist? What will be an exception in this case? MissingMethodException? 

Does not this mean if unused from the code properties of type can be removed by the linker then BinarySerialization and maybe other things won't work at all for types which properties are not referenced in the code?
Comment 12 Marek Safar 2017-03-06 17:17:05 UTC
The property won't be available, it'll be removed by Xamarin linker and yes linker is not very serialization friendly. Check https://developer.xamarin.com/guides/ios/advanced_topics/linker/ for more details
Comment 13 Sergey Zhukov 2017-03-07 07:15:27 UTC
So there are no ways to set `User-Agent` for Xamarin platform (except changing linker options)? Is this behaviour for IOS aot only or for Android property will be removed too?

So given for NetStandard 1.6:
1. Setting `User-Agent` header directly in `Headers` collection throws ArgumentException with the message that restricted headers must be set via corresponding property.
2. The property `UserAgent` is removed by linker if no referenced in the source code. It can't be referenced, because in NetStandard1.6 it is not exposed by API. So we can't set it via reflection, because it cannot be referenced.

Unbreakable cirle.
Comment 14 Marek Safar 2017-03-07 14:03:47 UTC
You only need to be careful and tell linker that it cannot remove the property because you are calling it via reflection