Bug 57106 - [Client Certificates] "System.Net.WebException: The request timed out" in iOS after Xamarin update
Summary: [Client Certificates] "System.Net.WebException: The request timed out" in iOS...
Alias: None
Product: iOS
Classification: Xamarin
Component: BCL Class Libraries (show other bugs)
Version: XI 10.10 (d15-2)
Hardware: PC Windows
: --- normal
Target Milestone: Future Cycle (TBD)
Assignee: Martin Baulig
Depends on: 58891
  Show dependency tree
Reported: 2017-06-02 13:59 UTC by Emiliano
Modified: 2018-01-10 19:42 UTC (History)
12 users (show)

Tags: Shiproom
Is this bug a regression?: Yes
Last known good build: XI 10.6 with MonoTLS

Test Solution (14.46 KB, application/x-zip-compressed)
2017-06-02 14:06 UTC, Emiliano

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report for Bug 57106 on Developer Community or GitHub if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: Developer Community HTML or GitHub Markdown
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:

Description Emiliano 2017-06-02 13:59:26 UTC
Hi all,
We have three apps that stopped working in the iOS version.
After the latest update Xamarin.iOS (10.10), when the app tries to connect the backend (HttpWebRequest with X509Certificate2 certificate), it stays freeze for a while and then I got a time out. I also have the app for Android but that one still working fine.
In our previous version, we had set Mono (managed) SSL/TSL but according to the release notes it was removed on 10.8 and it only supports AppleTLS.
I've upgraded the protocol support at our backend (Forefront TMG) so now we also have TLS 1.1 and 1.2 with forward secrecy (I've also been changing the cipher suites order) but I can't trace the request in the backend, I only get the timeout after a while waiting. We need a clue about what is not working on this communication after Mono TLS Removal.
Pls, find attached a test solution. We need some quick help because we need to deploy some changes and we can't because of it.

Thank you!

Comment 1 Emiliano 2017-06-02 14:06:46 UTC
Created attachment 22653 [details]
Test Solution
Comment 2 Manuel de la Peña [MSFT] 2017-06-07 10:10:13 UTC

Can you please provide the full environment information used to build the failing version? That would include the mono version and the xamarin version.

Easiest way to get exact version information:
 - On Visual Studio for Mac: "Visual Studio" menu, "About Visual Studio" item, "Show Details" button.
 - On Visual Studio for Windows: "Help menu", "About Microsoft Visual Studio" item.
 Then copy/paste the version information (you can use the "Copy Information" button).
Comment 3 Ammar Mheir 2017-06-12 16:12:20 UTC
Here is the version information, I am currently providing assistance and was able to gather some data after some troubleshooting: 

##Version Information

=== Xamarin Studio Community ===

Version 6.3 (build 864)
Installation UUID: 5188a613-2045-4b42-b1ad-0a4d4c3ddaf2
	Mono (2017-02/5077205) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 500010001

=== NuGet ===


=== Xamarin.Profiler ===

'/Applications/Xamarin Profiler.app' not found

=== Apple Developer Tools ===

Xcode 8.2.1 (11766.1)
Build 8C1002

=== Xamarin.iOS ===

Version: (Xamarin Studio Community)
Hash: d2270eec
Branch: d15-2
Build date: 2017-05-22 16:30:53-0400

=== Xamarin.Android ===

Version: (Xamarin Studio Community)
Android SDK: /Users/lagash/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		4.4 (API level 19)
		6.0 (API level 23)
		7.1 (API level 25)

SDK Tools Version: 25.2.5
SDK Platform Tools Version: 25.0.4
SDK Build Tools Version: 23.0.2

Java SDK: /usr
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

Android Designer EPL code available here:

=== Xamarin.Mac ===

Version: (Xamarin Studio Community)

=== Xamarin Inspector ===

Not Installed

=== Build Information ===

Release ID: 603000864
Git revision: 6c2f6737278ccc3e81e12276d49c0d92f975f189
Build date: 2017-04-24 11:26:01-04
Xamarin addins: d8d46e577d8507c35260ce9d73df3c33415bb214
Build lane: monodevelop-lion-d15-1

=== Operating System ===

Mac OS X 10.11.6
Darwin la-MacBook-Pro-de-Lagash.local 15.6.0 Darwin Kernel Version 15.6.0
    Tue Apr 11 16:00:51 PDT 2017
    root:xnu-3248. x86_64

=== Enabled user installed addins ===

Gorilla Player

##Additional Information

Even when switching to a different HttpClient implementation such as NSUrlSession we get the same error:

>System.Net.WebException: The request timed out
>  at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00059] in /Library/Frameworks/Xamarin.iOS.framework/Versions/ 
>  at System.Net.HttpWebRequest.GetResponse () [0x0000e] in /Library/Frameworks/Xamarin.iOS.framework/Versions/ 
>  at HttpsRequestsiOS.ViewController.MakeRequest () [0x00095] in /Users/lagash/Desktop/HttpsRequestsiOS/HttpsRequestsiOS 3/HttpsRequestsiOS 2/HttpsRequestsiOS/ViewController.cs:63
Comment 5 Ammar Mheir 2017-06-16 21:32:54 UTC
I have downloaded and downgraded Xamarin Studio to version 6.2.1 along with Xamarin.iOS to version 10.6.

After this, the menu option in iOS Build settings for SSL/TLS implementation has reappeared again and I set the TLS implementation to MonoTLS keeping the HttpClient implementation on Managed. 

After rebuilding, the connection completes successfully. However switching the TLS implementation back to AppleTLS results in the same error, Request Timed Out.
Comment 6 Marek Safar 2017-07-25 14:48:48 UTC
Martin, this looks like serious regression could you please investigate the underlying issue ASAP.
Comment 7 Xavier 2017-10-06 13:17:58 UTC
I'm having this issue too in Xamarin.Mac app. Any news on this?

I've been building my app against Mono 4.8.1 for a couple of months now because of this.
Comment 8 Ammar Mheir 2017-10-09 23:21:56 UTC

I just wanted to confirm with if you are also adding the client certificate via HttpWebRequest.ClientCertificate.Add(). 

We are creating our X509Certificate2 and adding it this way, and latest Mono versions will cause the web request to time out. Just wanted to see if you are seeing this same issue using the same functionality.
Comment 9 Xavier 2017-10-10 12:18:55 UTC
I might have been a bit to fast on this. I'm not sure I have the exact same problem and more of the same symptoms. I'm sorry if I'm confusing issues.

I've explained more in detail my issue on a forum post here: https://forums.xamarin.com/discussion/104118/timeout-exception-httpwebrequest-getresponse-in-mono-5-x#latest
Comment 10 Brendan Zagaeski (Xamarin Team, assistant) 2017-11-22 11:29:39 UTC
I have been taking a look at this issue to help provide more background information about why the new `MobileAuthenticatedStream` [1] + `AppleTlsContext` [2] implementation handles client certificates differently compared to the older `SslStreamBase` [3] implementation (from Xamarin.iOS 10.6 and lower).  In short, the problem is that the new implementation does not yet support scenarios where the server requests a new handshake (to change the TLS parameters) after the initial successful handshake.  I'll describe a few scenarios I have worked through to get a better understanding of this limitation.  I'll also plan to follow-up within the next few days to add a few notes about feature planning on the topic.

### Scenario: Client certificates requested during the _initial_ handshake work correctly.

For example, I set up a test Apache server configured to require client certificates (`SSLVerifyClient require` [4]) server-wide (as opposed to for a specific subset of pages).  In that case, Apache sends the "CertificateRequest" [5] message as part of the _initial_ handshake, _before_ the connection is encrypted, and Mono successfully replies with the client certificate.

This server-side settings change could be used as a workaround, but it might not be appropriate in all cases.  And of course, it's only possible when the server software allows it.  It looks like IIS can be configured to require client certificates site-wide (https://technet.microsoft.com/library/cc753983.aspx), but I haven't checked yet to see if that means the "CertificateRequest" gets sent during the initial handshake.

### Scenario: Re-handshaking is not supported yet, regardless of whether it is for client certificates or another scenario like cipher suite changes.

- Renegotiations for client certificates fail.

I moved the `SSLVerifyClient require` Apache setting so that it would apply to just one page (such as /index.html) instead of the whole server.  In that configuration, Apache no longer requests a client certificate during the _initial_ handshake.  Instead it waits until _after_ the connection is encrypted and the client has sent the HTTP "GET" request.  Apache then replies with a hello "HelloRequest" [6] message.  The old `SslStreamBase` implementation was able to identify this "HelloRequest" message and re-run the handshake process.  But the new implementation does not re-run the handshake (`AppleTlsContext.StartHandshake()`).  As usual, tools like Microsoft Message Analyzer [7] and Wireshark [8] are handy for inspecting these TLS protocol messages.  For example, Wireshark shows that the last packet from the server in this case is an "Encrypted Handshake Message", and the client simply never replies.  (If you set up a test server with a self-signed certificate and configure Wireshark to use the private key [9], Wireshark can decrypt that "Encrypted Handshake Message" and show it as a "Hello Request".  Message Analyzer works similarly [10].)

Unfortunately, the fix isn't as simple as just adding another call to `StartHandshake()`.  I'll plan to include a little more information about that in my next comment.

- Renegotiations to change cipher suites also fail, in the same way.

I used the following steps to test this scenario with Apache:

1. Disable client certificates (by removing the `SSLVerifyClient` setting).
2. Set `SSLCipherSuite AES128-SHA` server-wide.
3. Set `SSLCipherSuite AES256-SHA` for a single page.
4. Attempt to access the page from step 3 via an `HttpWebRequest` in Mono (on macOS), Xamarin.iOS, or Xamarin.Mac.

As in the client certificate scenario, the packet trace for this scenario stops at the encrypted "Hello Request" message, Mono never re-runs the handshake, and then the connection eventually times out.

## References 

[1] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs
[2] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.AppleTls/AppleTlsContext.cs
[3] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslStreamBase.cs
[4] https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslverifyclient
[5] Section "7.3.  Handshake Protocol Overview" on https://www.ietf.org/rfc/rfc5246.txt
[6] Section "  Hello Request" on https://www.ietf.org/rfc/rfc5246.txt
[7] https://technet.microsoft.com/library/jj649776.aspx
[8] https://www.wireshark.org/
[9] https://wiki.wireshark.org/SSL#SSL_dissection_in_Wireshark
[10] https://technet.microsoft.com/library/dn727244.aspx
Comment 11 Brendan Zagaeski (Xamarin Team, assistant) 2017-12-01 01:13:40 UTC
Here are a few additional notes to fill in a bit of context around feature planning for this issue.

> the new implementation does not re-run the handshake
> (`AppleTlsContext.StartHandshake()`)
It turns out `StartHandshake()` isn't exactly the missing piece.  The handshake currently gets stuck _after_ Apple's native Secure Transport `SSLRead()` function has already recognized the "HelloRequest".  At that point, `SSLRead()` _tries_ to perform the next handshake step by giving Mono an encrypted "ClientHello" message to send back to the server.  The issue is that rather than writing the "ClientHello" to the server, Mono currently discards it.  (To give a tiny extra bit of technical flavor about this behavior, the `MobileAuthenticatedStream.InternalWrite()` method currently specifically checks that the connection mode is an `SSLWrite()` or `SSLHandshake()` [2], so it discards the message that comes from `SSLRead()` [3].)  After `SSLRead()` has "sent" the "ClientHello" message, it goes back to trying to read the next reply from the server, but of course since the server is actually still waiting to receive the "ClientHello" message, nothing else happens, and the connection eventually times out.

In the end, altering `InternalWrite()` to handle messages from `SSLRead()` the same way as it handles messages from `SSLWrite()` might be a valid option, but it will need careful consideration.  The full renegotiation process involves some other steps too, and care will need to be taken with attention to security to ensure that none of the new steps break any assumptions of the existing code.  That work is planned, but it is not yet ready to be assigned to a specific release, so I will leave the current "Target Milestone" of this bug as "Future Cycle (TBD)" for the moment.  I will plan to check back on this bug during the first quarter of 2018 as the plans settle into place.

(On a side note, just to mention one example "historical" subtlety involving renegotiation, server-client pairs are recommended to use the TLS "Renegotiation Indication Extension" to avoid CVE-2009-3555 [4].)

## References

[1] https://developer.apple.com/documentation/security/1394324-sslread
[2] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs#L464
[3] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs#L477
[4] https://blogs.technet.microsoft.com/srd/2010/08/10/ms10-049-an-inside-look-at-cve-2009-3555-the-tls-renegotiation-vulnerability/
Comment 12 Emiliano 2018-01-10 18:00:14 UTC

I could not update the Xamarin version because this issue for the last 6 months. How longer should I wait? I cannot change the security mechanism that we already have in our apps.

Comment 13 Brendan Zagaeski (Xamarin Team, assistant) 2018-01-10 19:14:37 UTC
If it's a possibility for your use case, I would recommend adjusting the server side to require the client certificate during the initial negotiation rather than during renegotiation.  At this time, completing the new feature implementation for modern TLS renegotiation in the Mono's TLS stack is not targeted for a go-live release in the immediate future.  At the earliest, the feature could possibly be included by mid-to-late 2018.
Comment 14 Emiliano 2018-01-10 19:25:07 UTC
The server side is a reverse proxy that is configure for certificate pre-authentication. I don't see a way to adjust it when it requires the client certificate.
The hardest part is we used this configuration for years (we used to be Xamarin clients before MSFT) and stop working in a Xamarin update. Now I have to wait for another 6 months or more. Not good news at all.
Comment 15 Brendan Zagaeski (Xamarin Team, assistant) 2018-01-10 19:42:04 UTC
Apologies for the change in behavior between releases.  As one additional piece of information, it is perhaps worth noting that the old TLS renegotiation implementation never implemented the renegotiation indication extension [1, 2], so ideally from a security perspective, those old Mono APIs would have been considered obsolete as far back as 2009 [3].  (For example, Apache rejects renegotiations that do not use the renegotiation indication extension, so it has not been compatible with Mono for quite some time.  Ideally, Mono itself would have also output run time warnings for apps that called the renegotiation APIs starting back in 2009, but unfortunately that was never implemented.)  And of course adding a new implementation of renegotiation for the new TLS implementation is something that must be handled with care.

[1] https://tools.ietf.org/html/rfc5746
[2] Bug 18635
[3] https://nvd.nist.gov/vuln/detail/CVE-2009-3555