Bug 57106 - [Client Certificates] "System.Net.WebException: The request timed out" in iOS after Xamarin update
Summary: [Client Certificates] "System.Net.WebException: The request timed out" in iOS...
Status: ASSIGNED
Alias: None
Product: iOS
Classification: Xamarin
Component: BCL Class Libraries (show other bugs)
Version: XI 10.10 (d15-2)
Hardware: PC Windows
: --- normal
Target Milestone: Future Cycle (TBD)
Assignee: Martin Baulig
URL:
Depends on: 58891
Blocks:
  Show dependency tree
 
Reported: 2017-06-02 13:59 UTC by Emiliano
Modified: 2017-12-01 01:13 UTC (History)
12 users (show)

See Also:
Tags: Shiproom
Is this bug a regression?: Yes
Last known good build: XI 10.6 with MonoTLS


Attachments
Test Solution (14.46 KB, application/x-zip-compressed)
2017-06-02 14:06 UTC, Emiliano
Details

Description Emiliano 2017-06-02 13:59:26 UTC
Hi all,
We have three apps that stopped working in the iOS version.
After the latest update Xamarin.iOS (10.10), when the app tries to connect the backend (HttpWebRequest with X509Certificate2 certificate), it stays freeze for a while and then I got a time out. I also have the app for Android but that one still working fine.
In our previous version, we had set Mono (managed) SSL/TSL but according to the release notes it was removed on 10.8 and it only supports AppleTLS.
I've upgraded the protocol support at our backend (Forefront TMG) so now we also have TLS 1.1 and 1.2 with forward secrecy (I've also been changing the cipher suites order) but I can't trace the request in the backend, I only get the timeout after a while waiting. We need a clue about what is not working on this communication after Mono TLS Removal.
Pls, find attached a test solution. We need some quick help because we need to deploy some changes and we can't because of it.

Thank you!

Emiliano
Comment 1 Emiliano 2017-06-02 14:06:46 UTC
Created attachment 22653 [details]
Test Solution
Comment 2 Manuel de la Peña 2017-06-07 10:10:13 UTC
Hello,

Can you please provide the full environment information used to build the failing version? That would include the mono version and the xamarin version.

Easiest way to get exact version information:
 - On Visual Studio for Mac: "Visual Studio" menu, "About Visual Studio" item, "Show Details" button.
 - On Visual Studio for Windows: "Help menu", "About Microsoft Visual Studio" item.
 Then copy/paste the version information (you can use the "Copy Information" button).
Comment 3 Ammar Mheir 2017-06-12 16:12:20 UTC
Here is the version information, I am currently providing assistance and was able to gather some data after some troubleshooting: 

##Version Information

=== Xamarin Studio Community ===

Version 6.3 (build 864)
Installation UUID: 5188a613-2045-4b42-b1ad-0a4d4c3ddaf2
Runtime:
	Mono 5.0.1.1 (2017-02/5077205) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 500010001

=== NuGet ===

Version: 3.5.0.0

=== Xamarin.Profiler ===

'/Applications/Xamarin Profiler.app' not found

=== Apple Developer Tools ===

Xcode 8.2.1 (11766.1)
Build 8C1002

=== Xamarin.iOS ===

Version: 10.10.0.36 (Xamarin Studio Community)
Hash: d2270eec
Branch: d15-2
Build date: 2017-05-22 16:30:53-0400

=== Xamarin.Android ===

Version: 7.3.1.2 (Xamarin Studio Community)
Android SDK: /Users/lagash/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		4.4 (API level 19)
		6.0 (API level 23)
		7.1 (API level 25)

SDK Tools Version: 25.2.5
SDK Platform Tools Version: 25.0.4
SDK Build Tools Version: 23.0.2

Java SDK: /usr
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Xamarin.Mac ===

Version: 3.4.0.36 (Xamarin Studio Community)

=== Xamarin Inspector ===

Not Installed

=== Build Information ===

Release ID: 603000864
Git revision: 6c2f6737278ccc3e81e12276d49c0d92f975f189
Build date: 2017-04-24 11:26:01-04
Xamarin addins: d8d46e577d8507c35260ce9d73df3c33415bb214
Build lane: monodevelop-lion-d15-1

=== Operating System ===

Mac OS X 10.11.6
Darwin la-MacBook-Pro-de-Lagash.local 15.6.0 Darwin Kernel Version 15.6.0
    Tue Apr 11 16:00:51 PDT 2017
    root:xnu-3248.60.11.5.3~1/RELEASE_X86_64 x86_64

=== Enabled user installed addins ===

Gorilla Player 0.9.1.2


##Additional Information

Even when switching to a different HttpClient implementation such as NSUrlSession we get the same error:

>System.Net.WebException: The request timed out
>  at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00059] in /Library/Frameworks/Xamarin.iOS.framework/Versions/10.10.0.36/src/mono/mcs/class/System/System.Net/HttpWebRequest.cs:1031 
>  at System.Net.HttpWebRequest.GetResponse () [0x0000e] in /Library/Frameworks/Xamarin.iOS.framework/Versions/10.10.0.36/src/mono/mcs/class/System/System.Net/HttpWebRequest.cs:1043 
>  at HttpsRequestsiOS.ViewController.MakeRequest () [0x00095] in /Users/lagash/Desktop/HttpsRequestsiOS/HttpsRequestsiOS 3/HttpsRequestsiOS 2/HttpsRequestsiOS/ViewController.cs:63
Comment 5 Ammar Mheir 2017-06-16 21:32:54 UTC
I have downloaded and downgraded Xamarin Studio to version 6.2.1 along with Xamarin.iOS to version 10.6.

After this, the menu option in iOS Build settings for SSL/TLS implementation has reappeared again and I set the TLS implementation to MonoTLS keeping the HttpClient implementation on Managed. 

After rebuilding, the connection completes successfully. However switching the TLS implementation back to AppleTLS results in the same error, Request Timed Out.
Comment 6 Marek Safar 2017-07-25 14:48:48 UTC
Martin, this looks like serious regression could you please investigate the underlying issue ASAP.
Comment 7 Xavier 2017-10-06 13:17:58 UTC
I'm having this issue too in Xamarin.Mac app. Any news on this?

I've been building my app against Mono 4.8.1 for a couple of months now because of this.
Comment 8 Ammar Mheir 2017-10-09 23:21:56 UTC
@Xavier

I just wanted to confirm with if you are also adding the client certificate via HttpWebRequest.ClientCertificate.Add(). 

We are creating our X509Certificate2 and adding it this way, and latest Mono versions will cause the web request to time out. Just wanted to see if you are seeing this same issue using the same functionality.
Comment 9 Xavier 2017-10-10 12:18:55 UTC
I might have been a bit to fast on this. I'm not sure I have the exact same problem and more of the same symptoms. I'm sorry if I'm confusing issues.

I've explained more in detail my issue on a forum post here: https://forums.xamarin.com/discussion/104118/timeout-exception-httpwebrequest-getresponse-in-mono-5-x#latest
Comment 10 Brendan Zagaeski (Xamarin Support) 2017-11-22 11:29:39 UTC
I have been taking a look at this issue to help provide more background information about why the new `MobileAuthenticatedStream` [1] + `AppleTlsContext` [2] implementation handles client certificates differently compared to the older `SslStreamBase` [3] implementation (from Xamarin.iOS 10.6 and lower).  In short, the problem is that the new implementation does not yet support scenarios where the server requests a new handshake (to change the TLS parameters) after the initial successful handshake.  I'll describe a few scenarios I have worked through to get a better understanding of this limitation.  I'll also plan to follow-up within the next few days to add a few notes about feature planning on the topic.



### Scenario: Client certificates requested during the _initial_ handshake work correctly.

For example, I set up a test Apache server configured to require client certificates (`SSLVerifyClient require` [4]) server-wide (as opposed to for a specific subset of pages).  In that case, Apache sends the "CertificateRequest" [5] message as part of the _initial_ handshake, _before_ the connection is encrypted, and Mono successfully replies with the client certificate.

This server-side settings change could be used as a workaround, but it might not be appropriate in all cases.  And of course, it's only possible when the server software allows it.  It looks like IIS can be configured to require client certificates site-wide (https://technet.microsoft.com/library/cc753983.aspx), but I haven't checked yet to see if that means the "CertificateRequest" gets sent during the initial handshake.



### Scenario: Re-handshaking is not supported yet, regardless of whether it is for client certificates or another scenario like cipher suite changes.


- Renegotiations for client certificates fail.

I moved the `SSLVerifyClient require` Apache setting so that it would apply to just one page (such as /index.html) instead of the whole server.  In that configuration, Apache no longer requests a client certificate during the _initial_ handshake.  Instead it waits until _after_ the connection is encrypted and the client has sent the HTTP "GET" request.  Apache then replies with a hello "HelloRequest" [6] message.  The old `SslStreamBase` implementation was able to identify this "HelloRequest" message and re-run the handshake process.  But the new implementation does not re-run the handshake (`AppleTlsContext.StartHandshake()`).  As usual, tools like Microsoft Message Analyzer [7] and Wireshark [8] are handy for inspecting these TLS protocol messages.  For example, Wireshark shows that the last packet from the server in this case is an "Encrypted Handshake Message", and the client simply never replies.  (If you set up a test server with a self-signed certificate and configure Wireshark to use the private key [9], Wireshark can decrypt that "Encrypted Handshake Message" and show it as a "Hello Request".  Message Analyzer works similarly [10].)

Unfortunately, the fix isn't as simple as just adding another call to `StartHandshake()`.  I'll plan to include a little more information about that in my next comment.


- Renegotiations to change cipher suites also fail, in the same way.

I used the following steps to test this scenario with Apache:

1. Disable client certificates (by removing the `SSLVerifyClient` setting).
2. Set `SSLCipherSuite AES128-SHA` server-wide.
3. Set `SSLCipherSuite AES256-SHA` for a single page.
4. Attempt to access the page from step 3 via an `HttpWebRequest` in Mono (on macOS), Xamarin.iOS, or Xamarin.Mac.

As in the client certificate scenario, the packet trace for this scenario stops at the encrypted "Hello Request" message, Mono never re-runs the handshake, and then the connection eventually times out.




## References 

[1] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs
[2] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.AppleTls/AppleTlsContext.cs
[3] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslStreamBase.cs
[4] https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslverifyclient
[5] Section "7.3.  Handshake Protocol Overview" on https://www.ietf.org/rfc/rfc5246.txt
[6] Section "7.4.1.1.  Hello Request" on https://www.ietf.org/rfc/rfc5246.txt
[7] https://technet.microsoft.com/library/jj649776.aspx
[8] https://www.wireshark.org/
[9] https://wiki.wireshark.org/SSL#SSL_dissection_in_Wireshark
[10] https://technet.microsoft.com/library/dn727244.aspx
Comment 11 Brendan Zagaeski (Xamarin Support) 2017-12-01 01:13:40 UTC
Here are a few additional notes to fill in a bit of context around feature planning for this issue.

> the new implementation does not re-run the handshake
> (`AppleTlsContext.StartHandshake()`)
It turns out `StartHandshake()` isn't exactly the missing piece.  The handshake currently gets stuck _after_ Apple's native Secure Transport `SSLRead()` function has already recognized the "HelloRequest".  At that point, `SSLRead()` _tries_ to perform the next handshake step by giving Mono an encrypted "ClientHello" message to send back to the server.  The issue is that rather than writing the "ClientHello" to the server, Mono currently discards it.  (To give a tiny extra bit of technical flavor about this behavior, the `MobileAuthenticatedStream.InternalWrite()` method currently specifically checks that the connection mode is an `SSLWrite()` or `SSLHandshake()` [2], so it discards the message that comes from `SSLRead()` [3].)  After `SSLRead()` has "sent" the "ClientHello" message, it goes back to trying to read the next reply from the server, but of course since the server is actually still waiting to receive the "ClientHello" message, nothing else happens, and the connection eventually times out.

In the end, altering `InternalWrite()` to handle messages from `SSLRead()` the same way as it handles messages from `SSLWrite()` might be a valid option, but it will need careful consideration.  The full renegotiation process involves some other steps too, and care will need to be taken with attention to security to ensure that none of the new steps break any assumptions of the existing code.  That work is planned, but it is not yet ready to be assigned to a specific release, so I will leave the current "Target Milestone" of this bug as "Future Cycle (TBD)" for the moment.  I will plan to check back on this bug during the first quarter of 2018 as the plans settle into place.

(On a side note, just to mention one example "historical" subtlety involving renegotiation, server-client pairs are recommended to use the TLS "Renegotiation Indication Extension" to avoid CVE-2009-3555 [4].)




## References

[1] https://developer.apple.com/documentation/security/1394324-sslread
[2] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs#L464
[3] https://github.com/mono/mono/blob/3e1eeec6c3373b38688b714f354da72dc2d92247/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs#L477
[4] https://blogs.technet.microsoft.com/srd/2010/08/10/ms10-049-an-inside-look-at-cve-2009-3555-the-tls-renegotiation-vulnerability/

Note You need to log in before you can comment on or make changes to this bug.