Bug 59793 - NSURLSessionConfiguration Discretionary property breaks redirects on iOS Simulators.
Summary: NSURLSessionConfiguration Discretionary property breaks redirects on iOS Simu...
Status: CONFIRMED
Alias: None
Product: iOS
Classification: Xamarin
Component: Xamarin.iOS.dll (show other bugs)
Version: XI 11.0 (xcode9)
Hardware: Macintosh Mac OS
: Normal minor
Target Milestone: Future Cycle (TBD)
Assignee: Manuel de la Peña [MSFT]
URL:
Depends on:
Blocks:
 
Reported: 2017-09-27 15:39 UTC by Chris Mays
Modified: 2018-03-26 14:40 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report for Bug 59793 on Developer Community or GitHub if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: Developer Community HTML or GitHub Markdown
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:
Status:
CONFIRMED

Description Chris Mays 2017-09-27 15:39:15 UTC
I am running into a potential Xamarin bug involving background NSURLSessionConfiguration. Apple recommends that you set the Discretionary property to true so that the system can control when transfers occur. Whenever I turn this on in a Xamarin project and the session encounters a redirect I get this error `The operation couldn't be completed. Invalid argument`. After a lot of debugging I created an exact replica native project to see if I encountered the same error. The native project successfully follows the redirect and performs the download. The really unsettling thing about this is when I attach to the Xamarin project through lldb it successfully follows the redirect and the download succeeds (maybe race condition?). I have two example projects available if anyone wants to try this out:
https://github.com/chrsmys/NSURLSession-Discretionary-Xamarin
https://github.com/chrsmys/NSURLSession-Discretionary-Native
Comment 1 Chris Mays 2017-09-27 15:53:35 UTC
=== Visual Studio Community 2017 for Mac ===

Version 7.1.5 (build 2)
Installation UUID: d20c7b96-88fc-40f7-9bc7-7346b6036c58
Runtime:
	Mono 5.2.0.224 (d15-3/14f2c81) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 502000224

=== NuGet ===

Version: 4.3.0.2418

=== .NET Core ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
	1.1.1
	1.0.4
SDK: /usr/local/share/dotnet/sdk/1.0.3/Sdks
SDK Version: 1.0.3
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/5.2.0/lib/mono/msbuild/15.0/bin/Sdks

=== Xamarin.Profiler ===

Version: 1.5.5
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Xamarin.Android ===

Version: 7.4.5.1 (Visual Studio Community)
Android SDK: /Users/chrismays/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		4.0.3 (API level 15)
		4.4   (API level 19)
		5.0   (API level 21)
		5.1   (API level 22)
		6.0   (API level 23)
		7.0   (API level 24)
		7.1   (API level 25)

SDK Tools Version: 25.2.5
SDK Platform Tools Version: 25.0.5
SDK Build Tools Version: 25.0.3

Java SDK: /usr
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Apple Developer Tools ===

Xcode 9.0 (13247)
Build 9A235

=== Xamarin.iOS ===

Version: 11.0.0.0 (Visual Studio Community)
Hash: 152b654a
Branch: xcode9
Build date: 2017-09-15 02:25:56-0400

=== Xamarin.Mac ===

Version: 3.6.3.3 (Visual Studio Community)

=== Xamarin Inspector ===

Version: 1.3.0
Hash: 8c298a5
Branch: 1.3-release
Build date: Thu, 14 Sep 2017 21:21:26 GMT
Client compatibility: 1

=== Build Information ===

Release ID: 701050002
Git revision: 7afedcaef8e7542e70e3cf8f9bdb26938b8c0876
Build date: 2017-09-15 08:39:58-04
Xamarin addins: 3262aadf811a18c12eac6742532d052b0139a808
Build lane: monodevelop-lion-d15-3-xcode9

=== Operating System ===

Mac OS X 10.12.6
Darwin 16.7.0 Darwin Kernel Version 16.7.0
    Thu Jun 15 17:36:27 PDT 2017
    root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64
Comment 2 John Miller [MSFT] 2017-09-27 16:07:36 UTC
I am able to reproduce using the projects provided. 

The Xamarin.iOS project output:

> 2017-09-27 11:52:00.179 Discrectionary[1714:313567] Error Downloading: Error Domain=NSPOSIXErrorDomain Code=22 "Invalid argument" UserInfo={_kCFStreamErrorCodeKey=22, _kCFStreamErrorDomainKey=1}

The Xcode obj-c project output:

> 2017-09-27 11:59:29.425863-0400 Discretionary-Native[1860:320227] Downloaded to location file:///Users/johnmiller/Library/Developer/CoreSimulator/Devices/E665DD79-E8A0-49C9-A3ED-76BA92EA140A/data/Containers/Data/Application/49173AA0-2993-4DA0-95A3-32E92EDF8122/Library/Caches/com.apple.nsurlsessiond/Downloads/com.NSChris.Discretionary-Native/CFNetworkDownload_I9jDYW.tmp
Comment 3 Manuel de la Peña [MSFT] 2017-12-15 14:24:50 UTC
Adding here certain information about the requests performed by the native app and the xamarin app:

Native App:

    Header information:
        GET /episodes/439bd7ab-177b-4552-a102-eca19b7e50fd.mp3 HTTP/1.1
        Host: rss.art19.com
        Accept: */*
        User-Agent: Discretionary-Native/1 CFNetwork/893.10 Darwin/17.2.0
        Accept-Language: en-us
        Accept-Encoding: gzip, deflate
        Connection: keep-alive

    Extra info:

         URL	http://rss.art19.com/episodes/439bd7ab-177b-4552-a102-eca19b7e50fd.mp3
         Status	Complete
         Response Code	302 Found
         Protocol	HTTP/1.1
         Method	GET
         Kept Alive	No
         Content-Type	text/html; charset=utf-8
         Client Address	127.0.0.1:54356
         Remote Address	rss.art19.com/52.85.219.143:80

Xamarin App:

    Header information:
        GET /episodes/439bd7ab-177b-4552-a102-eca19b7e50fd.mp3 HTTP/1.1
        Host: rss.art19.com
        Accept: */*
        User-Agent: Discrectionary/1.0 CFNetwork/893.14 Darwin/17.2.0
        Accept-Language: en-us
        Accept-Encoding: gzip, deflate
        Connection: keep-alive

    Extra info:
       URL	http://rss.art19.com/episodes/439bd7ab-177b-4552-a102-eca19b7e50fd.mp3
       Status	Complete
       Response Code	302 Found
       Protocol	HTTP/1.1
       Method	GET
       Kept Alive	No
       Content-Type	audio/mpeg; charset=utf-8
       Content-Type	text/html; charset=utf-8
       Client Address	127.0.0.1:53828
       Remote Address	rss.art19.com/52.85.219.43:80

Both request are nearly identical EXCEP for the content type! The xamarin app does use "audio/mpeg" while the native app uses "text/html".

I'll keep investigating to see if this difference could be the issue and when/why we are setting the content type.

Note (I edited the apps to allow using http so that we can see the request info).

More info: 

Xcode version: Version 9.2 beta (9C34b)
Xamarin env:

=== Visual Studio Community 2017 for Mac ===

Version Preview - Internal Dogfood (7.3.2 build 10)
Installation UUID: 8d12e55e-3489-463f-ac52-8cb4573c5a81
Runtime:
	Mono 5.8.0.39 (2017-10/99db2351979) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 508000039

=== NuGet ===

Version: 4.3.1.4445

=== .NET Core ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
	2.0.0
	1.1.2
	1.0.5
	1.0.0
SDK: /usr/local/share/dotnet/sdk/2.0.0/Sdks
SDK Versions:
	2.0.0
	1.0.4
	1.0.0-preview2-003121
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/5.8.0/lib/mono/msbuild/15.0/bin/Sdks

=== Xamarin.Profiler ===

Version: 1.6.0
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Xamarin.Android ===

Version: 8.1.0.24 (Visual Studio Community)
Android SDK: /Users/mandel/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		4.0.3 (API level 15)
		4.3   (API level 18)
		4.4   (API level 19)
		5.0   (API level 21)
		6.0   (API level 23)
		7.0   (API level 24)
		7.1   (API level 25)

SDK Tools Version: 25.2.5
SDK Platform Tools Version: 25.0.4
SDK Build Tools Version: 25.0.1

Java SDK: /usr
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Xamarin Inspector ===

Version: 1.3.2
Hash: 461f09a
Branch: 1.3-release
Build date: Tue, 03 Oct 2017 18:26:57 GMT
Client compatibility: 1

=== Apple Developer Tools ===

Xcode 9.2 (13772)
Build 9C40b

=== Xamarin.iOS ===

Version: 11.7.0.252 (Visual Studio Community)
Hash: 68b321df
Branch: gpath-make-copy
Build date: 2017-12-15 14:03:33+0100

=== Xamarin.Mac ===

Version: 4.1.0.540 (Visual Studio Community)

=== Build Information ===

Release ID: 703020010
Git revision: d203af94011a0bfba0685a67e73281a53c25e393
Build date: 2017-12-07 11:00:27-05
Xamarin addins: 1981a9e7c28d5c3839bb5dc2a0dcaf56fe640906
Build lane: monodevelop-lion-dogfood-vNext

=== Operating System ===

Mac OS X 10.13.1
Darwin 17.2.0 Darwin Kernel Version 17.2.0
    Fri Sep 29 18:27:05 PDT 2017
    root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64

=== Enabled user installed addins ===

Internet of Things (IoT) development (Preview) 7.1
Comment 4 Manuel de la Peña [MSFT] 2018-01-10 19:10:56 UTC
Some more extra info to clarify what is going on.

Xamarin Application

    Request data: https://gist.github.com/mandel-macaque/2158e3bf813f216dccc6462444dfbedd
    Environment: https://gist.github.com/mandel-macaque/8616e63f2d81375264889b362a1656b0

Xcode Application:

    Request data: https://gist.github.com/mandel-macaque/2158e3bf813f216dccc6462444dfbedd
    Environment: Version 9.2 beta (9C34b)


The applications have been modified to use http, and the changes can be found here:

https://github.com/mandel-macaque/NSURLSession-Discretionary-Xamarin
https://github.com/mandel-macaque/NSURLSession-Discretionary-Native

Known facts:

1. It is not a https related issue.
2. Headers have small differences:
   2.a
       Xamarin: Content-Type: text/html; charset=utf-8
       Native: Content-Type: audio/mpeg; charset=utf-8
   2.b 
       Xamarin: ETag: W/"VGY4UUFySk5qRzZqYnhyMllqNXU5dzFDd0R2aEN1dHo0YVFjbjdRaCtkYU5iRGN3R0haWUYyUmpKNnFmUnFOOC0tS3gvcEoybFgxNDBnMTRYbE1keFVodz09--e5370e86f315354f7b2e7a09ff4424fce47bc702"
       Native: No ETag
3. The issue is not related to the following code: https://github.com/xamarin/xamarin-macios/blob/master/src/Foundation/NSUrlSession.cs 
4. Plist differences:
    4.a Info.plist -> No major diff.
    4.b Entitlements.plist -> No major diff.
Comment 5 Manuel de la Peña [MSFT] 2018-01-10 19:26:40 UTC
Some extra info, setting the headers as follows:

https://gist.github.com/mandel-macaque/ffdba7584934ecab9bf273f4872ac632

Generates the following request

https://gist.github.com/mandel-macaque/e05c65e4ef81a7b2ddce489bcd737965

Which also has the same error/issue.
Comment 6 Manuel de la Peña [MSFT] 2018-01-11 12:15:04 UTC
Some extra information of the progress os far:

When creating a background task, objc creates a __NSCFBackgroundDownloadTask which already gave us some problems:

1. Bug report: https://bugzilla.xamarin.com/show_bug.cgi?id=23421
2. Code: https://github.com/xamarin/xamarin-macios/blob/master/runtime/runtime.m#L684

That explains why https://github.com/xamarin/xamarin-macios/blob/master/src/Foundation/NSUrlSession.cs does not have an effect at all when creating the background task. Need to talk with @rolf about this to see what else we can do with the debugging.
Comment 7 Manuel de la Peña [MSFT] 2018-01-12 12:18:46 UTC
And we continue with the investigation, as well pointed by @Seb in a chat, all the above information is just useful for the very first request, since the server side is sending a redirect, we do have a second request, the following information is both the first and second request for both applications:

Native:

1. First request: https://gist.github.com/mandel-macaque/1e88ed7a0310147e890482d1c01694ec
2. Second request: https://gist.github.com/mandel-macaque/6e4e5715f6899ac514771fd08cbc93da

Xamarin:

1. First Request: https://gist.github.com/173e595337f93ffb86a292c0b089de6b
2. Second Request: never happens.

This could explain the issue. I'm looking at why we do not automatically follow redirects.
Comment 8 Manuel de la Peña [MSFT] 2018-01-16 16:34:32 UTC
I have tested the application in a device and the application works as expected. This issue just happens on the simulator:

Launched application 'com.xamarin.Discrectionary' on 'iPhone 7 (mandel)' with pid 499
2018-01-16 17:32:01.535 Discrectionary[499:349399] Xamarin.iOS: IDE Port: 19505 Transport: USB
2018-01-16 17:32:01.552 Discrectionary[499:349399] Xamarin.iOS: Successfully received USB connection from the IDE on port 19505, fd: 4
2018-01-16 17:32:01.553 Discrectionary[499:349399] Xamarin.iOS: Processing: 'start debugger: sdb'
2018-01-16 17:32:01.553 Discrectionary[499:349379] Xamarin.iOS: Debugger loaded with custom transport (fd: 4)
2018-01-16 17:32:01.554 Discrectionary[499:349399] Xamarin.iOS: Successfully received USB connection from the IDE on port 19505, fd: 5
2018-01-16 17:32:01.555 Discrectionary[499:349399] Xamarin.iOS: Processing: 'connect output'
2018-01-16 17:32:01.555 Discrectionary[499:349399] Xamarin.iOS: Successfully received USB connection from the IDE on port 19505, fd: 6
2018-01-16 17:32:01.556 Discrectionary[499:349399] Xamarin.iOS: Processing: 'start profiler: no'
2018-01-16 17:32:01.556 Discrectionary[499:349379] Xamarin.iOS: Profiler not loaded (disabled)
Loaded assembly: /private/var/containers/Bundle/Application/9B20D719-00B3-4852-9FF0-B1E4B66817A3/Discrectionary.app/Xamarin.iOS.dll
Loaded assembly: /private/var/containers/Bundle/Application/9B20D719-00B3-4852-9FF0-B1E4B66817A3/Discrectionary.app/Discrectionary.exe
Thread started:  #2
Resolved pending breakpoint at 'ViewController.cs:26,1' to void Discrectionary.MyDownloadDelegate.DidFinishDownloading (Foundation.NSUrlSession session, Foundation.NSUrlSessionDownloadTask downloadTask, Foundation.NSUrl location) [0x00001].
Resolved pending breakpoint at 'ViewController.cs:13,1' to void Discrectionary.MyDownloadDelegate.WillPerformHttpRedirection (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSHttpUrlResponse response, Foundation.NSUrlRequest newRequest, System.Action<Foundation.NSUrlRequest> completionHandler) [0x00001].
Resolved pending breakpoint at 'ViewController.cs:20,1' to void Discrectionary.MyDownloadDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) [0x0000a].
Thread started:  #3
2018-01-16 17:32:15.580 Discrectionary[499:349397] Downloaded to location file:///private/var/mobile/Containers/Data/Application/240BFDB2-68A5-484E-A4DB-198ADB036E22/Library/Caches/com.apple.nsurlsessiond/Downloads/com.xamarin.Discrectionary/CFNetworkDownload_egj2Rc.tmp

I will keep digging, @Chris could you confirm that the application does work on a device in your side?
Comment 9 Manuel de la Peña [MSFT] 2018-01-17 10:56:35 UTC
Some more info, adding the following to the Info.plist has no effect (in the simulator error):

<key>UIBackgroundModes</key>
<array>
    <string>fetch</string>
</array>
Comment 10 Manuel de la Peña [MSFT] 2018-02-01 11:42:36 UTC
Logs of the sim running the diff applications: https://gist.github.com/mandel-macaque/04d17914834a431f8fa0680aa351b613
Comment 11 Manuel de la Peña [MSFT] 2018-02-09 15:41:35 UTC
Setting bug to minor since we do not experience this issue in devices.
Comment 12 Manuel de la Peña [MSFT] 2018-03-23 14:44:25 UTC
Reopening since we do know it happens.