Bug 42656 - Tests for recently added assemblies
Summary: Tests for recently added assemblies
Status: IN_PROGRESS
Alias: None
Product: iOS
Classification: Xamarin
Component: BCL Class Libraries (show other bugs)
Version: XI 9.99 (iOS 10 previews)
Hardware: PC Mac OS
: Normal enhancement
Target Milestone: Future Cycle (TBD)
Assignee: Manuel de la Peña [MSFT]
URL:
Depends on:
Blocks:
 
Reported: 2016-07-19 14:11 UTC by Marek Safar
Modified: 2018-01-26 15:43 UTC (History)
8 users (show)

Tags:
Is this bug a regression?: No
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report for Bug 42656 on Developer Community or GitHub if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: Developer Community HTML or GitHub Markdown
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:
Status:
IN_PROGRESS

Description Marek Safar 2016-07-19 14:11:38 UTC
There have been few recently added assemblies, mainly as part of netstandard, which would benefit from having unit tests added to build process.
Comment 1 Vincent Dondain [MSFT] 2016-07-19 15:30:40 UTC
From Sebastien: "the tests exists but we don’t have them enabled in our builds and bots".
Comment 2 Sebastien Pouliot 2016-08-15 13:43:44 UTC
That will happen on master and might be backported to the active/release branch afterward
Comment 3 Sebastien Pouliot 2016-09-19 14:36:09 UTC
@Manuel, not urgent but please look at this once you have completed the jenkins works.
Comment 4 Sebastien Pouliot 2016-10-03 14:22:24 UTC
To be done in master (for C10) and potentially backported to C9 (to run more tests).
Comment 9 Manuel de la Peña [MSFT] 2017-09-01 11:41:32 UTC
PR that adds the tests (but failing) for Mono.Data.Tds can be found here: https://github.com/xamarin/xamarin-macios/pull/2580

PR that adds the tests (but failing) for System.IO.Compression.FileSystem can be found here: https://github.com/xamarin/xamarin-macios/pull/2582
Comment 10 Manuel de la Peña [MSFT] 2017-09-01 11:58:16 UTC
PR that adds the tests (but failing) for System.IO.Compression can be found here: https://github.com/xamarin/xamarin-macios/pull/2583
Comment 11 Manuel de la Peña [MSFT] 2017-10-26 10:34:27 UTC
New tests landed for

Mono.Data.Tds
System.IO.Compression
System.IO.Compression.FileSystem

Missing:

Microsoft.CSharp.dll
Mono.CSharp.dll
System.ComponentModel.Composition.dll
System.Data.Services.Client.dll
System.IdentityModel.dll
System.Net.Http.WinHttpHandler.dll
System.Net.dll
System.Numerics.Vectors.dll
System.Reflection.Context.dll
System.Security.dll
System.ServiceModel.Internals.dll
System.ServiceModel.dll
System.Windows.dll
System.Xml.Serialization.dll
System.Xml.dll
Comment 12 Sebastien Pouliot 2017-10-26 14:21:32 UTC
> System.Xml.dll

That's already covered. Typo for something else ?
Comment 13 Manuel de la Peña [MSFT] 2017-10-30 17:02:00 UTC
Correct, that was a typo. System.Xml was already present.

The new missing tests are:

Microsoft.CSharp.dll
System.ComponentModel.Composition.dll
System.Data.Services.Client.dll
System.IdentityModel.dll
System.Net.Http.WinHttpHandler.dll
System.Net.dll
System.Numerics.Vectors.dll
System.Reflection.Context.dll
System.Security.dll
System.ServiceModel.Internals.dll
System.ServiceModel.dll
System.Windows.dll
System.Xml.Serialization.dll

But we know that ->

Microsoft.CSharp -> We do not know how are tests ran, we need to talk with the mono team.
System.Data.Services -> We cannot run tests because it starts a service and we only have System.Data.Services.Client
System.Net -> missing tests.
System.Reflection.Context => missing tests.

So atm we could say we have missing ->

System.ComponentModel.Composition.dll
System.Data.Services.Client.dll
System.IdentityModel.dll
System.Net.Http.WinHttpHandler.dll
System.Numerics.Vectors.dll
System.Security.dll
System.ServiceModel.Internals.dll
System.ServiceModel.dll
System.Windows.dll
System.Xml.Serialization.dll

We are working on the System.Security tests, this might need to be landed in a diff branch since the runtime time has proposed a PR for the mono-2017-10 branch => https://github.com/xamarin/xamarin-macios/pull/2905
Comment 14 Manuel de la Peña [MSFT] 2017-10-31 12:08:03 UTC
System.Security tests => https://github.com/xamarin/xamarin-macios/pull/2939
Comment 15 Manuel de la Peña [MSFT] 2018-01-12 15:35:09 UTC
Updated info regarding present tests and missing ones:

Currently missing ones are:

Microsoft.CSharp.dll -> We do not know how are tests ran, we need to talk with the mono team.
System.ComponentModel.Composition.dll -> missing tests in mono. they are coming from .net core. We should run them accordingly. -> tests are present. Should find a subset of those that pass.
System.Data.Services.Client.dll -> We cannot run tests because it starts a service and we only have System.Data.Services.Client
System.IdentityModel.dll -> tests can be added. Some of the will fail. We need to find a subset.
System.Net.Http.WinHttpHandler.dll -> do not make much sense.
System.Net.dll -> missing tests in mono. they are coming from .net core. We should run them accordingly. 
System.Numerics.Vectors.dll -> missing tests in mono. they are coming from .net core. We should run them accordingly. 
System.Reflection.Context.dll -> missing tests in mono. they are coming from .net core. We should run them accordingly. 
System.ServiceModel.Internals.dll
System.ServiceModel.dll -> some tests are present. We need to create a subset of the tests to be ran in the iOS devices. We need to find those that do not need a service running.
System.Windows.dll -> do not make much sense 
System.Xml.Serialization.dll -> missing tests in mono. they are coming from .net core. We should run them accordingly.
Comment 16 Manuel de la Peña [MSFT] 2018-01-17 09:18:16 UTC
System.ServiceModel tests added in https://github.com/mono/mono/pull/6539 and https://github.com/xamarin/xamarin-macios/pull/3238
Comment 17 Manuel de la Peña [MSFT] 2018-01-18 12:05:51 UTC
Looked into System.Windows since it does have some classes we are interested in yet those tests have bene moved outside mono.
Comment 18 Manuel de la Peña [MSFT] 2018-01-18 12:39:31 UTC
Same with System.ServiceModel.internals.
Comment 19 Manuel de la Peña [MSFT] 2018-01-18 15:15:00 UTC
System.IdentityModel will be added. One test is added in iOS and Mac Modern and 158 on mac full.
Comment 20 Marek Safar 2018-01-19 15:44:35 UTC
Few comments

Microsoft.CSharp.dll - many xunit tests are available in mono
System.ComponentModel.Composition.dll - no tests and nothing planed in the short term
System.Net.Http.WinHttpHandler.dll - ok to ignore
System.Net.dll - forwarding assembly, ok to ignore
System.Numerics.Vectors.dll - many xunit tests are available in mono
System.Reflection.Context.dll - ok to ignore, the code will be NIE anyway
System.Xml.Serialization.dll - forwarding assembly, ok to ignore
Comment 21 Manuel de la Peña [MSFT] 2018-01-24 16:40:12 UTC
PR for System.IdentityModel -> https://github.com/xamarin/xamarin-macios/pull/3295
Comment 22 Manuel de la Peña [MSFT] 2018-01-24 16:45:53 UTC
Taking into account Marek comments and the new added tests, we have to get the following assemblies to run the unit tests since we do not have more assemblies to be added using nunit after landing https://github.com/xamarin/xamarin-macios/pull/3295

Xunit tests:
Microsoft.CSharp.dll
System.Numerics.Vectors.dll

Ignored assemblies:
System.ComponentModel.Composition.dll
System.Net.Http.WinHttpHandler.dll 
System.Reflection.Context.dll
System.Xml.Serialization.dll

Present tests (NUnit)
System
System.Runtime.Serialization
System.ComponentModel.DataAnnotations
System.Security
System.Core
System.ServiceModel
System.Data
System.ServiceModel.Web
Mono.CSharp
System.IO.Compression
System.Transactions
Mono.Data.Sqlite
System.IO.Compression.FileSystem
System.Web.Services
Mono.Data.Tds
System.Json
System.Xml
Mono.Security
System.Net.Http
System.Xml.Linq
SyncTestResources.targets
System.Numerics	
mscorlib

Unknown:
System.Data.Services.Client -> Will take a look if a subset of the tests can be ran on devices.
Comment 23 Marek Safar 2018-01-24 18:06:49 UTC
Manuel,

About the xnunit test. They are extensions to existing nunit tests (they cover different code) and the goal is to have xunit tests for all assemblies. Right now we have about 6 but the end goal is to have many more
Comment 24 Manuel de la Peña [MSFT] 2018-01-25 10:57:06 UTC
System.Data.Services.Client => Seems not to have tests present in mono:

[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.FileNotFoundException: Could not find file "/Users/mandel/Xamarin/macios/master/xamarin-macios/external/mono/mcs/class/System.Data.Services.Client/System.Data.Services.Client_test.dll.sources"

@marek yes, I know about the xunit goals we will make that harness is ready for that goal :)
Comment 25 Sebastien Pouliot 2018-01-26 15:43:18 UTC
Great, let's do the xunit ones from mono-2017-12 (mono 2017-10) and land them, one by one, once we have bumped to the new mono