Bug 36128

Summary: Native interop: LPArray output parameter becomes invalid after call
Product: [Mono] Runtime Reporter: Thomas Chust <chust>
Component: InteropAssignee: Bugzilla <bugzilla>
Severity: normal CC: mono-bugs+mono, mono-bugs+runtime, vargaz
Priority: ---    
Version: 4.2.0 (C6)   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Tags: Is this bug a regression?: ---
Last known good build:
Attachments: Sample program triggering the bug when compiled without -D:PINNED

Description Thomas Chust 2015-11-21 20:25:31 UTC
Created attachment 13932 [details]
Sample program triggering the bug when compiled without -D:PINNED

When passing an array to a native method that writes data into the array, the reference to the array can apparently get destroyed in the marshalling process.

The attached C# program uses the memset function from the standard C library to demonstrate the problem. When I compile the program and run it with Mono (I'm using the version for Debian derivatives from Xamarin's package repository), I get an exception:

$ mcs -debug arrayfill.cs
$ mono --gc=sgen --debug arrayfill.exe 

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.String.Join[T] (System.String separator, IEnumerable`1 values) <0x416bb340 + 0x0005d> in <filename unknown>:0 
  at ArrayFill.Main () [0x0001b] in arrayfill.cs:40 

It looks like the reference to the array that is passed to the C function through P/Invoke as an LPArray output parameter somehow ends up being null after the call!

Using the Boehm-Demers-Weiser garbage collector the runtime crashes with a segmentation fault.

The problem can be circumvented by manually pinning the array in memory and passing the native pointer provided by the pinned GCHandle to the native function through P/Invoke. I have included an alternative code path in the source to demonstrate this:

$ mcs -debug -D:PINNED arrayfill.cs
$ mono --gc=sgen --debug arrayfill.exe 
buf = {42, 42, 42, 42, 42, 42, 42, 42}
$ mono --gc=boehm --debug arrayfill.exe 
buf = {42, 42, 42, 42, 42, 42, 42, 42}

I have checked that the problem does not occur -- and both versions of the program run correctly -- on Mono (the previous version available from Xamarin's package repository), on Mono 3.2.8 (the version available from Ubuntu's package repository) and on the Microsoft CLR that comes with Windows 7.
Comment 1 Zoltan Varga 2015-11-22 09:05:16 UTC
Fixed in mono master a425ee9491e69a658363b321ff5d32c720661c8e. As a workaround, avoid LPArray marshalling, i.e. simply do:

extern static private IntPtr memset([Out] byte[] s, int c, int n);