Bug 25095

Summary: Struct w/nested structs as return value from P/Invoke is incorrectly marshalled on x86_64/OSX
Product: [Mono] Runtime Reporter: Rolf Bjarne Kvinge [MSFT] <rolf>
Component: InteropAssignee: Bugzilla <bugzilla>
Status: RESOLVED FIXED    
Severity: normal CC: mono-bugs+mono, mono-bugs+runtime, vargaz
Priority: ---    
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Mac OS   
Tags: Is this bug a regression?: ---
Last known good build:
Attachments: testcase.zip

Description Rolf Bjarne Kvinge [MSFT] 2014-12-05 06:05:03 UTC
Created attachment 8974 [details]
testcase.zip

Repro:
* Download & extract provided test case.
* Fixup Makefile ('test' target) to use a 64-bit capable mono.
* 'make test'

Result:
  Flat:
  0xbb113311 0x553311
  0 0 0 0

  Nested:
  0x7fff5626c618 0xbb113311
  0 0 5.712068E-33 1.401298E-45
  **** FAILED **** 

The problem is that the nested structure {{float,float,float},float} is marshalled differently from a flat structure {float,float,float,float} - Mono thinks the native function requires a pointer to the memory for the struct as the first argument, when the native function will return the struct in xmm0/xmm1. This effectively shifts the rest of the arguments by one register.
Comment 1 Zoltan Varga 2014-12-06 06:18:28 UTC
Fixed in mono master 2a8f86f22388107e1c978d28179832b71295afac.