Bug 28679 - Cannot bootstrap on POWER8/ppc64 using make get-monolite-latest (exception is thrown in System.Console..cctor)
Summary: Cannot bootstrap on POWER8/ppc64 using make get-monolite-latest (exception is...
Status: RESOLVED NORESPONSE
Alias: None
Product: Runtime
Classification: Mono
Component: General (show other bugs)
Version: 3.12.0
Hardware: Other Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2015-04-02 06:14 UTC by farfalle75
Modified: 2018-04-05 19:44 UTC (History)
7 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NORESPONSE

Description farfalle75 2015-04-02 06:14:43 UTC
When trying to build mono from github on POWER8 (RedHat ppc64, big endian)

Some exception is thrown while executing the static constructor of System.Console

To reproduce

> checkout from github (any of master, 3.12.0, 3.10.0, 4.0.0)
> autogen.sh --prefix=blah
> make get-monolite-latest

Everything compiles fine on native side, but when starting to build mcs profiles, running mono fails

> make V=1 PROFILE=basic

or

> make V=1 PROFILE=net_4_5



Stacktrace:

  at <unknown> <0xffffffff>
  at System.Console.SetupStreams (System.Text.Encoding,System.Text.Encoding) <0x00094>
  at System.Console..cctor () <0x003b8>
  at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at <unknown> <0xffffffff>
  at Mono.CSharp.Driver.Main (string[]) <0x000e8>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int_object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

        /root/mono/mono/mini/mono() [0x100c3344]
        /root/mono/mono/mini/mono() [0x10106160]
        /root/mono/mono/mini/mono() [0x1002fe94]
        [0x3fff8b430478]
        [0x3fff89409bf4]
        [0x3fff893fedd4]
        [0x3fff893fe7d4]
        [0x3fff893fd638]
        [0x3fff893fd41c]
        [0x3fff893fcc9c]
        [0x3fff893f4b10]
        /root/mono/mono/mini/mono() [0x10032988]
        /root/mono/mono/mini/mono(mono_runtime_invoke-0x1ab464) [0x101dc704]
        /root/mono/mono/mini/mono() [0x101dcbec]
        /root/mono/mono/mini/mono() [0x100320c0]
        /root/mono/mono/mini/mono() [0x10032828]
        /root/mono/mono/mini/mono(mono_compile_method-0x1aec4c) [0x101d8dac]
        /root/mono/mono/mini/mono() [0x100c4b10]
        [0x3fff8afc01d4]
        [0x3fff893fbbbc]
        [0x3fff893f62e4]
        [0x3fff893f4b10]
        /root/mono/mono/mini/mono() [0x10032988]
        /root/mono/mono/mini/mono(mono_runtime_invoke-0x1ab464) [0x101dc704]
        /root/mono/mono/mini/mono() [0x101dcbec]
        /root/mono/mono/mini/mono() [0x100320c0]
        /root/mono/mono/mini/mono() [0x10032828]
        /root/mono/mono/mini/mono(mono_compile_method-0x1aec4c) [0x101d8dac]
        /root/mono/mono/mini/mono() [0x100c4b10]
        [0x3fff8afc01d4]
        [0x3fff893f3960]
        [0x3fff893f4478]
        /root/mono/mono/mini/mono() [0x10032988]
        /root/mono/mono/mini/mono(mono_runtime_invoke-0x1ab464) [0x101dc704]
        /root/mono/mono/mini/mono(mono_runtime_exec_main-0x1a8d48) [0x101df0b0]
        /root/mono/mono/mini/mono(mono_jit_exec-0x2ecbb8) [0x1008ea80]
        /root/mono/mono/mini/mono(mono_main-0x2eb3e4) [0x10090274]
        /root/mono/mono/mini/mono() [0x100231b0]
        /lib64/power8/libc.so.6(+0x4454c) [0x3fff8b0e454c]
        /lib64/power8/libc.so.6(__libc_start_main-0x19ce54) [0x3fff8b0e4774]

Debug info from gdb:


=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Note: Was able to compile and run everything on another virtual machine (RedHat ppc64le, little endian)
Comment 1 Timotheus Pokorra 2015-05-27 03:37:18 UTC
We have now got this problem also for the Fedora build for ppc64: https://bugzilla.redhat.com/show_bug.cgi?id=1225035

Looking at the code of method SetupStreams:
https://github.com/mono/mono/blob/master/mcs/class/corlib/System/Console.cs#L140

I cannot see what would be wrong?
anyone with ideas?
Comment 2 Bill Seurer 2015-06-04 11:47:28 UTC
I have successfully built Mono on Ubuntu 14.04 LE using gcc 5.0.1

gcc version 5.0.1 20150417 (prerelease) [gcc-5-branch revision 222193] (GCC)
Comment 3 Bill Seurer 2015-06-04 11:55:33 UTC
BTW, I have some "generic" PPC64 LE patches in progress that will hopefully soon be available.
Comment 4 Timotheus Pokorra 2015-06-04 11:57:26 UTC
That sounds good!
Comment 5 than 2015-06-11 07:53:16 UTC
Bill, it's great! it would be great if you could attach the patches here so we can test?

Thanks
Comment 6 Bill Seurer 2015-06-12 11:24:24 UTC
I have already submitted a couple of minor patches to test things out.  That said, you should have been able to get it to at least build without any patches.

What level of gcc are you using?  Which OS (precisely)?  I can try to find a similar system and try it there myself maybe.
Comment 7 than 2015-06-18 07:22:15 UTC
Hi Bill

it's fedora23 and we use gcc-5.1.1-1.fc23 to build mono, Could you please attach the url so i can have a look.

Thanks
Comment 8 Bill Seurer 2015-06-18 13:45:32 UTC
I got on a big endian system and tried this and saw the same issue.

The specific problem here is that the handling of "nullified_class_init_trampoline"s has change.

mono_arch_get_nullified_class_init_trampoline returns a pointer to a function *but* the code that uses it is expecting a function descriptor instead of a direct function pointer.  Function descriptors are pointers to pointers to functions and so it creates an invalid branch instruction which when executed causes a segmentation violation.  

I am still looking into what changed here from Mono 3.2.8 where this worked.
Comment 9 Bill Seurer 2015-06-18 16:08:47 UTC
There are two changes to the code that are needed to get this to work on BE.  I haven't gotten the CLA through our legal process yet so I can't contribute patches yet.  Maybe someone else can do these?

#1 is an added line and comment in mono/mini/tramp.ppc.c around line 629:

gpointer
mono_arch_get_nullified_class_init_trampoline (MonoTrampInfo **info)
{
	guint8 *code, *buf;
	guint32 tramp_size = 64;

	code = buf = mono_global_codeman_reserve (tramp_size);
	ppc_blr (code);

	mono_arch_flush_icache (buf, code - buf);

	g_assert (code - buf <= tramp_size);

	*info = mono_tramp_info_create ("nullified_class_init_trampoline", buf, code - buf, NULL, NULL);
+	/* It is expected to be a function descriptor on power pre-v2 ABI */
+	buf = mono_create_ftnptr (mono_domain_get (), buf);

	return buf;
}

#2 is a changed #if in mono/sgen/sgen-archdep.h around line 92:

/* MS_BLOCK_SIZE must be a multiple of the system pagesize, which for some
   archs is 64k.  */
- #if defined(TARGET_POWERPC64) && _CALL_ELF == 2
+ #if defined(TARGET_POWERPC64)
#define ARCH_MIN_MS_BLOCK_SIZE	(64*1024)
#define ARCH_MIN_MS_BLOCK_SIZE_SHIFT	16
#endif

These will allow mono to build.  There are still some issues in the thread support but most of the tests will pass.
Comment 10 Bill Seurer 2015-07-01 10:32:04 UTC
I pushed some fixes (including the above) that will get Mono building and running on BE power.

https://github.com/mono/mono/pull/1905
Comment 11 than 2015-07-02 07:26:49 UTC
Hi Bill,

i applied the fixes from above in mono-4.0.2 and still got the build failure. i used gcc-5.1.1-3.fc22.ppc64 to build mono-4.0.2


make[6]: mcs: Command not found
build/profiles/basic.make:93: recipe for target 'build/deps/basic-profile-check.exe' failed
make[6]: *** [build/deps/basic-profile-check.exe] Error 127
*** The compiler 'mcs' doesn't appear to be usable.
*** Trying the 'monolite' directory.
* Assertion at mini-codegen.c:269, condition `vreg >= MONO_MAX_IREGS' not met

Stacktrace:

  at <unknown> <0xffffffff>
  at System.ConsoleDriver.CreateTermInfoDriver (string) <0x00074>
  at System.ConsoleDriver..cctor () <0x001d8>
  at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at <unknown> <0xffffffff>
  at System.Console.SetupStreams (System.Text.Encoding,System.Text.Encoding) <0x00094>
  at System.Console..cctor () <0x003b8>
  at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at <unknown> <0xffffffff>
  at Mono.CSharp.Driver.Main (string[]) <0x000e8>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int_object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100c18ec]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10106758]
        linux-vdso64.so.1(__kernel_sigtramp_rt64+0) [0x3fff90820478]
        /lib64/libc.so.6(gsignal-0x1a31e0) [0x3fff9044f928]
        /lib64/libc.so.6(abort-0x1a0bcc) [0x3fff9045223c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x1028e854]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x1028eb24]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x1028ec1c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100b7fa4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100f3cdc]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10078870]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10031b88]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100353e0]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10036218]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_runtime_invoke-0x1f877c) [0x101df0a4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x101df68c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10035730]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10035f50]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_compile_method-0x1fbf64) [0x101db74c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100c3260]
        [0x3fff89c701d4]
        [0x3fff89889fdc]
        [0x3fff8988985c]
        [0x3fff89884b10]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100360c4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_runtime_invoke-0x1f877c) [0x101df0a4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x101df68c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10035730]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10035f50]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_compile_method-0x1fbf64) [0x101db74c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100c3260]
        [0x3fff89c701d4]
        [0x3fff8988877c]
        [0x3fff898862e4]
        [0x3fff89884b10]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100360c4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_runtime_invoke-0x1f877c) [0x101df0a4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x101df68c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10035730]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x10035f50]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_compile_method-0x1fbf64) [0x101db74c]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100c3260]
        [0x3fff89c701d4]
        [0x3fff89883960]
        [0x3fff89884478]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x100360c4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_runtime_invoke-0x1f877c) [0x101df0a4]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_runtime_exec_main-0x1f53d0) [0x101e2720]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_jit_exec-0x33d4e0) [0x1008df20]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono(mono_main-0x33bc88) [0x1008f798]
        /home/than/rpmbuild/BUILD/mono-4.0.2/mono/mini/mono() [0x1002674c]
        /lib64/libc.so.6(+0x44d08) [0x3fff90434d08]
        /lib64/libc.so.6(__libc_start_main-0x1bc950) [0x3fff90434f38]

Debug info from gdb:

ptrace: Operation not permitted.
No threads.
Comment 12 than 2015-07-07 16:18:18 UTC
Hi Bill,

i finally got mono-4.0.2 built correctly on fedora 23. I did backported a part of your patch for mono-4.0.2 and attached it in this bugzilla. Thanks again for your great help.


diff -up mono-4.0.2/mono/mini/mini-ppc.c.than mono-4.0.2/mono/mini/mini-ppc.c
--- mono-4.0.2/mono/mini/mini-ppc.c.than        2015-07-07 18:40:56.066059552 +0200
+++ mono-4.0.2/mono/mini/mini-ppc.c     2015-07-07 18:50:44.867880449 +0200
@@ -2147,11 +2147,7 @@ mono_arch_decompose_opts (MonoCompile *c
 {
        switch (ins->opcode) {
        case OP_ICONV_TO_R_UN: {
-#if G_BYTE_ORDER == G_BIG_ENDIAN
                static const guint64 adjust_val = 0x4330000000000000ULL;
-#else
-               static const guint64 adjust_val = 0x0000000000003043ULL;
-#endif
                int msw_reg = mono_alloc_ireg (cfg);
                int adj_reg = mono_alloc_freg (cfg);
                int tmp_reg = mono_alloc_freg (cfg);
@@ -2162,8 +2158,14 @@ mono_arch_decompose_opts (MonoCompile *c
                        basereg = mono_alloc_ireg (cfg);
                        MONO_EMIT_NEW_BIALU_IMM (cfg, OP_IADD_IMM, basereg, cfg->frame_reg, offset);
                }
+#if G_BYTE_ORDER == G_BIG_ENDIAN
                MONO_EMIT_NEW_STORE_MEMBASE (cfg, OP_STOREI4_MEMBASE_REG, basereg, offset, msw_reg);
                MONO_EMIT_NEW_STORE_MEMBASE (cfg, OP_STOREI4_MEMBASE_REG, basereg, offset + 4, ins->sreg1);
+#else
+               // For little endian the words are reversed
+               MONO_EMIT_NEW_STORE_MEMBASE (cfg, OP_STOREI4_MEMBASE_REG, basereg, offset + 4, msw_reg);
+               MONO_EMIT_NEW_STORE_MEMBASE (cfg, OP_STOREI4_MEMBASE_REG, basereg, offset, ins->sreg1);
+#endif
                MONO_EMIT_NEW_LOAD_R8 (cfg, adj_reg, &adjust_val);
                MONO_EMIT_NEW_LOAD_MEMBASE_OP (cfg, OP_LOADR8_MEMBASE, tmp_reg, basereg, offset);
                MONO_EMIT_NEW_BIALU (cfg, OP_FSUB, ins->dreg, tmp_reg, adj_reg);
@@ -2213,7 +2215,11 @@ mono_arch_decompose_opts (MonoCompile *c
                        MONO_EMIT_NEW_BIALU_IMM (cfg, OP_IADD_IMM, basereg, cfg->frame_reg, offset);
                }
                MONO_EMIT_NEW_STORE_MEMBASE (cfg, OP_STORER8_MEMBASE_REG, basereg, offset, ins->sreg1);
+#if G_BYTE_ORDER == G_BIG_ENDIAN
                MONO_EMIT_NEW_LOAD_MEMBASE_OP (cfg, OP_LOADI4_MEMBASE, msw_reg, basereg, offset);
+#else
+               MONO_EMIT_NEW_LOAD_MEMBASE_OP (cfg, OP_LOADI4_MEMBASE, msw_reg, basereg, offset+4);
+#endif
                MONO_EMIT_NEW_UNALU (cfg, OP_CHECK_FINITE, -1, msw_reg);
                MONO_EMIT_NEW_UNALU (cfg, OP_FMOVE, ins->dreg, ins->sreg1);
                ins->opcode = OP_NOP;
diff -up mono-4.0.2/mono/mini/mini-ppc.h.than mono-4.0.2/mono/mini/mini-ppc.h
--- mono-4.0.2/mono/mini/mini-ppc.h.than        2015-07-07 18:50:55.337912865 +0200
+++ mono-4.0.2/mono/mini/mini-ppc.h     2015-07-07 18:51:20.637991106 +0200
@@ -145,6 +145,7 @@ typedef struct MonoCompileArch {
 #define PPC_LAST_FPARG_REG ppc_f13
 #define PPC_PASS_STRUCTS_BY_VALUE 1
 #define PPC_THREAD_PTR_REG ppc_r13
+#define MONO_ARCH_HAVE_SIGCTX_TO_MONOCTX 1
 #else
 #define PPC_RET_ADDR_OFFSET 4
 #define PPC_STACK_PARAM_OFFSET 8
diff -up mono-4.0.2/mono/tests/Makefile.am.than mono-4.0.2/mono/tests/Makefile.am
--- mono-4.0.2/mono/tests/Makefile.am.than      2015-07-07 18:51:39.508049497 +0200
+++ mono-4.0.2/mono/tests/Makefile.am   2015-07-07 18:52:18.098168853 +0200
@@ -454,6 +454,12 @@ if POWERPC
 PLATFORM_DISABLED_TESTS=finalizer-abort.exe finalizer-exception.exe finalizer-exit.exe
 endif
 
+
+if POWERPC64
+# These tests hang
+PLATFORM_DISABLED_TESTS=monitor.exe finalizer-abort.exe finalizer-exception.exe finalizer-exit.exe
+endif
+
 if ARM
 PLATFORM_DISABLED_TESTS=filter-stack.exe
 endif
diff -up mono-4.0.2/mono/metadata/sgen-archdep.h.than mono-4.0.2/mono/metadata/sgen-archdep.h
--- mono-4.0.2/mono/metadata/sgen-archdep.h.than        2015-07-07 19:02:32.370056671 +0200
+++ mono-4.0.2/mono/metadata/sgen-archdep.h     2015-07-07 19:03:19.740200462 +0200
@@ -89,7 +89,7 @@
 
 /* MS_BLOCK_SIZE must be a multiple of the system pagesize, which for some
    archs is 64k.  */
-#if defined(TARGET_POWERPC64) && _CALL_ELF == 2
+#if defined(TARGET_POWERPC64)
 #define ARCH_MIN_MS_BLOCK_SIZE (64*1024)
 #define ARCH_MIN_MS_BLOCK_SIZE_SHIFT   16
 #endif
diff -up mono-4.0.2/mono/mini/tramp-ppc.c.than mono-4.0.2/mono/mini/tramp-ppc.c
--- mono-4.0.2/mono/mini/tramp-ppc.c.than       2015-07-07 19:04:29.860413420 +0200
+++ mono-4.0.2/mono/mini/tramp-ppc.c    2015-07-07 19:05:47.230648113 +0200
@@ -691,6 +691,9 @@ mono_arch_get_nullified_class_init_tramp
        if (info)
                *info = mono_tramp_info_create ("nullified_class_init_trampoline", buf, code - buf, NULL, NULL);
 
+       /* It is expected to be a function descriptor on power pre-v2 ABI */
+       buf = mono_create_ftnptr (mono_domain_get (), buf);
+
        return buf;
 }
Comment 13 Ludovic Henry 2018-03-01 00:01:57 UTC
Hello,

Thank you very much for all your contributions! We now have a PowerPC as part of our CI so we shouldn't regress as much as we used to.

So do you still have the same issue with latest version of Mono? If so, could you please provide us with reproduction steps.

Thank you
Comment 14 Timotheus Pokorra 2018-03-02 09:41:55 UTC
Thank you Ludovic!

For Fedora, we are still on Mono 4.8, because we cannot build Mono 5.x with roslyn packages and msbuild because those are not properly packaged according to the Fedora standards (external binaries only allowed for initial bootstrap, after that it needs to build from itself without external binaries; for details see https://bugzilla.redhat.com/show_bug.cgi?id=1436896#c8).

back to the topic:

The only patch we have for Mono 4.8 for PPC is https://src.fedoraproject.org/rpms/mono/blob/master/f/mono-4.2.1-ppc.patch, which looks more like a workaround than a proper fix, for https://github.com/mono/mono/blob/master/mcs/class/corlib/System/TimeZoneInfo.cs#L1264
Comment 15 Ludovic Henry 2018-03-02 16:04:32 UTC
Hi Timotheus,

We do support building Mono from source without roslyn and msbuild. To do so, just pass `--with-csc=mcs` to `./configure` and it will default to mcs which we  build from source as part of the build process.

For the workaround, can you figure out exactly why `ParseTZBuffer` throws an exception? Maybe you could submit an issue at https://github.com/mono/mono/issues/ with the file that fails to get parsed, so we can take a look at it.

Thank you,
Comment 16 Ludovic Henry 2018-04-05 19:44:05 UTC
Please reopen if you can still reproduce. Thank you