Bug 45925 - Out of memory and Could not allocate bytes error when running mono application and 2GB mem free
Summary: Out of memory and Could not allocate bytes error when running mono applicatio...
Status: RESOLVED ANSWERED
Alias: None
Product: Runtime
Classification: Mono
Component: GC (show other bugs)
Version: 4.4.1 (C7SR0)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-10-26 15:01 UTC by Albert
Modified: 2017-10-11 17:49 UTC (History)
4 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Albert 2016-10-26 15:01:51 UTC
I have 2GB free on my system but cant run mono a small application because out of memory:

Free memory:

[root@Src01 ]# free -m
             total       used       free     shared    buffers     cached
Mem:          3832       1757       2075          0         17        774
-/+ buffers/cache:        964       2867
Swap:         2047          0       2047


Exception when using Process.Start:

20161026.log:2016-10-26 11:01:00.862 -04:00 [Error] JobManager.cs(109): RunJob(): ApplicationName='/usr/bp/app.exe', CommandLine='1 Test', CurrentDirectory='/usr/bp', Native error= Out of memory ->   at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) <0x404c1ab0 + 0x00637> in <filename unknown>:0
20161026.log:2016-10-26 11:06:00.678 -04:00 [Error] JobManager.cs(109): RunJob(): ApplicationName='/usr/bp/app.exe', CommandLine='1 Test', CurrentDirectory='/usr/bp', Native error= Out of memory ->   at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) <0x404c1ab0 + 0x00637> in <filename unknown>:0


Exception when running from command line:

Could not allocate 8192 bytes
Stacktrace:

  at <unknown> <0xffffffff>
  at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVimService.Read2229_DynamicProperty (bool,bool) <0x002ff>
  at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVimService.Read2231_ObjectContent (bool,bool) <0x0052b>
  at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVimService.Read4757_RetrievePropertiesResponse () <0x0021f>
  at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer2825.Deserialize (System.Xml.Serialization.XmlSerializationReader) <0x00047>
  at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.XmlReader,string,System.Xml.Serialization.XmlDeserializationEvents) <0x00283>
  at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.XmlReader,string) <0x0007f>
  at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse (System.Web.Services.Protocols.SoapClientMessage,System.Net.WebResponse,System.IO.Stream,bool) <0x00af7>
  at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke (string,object[]) <0x00177>
  at VimApi_60.VimService.RetrieveProperties (VimApi_60.ManagedObjectReference,VimApi_60.PropertyFilterSpec[]) <0x00073>
  at (wrapper remoting-invoke-with-check) VimApi_60.VimService.RetrieveProperties (VimApi_60.ManagedObjectReference,VimApi_60.PropertyFilterSpec[]) <0x00073>
  at VMware.Vim.PropertyCollector.RetrieveProperties (VMware.Vim.PropertyFilterSpec[]) <0x001d7>
  at VMware.Vim.VimClientImpl.GetViewsByMorefs (System.Collections.Generic.IEnumerable`1<VMware.Vim.ManagedObjectReference>,string[]) <0x00393>
  at VMware.Vim.VimClientImpl.FindEntityViews (System.Type,VMware.Vim.ManagedObjectReference,System.Collections.Specialized.NameValueCollection,string[]) <0x005d7>
  at unitrends.rdr.vmware.VimWrapper.ConnectionWrapper () <0x0009d>
  at unitrends.rdr.vmware.VimWrapper.CheckConnection () <0x0000f>
  at unitrends.rdr.vmware.VimWrapper.ReconfigureResources (string,string,string,string,string,string,int,int) <0x00077>
  at rdrjob.Management.Operations/<>c__DisplayClass5d/<>c__DisplayClass5f/<>c__DisplayClass61.<ReconfigureResources>b__5a () <0x0045b>
  at rdrjob.Management.Common.RetryExecution<unitrends.rdr.domain.Models.JobResult> (System.Func`1<unitrends.rdr.domain.Models.JobResult>,System.Nullable`1<int16>,bool,string,string,int) <0x003bc>
  at rdrjob.Management.Common.ControlExecution (rdrjob.Management.JobLogTaskStep,System.Func`1<unitrends.rdr.domain.Models.JobResult>,System.Nullable`1<int16>,bool,bool,unitrends.rdr.domain.Models.vm,string,string,int) <0x0007b>
  at rdrjob.Management.Operations/<>c__DisplayClass5d/<>c__DisplayClass5f.<ReconfigureResources>b__59 () <0x0023b>
  at System.Threading.Tasks.Task`1<unitrends.rdr.domain.Models.JobResult>.InnerInvoke () <0x0004a>
  at System.Threading.Tasks.Task.Execute () <0x0005b>
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) <0x0004f>
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x001f9>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00023>
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) <0x0013b>
  at System.Threading.Tasks.Task.ExecuteEntry (bool) <0x000bf>
  at System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork (object) <0x0004f>
  at System.Threading.ThreadHelper.ThreadStart_Context (object) <0x000b0>
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x001f9>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00023>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) <0x0005b>
  at System.Threading.ThreadHelper.ThreadStart (object) <0x00067>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <0x000d6>

Native stacktrace:

        /usr/bin/mono() [0x49ffe0]
        /lib64/libpthread.so.0(+0xf7e0) [0x7ffff753f7e0]
        /lib64/libc.so.6(gsignal+0x35) [0x7ffff6fb8625]
        /lib64/libc.so.6(abort+0x175) [0x7ffff6fb9e05]
        /usr/bin/mono() [0x645679]
        /usr/bin/mono() [0x645707]
        /usr/bin/mono() [0x645853]
        /usr/bin/mono() [0x64547f]
        /usr/bin/mono(mono_mempool_alloc+0xc8) [0x560cf8]
        /usr/bin/mono() [0x41a384]
        /usr/bin/mono() [0x45ceca]
        /usr/bin/mono() [0x502216]
        /usr/bin/mono() [0x5038c9]
        /usr/bin/mono() [0x418856]
        /usr/bin/mono() [0x418bab]
        /usr/bin/mono() [0x4a3185]
        /usr/bin/mono() [0x4a39c9]
        [0x40000289]

Debug info from gdb:
Comment 1 Vlad Brezae 2016-10-27 17:45:34 UTC
Hello Albert,

    Can you share a repro for this ?  How does the memory usage evolves for the app (htop) ? Does it increase very slowly, suggesting a memory leak ? Or maybe your app is allocating too much memory (you can try to use MONO_GC_PARAMS=soft-heap-limit=1G to trigger more collections after the heap grows too much) ?
Comment 2 Albert 2016-10-27 18:04:19 UTC
No, I dont know how to reproduce, if i reboot the system it works for some time and then fails again.

The app doesnt run for long time, it is a command line and run for less than a minute so I dont think it is a memory leak.

application memory from top is:

VIRT: 1200m  
RES: 75m  
SHR: 14m 

If I upgrade system from 4GB to 8GB RAM i dont get this errors, but not sure why having 2GB free it cant allocate the memory.
Comment 3 Vlad Brezae 2016-10-27 18:46:26 UTC
This doesn't make much sense. Have you checked the memory usage at the actual time when mono crashes. I suggest always passing MONO_DEBUG=suspend-on-sigsegv so you can catch the crash in action. Then, when it is stopped, you can see if it still uses a sane amount of memory at crash time. If so, maybe you can attach to the process and try to call malloc from the debugger to see whether it works or not. Maybe the OS just runs out of memory for a short period of time ?
Comment 4 Albert 2016-11-05 13:09:20 UTC
Vlad, would you be willing to do a teamviewer/webex session next week and i will show you in few minutes and you can check whatever you want?

Now, i am seeing also i have a mono process that has been running for few days and is running at 100% cpu constantly since 4 days ago and it is not doing anything or replying to API requests.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12074 root      20   0 1061m  64m  14m S 99.9  1.7   7138:15 app.exe
Comment 5 Vlad Brezae 2016-11-06 17:42:40 UTC
Sure, Feel free to email me directly
Comment 6 Albert 2017-01-20 22:59:06 UTC
Still getting out of memory errors although i have 5GB free on buffers/cached and 0 swap usage. Any idea?


[root@VMware-UB rdr]# free -m
             total       used       free     shared    buffers     cached
Mem:          7872       7656        215          0        284       4733
-/+ buffers/cache:       2639       5233
swap:         2047          0       2047


 CurrentDomain_UnhandledException(): Out of memory ->   at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr)
  at (wrapper alloc) System.Object:AllocVector (intptr,intptr)
  at System.IO.Stream+<CopyToAsyncInternal>c__async0.MoveNext () <0x40466ca0 + 0x000a3> in <filename unknown>:0
--- End of stack trace from previous location where exception was thrown ---
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr)
  at (wrapper alloc) System.Object:AllocVector (intptr,intptr)
  at System.Net.HttpConnection.BeginReadRequest () <0x403c5ea0 + 0x00033> in <filename unknown>:0
  at System.Net.EndPointListener.OnAccept (System.Object sender, System.EventArgs e) <0x403c0000 + 0x003df> in <filename unknown>:0
  at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted (System.Net.Sockets.SocketAsyncEventArgs e) <0x403bfda0 + 0x0002e> in <filename unknown>:0
  at System.Net.Sockets.SocketAsyncEventArgs.Complete () <0x403bfd20 + 0x00013> in <filename unknown>:0
  at System.Net.Sockets.Socket.<AcceptAsyncCallback>m__0 (IAsyncResult ares) <0x403bf6e0 + 0x0037f> in <filename unknown>:0
  at System.Net.Sockets.SocketAsyncResult+<Complete>c__AnonStorey0.<>m__0 (System.Object _) <0x40319170 + 0x0001d> in <filename unknown>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () <0x40319110 + 0x0002f> in <filename unknown>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x4012e6e0 + 0x001e4> in <filename unknown>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x4012e450 + 0x0000b> in <filename unknown>:0
Comment 7 Vlad Brezae 2017-01-21 00:52:15 UTC
From the stacktrace it looks like you are using a 32 bit mono. On a 32 bit process I wouldn't count on more than 2 GB for an application due to lack of address space. So it doesn't matter if the system has 5 GB free if the mono process has a small address space.
Comment 8 Albert 2017-01-21 01:06:19 UTC
Vlad, that is right process is 32bit. But i have done test program to fill memory with 32-bit and 64-bit and seeing the same result, 64-bit process crashes oom without using more memory. In windows works fine.

I explained in this issue:

https://bugzilla.xamarin.com/show_bug.cgi?id=51677

Thanks,
Alberto
Comment 9 Albert 2017-02-24 13:57:11 UTC
I have run this with new mono 4.8 instead of 4.4 and error is different, i think the out of mem error is more clear,  why getting value cannot be null?

Unhandled Exception:
System.ArgumentNullException: Value cannot be null.
Parameter name: destinationArray
  at System.Array.Copy (System.Array sourceArray, System.Int32 sourceIndex, System.Arr               ay destinationArray, System.Int32 destinationIndex, System.Int32 length) [0x00017] in                <373b6e083d6e45e498c9082a8eebd27f>:0
  at System.Collections.Generic.List`1[T].set_Capacity (System.Int32 value) [0x0003d]                in <373b6e083d6e45e498c9082a8eebd27f>:0
  at System.Collections.Generic.List`1[T].EnsureCapacity (System.Int32 min) [0x00046]                in <373b6e083d6e45e498c9082a8eebd27f>:0
  at System.Collections.Generic.List`1[T].Add (T item) [0x00013] in <373b6e083d6e45e49               8c9082a8eebd27f>:0
  at outofmem.Program.Main (System.String[] args) [0x0001f] in <793a908f11c944079a5375               2dad9faaf0>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Value cannot be null.
Parameter name: destinationArray
  at System.Array.Copy (System.Array sourceArray, System.Int32 sourceIndex, System.Arr               ay destinationArray, System.Int32 destinationIndex, System.Int32 length) [0x00017] in                <373b6e083d6e45e498c9082a8eebd27f>:0
  at System.Collections.Generic.List`1[T].set_Capacity (System.Int32 value) [0x0003d]                in <373b6e083d6e45e498c9082a8eebd27f>:0
  at System.Collections.Generic.List`1[T].EnsureCapacity (System.Int32 min) [0x00046]                in <373b6e083d6e45e498c9082a8eebd27f>:0
  at System.Collections.Generic.List`1[T].Add (T item) [0x00013] in <373b6e083d6e45e49               8c9082a8eebd27f>:0
  at outofmem.Program.Main (System.String[] args) [0x0001f] in <793a908f11c944079a5375               2dad9faaf0>:0
Comment 10 Albert 2017-05-12 12:42:41 UTC
I have tried this in mono 5.0 and it works better process as process start using cache memory and can get more memory, has been any change for that?  but after some some time i got a native error and next run a proper out of meemory error:

native error:

7fffc5400000-7fffc5500000 rw-p 00000000 00:00 0
7fffc5600000-7fffc5700000 rw-p 00000000 00:00 0
7fffc5800000-7fffc5900000 rw-p 00000000 00:00 0
7fffc5a00000-7fffc5b00000 rw-p 00000000 00:00 0
7fffc5c00000-7fffc5d00000 rw-p 00000000 00:00 0
7fffc5e00000-7fffc5f00000 rw-p 00000000 00:00 0
7fffc6000000-7fffc6100000 rw-p 00000000 00:00 0
7fffc6200000-7fffc6300000 rw-p 00000000 00:00 0
7fffc6400000-7fffc6500000 rw-p 00000000 00:00 0
7fffc6600000-7fffc6700000 rw-p 00000000 00:00 0
7fffc6800000-7fffc6900000 rw-p 00000000 00:00 0
7fffc6a00000-7fffc6b00000 rw-p 00000000 00:00 0
7fffc6c00000-7fffc6d00000 rw-p 00000000 00:00 0
7fffc6e00000-7fffc6f00000 rw-p 00000000 00:00 0
7fffc7000000-7fffc7100000 rw-p 00000000 00:00 0
7fffc7200000-7fffc7300000 rw-p 00000000 00:00 0
7fffc7400000-7fffc7500000 rw-p 00000000 00:00 0
7fffc7600000-7fffc7700000 rw-p 00000000 00:00 0
7fffc7800000-7fffc7900000 rw-p 00000000 00:00 0
7fffc7a00000-7fffc7b00000 rw-p 00000000 00:00 0
7fffc7c00000-7fffc7d00000 rw-p 00000000 00:00 0
7fffc7e00000-7fffc7f00000 rw-p 00000000 00:00 0
7fffc8000000-7fffc8100000 rw-p 00000000 00:00 0
7fffc8200000-7fffc8300000 rw-p 00000000 00:00 0
7fffc8400000-7fffc8500000 rw-p 00000000 00:00 0
7fffc8600000-7fffc8700000 rw-p 00000000 00:00 0
7fffc8800000-7fffc8900000 rw-p 00000000 00:00 0
7fffc8a00000-7fffc8b00000 rw-p 00000000 00:00 0
7fffc8c00000-7fffc8d00000 rw-p 00000000 00:00 0
7fffc8e00000-7fffc8f00000 rw-p 00000000 00:00 0
7fffc9000000-7fffc9100000 rw-p 00000000 00:00 0
7fffc9200000-7fffc9300000 rw-p 00000000 00:00 0
7fffc9400000-7fffc9500000 rw-p 00000000 00:00 0
7fffc9600000-7fffc9700000 rw-p 00000000 00:00 0
7fffc9800000-7fffc9900000 rw-p 00000000 00:00 0
7fffc9a00000-7fffc9b00000 rw-p 00000000 00:00 0
7fffc9c00000-7fffc9d00000 rw-p 00000000 00:00 0
7fffc9e00000-7fffc9f00000 rw-p 00000000 00:00 0
7fffca000000-7fffca100000 rw-p 00000000 00:00 0
7fffca200000-7fffca300000 rw-p 00000000 00:00 0
7fffca400000-7fffca500000 rw-p 00000000 00:00 0
7fffca600000-7fffca700000 rw-p 00000000 00:00 0
7fffca800000-7fffca900000 rw-p 00000000 00:00 0
7fffcaa00000-7fffcab00000 rw-p 00000000 00:00 0
7fffcac00000-7fffcad00000 rw-p 00000000 00:00 0
7fffcae00000-7fffcaf00000 rw-p 00000000 00:00 0
7fffcb000000-7fffcb100000 rw-p 00000000 00:00 0
7fffcb200000-7fffcb300000 rw-p 00000000 00:00 0
7fffcb400000-7fffcb500000 rw-p 00000000 00:00 0
7fffcb600000-7fffcb700000 rw-p 00000000 00:00 0
7fffcb800000-7fffcb900000 rw-p 00000000 00:00 0
7fffcba00000-7fffcbb00000 rw-p 00000000 00:00 0
7fffcbc00000-7fffcbd00000 rw-p 00000000 00:00 0
7fffcbe00000-7fffcbf00000 rw-p 00000000 00:00 0
7fffcc000000-7fffcc100000 rw-p 00000000 00:00 0
7fffcc200000-7fffcc300000 rw-p 00000000 00:00 0
7fffcc400000-7fffcc500000 rw-p 00000000 00:00 0
7fffcc600000-7fffcc700000 rw-p 00000000 00:00 0
7fffcc800000-7fffcc900000 rw-p 00000000 00:00 0
7fffcca00000-7fffccb00000 rw-p 00000000 00:00 0
7fffccc00000-7fffccd00000 rw-p 00000000 00:00 0
7fffcce00000-7fffccf00000 rw-p 00000000 00:00 0
7fffcd000000-7fffcd100000 rw-p 00000000 00:00 0
7fffcd200000-7fffcd300000 rw-p 00000000 00:00 0
7fffcd400000-7fffcd500000 rw-p 00000000 00:00 0
7fffcd600000-7fffcd700000 rw-p 00000000 00:00 0
7fffcd800000-7fffcd900000 rw-p 00000000 00:00 0
7fffcda00000-7fffcdb00000 rw-p 00000000 00:00 0
7fffcdc00000-7fffcdd00000 rw-p 00000000 00:00 0
7fffcde00000-7fffcdf00000 rw-p 00000000 00:00 0
7fffce000000-7fffce100000 rw-p 00000000 00:00 0
7fffce200000-7fffce300000 rw-p 00000000 00:00 0
7fffce400000-7fffce500000 rw-p 00000000 00:00 0
7fffce600000-7fffce700000 rw-p 00000000 00:00 0
7fffce800000-7fffce900000 rw-p 00000000 00:00 0
7fffcea00000-7fffceb00000 rw-p 00000000 00:00 0
7fffcec00000-7fffced00000 rw-p 00000000 00:00 0
7fffcee00000-7fffcef00000 rw-p 00000000 00:00 0
7fffcf000000-7fffcf100000 rw-p 00000000 00:00 0
7fffcf200000-7fffcf300000 rw-p 00000000 00:00 0
7fffcf400000-7fffcf500000 rw-p 00000000 00:00 0
7fffcf600000-7fffcf700000 rw-p 00000000 00:00 0
7fffcf800000-7fffcf900000 rw-p 00000000 00:00 0
7fffcfa00000-7fffcfb00000 rw-p 00000000 00:00 0
7fffcfc00000-7fffcfd00000 rw-p 00000000 00:00 0
7fffcfe00000-7fffcff00000 rw-p 00000000 00:00 0
7fffd0000000-7fffd0100000 rw-p 00000000 00:00 0
7fffd0200000-7fffd0300000 rw-p 00000000 00:00 0
7fffd0400000-7fffd0500000 rw-p 00000000 00:00 0
7fffd0600000-7fffd0700000 rw-p 00000000 00:00 0
7fffd0800000-7fffd0900000 rw-p 00000000 00:00 0
7fffd0a00000-7fffd0b00000 rw-p 00000000 00:00 0
7fffd0c00000-7fffd0d00000 rw-p 00000000 00:00 0
7fffd0e00000-7fffd0f00000 rw-p 00000000 00:00 0
7fffd1000000-7fffd1100000 rw-p 00000000 00:00 0
7fffd1200000-7fffd1300000 rw-p 00000000 00:00 0
7fffd1400000-7fffd1500000 rw-p 00000000 00:00 0
7fffd1600000-7fffd1700000 rw-p 00000000 00:00 0
7fffd1800000-7fffd1900000 rw-p 00000000 00:00 0
7fffd1a00000-7fffd1b00000 rw-p 00000000 00:00 0
7fffd1c00000-7fffd1d00000 rw-p 00000000 00:00 0
7fffd1e00000-7fffd1f00000 rw-p 00000000 00:00 0
7fffd2000000-7fffd2100000 rw-p 00000000 00:00 0
7fffd2200000-7fffd2300000 rw-p 00000000 00:00 0
7fffd2400000-7fffd2500000 rw-p 00000000 00:00 0
7fffd2600000-7fffd2700000 rw-p 00000000 00:00 0
7fffd2800000-7fffd2900000 rw-p 00000000 00:00 0
7fffd2a00000-7fffd2b00000 rw-p 00000000 00:00 0
7fffd2c00000-7fffd2d00000 rw-p 00000000 00:00 0
7fffd2e00000-7fffd2f00000 rw-p 00000000 00:00 0
7fffd3000000-7fffd3100000 rw-p 00000000 00:00 0
7fffd3200000-7fffd3300000 rw-p 00000000 00:00 0
7fffd3400000-7fffd3500000 rw-p 00000000 00:00 0
7fffd3600000-7fffd3700000 rw-p 00000000 00:00 0
7fffd3800000-7fffd3900000 rw-p 00000000 00:00 0
7fffd3a00000-7fffd3b00000 rw-p 00000000 00:00 0
7fffd3c00000-7fffd3d00000 rw-p 00000000 00:00 0
7fffd3e00000-7fffd3f00000 rw-p 00000000 00:00 0
7fffd4000000-7fffd4100000 rw-p 00000000 00:00 0
7fffd4200000-7fffd4300000 rw-p 00000000 00:00 0
7fffd4400000-7fffd4500000 rw-p 00000000 00:00 0
7fffd4600000-7fffd4700000 rw-p 00000000 00:00 0
7fffd4800000-7fffd4900000 rw-p 00000000 00:00 0
7fffd4a00000-7fffd4b00000 rw-p 00000000 00:00 0
7fffd4c00000-7fffd4d00000 rw-p 00000000 00:00 0
7fffd4e00000-7fffd4f00000 rw-p 00000000 00:00 0
7fffd5000000-7fffd5100000 rw-p 00000000 00:00 0
7fffd5200000-7fffd5300000 rw-p 00000000 00:00 0
7fffd5400000-7fffd5500000 rw-p 00000000 00:00 0
7fffd5600000-7fffd5700000 rw-p 00000000 00:00 0
7fffd5800000-7fffd5900000 rw-p 00000000 00:00 0
7fffd5a00000-7fffd5b00000 rw-p 00000000 00:00 0
7fffd5c00000-7fffd5d00000 rw-p 00000000 00:00 0
7fffd5e00000-7fffd5f00000 rw-p 00000000 00:00 0
7fffd6000000-7fffd6100000 rw-p 00000000 00:00 0
7fffd6200000-7fffd6300000 rw-p 00000000 00:00 0
7fffd6400000-7fffd6500000 rw-p 00000000 00:00 0
7fffd6600000-7fffd6700000 rw-p 00000000 00:00 0
7fffd6800000-7fffd6900000 rw-p 00000000 00:00 0
7fffd6a00000-7fffd6b00000 rw-p 00000000 00:00 0
7fffd6c00000-7fffd6d00000 rw-p 00000000 00:00 0
7fffd6e00000-7fffd6f00000 rw-p 00000000 00:00 0
7fffd7000000-7fffd7100000 rw-p 00000000 00:00 0
7fffd7200000-7fffd7300000 rw-p 00000000 00:00 0
7fffd7400000-7fffd7500000 rw-p 00000000 00:00 0
7fffd7600000-7fffd7700000 rw-p 00000000 00:00 0
7fffd7800000-7fffd7900000 rw-p 00000000 00:00 0
7fffd7a00000-7fffd7b00000 rw-p 00000000 00:00 0
7fffd7c00000-7fffd7d00000 rw-p 00000000 00:00 0
7fffd7e00000-7fffd7f00000 rw-p 00000000 00:00 0
7fffd8000000-7fffd8100000 rw-p 00000000 00:00 0
7fffd8200000-7fffd8300000 rw-p 00000000 00:00 0
7fffd8400000-7fffd8500000 rw-p 00000000 00:00 0
7fffd8600000-7fffd8700000 rw-p 00000000 00:00 0
7fffd8800000-7fffd8900000 rw-p 00000000 00:00 0
7fffd8a00000-7fffd8b00000 rw-p 00000000 00:00 0
7fffd8c00000-7fffd8d00000 rw-p 00000000 00:00 0
7fffd8e00000-7fffd8f00000 rw-p 00000000 00:00 0
7fffd9000000-7fffd9100000 rw-p 00000000 00:00 0
7fffd9200000-7fffd9300000 rw-p 00000000 00:00 0
7fffd9400000-7fffd9500000 rw-p 00000000 00:00 0
7fffd9600000-7fffd9700000 rw-p 00000000 00:00 0
7fffd9800000-7fffd9900000 rw-p 00000000 00:00 0
7fffd9a00000-7fffd9b00000 rw-p 00000000 00:00 0
7fffd9c00000-7fffd9d00000 rw-p 00000000 00:00 0
7fffd9e00000-7fffd9f00000 rw-p 00000000 00:00 0
7fffda000000-7fffda100000 rw-p 00000000 00:00 0
7fffda200000-7fffda300000 rw-p 00000000 00:00 0
7fffda400000-7fffda500000 rw-p 00000000 00:00 0
7fffda600000-7fffda700000 rw-p 00000000 00:00 0
7fffda800000-7fffda900000 rw-p 00000000 00:00 0
7fffdaa00000-7fffdab00000 rw-p 00000000 00:00 0
7fffdac00000-7fffdad00000 rw-p 00000000 00:00 0
7fffdae00000-7fffdaf00000 rw-p 00000000 00:00 0
7fffdb000000-7fffdb100000 rw-p 00000000 00:00 0
7fffdb200000-7fffdb300000 rw-p 00000000 00:00 0
7fffdb400000-7fffdb500000 rw-p 00000000 00:00 0
7fffdb600000-7fffdb700000 rw-p 00000000 00:00 0
7fffdb800000-7fffdb900000 rw-p 00000000 00:00 0
7fffdba00000-7fffdbb00000 rw-p 00000000 00:00 0
7fffdbc00000-7fffdbd00000 rw-p 00000000 00:00 0
7fffdbe00000-7fffdbf00000 rw-p 00000000 00:00 0
7fffdc000000-7fffdc100000 rw-p 00000000 00:00 0
7fffdc200000-7fffdc300000 rw-p 00000000 00:00 0
7fffdc400000-7fffdc500000 rw-p 00000000 00:00 0
7fffdc600000-7fffdc700000 rw-p 00000000 00:00 0
7fffdc800000-7fffdc900000 rw-p 00000000 00:00 0
7fffdca00000-7fffdcb00000 rw-p 00000000 00:00 0
7fffdcc00000-7fffdcd00000 rw-p 00000000 00:00 0
7fffdce00000-7fffdcf00000 rw-p 00000000 00:00 0
7fffdd000000-7fffdd100000 rw-p 00000000 00:00 0
7fffdd200000-7fffdd300000 rw-p 00000000 00:00 0
7fffdd400000-7fffdd500000 rw-p 00000000 00:00 0
7fffdd600000-7fffdd700000 rw-p 00000000 00:00 0
7fffdd800000-7fffdd900000 rw-p 00000000 00:00 0
7fffdda00000-7fffddb00000 rw-p 00000000 00:00 0
7fffddc00000-7fffddd00000 rw-p 00000000 00:00 0
7fffdde00000-7fffddf00000 rw-p 00000000 00:00 0
7fffde000000-7fffde100000 rw-p 00000000 00:00 0
7fffde200000-7fffde300000 rw-p 00000000 00:00 0
7fffde400000-7fffde500000 rw-p 00000000 00:00 0
7fffde600000-7fffde700000 rw-p 00000000 00:00 0
7fffde800000-7fffde900000 rw-p 00000000 00:00 0
7fffdea00000-7fffdeb00000 rw-p 00000000 00:00 0
7fffdec00000-7fffded00000 rw-p 00000000 00:00 0
7fffdee00000-7fffdef00000 rw-p 00000000 00:00 0
7fffdf000000-7fffdf100000 rw-p 00000000 00:00 0
7fffdf200000-7fffdf300000 rw-p 00000000 00:00 0
7fffdf400000-7fffdf500000 rw-p 00000000 00:00 0
7fffdf600000-7fffdf700000 rw-p 00000000 00:00 0
7fffdf800000-7fffdf900000 rw-p 00000000 00:00 0
7fffdfa00000-7fffdfb00000 rw-p 00000000 00:00 0
7fffdfc00000-7fffdfd00000 rw-p 00000000 00:00 0
7fffdfe00000-7fffdff00000 rw-p 00000000 00:00 0
7fffe0000000-7fffe0100000 rw-p 00000000 00:00 0
7fffe0200000-7fffe0300000 rw-p 00000000 00:00 0
7fffe0400000-7fffe0500000 rw-p 00000000 00:00 0
7fffe0600000-7fffe0700000 rw-p 00000000 00:00 0
7fffe0800000-7fffe0900000 rw-p 00000000 00:00 0
7fffe0a00000-7fffe0b00000 rw-p 00000000 00:00 0
7fffe0c00000-7fffe0d00000 rw-p 00000000 00:00 0
7fffe0e00000-7fffe0f00000 rw-p 00000000 00:00 0
7fffe1000000-7fffe1100000 rw-p 00000000 00:00 0
7fffe1200000-7fffe1300000 rw-p 00000000 00:00 0
7fffe1400000-7fffe1500000 rw-p 00000000 00:00 0
7fffe1600000-7fffe1700000 rw-p 00000000 00:00 0
7fffe1800000-7fffe1900000 rw-p 00000000 00:00 0
7fffe1a00000-7fffe1b00000 rw-p 00000000 00:00 0
7fffe1c00000-7fffe1d00000 rw-p 00000000 00:00 0
7fffe1e00000-7fffe1f00000 rw-p 00000000 00:00 0
7fffe2000000-7fffe2100000 rw-p 00000000 00:00 0
7fffe2200000-7fffe2300000 rw-p 00000000 00:00 0
7fffe2400000-7fffe2500000 rw-p 00000000 00:00 0
7fffe2600000-7fffe2700000 rw-p 00000000 00:00 0
7fffe2800000-7fffe2900000 rw-p 00000000 00:00 0
7fffe2a00000-7fffe2b00000 rw-p 00000000 00:00 0
7fffe2c00000-7fffe2d00000 rw-p 00000000 00:00 0
7fffe2e00000-7fffe2f00000 rw-p 00000000 00:00 0
7fffe3000000-7fffe3100000 rw-p 00000000 00:00 0
7fffe3200000-7fffe3300000 rw-p 00000000 00:00 0
7fffe3400000-7fffe3500000 rw-p 00000000 00:00 0
7fffe3600000-7fffe3700000 rw-p 00000000 00:00 0
7fffe3800000-7fffe3900000 rw-p 00000000 00:00 0
7fffe3a00000-7fffe3b00000 rw-p 00000000 00:00 0
7fffe3c00000-7fffe3d00000 rw-p 00000000 00:00 0
7fffe3e00000-7fffe3f00000 rw-p 00000000 00:00 0
7fffe4000000-7fffe4100000 rw-p 00000000 00:00 0
7fffe4200000-7fffe4300000 rw-p 00000000 00:00 0
7fffe4400000-7fffe4500000 rw-p 00000000 00:00 0
7fffe4600000-7fffe4700000 rw-p 00000000 00:00 0
7fffe4800000-7fffe4900000 rw-p 00000000 00:00 0
7fffe4a00000-7fffe4b00000 rw-p 00000000 00:00 0
7fffe4c00000-7fffe4d00000 rw-p 00000000 00:00 0
7fffe4e00000-7fffe4f00000 rw-p 00000000 00:00 0
7fffe5000000-7fffe5100000 rw-p 00000000 00:00 0
7fffe5200000-7fffe5300000 rw-p 00000000 00:00 0
7fffe5400000-7fffe5500000 rw-p 00000000 00:00 0
7fffe5600000-7fffe5700000 rw-p 00000000 00:00 0
7fffe5800000-7fffe5900000 rw-p 00000000 00:00 0
7fffe5a00000-7fffe5b00000 rw-p 00000000 00:00 0
7fffe5c00000-7fffe5d00000 rw-p 00000000 00:00 0
7fffe5e00000-7fffe5f00000 rw-p 00000000 00:00 0
7fffe6000000-7fffe6100000 rw-p 00000000 00:00 0
7fffe6200000-7fffe6300000 rw-p 00000000 00:00 0
7fffe6400000-7fffe6500000 rw-p 00000000 00:00 0
7fffe6600000-7fffe6700000 rw-p 00000000 00:00 0
7fffe6800000-7fffe6900000 rw-p 00000000 00:00 0
7fffe6a00000-7fffe6b00000 rw-p 00000000 00:00 0
7fffe6c00000-7fffe6d00000 rw-p 00000000 00:00 0
7fffe6e00000-7fffe6f00000 rw-p 00000000 00:00 0
7fffe7000000-7fffe7100000 rw-p 00000000 00:00 0
7fffe7200000-7fffe7300000 rw-p 00000000 00:00 0
7fffe7400000-7fffe7500000 rw-p 00000000 00:00 0
7fffe7600000-7fffe7700000 rw-p 00000000 00:00 0
7fffe7800000-7fffe7900000 rw-p 00000000 00:00 0
7fffe7a00000-7fffe7b00000 rw-p 00000000 00:00 0
7fffe7c00000-7fffe7d00000 rw-p 00000000 00:00 0
7fffe7e00000-7fffe7f00000 rw-p 00000000 00:00 0
7fffe8000000-7fffe8021000 rw-p 00000000 00:00 0
7fffe8021000-7fffec000000 ---p 00000000 00:00 0
7fffec000000-7fffec100000 rw-p 00000000 00:00 0
7fffec200000-7fffec300000 rw-p 00000000 00:00 0
7fffec400000-7fffec500000 rw-p 00000000 00:00 0
7fffec600000-7fffec700000 rw-p 00000000 00:00 0
7fffec800000-7fffec900000 rw-p 00000000 00:00 0
7fffeca00000-7fffecb00000 rw-p 00000000 00:00 0
7fffecc00000-7fffecd00000 rw-p 00000000 00:00 0
7fffece00000-7fffecf00000 rw-p 00000000 00:00 0
7fffed000000-7fffed100000 rw-p 00000000 00:00 0
7fffed200000-7fffed300000 rw-p 00000000 00:00 0
7fffed400000-7fffed500000 rw-p 00000000 00:00 0
7fffed600000-7fffed700000 rw-p 00000000 00:00 0
7fffed800000-7fffed900000 rw-p 00000000 00:00 0
7fffeda00000-7fffedb00000 rw-p 00000000 00:00 0
7fffedc00000-7fffedd00000 rw-p 00000000 00:00 0
7fffede00000-7fffedf00000 rw-p 00000000 00:00 0
7fffee000000-7fffee100000 rw-p 00000000 00:00 0
7fffee200000-7fffee300000 rw-p 00000000 00:00 0
7fffee36e000-7fffee377000 ---p 00000000 00:00 0
7fffee377000-7fffee56f000 rw-p 00000000 00:00 0
7fffee56f000-7fffeec39000 r-xp 00000000 08:05 133883                     /usr/lib/mono/4.5/mscorlib.dll.so
7fffeec39000-7fffeee38000 ---p 006ca000 08:05 133883                     /usr/lib/mono/4.5/mscorlib.dll.so
7fffeee38000-7fffeee39000 rw-p 006c9000 08:05 133883                     /usr/lib/mono/4.5/mscorlib.dll.so
7fffeee39000-7fffeee56000 rw-p 00000000 00:00 0
7fffeee56000-7fffef1ff000 r--p 00000000 08:05 133882                     /usr/lib/mono/4.5/mscorlib.dll
7fffef1ff000-7ffff01ff000 rw-p 00000000 00:00 0
7ffff01ff000-7ffff0200000 ---p 00000000 00:00 0
7ffff0200000-7ffff1000000 rw-p 00000000 00:00 0
7ffff10f0000-7ffff10f5000 rw-p 00000000 00:00 0
7ffff10f5000-7ffff6f86000 r--p 00000000 08:05 474                        /usr/lib/locale/locale-archive
7ffff6f86000-7ffff7110000 r-xp 00000000 08:02 398                        /lib64/libc-2.12.so
7ffff7110000-7ffff7310000 ---p 0018a000 08:02 398                        /lib64/libc-2.12.so
7ffff7310000-7ffff7314000 r--p 0018a000 08:02 398                        /lib64/libc-2.12.so
7ffff7314000-7ffff7315000 rw-p 0018e000 08:02 398                        /lib64/libc-2.12.so
7ffff7315000-7ffff731a000 rw-p 00000000 00:00 0
7ffff731a000-7ffff7330000 r-xp 00000000 08:02 32                         /lib64/libgcc_s-4.4.7-20120601.so.1
7ffff7330000-7ffff752f000 ---p 00016000 08:02 32                         /lib64/libgcc_s-4.4.7-20120601.so.1
7ffff752f000-7ffff7530000 rw-p 00015000 08:02 32                         /lib64/libgcc_s-4.4.7-20120601.so.1
7ffff7530000-7ffff7547000 r-xp 00000000 08:02 422                        /lib64/libpthread-2.12.so
7ffff7547000-7ffff7747000 ---p 00017000 08:02 422                        /lib64/libpthread-2.12.so
7ffff7747000-7ffff7748000 r--p 00017000 08:02 422                        /lib64/libpthread-2.12.so
7ffff7748000-7ffff7749000 rw-p 00018000 08:02 422                        /lib64/libpthread-2.12.so
7ffff7749000-7ffff774d000 rw-p 00000000 00:00 0
7ffff774d000-7ffff774f000 r-xp 00000000 08:02 404                        /lib64/libdl-2.12.so
7ffff774f000-7ffff794f000 ---p 00002000 08:02 404                        /lib64/libdl-2.12.so
7ffff794f000-7ffff7950000 r--p 00002000 08:02 404                        /lib64/libdl-2.12.so
7ffff7950000-7ffff7951000 rw-p 00003000 08:02 404                        /lib64/libdl-2.12.so
7ffff7951000-7ffff7958000 r-xp 00000000 08:02 426                        /lib64/librt-2.12.so
7ffff7958000-7ffff7b57000 ---p 00007000 08:02 426                        /lib64/librt-2.12.so
7ffff7b57000-7ffff7b58000 r--p 00006000 08:02 426                        /lib64/librt-2.12.so
7ffff7b58000-7ffff7b59000 rw-p 00007000 08:02 426                        /lib64/librt-2.12.so
7ffff7b59000-7ffff7bdc000 r-xp 00000000 08:02 406                        /lib64/libm-2.12.so
7ffff7bdc000-7ffff7ddb000 ---p 00083000 08:02 406                        /lib64/libm-2.12.so
7ffff7ddb000-7ffff7ddc000 r--p 00082000 08:02 406                        /lib64/libm-2.12.so
7ffff7ddc000-7ffff7ddd000 rw-p 00083000 08:02 406                        /lib64/libm-2.12.so
7ffff7ddd000-7ffff7dfd000 r-xp 00000000 08:02 391                        /lib64/ld-2.12.so
7ffff7e15000-7ffff7e18000 rw-p 00000000 00:00 0
7ffff7e53000-7ffff7e54000 rw-p 00000000 00:00 0
7ffff7e55000-7ffff7e58000 rw-p 00000000 00:00 0
7ffff7e62000-7ffff7ea1000 rw-p 00000000 00:00 0
7ffff7ea1000-7ffff7ea3000 r--p 00000000 fd:01 2542197                    /_Stateless/backups/samba/outofmem.exe
7ffff7ea3000-7ffff7f8b000 rw-p 00000000 00:00 0
7ffff7f8b000-7ffff7fea000 ---p 00000000 00:00 0
7ffff7fea000-7ffff7ff3000 rw-p 00000000 00:00 0
7ffff7ff3000-7ffff7ff9000 rw-p 00000000 00:00 0
7ffff7ff9000-7ffff7ffa000 rw-s 00000000 00:10 16120805                   /dev/shm/mono.25017
7ffff7ffa000-7ffff7ffb000 rw-p 00000000 00:00 0
7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 0001f000 08:02 391                        /lib64/ld-2.12.so
7ffff7ffd000-7ffff7ffe000 rw-p 00020000 08:02 391                        /lib64/ld-2.12.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7fffff600000-7fffff608000 ---p 00000000 00:00 0
7ffffffea000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Native stacktrace:

        mono() [0x49e91b]
        /lib64/libpthread.so.0(+0xf7e0) [0x7ffff753f7e0]
        /lib64/libc.so.6(gsignal+0x35) [0x7ffff6fb8625]
        /lib64/libc.so.6(abort+0x175) [0x7ffff6fb9e05]
        mono() [0x66aab9]
        mono() [0x67f818]
        mono() [0x67f973]
        mono() [0x67f5df]
        mono(mono_mempool_alloc+0xc8) [0x584998]
        mono() [0x4e8523]
        mono(mono_emit_native_call+0x180) [0x421720]
        mono() [0x440ff8]
        mono() [0x5129f0]
        mono() [0x5146b1]
        mono() [0x41a501]
        mono() [0x514a59]
        mono() [0x41a501]
        mono() [0x4a1ed7]
        mono() [0x4a27cf]
        [0x4000178c]

Debug info from gdb:


=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted (core dumped)



oom error:
Unhandled Exception:
System.OutOfMemoryException: Out of memory
  at (wrapper alloc) System.Object:AllocVector (intptr,intptr)
  at outofmem.Program.Main (System.String[] args) [0x0001f] in <793a908f11c944079a53752dad9faaf0>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.OutOfMemoryException: Out of memory
  at (wrapper alloc) System.Object:AllocVector (intptr,intptr)
  at outofmem.Program.Main (System.String[] args) [0x0001f] in <793a908f11c944079a53752dad9faaf0>:0
You have new mail in /var/spool/mail/root
Comment 11 Rodrigo Kumpera 2017-10-11 17:49:03 UTC
Once a process runs out of memory, most of the time it's not possible for the runtime to recover.

If you're running into out-of-memory issues, I suggest setting up an artificial limit with MONO_GC_PARAMS so native code can still function once you hit the artificial managed heap limit.

Note You need to log in before you can comment on or make changes to this bug.