Insufficient JVM memory causes the process to die. Native memory allocation (mmap) failed to map

Deploying many JAVA processes on one server is already the norm for microservices, but there are also some pitfalls.

In the background, some JAVA processes on the test server suddenly hang up. Check the call back log and find the following:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1786867712 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2673), pid=28610, tid=139813184919296
#
# JRE version: Java(TM) SE Runtime Environment (8.0_40-b26) (build 1.8.0_40-b26)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode linux-amd64 compressed oops)
#

Apparently it was caused by the server running out of memory

There are two cases of insufficient memory

Speculation: There are two cases of insufficient memory

Speculation 1 , the memory application used by the JAVA program exceeds the memory allocated by the JVM, and this class will only throw

java.lang.OutOfMemoryError: Java heap space, the JAVA process will not be affected, and you can continue to accept requests to provide services next time

Speculation 2 , when the JVM tries to apply for a piece of memory that exceeds the available (RSS) memory of the system, linux will end the JAVA process and throw the above error.

This scenario is also common: For example, two services are running on a server with 16G memory, and the two services are started at the same time and each allocates 10G memory, because the garbage collection mechanism of the JVM is generally not problematic and can serve normally. When the memory required by two services at the same time exceeds 16G, one service will be killed by linux.

(Note: When the JVM starts, the memory allocated by parameters such as -Xmx will only affect VIRT. For details, see the previous blog " The Size Relationship Between JAVA Process and Linux Memory "

Confirm:

To confirm is the most important, the above are all my speculations

A linux server with 1G memory

 total used free shared buffers cached 
Mem:      984 119 864 0 2 20 
-/+         buffers/cache: 97 886 
Swap:      2044 45 1999

物理内存+swap内存在 3G左右 (886+1999)

加入JAVA程序，因为是要观察JAVA的内存超出是否会终止JAVA的进程，故思路如下设计

1，用一个线程去申请分配内存

2，主线程不停报告存活状态

程序如下

package org.hejinbin.memory.test;

import java.io.IOException;

/**
 * 用于演示内存不足导致JAVA进程退出, 注意:输入的size勿导致超出int （1024 * 1024 * size）
 * 
 * @author 何锦彬 QQ 277803242（包子） 2016.12.29
 */
public class Test {
    public static void main(String[] args) throws IOException, InterruptedException {

        new Thread(new Runnable() {

            public void run() {

                try {
                    //接收测试内存的大小,单位m
                    byte[] byteSize = new byte[50];
                    System.out.print("input want to allocation memory:");
                    int temp = System.in.read(byteSize);
                    String sizeStr = new String(byteSize, 0, temp);
                    int size = Integer.parseInt(sizeStr.trim());
                    System.out.println(1024 * 1024 * size);
                    byte[] bt = new byte[1024 * 1024 * size];
                    System.out.println("succ..allocation: " + size + "m");
                    //阻30分,防止垃圾回收
                    Thread.sleep(1000 * 60 * 30);
                } catch (Exception e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }

            }
        }).start();

        while (true) {
            //1分钟后报告
            Thread.sleep(1000 * 60);
            System.out.println(Thread.currentThread().getId() + ": i‘m alive");
        }
    }
}

证实推测1：

运行程序，输入1000 (这里只要超过了年龄代内存其实就会抛出异常)

技术分享

输出 alive, 证明了推测1，使用内存超出了并不会导致JAVA进程死掉

证实推测2：

打开窗口1 ，运行，输入

root@ubuntu:/home/hejinbin# java -Xmx3048m -Xms3048m org.hejinbin.memory.test.Test

input want to allocation memory:

打开窗口2 ，运行，输入

root@ubuntu:/home/hejinbin# java -Xmx3048m -Xms3048m org.hejinbin.memory.test.Test

input want to allocation memory:

然后在窗口1输入1500

运行如下：

input want to allocation memory:1500
succ..allocation: 1500m

然后在窗口2，输入1500

input want to allocation memory:1500
succ..allocation: 1500m

此时发现窗口1的JAVA进程被杀掉了，完整如下

root@ubuntu:/home/hejinbin# java -Xmx3048m -Xms3048m org.hejinbin.memory.test.Test
input want to allocation memory:1500
succ..allocation: 1500m
Killed

证实了推测2

结论：如今微服务流行（趁点热度）。很多进程同时在一台服务器上跑，必须注意，分配给JAVA的内存只和一定在服务器的可用内存之内。不然很有可能突然被linux干掉一个进程

最后还有个以为，为啥我的进程被killed之后，并没有产生hs_err_pid*.log的日志呢?

转自:http://www.mamicode.com/info-detail-1663276.html

Insufficient JVM memory causes the process to die. Native memory allocation (mmap) failed to map

Guess you like