Kkjcre1p: unable to spawn jobq slave process Happened Intermittently When Memory_target Is Set and Swap Size Is Not Big (Doc ID 2356025.1) | To Bottom |
In this Document
APPLIES TO:Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.2.0.1 [Release 11.2 to 12.2]Information in this document applies to any platform. SYMPTOMSDatabase is running, and the processes which previously connected to database are working fine; however new sessions cannot be created with below symptom:
SQL*Plus: Release 11.2.0.4.0 Production on Wed Sep 28 02:24:24 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to an idle instance. <============ sqlplus shows that it connects to an idle instance SQL> show parameter cpu ORA-01034: ORACLE not available Alert log shows that "Process J000 died, see its trace file" and "kkjcre1p: unable to spawn jobq slave process" at bad time frame. The problem happened intermittently. Memory_target is being used. Strace output shows that a mmap() OS system call failed, the mmap() is trying to "map"/"claim" it will use memory as big as SGA_MAX_SIZE/MEMORY_TARGET size, and OS returned error ENOMEM (Cannot allocate memory).
[pid 16253] 15:38:39.787948 shmat(262149, 0, 0) = ?
[pid 16253] 15:38:39.787980 shmdt(0x7fd939905000) = 0 [pid 16253] 15:38:39.788011 mmap(0x66000000, 17079205888, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 ENOMEM (Cannot allocate memory) <===== map 15GB memory [pid 16253] 15:38:39.788044 munmap(0x60000000, 33554432) = 0 [pid 16253] 15:38:39.788072 munmap(0x62000000, 33554432) = 0 [pid 16253] 15:38:39.788098 munmap(0x64000000, 33554432) = 0 However the free memory is more than 4GB. CAUSESwap size is under configured: When memory_target is configured, server process will try to mmap a memory as big as sga_max_size/memory_target when it is created. (It is not to allocate so much memory, but only to "claim" it could use so much memory in the future, given sga_max_size/memory_target is set). If free memory + free swap is smaller than sga_max_size/memory_target, operating system will fail with ENOMEM (Cannot allocate memory). So if swap size is under configured, the issue will be likely happened. SOLUTIONAdd more swap size (eg, 0.75 x memory size, ideally swap size should at least be equal to memory size) |