[oracle@db trace]$ tail -100 alert_test.log |more
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 32
Number of processor cores in the system is 32
Number of processor sockets in the system is 32
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /app/oracle/product/11.2.0/dbhome_1/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =168
LICENSE_MAX_USERS = 0
SYS auditing is disabled
NUMA system with 3 nodes detected
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
ORACLE_HOME = /app/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: db
Release: 3.10.0-693.el7.x86_64
Version: #1 SMP Tue Aug 22 21:09:27 UTC 2017
Machine: x86_64
VM name: VMWare Version: 6
Using parameter settings in server-side spfile /app/oracle/product/11.2.0/dbhome_1/dbs/spfiletest.ora
System parameters with non-default values:
processes = 1000
sga_target = 20G
control_files = "/app/oracle/oradata/test/control01.ctl"
db_block_size = 8192
compatible = "11.2.0.4.0"
undo_tablespace = "UNDOTBS1"
recyclebin = "OFF"
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=testXDB)"
java_jit_enabled = TRUE
audit_file_dest = "/app/oracle/admin/test/adump"
audit_trail = "NONE"
db_name = "test"
open_cursors = 300
pga_aggregate_target = 8G
diagnostic_dest = "/app/oracle"
Thu Oct 15 13:43:42 2020
PMON started with pid=2, OS id=18304
Thu Oct 15 13:43:42 2020
PSP0 started with pid=3, OS id=18306
Thu Oct 15 13:43:43 2020
VKTM started with pid=4, OS id=18311 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Thu Oct 15 13:43:43 2020
GEN0 started with pid=5, OS id=18315
Thu Oct 15 13:43:43 2020
DIAG started with pid=6, OS id=18317
Thu Oct 15 13:43:43 2020
DBRM started with pid=7, OS id=18319
Thu Oct 15 13:43:43 2020
DIA0 started with pid=8, OS id=18321
Thu Oct 15 13:43:43 2020
MMAN started with pid=9, OS id=18323
Thu Oct 15 13:43:43 2020
DBW0 started with pid=10, OS id=18325
[oracle@db trace]$ tail -200 alert_test.log |more
Current log# 1 seq# 17770 mem# 0: /app/oracle/oradata/test/redo01.log
Thread 1 advanced to log sequence 17771 (LGWR switch)
Current log# 2 seq# 17771 mem# 0: /app/oracle/oradata/test/redo02.log
Thread 1 cannot allocate new log, sequence 17772
Checkpoint not complete
Current log# 2 seq# 17771 mem# 0: /app/oracle/oradata/test/redo02.log
Wed Sep 30 03:21:43 2020
Thread 1 advanced to log sequence 17772 (LGWR switch)
Current log# 3 seq# 17772 mem# 0: /app/oracle/oradata/test/redo03.log
Wed Sep 30 03:33:59 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 03:54:17 2020
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /app/oracle/diag/rdbms/test/test/trace/test_cjq0_7090.trc:
Wed Sep 30 04:58:27 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 06:00:05 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 07:01:31 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 08:03:01 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 09:04:34 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 10:09:09 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 11:41:12 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 11:41:12 2020
Errors in file /app/oracle/diag/rdbms/test/test/trace/test_cjq0_7090.trc (incident=42065):
ORA-00445: background process "J000" did not start after 120 seconds
Incident details in: /app/oracle/diag/rdbms/test/test/incident/incdir_42065/test_cjq0_7090_i42065.trc
Wed Sep 30 11:41:12 2020
Errors in file /app/oracle/diag/rdbms/test/test/trace/test_qmnc_7076.trc (incident=42033):
ORA-00445: background process "q003" did not start after 120 seconds
Incident details in: /app/oracle/diag/rdbms/test/test/incident/incdir_42033/test_qmnc_7076_i42033.trc
Wed Sep 30 13:11:44 2020
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Sep 30 13:11:46 2020
kkjcre1p: unable to spawn jobq slave process
Errors in file /app/oracle/diag/rdbms/test/test/trace/test_cjq0_7090.trc:
Wed Sep 30 13:11:46 2020
Sweep [inc][42065]: completed
Sweep [inc][42033]: completed
Sweep [inc2][42065]: completed
Sweep [inc2][42033]: completed
Wed Sep 30 13:11:44 2020
Dumping diagnostic data in directory=[cdmp_20200930131144], requested by (instance=1, osid=7076 (QMNC)), summary=[incident=42033].
Wed Sep 30 13:11:54 2020
Thread 1 cannot allocate new log, sequence 17773
Checkpoint not complete
Current log# 3 seq# 17772 mem# 0: /app/oracle/oradata/test/redo03.log
Thread 1 advanced to log sequence 17773 (LGWR switch)
Current log# 1 seq# 17773 mem# 0: /app/oracle/oradata/test/redo01.log
Wed Sep 30 13:13:15 2020
KCF: read, write or open error, block=0x2ef94e online=1
file=7 '/app/oracle/oradata/test/users04.dbf'
error=27072 txt: 'Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 3078478
Additional information: -1'
Errors in file /app/oracle/diag/rdbms/test/test/trace/test_dbw2_6981.trc:
Errors in file /app/oracle/diag/rdbms/test/test/trace/test_dbw2_6981.trc:
ORA-63999: data file suffered media failure
ORA-01114: IO error writing block to file 7 (block # 3078478)
ORA-01110: data file 7: '/app/oracle/oradata/test/users04.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 3078478
Additional information: -1
DBW2 (ospid: 6981): terminating the instance due to error 63999
Wed Sep 30 13:13:16 2020
System state dump requested by (instance=1, osid=6981 (DBW2)), summary=[abnormal instance termination].
System State dumped to trace file /app/oracle/diag/rdbms/test/test/trace/test_diag_6969_20200930131316.trc
Dumping diagnostic data in directory=[cdmp_20200930131316], requested by (instance=1, osid=6981 (DBW2)), summary=[abnormal instance terminat
ion].
Instance terminated by DBW2, pid = 6981
Thu Oct 15 13:43:39 2020
Adjusting the default value of parameter parallel_max_servers
from 1280 to 970 due to the value of parameter processes (1000)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = UNLIMITED
Total Shared Global Region in Large Pages = 20 GB (100%)
Large Pages used by this instance: 10241 (20 GB)
Large Pages unused system wide = 4759 (9518 MB)
Large Pages configured system wide = 15000 (29 GB)
Large Page size = 2048 KB
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 32
Number of processor cores in the system is 32
Number of processor sockets in the system is 32
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /app/oracle/product/11.2.0/dbhome_1/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =168
LICENSE_MAX_USERS = 0
SYS auditing is disabled
NUMA system with 3 nodes detected
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
ORACLE_HOME = /app/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: db
Release: 3.10.0-693.el7.x86_64
Version: #1 SMP Tue Aug 22 21:09:27 UTC 2017
Machine: x86_64
VM name: VMWare Version: 6
Using parameter settings in server-side spfile /app/oracle/product/11.2.0/dbhome_1/dbs/spfiletest.ora
System parameters with non-default values:
processes = 1000
sga_target = 20G
control_files = "/app/oracle/oradata/test/control01.ctl"
db_block_size = 8192
compatible = "11.2.0.4.0"
undo_tablespace = "UNDOTBS1"
recyclebin = "OFF"
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=testXDB)"
java_jit_enabled = TRUE
audit_file_dest = "/app/oracle/admin/test/adump"
audit_trail = "NONE"
db_name = "test"
open_cursors = 300
pga_aggregate_target = 8G
diagnostic_dest = "/app/oracle"
Thu Oct 15 13:43:42 2020
PMON started with pid=2, OS id=18304
Thu Oct 15 13:43:42 2020
PSP0 started with pid=3, OS id=18306
Thu Oct 15 13:43:43 2020
VKTM started with pid=4, OS id=18311 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Thu Oct 15 13:43:43 2020
GEN0 started with pid=5, OS id=18315
Thu Oct 15 13:43:43 2020
DIAG started with pid=6, OS id=18317
Thu Oct 15 13:43:43 2020
DBRM started with pid=7, OS id=18319
Thu Oct 15 13:43:43 2020
DIA0 started with pid=8, OS id=18321
Thu Oct 15 13:43:43 2020
MMAN started with pid=9, OS id=18323
Thu Oct 15 13:43:43 2020
DBW0 started with pid=10, OS id=18325
Thu Oct 15 13:43:43 2020
DBW1 started with pid=11, OS id=18327
Thu Oct 15 13:43:43 2020
DBW2 started with pid=12, OS id=18329
Thu Oct 15 13:43:43 2020
DBW3 started with pid=13, OS id=18331
Thu Oct 15 13:43:43 2020
LGWR started with pid=14, OS id=18333
Thu Oct 15 13:43:43 2020
CKPT started with pid=15, OS id=18335
Thu Oct 15 13:43:43 2020
SMON started with pid=16, OS id=18337
Thu Oct 15 13:43:43 2020
RECO started with pid=17, OS id=18339
Thu Oct 15 13:43:43 2020
MMON started with pid=18, OS id=18341
Thu Oct 15 13:43:43 2020
MMNL started with pid=19, OS id=18343
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
ORACLE_BASE from environment = /app/oracle
Thu Oct 15 13:43:43 2020
ALTER DATABASE MOUNT
Successful mount of redo thread 1, with mount id 2458952015
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
Thu Oct 15 13:43:48 2020
ALTER DATABASE OPEN
Errors in file /app/oracle/diag/rdbms/test/test/trace/test_ora_18357.trc:
ORA-01114: IO error writing block to file 6 (block # 1)
ORA-01110: data file 6: '/app/oracle/oradata/test/users03.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1
Additional information: -1
ORA-1114 signalled during: ALTER DATABASE OPEN...
[oracle@db trace]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-root 20G 3.6G 17G 18% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 137M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/VolGroup-opt 40G 34M 40G 1% /opt
/dev/mapper/VolGroup-tmp 10G 33M 10G 1% /tmp
/dev/mapper/VolGroup-home 10G 38M 10G 1% /home
/dev/mapper/VolGroup-var 10G 295M 9.8G 3% /var
/dev/sda1 1011M 179M 833M 18% /boot
/dev/mapper/VolGroup-app 890G 309G 582G 35% /app
tmpfs 6.3G 32K 6.3G 1% /run/user/0
tmpfs 6.3G 0 6.3G 0% /run/user/1000
10.62.226.233:/outFile 350G 339G 12G 97% /app/p_znzc/outfile
[oracle@db trace]$ cd /app/oracle/oradata/test
[oracle@db test]$ ll
total 290259016
-rw-r----- 1 oracle oracle 10764288 Oct 15 13:48 control01.ctl
-rw-r----- 1 oracle oracle 209715712 Sep 30 13:12 redo01.log
-rw-r----- 1 oracle oracle 209715712 Sep 30 03:21 redo02.log
-rw-r----- 1 oracle oracle 209715712 Sep 30 13:11 redo03.log
-rw-r----- 1 oracle oracle 2147491840 Oct 15 13:43 sysaux01.dbf
-rw-r----- 1 oracle oracle 2147491840 Oct 15 13:43 system01.dbf
-rw-r----- 1 oracle oracle 34358697984 Sep 29 22:06 temp01.dbf
-rw-r----- 1 oracle oracle 11544829952 Oct 15 13:43 undotbs01.dbf
-rw-r----- 1 oracle oracle 13371449344 Sep 30 13:11 undotbs02.dbf
-rw-r----- 1 oracle oracle 31331459072 Oct 15 13:43 users01.dbf
-rw-r----- 1 oracle oracle 31541174272 Oct 15 13:43 users02.dbf
-rw-r----- 1 oracle oracle 31331459072 Oct 15 13:43 users03.dbf
-rw-r----- 1 oracle oracle 26249011200 Sep 30 13:13 users04.dbf
-rw-r----- 1 oracle oracle 26247962624 Sep 30 13:12 users05.dbf
-rw-r----- 1 oracle oracle 25216163840 Sep 30 13:12 users06.dbf
-rw-r----- 1 oracle oracle 30871134208 Sep 30 13:12 users07.dbf
-rw-r----- 1 oracle oracle 30870085632 Sep 30 13:12 users08.dbf
[oracle@db test]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Thu Oct 15 13:58:26 2020
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> set line 132
SQL> set wrap off
SQL> col name for a80
SQL> select name,file#,status from v$datafile
NAME FILE# STATUS
-------------------------------------------------------------------------------- ---------- -------
/app/oracle/oradata/test/system01.dbf 1 SYSTEM
/app/oracle/oradata/test/sysaux01.dbf 2 ONLINE
/app/oracle/oradata/test/undotbs01.dbf 3 ONLINE
/app/oracle/oradata/test/users01.dbf 4 ONLINE
/app/oracle/oradata/test/users02.dbf 5 ONLINE
/app/oracle/oradata/test/users03.dbf 6 ONLINE
/app/oracle/oradata/test/users04.dbf 7 ONLINE
/app/oracle/oradata/test/users05.dbf 8 ONLINE
/app/oracle/oradata/test/undotbs02.dbf 9 ONLINE
/app/oracle/oradata/test/users06.dbf 10 ONLINE
/app/oracle/oradata/test/users07.dbf 11 ONLINE
NAME FILE# STATUS
-------------------------------------------------------------------------------- ---------- -------
/app/oracle/oradata/test/users08.dbf 12 ONLINE
12 rows selected.
SQL> Select * from v$recover_file ;
no rows selected
SQL> select status from v$instance;
STATUS
------------
MOUNTED
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01114: IO error writing block to file 8 (block # 1)
ORA-01110: data file 8: '/app/oracle/oradata/test/users05.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1
Additional information: -1
SQL> recover datafile 8;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01114: IO error writing block to file 10 (block # 1)
ORA-01110: data file 10: '/app/oracle/oradata/test/users06.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1
Additional information: -1
SQL> recover datafile 10;
Media recovery complete.
SQL> alter database open;
Database altered.
SQL> select name,file#,status from v$datafile;
NAME FILE# STATUS
-------------------------------------------------------------------------------- ---------- -------
/app/oracle/oradata/test/system01.dbf 1 SYSTEM
/app/oracle/oradata/test/sysaux01.dbf 2 ONLINE
/app/oracle/oradata/test/undotbs01.dbf 3 ONLINE
/app/oracle/oradata/test/users01.dbf 4 ONLINE
/app/oracle/oradata/test/users02.dbf 5 ONLINE
/app/oracle/oradata/test/users03.dbf 6 ONLINE
/app/oracle/oradata/test/users04.dbf 7 ONLINE
/app/oracle/oradata/test/users05.dbf 8 ONLINE
/app/oracle/oradata/test/undotbs02.dbf 9 ONLINE
/app/oracle/oradata/test/users06.dbf 10 ONLINE
/app/oracle/oradata/test/users07.dbf 11 ONLINE
NAME FILE# STATUS
-------------------------------------------------------------------------------- ---------- -------
/app/oracle/oradata/test/users08.dbf 12 ONLINE
12 rows selected.
SQL> archive log list;
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination /app/oracle/product/11.2.0/dbhome_1/dbs/arch
Oldest online log sequence 17772
Current log sequence 17774
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 2.1379E+10 bytes
Fixed Size 2262656 bytes
Variable Size 8522828160 bytes
Database Buffers 1.2818E+10 bytes
Redo Buffers 36073472 bytes
Database mounted.
Database opened.
SQL> exit