我在一个虚拟机上（4核，16G内存）上跑Tomcat应用，压力到了1000qps的时候，经过了20分钟，JVM就crash了。

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa1e0775b5c, pid=29733, tid=140333024683776
#
# JRE version: 6.0_45-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.45-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x3cfb5c]  Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x21c
#
# If you would like to submit a bug report, please visit:

具体信息在后面，请帮忙看一下吧，多谢

下边是我的JVM配置：

CATALINA_OPTS="$CATALINA_OPTS -Xss256k -Xms4G -Xmx4G -Xmn1G -XX:PermSize=60M -XX:MaxPermSize=60M -XX:SurvivorRatio=8 -XX:CMSFullGCsBeforeCompaction=3 -XX:CMSInitiatingOccupancyFraction=60 -XX:MaxTenuringThreshold=30 -XX:+CMSScavengeBeforeRemark -XX:+ScavengeBeforeFullGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:+TieredCompilation -XX:ParallelCMSThreads=15 -XX:+UseFastAccessorMethods -XX:LargePageSizeInBytes=128M -XX:+UseFastAccessorMethods -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -Xloggc:/home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/logs/gc.log"


================================================



#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa1e0775b5c, pid=29733, tid=140333024683776
#
# JRE version: 6.0_45-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.45-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x3cfb5c]  Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x21c
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007fa1dc0b2000):  GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=29743]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x000000020a7a19f0

Registers:
RAX=0x0000000000000001, RBX=0x00007fa1e0e3a050, RCX=0x0000000000000003, RDX=0x000000020a7a19e0
RSP=0x00007fa1d4161840, RBP=0x00007fa1d4161910, RSI=0x00000007b10e8f68, RDI=0x00007fa1e0e4c300
R8 =0x00007fa1d4161820, R9 =0x000b84a30000395b, R10=0x000b84a40000395b, R11=0x00007fa1bcff0010
R12=0x00007fa1d41619c0, R13=0x00007fa1d4161850, R14=0x00007fa1e0e2f7b0, R15=0x00007fa1d41619c0
RIP=0x00007fa1e0775b5c, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fa1d4161840)
0x00007fa1d4161840:   0000003f25d8de80 00000007b10e8f68
0x00007fa1d4161850:   00007fa1e0e29090 0000000025e0b800
0x00007fa1d4161860:   00007fa1dc109420 00007fa1dc0a9a30
0x00007fa1d4161870:   00007fa1dc0a9e98 0000000000000001
0x00007fa1d4161880:   000000073c400000 0000000018780000
0x00007fa1d4161890:   00000007b1401e60 000000000001fc34
0x00007fa1d41618a0:   00007fa1dc0a9c48 00007fa1dc0c6100
0x00007fa1d41618b0:   00007fa1dc0a9d88 00000007b1427ca8
0x00007fa1d41618c0:   00007fa1bcdec860 00007fa1d41619c0
0x00007fa1d41618d0:   00000007b1401e60 000000000001fc34
0x00007fa1d41618e0:   00022c9897bfbd82 00007fa1d4161910
0x00007fa1d41618f0:   00007fa1d41619c0 00000007b1427c80
0x00007fa1d4161900:   000000000ea20000 00007fa1dc0a9cd0
0x00007fa1d4161910:   00007fa1d4161930 00007fa1e0775921
0x00007fa1d4161920:   000000000ea04f90 00000000003a813e
0x00007fa1d4161930:   00007fa1d4161970 00007fa1e061e73b
0x00007fa1d4161940:   00000000003a8801 00007fa1dc01fb30
0x00007fa1d4161950:   000000000ea20000 000000073c400000
0x00007fa1d4161960:   00007fa1bcdec7e0 0000000000020000
0x00007fa1d4161970:   00007fa1d4161aa0 00007fa1e076bef6
0x00007fa1d4161980:   00007fa1d4161990 00007fa1dc0c6100
0x00007fa1d4161990:   00007fa1d41619c0 00007fa1d4161a40
0x00007fa1d41619a0:   00007fa1d4161a50 00007fa1d4161a60
0x00007fa1d41619b0:   00007fa1dc01fb98 000007500000000c
0x00007fa1d41619c0:   00007fa1e0e28890 00007fa1dc0a9a30
0x00007fa1d41619d0:   000000073c400000 0000000018780000
0x00007fa1d41619e0:   00000007b1401e60 000000000001fc34
0x00007fa1d41619f0:   00007fa1dc0a9c48 00007fa1dc0a9ce8
0x00007fa1d4161a00:   00007fa1dc0c6100 00007fa1dc0a9d88
0x00007fa1d4161a10:   00007fa1dc0a9e98 0000000067bff801
0x00007fa1d4161a20:   00000007b1427ca8 00000007b1401e60
0x00007fa1d4161a30:   00007fa1bcdec7e0 00007fff67bff8bc

Instructions: (pc=0x00007fa1e0775b5c)
0x00007fa1e0775b3c:   48 8b b5 38 ff ff ff 41 80 3e 00 74 57 48 8b 3d
0x00007fa1e0775b4c:   88 e9 7f 00 8b 56 08 8b 4f 08 48 d3 e2 48 03 17
0x00007fa1e0775b5c:   48 8b 5a 10 48 8d 7a 10 4c 89 ea ff 93 58 02 00
0x00007fa1e0775b6c:   00 48 8b 35 bc d7 7f 00 8b 06 85 c0 7e a6 4c 8b

Register to memory mapping:

RAX=0x0000000000000001 is an unknown value
RBX=0x00007fa1e0e3a050: _ZTV16instanceRefKlass+0x10 in /home/admin/jdk1.6.0_45/jre/lib/amd64/server/libjvm.so at 0x00007fa1e03a6000
RCX=0x0000000000000003 is an unknown value
RDX=0x000000020a7a19e0 is an unknown value
RSP=0x00007fa1d4161840 is an unknown value
RBP=0x00007fa1d4161910 is an unknown value
RSI=0x00000007b10e8f68 is an oop
[B
- klass: {type array byte}
- length: 8192
RDI=0x00007fa1e0e4c300:  in /home/admin/jdk1.6.0_45/jre/lib/amd64/server/libjvm.so at 0x00007fa1e03a6000
R8 =0x00007fa1d4161820 is an unknown value
R9 =0x000b84a30000395b is an unknown value
R10=0x000b84a40000395b is an unknown value
R11=0x00007fa1bcff0010 is an unknown value
R12=0x00007fa1d41619c0 is an unknown value
R13=0x00007fa1d4161850 is an unknown value
R14=0x00007fa1e0e2f7b0:  in /home/admin/jdk1.6.0_45/jre/lib/amd64/server/libjvm.so at 0x00007fa1e03a6000
R15=0x00007fa1d41619c0 is an unknown value


Stack: [0x0000000000000000,0x0000000000000000],  sp=0x00007fa1d4161840,  free space=137043969414k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x3cfb5c]  Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x21c
V  [libjvm.so+0x3cf921]  Par_MarkFromRootsClosure::do_bit(unsigned long)+0xf1
V  [libjvm.so+0x27873b]  BitMap::iterate(BitMapClosure*, unsigned long, unsigned long)+0xbb
V  [libjvm.so+0x3c5ef6]  CMSConcMarkingTask::do_scan_and_mark(int, CompactibleFreeListSpace*)+0x2e6
V  [libjvm.so+0x3c5989]  CMSConcMarkingTask::work(int)+0x89
V  [libjvm.so+0x87b80c]  YieldingFlexibleGangWorker::loop()+0xbc
V  [libjvm.so+0x8789e4]  GangWorker::run()+0x24
V  [libjvm.so+0x712eef]  java_start(Thread*)+0x13f


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007fa155210000 JavaThread "ActiveMQ InactivityMonitor Worker" daemon [_thread_blocked, id=5968, stack(0x00007fa12aa20000,0x00007fa12aa61000)]
  0x00007fa1b46c2800 JavaThread "catalina-exec-2566" daemon [_thread_in_native, id=5089, stack(0x00007fa125896000,0x00007fa1258d7000)]
  0x00007fa1b46c0800 JavaThread "catalina-exec-2565" daemon [_thread_in_native, id=5088, stack(0x00007fa1258d7000,0x00007fa125918000)]
  0x00007fa1b46be800 JavaThread "catalina-exec-2564" daemon [_thread_blocked, id=5087, stack(0x00007fa125918000,0x00007fa125959000)]
  0x00007fa1b46bc800 JavaThread "catalina-exec-2563" daemon [_thread_in_native, id=5086, stack(0x00007fa125959000,0x00007fa12599a000)]
  0x00007fa1b46ba800 JavaThread "catalina-exec-2562" daemon [_thread_blocked, id=5085, stack(0x00007fa12599a000,0x00007fa1259db000)]
  0x00007fa1b46b8800 JavaThread "catalina-exec-2561" daemon [_thread_in_native, id=5084, stack(0x00007fa1259db000,0x00007fa125a1c000)]
  0x00007fa1b46b6800 JavaThread "catalina-exec-2560" daemon [_thread_in_native, id=5083, stack(0x00007fa125a1c000,0x00007fa125a5d000)]
  0x00007fa1b46b4800 JavaThread "catalina-exec-2559" daemon [_thread_in_native, id=5082, stack(0x00007fa125a5d000,0x00007fa125a9e000)]
  0x00007fa1b46b2800 JavaThread "catalina-exec-2558" daemon [_thread_in_native, id=5081, stack(0x00007fa125a9e000,0x00007fa125adf000)]
  0x00007fa1b46b0800 JavaThread "catalina-exec-2557" daemon [_thread_in_native, id=5080, stack(0x00007fa125adf000,0x00007fa125b20000)]
  0x00007fa1b46ae800 JavaThread "catalina-exec-2556" daemon [_thread_blocked, id=5079, stack(0x00007fa125b20000,0x00007fa125b61000)]
  ******中间省略一大批类似的东西
  0x00007fa1dc353800 JavaThread "GC Daemon" daemon [_thread_blocked, id=29765, stack(0x00007fa1bc095000,0x00007fa1bc0d6000)]
  0x00007fa1dc144000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=29763, stack(0x00007fa1bc313000,0x00007fa1bc354000)]
  0x00007fa1dc139800 JavaThread "C1 CompilerThread2" daemon [_thread_blocked, id=29762, stack(0x00007fa1bc354000,0x00007fa1bc455000)]
  0x00007fa1dc137000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=29761, stack(0x00007fa1bc455000,0x00007fa1bc556000)]
  0x00007fa1dc135000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=29760, stack(0x00007fa1bc556000,0x00007fa1bc657000)]
  0x00007fa1dc132800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=29759, stack(0x00007fa1bc657000,0x00007fa1bc698000)]
  0x00007fa1dc131000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=29758, stack(0x00007fa1bc698000,0x00007fa1bc6d9000)]
  0x00007fa1dc114800 JavaThread "Finalizer" daemon [_thread_blocked, id=29757, stack(0x00007fa1d4021000,0x00007fa1d4062000)]
  0x00007fa1dc112800 JavaThread "Reference Handler" daemon [_thread_blocked, id=29756, stack(0x00007fa1bc6d9000,0x00007fa1bc71a000)]
  0x00007fa1dc007800 JavaThread "main" [_thread_in_native, id=29734, stack(0x00007fa1e0365000,0x00007fa1e03a6000)]

Other Threads:
  0x00007fa1dc10c000 VMThread [stack: 0x00007fa1bc71a000,0x00007fa1bc81b000] [id=29755]
  0x00007fa1dc147000 WatcherThread [stack: 0x00007fa1bc212000,0x00007fa1bc313000] [id=29764]

=>0x00007fa1dc0b2000 (exited) GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=29743]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
par new generation   total 943744K, used 602260K [0x00000006fc400000, 0x000000073c400000, 0x000000073c400000)
  eden space 838912K,  63% used [0x00000006fc400000, 0x000000071c92cef0, 0x000000072f740000)
  from space 104832K,  75% used [0x000000072f740000, 0x0000000734446400, 0x0000000735da0000)
  to   space 104832K,   0% used [0x0000000735da0000, 0x0000000735da0000, 0x000000073c400000)
concurrent mark-sweep generation total 3145728K, used 1921475K [0x000000073c400000, 0x00000007fc400000, 0x00000007fc400000)
concurrent-mark-sweep perm gen total 61440K, used 50212K [0x00000007fc400000, 0x0000000800000000, 0x0000000800000000)

Code Cache  [0x00007fa1d5df2000, 0x00007fa1d73f2000, 0x00007fa1dbdf2000)
total_blobs=5971 nmethods=5582 adapters=310 free_code_cache=77779136 largest_free_block=11328

Dynamic libraries:
40000000-40009000 r-xp 00000000 ca:01 533706                             /home/admin/jdk1.6.0_45/bin/java
40108000-4010a000 rwxp 00008000 ca:01 533706                             /home/admin/jdk1.6.0_45/bin/java
41876000-41903000 rwxp 00000000 00:00 0                                  [heap]
6fc400000-800000000 rwxp 00000000 00:00 0
3f25200000-3f25220000 r-xp 00000000 ca:01 786818                         /lib64/ld-2.12.so
3f2541f000-3f25420000 r-xp 0001f000 ca:01 786818                         /lib64/ld-2.12.so
3f25420000-3f25421000 rwxp 00020000 ca:01 786818                         /lib64/ld-2.12.so
3f25421000-3f25422000 rwxp 00000000 00:00 0
3f25600000-3f25602000 r-xp 00000000 ca:01 786822                         /lib64/libdl-2.12.so
3f25602000-3f25802000 ---p 00002000 ca:01 786822                         /lib64/libdl-2.12.so
3f25802000-3f25803000 r-xp 00002000 ca:01 786822                         /lib64/libdl-2.12.so
3f25803000-3f25804000 rwxp 00003000 ca:01 786822                         /lib64/libdl-2.12.so
3f25a00000-3f25b89000 r-xp 00000000 ca:01 786819                         /lib64/libc-2.12.so
3f25b89000-3f25d88000 ---p 00189000 ca:01 786819                         /lib64/libc-2.12.so
3f25d88000-3f25d8c000 r-xp 00188000 ca:01 786819                         /lib64/libc-2.12.so
3f25d8c000-3f25d8d000 rwxp 0018c000 ca:01 786819                         /lib64/libc-2.12.so
3f25d8d000-3f25d92000 rwxp 00000000 00:00 0
3f25e00000-3f25e17000 r-xp 00000000 ca:01 786527                         /lib64/libpthread-2.12.so
3f25e17000-3f26017000 ---p 00017000 ca:01 786527                         /lib64/libpthread-2.12.so
3f26017000-3f26018000 r-xp 00017000 ca:01 786527                         /lib64/libpthread-2.12.so
3f26018000-3f26019000 rwxp 00018000 ca:01 786527                         /lib64/libpthread-2.12.so
3f26019000-3f2601d000 rwxp 00000000 00:00 0
3f26600000-3f26683000 r-xp 00000000 ca:01 786529                         /lib64/libm-2.12.so
3f26683000-3f26882000 ---p 00083000 ca:01 786529                         /lib64/libm-2.12.so
3f26882000-3f26883000 r-xp 00082000 ca:01 786529                         /lib64/libm-2.12.so
3f26883000-3f26884000 rwxp 00083000 ca:01 786529                         /lib64/libm-2.12.so
3f26a00000-3f26a07000 r-xp 00000000 ca:01 786830                         /lib64/librt-2.12.so
3f26a07000-3f26c06000 ---p 00007000 ca:01 786830                         /lib64/librt-2.12.so
3f26c06000-3f26c07000 r-xp 00006000 ca:01 786830                         /lib64/librt-2.12.so
3f26c07000-3f26c08000 rwxp 00007000 ca:01 786830                         /lib64/librt-2.12.so
3f28a00000-3f28a16000 r-xp 00000000 ca:01 786559                         /lib64/libnsl-2.12.so
3f28a16000-3f28c15000 ---p 00016000 ca:01 786559                         /lib64/libnsl-2.12.so
3f28c15000-3f28c16000 r-xp 00015000 ca:01 786559                         /lib64/libnsl-2.12.so
3f28c16000-3f28c17000 rwxp 00016000 ca:01 786559                         /lib64/libnsl-2.12.so
3f28c17000-3f28c19000 rwxp 00000000 00:00 0
7fa125603000-7fa12560c000 r-xs 00061000 ca:01 666889                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/webapps/v3/WEB-INF/lib/httpclient-4.2.3.jar
7fa12560c000-7fa125615000 r-xs 00061000 ca:01 666889                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/webapps/v3/WEB-INF/lib/httpclient-4.2.3.jar
7fa125615000-7fa12561e000 r-xs 00061000 ca:01 666889                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/webapps/v3/WEB-INF/lib/httpclient-4.2.3.jar
7fa12561e000-7fa125627000 r-xs 00061000 ca:01 666889                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/webapps/v3/WEB-INF/lib/httpclient-4.2.3.jar
7fa125627000-7fa125630000 r-xs 00061000 ca:01 666889                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/webapps/v3/WEB-INF/lib/httpclient-4.2.3.jar
7fa125630000-7fa125639000 r-xs 00061000 ca:01 666889                    


7fa125843000-7fa12584c000 r-xs 00061000 ca:01 666889                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/webapps/v3/WEB-INF/lib/httpclient-4.2.3.jar
7fa125855000-7fa125858000 ---p 00000000 00:00 0
7fa125858000-7fa125896000 rwxp 00000000 00:00 0
7fa125896000-7fa125899000 ---p 00000000 00:00 0
7fa125899000-7fa1258d7000 rwxp 00000000 00:00 0
7fa1258d7000-7fa1258da000 ---p 00000000 00:00 0
7fa1258da000-7fa125918000 rwxp 00000000 00:00 0
7fa125918000-7fa12591b000 ---p 00000000 00:00 0
7fa12591b000-7fa125959000 rwxp 00000000 00:00 0
7fa125959000-7fa12595c000 ---p 00000000 00:00 0

  ******中间省略一大批类似的东西

7fa1dc000000-7fa1dc3c6000 rwxp 00000000 00:00 0
7fa1dc3c6000-7fa1e0000000 ---p 00000000 00:00 0
7fa1e0000000-7fa1e0001000 r-xs 00003000 ca:01 534080                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/lib/annotations-api.jar
7fa1e0001000-7fa1e0003000 r-xs 0001f000 ca:01 534074                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/lib/catalina-ha.jar
7fa1e0003000-7fa1e0006000 r-xs 0001c000 ca:01 534085                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/lib/jasper-el.jar
7fa1e0006000-7fa1e0013000 r-xs 000c2000 ca:01 534084                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/lib/tomcat-coyote.jar
7fa1e0013000-7fa1e0021000 r-xp 00000000 ca:01 661353                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libzip.so
7fa1e0021000-7fa1e0123000 ---p 0000e000 ca:01 661353                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libzip.so
7fa1e0123000-7fa1e0126000 rwxp 00010000 ca:01 661353                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libzip.so
7fa1e0126000-7fa1e0127000 rwxp 00000000 00:00 0
7fa1e0127000-7fa1e0150000 r-xp 00000000 ca:01 661325                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libjava.so
7fa1e0150000-7fa1e024f000 ---p 00029000 ca:01 661325                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libjava.so
7fa1e024f000-7fa1e0256000 rwxp 00028000 ca:01 661325                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libjava.so
7fa1e0256000-7fa1e0263000 r-xp 00000000 ca:01 661319                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libverify.so
7fa1e0263000-7fa1e0362000 ---p 0000d000 ca:01 661319                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libverify.so
7fa1e0362000-7fa1e0365000 rwxp 0000c000 ca:01 661319                     /home/admin/jdk1.6.0_45/jre/lib/amd64/libverify.so
7fa1e0365000-7fa1e0368000 ---p 00000000 00:00 0
7fa1e0368000-7fa1e03a6000 rwxp 00000000 00:00 0
7fa1e03a6000-7fa1e0cc4000 r-xp 00000000 ca:01 661309                     /home/admin/jdk1.6.0_45/jre/lib/amd64/server/libjvm.so
7fa1e0cc4000-7fa1e0dc6000 ---p 0091e000 ca:01 661309                     /home/admin/jdk1.6.0_45/jre/lib/amd64/server/libjvm.so
7fa1e0dc6000-7fa1e0f7c000 rwxp 00920000 ca:01 661309                     /home/admin/jdk1.6.0_45/jre/lib/amd64/server/libjvm.so
7fa1e0f7c000-7fa1e0fb9000 rwxp 00000000 00:00 0
7fa1e0fb9000-7fa1e0fc0000 r-xp 00000000 ca:01 661311                     /home/admin/jdk1.6.0_45/jre/lib/amd64/jli/libjli.so
7fa1e0fc0000-7fa1e10c1000 ---p 00007000 ca:01 661311                     /home/admin/jdk1.6.0_45/jre/lib/amd64/jli/libjli.so
7fa1e10c1000-7fa1e10c3000 rwxp 00008000 ca:01 661311                     /home/admin/jdk1.6.0_45/jre/lib/amd64/jli/libjli.so
7fa1e10c3000-7fa1e10c4000 rwxp 00000000 00:00 0
7fa1e10c4000-7fa1e10c5000 r-xs 0000a000 ca:01 534092                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/lib/tomcat-i18n-fr.jar
7fa1e10c5000-7fa1e10c6000 r-xs 00001000 ca:01 534076                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/lib/tomcat-api.jar
7fa1e10c6000-7fa1e10c8000 r-xs 00008000 ca:01 534012                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/bin/tomcat-juli.jar
7fa1e10c8000-7fa1e10c9000 r-xs 00005000 ca:01 534014                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/bin/commons-daemon.jar
7fa1e10c9000-7fa1e10ca000 r-xs 00006000 ca:01 534018                     /home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/bin/bootstrap.jar
7fa1e10ca000-7fa1e10d2000 rwxs 00000000 ca:01 1077909                    /tmp/hsperfdata_admin/29733
7fa1e10d2000-7fa1e10d3000 rwxp 00000000 00:00 0
7fa1e10d3000-7fa1e10d4000 r-xp 00000000 00:00 0
7fa1e10d4000-7fa1e10d5000 rwxp 00000000 00:00 0
7fff67bb6000-7fff67bcb000 rwxp 00000000 00:00 0                          [stack]
7fff67bff000-7fff67c00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Djava.util.logging.config.file=/home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xss256k -Xms4G -Xmx4G -Xmn1G -XX:PermSize=60M -XX:MaxPermSize=60M -XX:SurvivorRatio=8 -XX:CMSFullGCsBeforeCompaction=3 -XX:CMSInitiatingOccupancyFraction=60 -XX:MaxTenuringThreshold=30 -XX:+CMSScavengeBeforeRemark -XX:+ScavengeBeforeFullGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:+TieredCompilation -XX:ParallelCMSThreads=15 -XX:+UseFastAccessorMethods -XX:LargePageSizeInBytes=128M -XX:+UseFastAccessorMethods -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -Xloggc:/home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/logs/gc.log -Djava.endorsed.dirs=/home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/endorsed -Dcatalina.base=/home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01 -Dcatalina.home=/home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01 -Djava.io.tmpdir=/home/admin/tomcat-hosts/tomcat7.0.55-apilocate-01/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin
LD_LIBRARY_PATH=/home/admin/jdk1.6.0_45/jre/lib/amd64/server:/home/admin/jdk1.6.0_45/jre/lib/amd64:/home/admin/jdk1.6.0_45/jre/../lib/amd64
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x862a30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x862a30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x7106f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x7106f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x7106f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x7106f0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x713520], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x713120], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x713120], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x713120], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


---------------  S Y S T E M  ---------------
Linux Server release 6.2


uname:Linux 2.6.32-220.23.2.ali878.el6.x86_64 #1 SMP Mon Jan 28 17:12:52 CST 2013 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 127440, NOFILE 65535, AS infinity
load average:14.65 13.85 16.18

/proc/meminfo:
MemTotal:       16331532 kB
MemFree:           87696 kB
Buffers:          158152 kB
Cached:          9692380 kB
SwapCached:            0 kB
Active:          7270852 kB
Inactive:        8363636 kB
Active(anon):    5780112 kB
Inactive(anon):     4540 kB
Active(file):    1490740 kB
Inactive(file):  8359096 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              4816 kB
Writeback:          5336 kB
AnonPages:       5784220 kB
Mapped:            18144 kB
Shmem:               228 kB
Slab:             390348 kB
SReclaimable:     352004 kB
SUnreclaim:        38344 kB
KernelStack:       22416 kB
PageTables:        15328 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8165764 kB
Committed_AS:    9928900 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       40504 kB
VmallocChunk:   34359695484 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6144 kB
DirectMap2M:    16771072 kB


CPU:total 4 (32 cores per cpu, 2 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht

/proc/cpuinfo:
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 45
model name    : Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz
stepping    : 7
cpu MHz        : 2200.441
cache size    : 15360 KB
physical id    : 0
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx lm rep_good unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips    : 4400.88
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 16331532k(87696k free), swap 0k(0k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (20.45-b01) for linux-amd64 JRE (1.6.0_45-b06), built on Mar 26 2013 14:07:02 by "java_re" with gcc 3.2.2 (SuSE Linux)

time: Mon Nov  3 23:09:27 2014
elapsed time: 9483 seconds

楼主的全方位轰炸让人很难受…请不要这样。 

===========================================

我在这边回复。先问几个问题：
1、楼主是在 JDK6u45 上跑的。遇到的这个crash是否容易重现？是否稳定重现？
2、如果1的答案是“是”，那么请问在 JDK7u45 上是否也可以稳定重现？
3、在 JDK7u71 或 JDK7u72 上是否可以重现？

问这几个问题主要是因为：
a、楼主遇到的问题可能重现率非常非常低。是的话那就先别管它了。不重现的bug没法修。
b、如果是可以重现的bug，那最好先看看是不是已经有已发布的版本修好了。
c、如果在最新版上仍然可以重现，那就值得仔细看看了。

上面说的问题如果不全部跟进完，那别人也很难帮上忙…

===========================================

唯一一个看起来一致的bug是这个：
JDK-8020236： SIGSEGV in void Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)
这个bug在JDK7u40上发生过但是非常难重现，所以没有修。那么JDK6u45也应该有同样的问题但是没有修。

如果楼主可以在自己的测试环境里稳定重现问题那是最好不过。那样就可以重新给OpenJDK提交一个bug了。

===========================================

看起来像但不一样的bug也有若干，例如：

JDK-6668573: CMS: reference processing crash if ParallelCMSThreads > ParallelGCThreads
这个在JDK6u45应该已经修好了。没修好的话也可以试试这个bug的评论里提到的workaround。

JDK-6354939: CMS: crashes/asserts in Xconc nightly testing with rt_plumhall/rt_quick
这个很老了，肯定不是的。

首先对我全方位，立体式的轰炸表示非常抱歉。打扰到的，请见谅。 

crash可以重现，20分钟左右压测就能出现。至少压测了两次，都复现了。
但如果把ParallelCMSThreads这个参数去掉了，也就是默认（4+3）/ 4 =1了吧，压了很久都没有问题了。

我昨天也找了一些资料，看JDK的bug描述，6u45应该是修复好了ParallelCMSThreads > ParallelGCThreads ，但15这个参数远大于CPU核数4。是不是跟这个有一定的关系呢？

我找了一篇文章，请问这个是对的吗？http://hiroshiyamauchi.blogspot.com/2009/12/gc-threads.html

新版本的压测我再进行跟进。
多谢 

Crash log里说您的测试环境是Xeon E5-2430，而规格书说这个CPU有6个核。楼主为啥说是4核的？
6核的话，ParallelCMSThreads参数去掉之后默认应该是2吧。而对应的ParallelGCThreads是6。

没有具体环境我很难深入分析，光看crash log里的信息我无法确定楼主看到的状况跟ParallelCMSThreads太大有没有关系。不过既然楼主的测试环境能重现这问题，那就有搞头：把ParallelCMSThreads从2到7这么逐个测上去，看是不是还是在ParallelCMSThreads > ParallelGCThreads时会出问题而反之不会，就可以确定（或排除）一些可能性了。



Hiroshi是在Google做JVM开发的大神。他知道他在写什么