[讨论] Red大 ,帮解决一下啊。
xgj1988
2015-02-04
情况是这样的。我在压一个JAVA写的游戏服务器。
压测服务器环境 硬件: CPU: intel core i3-3240 3.40GHz 3.40GHz 内存:4.0G 硬盘:希捷 ST500DM002 500G 7200转/分 固态硬盘:无 软件: 操作系统:windows7 旗舰版 JAVA: 1.7.0_51 MySQL: 5.5.36 400个连接连上。 CPU不高,过一会之后CPU只有0.01%,内存1.1G,过10多分钟后,导致 服务器卡死。。不能响应任何socket请求。关闭压测的客户端之后,客户端的连接并没有关闭,出现大量的FIN_WAIT_1(应该是服务器没有响应ACK)。这个时候jvisualvm连接不上服务器(一直loading状态)。由于不能jvisualvm上去,我就直接用jstack dump 线程信息。 jstack出来的结果如下(不带-F): 2015-02-04 15:07:40 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode): "RMI TCP Connection(27)-192.168.60.248" daemon prio=6 tid=0x000000000b5fe000 nid=0x1c20 in Object.wait() [0x000000000f79d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - locked <0x00000000f34743f0> (a java.io.ObjectStreamClass$EntryFuture) ... "RMI TCP Connection(26)-192.168.60.248" daemon prio=6 tid=0x000000000b5fd800 nid=0x12e0 in Object.wait() [0x000000001436d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - locked <0x00000000f34743f0> (a java.io.ObjectStreamClass$EntryFuture) ... "RMI TCP Connection(25)-192.168.60.248" daemon prio=6 tid=0x000000000b5fc800 nid=0x2714 in Object.wait() [0x0000000013ced000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - locked <0x00000000f34743f0> (a java.io.ObjectStreamClass$EntryFuture) ... "RMI Scheduler(0)" daemon prio=6 tid=0x000000000e93d000 nid=0x1c04 waiting on condition [0x0000000013b3e000] java.lang.Thread.State: WAITING (parking) - parking to wait for <0x00000000d93b3728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "SCENE-MSG-PROCESS-THREADS-4" prio=6 tid=0x0000000009f99000 nid=0x760 waiting on condition [0x0000000011aef000] java.lang.Thread.State: WAITING (parking) - parking to wait for <0x00000000c266e890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "SCENE-MSG-PROCESS-THREADS-3" prio=6 tid=0x0000000009f98000 nid=0x838 waiting on condition [0x00000000119be000] java.lang.Thread.State: WAITING (parking) - parking to wait for <0x00000000c266e890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "MODULE-PROCESS-THREADS-2" prio=6 tid=0x000000000b710000 nid=0x19b8 waiting on condition [0x000000000ff1f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "MODULE-PROCESS-THREADS-1" prio=6 tid=0x000000000b70f000 nid=0x4bc waiting on condition [0x00000000107df000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "HashSessionScavenger-0" daemon prio=6 tid=0x000000000b2ae000 nid=0x2184 in Object.wait() [0x000000001033f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000c25c2550> (a java.util.TaskQueue) ... "qtp1505546523-57" prio=6 tid=0x000000000b2ad000 nid=0x235c waiting on condition [0x000000001047f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "qtp1505546523-56" prio=6 tid=0x000000000b2ac800 nid=0x21d0 waiting on condition [0x000000001021f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "qtp1505546523-55" prio=6 tid=0x000000000b2ab800 nid=0x21f0 waiting on condition [0x000000001002f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "qtp1505546523-54" prio=6 tid=0x000000000b2ab000 nid=0x18a0 waiting on condition [0x000000000fdde000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "qtp1505546523-53" prio=6 tid=0x000000000b2aa000 nid=0x11b4 waiting on condition [0x000000000fcbf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "qtp1505546523-52" prio=6 tid=0x000000000b2a9800 nid=0xefc waiting on condition [0x000000000fb5f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e8290> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "qtp1505546523-51" prio=6 tid=0x000000000b2a8800 nid=0x8d0 runnable [0x000000000f98e000] java.lang.Thread.State: RUNNABLE ... "qtp1505546523-50 - Acceptor0 SelectChannelConnector@0.0.0.0:8088" prio=6 tid=0x000000000b2a8000 nid=0x1c38 runnable [0x000000000e01f000] java.lang.Thread.State: RUNNABLE ... "BoneCP-pool-watch-thread" daemon prio=6 tid=0x000000000b271000 nid=0x2a14 waiting on condition [0x000000000e5ae000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1e456d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "BoneCP-keep-alive-scheduler" daemon prio=6 tid=0x000000000b270000 nid=0x127c waiting on condition [0x000000000e42f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec04d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b26f800 nid=0x20e4 waiting on condition [0x000000000e2ae000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec50e8> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b26e800 nid=0x1d2c waiting on condition [0x000000000e17f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec50e8> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b26e000 nid=0x2444 waiting on condition [0x000000000d8de000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec50e8> (a jsr166y.LinkedTransferQueue) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) ... "BoneCP-pool-watch-thread" daemon prio=6 tid=0x000000000b145800 nid=0x2028 waiting on condition [0x000000000dc3e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec0380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "BoneCP-keep-alive-scheduler" daemon prio=6 tid=0x000000000b144800 nid=0x27b0 waiting on condition [0x000000000deee000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec04d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b144000 nid=0x2be4 waiting on condition [0x000000000dd3f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec0298> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b143000 nid=0x2be0 waiting on condition [0x000000000db0e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b142800 nid=0x1884 waiting on condition [0x000000000d9de000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ec0298> (a jsr166y.LinkedTransferQueue) ... "com.google.common.base.internal.Finalizer" daemon prio=6 tid=0x000000000b141800 nid=0x2bd4 in Object.wait() [0x000000000d08e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000c1e951c0> (a java.lang.ref.ReferenceQueue$Lock) ... "BoneCP-pool-watch-thread" daemon prio=6 tid=0x000000000b141000 nid=0x2bd0 waiting on condition [0x000000000d79e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25e9890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b13f800 nid=0x2954 waiting on condition [0x000000000d47e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25ea1b0> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b13e800 nid=0x2950 waiting on condition [0x000000000d37e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25ea1b0> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000afb9000 nid=0x1cb0 waiting on condition [0x000000000d1ff000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c25ea1b0> (a jsr166y.LinkedTransferQueue) ... "com.google.common.base.internal.Finalizer" daemon prio=6 tid=0x000000000afb8800 nid=0x1768 in Object.wait() [0x000000000ce6f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c25ea750> (a java.lang.ref.ReferenceQueue$Lock) ... - locked <0x00000000c25ea750> (a java.lang.ref.ReferenceQueue$Lock) ... "BoneCP-pool-watch-thread" daemon prio=6 tid=0x000000000afb7800 nid=0x2bc8 waiting on condition [0x000000000cf8e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1eee018> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "BoneCP-keep-alive-scheduler" daemon prio=6 tid=0x000000000afb7000 nid=0x2bcc waiting on condition [0x000000000c96e000] java.lang.Thread.State: TIMED_WAITING (sleeping) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000afb6000 nid=0x2bc4 waiting on condition [0x000000000cd4e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1eedf90> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000afb5800 nid=0x2bf4 waiting on condition [0x000000000cbdf000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1eedf90> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000afb4800 nid=0x2bec waiting on condition [0x000000000caaf000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1eedf90> (a jsr166y.LinkedTransferQueue) ... "BoneCP-pool-watch-thread" daemon prio=6 tid=0x000000000afb4000 nid=0x2bf0 waiting on condition [0x000000000c4fe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ea33b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) ... "BoneCP-keep-alive-scheduler" daemon prio=6 tid=0x000000000afb3000 nid=0x2bbc waiting on condition [0x000000000c7ef000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000afb2800 nid=0x2bb8 waiting on condition [0x000000000c66e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ea3328> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b10b000 nid=0x2be8 waiting on condition [0x000000000bf8f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ea3328> (a jsr166y.LinkedTransferQueue) ... "BoneCP-release-thread-helper-thread" daemon prio=6 tid=0x000000000b0ad800 nid=0x1ce0 waiting on condition [0x000000000c37f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1ea3328> (a jsr166y.LinkedTransferQueue) ... "com.google.common.base.internal.Finalizer" daemon prio=6 tid=0x000000000b063000 nid=0x2424 in Object.wait() [0x000000000c22f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c1e33e68> (a java.lang.ref.ReferenceQueue$Lock) ... - locked <0x00000000c1e33e68> (a java.lang.ref.ReferenceQueue$Lock) ... "Abandoned connection cleanup thread" daemon prio=6 tid=0x000000000aff7000 nid=0x2680 in Object.wait() [0x000000000bd6f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) - locked <0x00000000c20a7378> (a java.lang.ref.ReferenceQueue$Lock) ... "Timer-0" daemon prio=6 tid=0x000000000a06c800 nid=0x1f98 in Object.wait() [0x000000000c0af000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c258b920> (a java.util.TaskQueue) ... - locked <0x00000000c258b920> (a java.util.TaskQueue) ... "GAME_QUARTZ_QuartzSchedulerThread" prio=6 tid=0x000000000b018800 nid=0x28e4 runnable [0x000000000be6e000] java.lang.Thread.State: RUNNABLE - locked <0x00000000c258b020> (a java.lang.Object) ... "GAME_QUARTZ_Worker-5" prio=6 tid=0x0000000009ff3000 nid=0x28e0 in Object.wait() [0x000000000b89f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568) - locked <0x00000000c1ec5700> (a java.lang.Object) "GAME_QUARTZ_Worker-4" prio=6 tid=0x0000000009efa800 nid=0x2078 in Object.wait() [0x000000000bb0f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - locked <0x00000000c1ec5710> (a java.lang.Object) "GAME_QUARTZ_Worker-3" prio=6 tid=0x0000000009fee000 nid=0x2738 in Object.wait() [0x000000000ab8f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - locked <0x00000000c1ec5720> (a java.lang.Object) "GAME_QUARTZ_Worker-2" prio=6 tid=0x000000000b033000 nid=0x1594 in Object.wait() [0x000000000bc1f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - locked <0x00000000c1ec5730> (a java.lang.Object) "GAME_QUARTZ_Worker-1" prio=6 tid=0x000000000a145000 nid=0x2390 in Object.wait() [0x000000000b99f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - locked <0x00000000c1ec5740> (a java.lang.Object) "Service Thread" daemon prio=6 tid=0x0000000009d80000 nid=0x2448 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x0000000009d76800 nid=0x1dbc waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x0000000009d72800 nid=0x1fc4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "JDWP Command Reader" daemon prio=6 tid=0x0000000009d64800 nid=0xc84 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "JDWP Event Helper Thread" daemon prio=6 tid=0x0000000009d64000 nid=0x2a00 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "JDWP Transport Listener: dt_socket" daemon prio=6 tid=0x0000000009d52000 nid=0x2040 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Attach Listener" daemon prio=10 tid=0x000000000864b800 nid=0x1e58 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00000000086ab000 nid=0x297c runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=8 tid=0x0000000008634800 nid=0x1bcc in Object.wait() [0x0000000009bcf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000c23379f0> (a java.lang.ref.ReferenceQueue$Lock) ... "Reference Handler" daemon prio=10 tid=0x000000000862b800 nid=0x1fb0 in Object.wait() [0x0000000009a5e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000000c2389e38> (a java.lang.ref.Reference$Lock) JNI global references: 7489 我用jstack -F (带-F) dump出来的结果如下,这里由于文本限制,少了很多东西: Deadlock Detection: No deadlocks found. Thread 7: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise) - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Interpreted frame) 这里有几个简单的问题问一下: 1:为什么jstack -F 抓到的包都是BLOCKED的呢? 2:jstack -F 打出的thread X 。比如 Thread 7: (state = BLOCKED) 这里的 id为7的线程和jstack(不带-F)的哪个线程对应呢?而且发现jstack -F 所打的信息感觉不多啊?又都是block 没有什么价值。 上面是几个简单的问题,其实更进一步的问题是。 1:为什么我服务器会无响应。我开始以为是死锁。但是jstack打出来的信息没有死锁信息。 2:为什么压测的客户端到了一定数量就和服务器断开连接? 出现的异常如下: java.io.IOException: 远程主机强迫关闭了一个现有的连接。 at sun.nio.ch.SocketDispatcher.read0(Native Method) 2015-01-30 16:36:56 [ERROR]MinaSocketHandler的exceptionCaught方法 org.apache.mina.core.write.WriteToClosedSessionException 3:解决这种问题,一般的思路是什么。其实我更像让Red大教我们一下思路,一般这种虚拟机方面的问题,分几步解决。分别怎么查找问题。 谢谢了。(开始发了一版。超过限制,把一些内容修改后再发一次) |
|
xgj1988
2015-02-04
我怀疑还是死锁问题,但是jstack -F(加了-F) 打出来是No deadlocks found. 。但是我总感觉jstack -F打出来的都是block。我总觉得不对。所以我觉得他打出来的No deadlocks found这个也是不可信的。
|
|
xgj1988
2015-02-04
还有个问题:
比如 - locked <0x00000000c2389e38> in Object.wait() [0x0000000009a5e000] waiting on condition [0x000000000fb5f000] 红色标注是对象ID吧。如果是,怎么看是哪个对象呢?如果不是,那是什么呢? |
|
huangriyan
2015-04-20
你好
这个问题解决没有,想学习一下 |
|
xgj1988
2015-07-20
huangriyan 写道 你好
这个问题解决没有,想学习一下 解决了! |