java crash 不定期出现，一直查不到原因，求解！
 |
subchen
2013-08-14
#
# A fatal error has been detected by the Java Runtime Environment: # # Internal Error (safepoint.cpp:308), pid=12262, tid=140481717974784 # guarantee(PageArmed == 0) failed: invariant # # JRE version: 6.0_29-b11 # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.4-b02 mixed mode linux-amd64 compressed oops) # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x0000000040191800): VMThread [stack: 0x00007fc472d5b000,0x00007fc472e5c000] [id=12288] Stack: [0x00007fc472d5b000,0x00007fc472e5c000], sp=0x00007fc472e5a9b0, free space=1022k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x85eb55] VMError::report_and_die()+0x265 V [libjvm.so+0x3e4066] report_vm_error(char const*, int, char const*, char const*)+0x56 V [libjvm.so+0x783616] SafepointSynchronize::begin()+0x4f6 V [libjvm.so+0x86cfc7] VMThread::loop()+0x187 V [libjvm.so+0x86cade] VMThread::run()+0x6e V [libjvm.so+0x710b8f] java_start(Thread*)+0x13f VM_Operation (0x00007fc46b157aa0): RevokeBias, mode: safepoint, requested by thread 0x000000004023f000 --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x000000004023f000 JavaThread "Thread-1084" daemon [_thread_blocked, id=6636, stack(0x00007fc46b058000,0x00007fc46b159000)] 0x0000000040ee2800 JavaThread "RMI RenewClean-[localhost:56822,com.cisco.unicorn.server.LocalSocketFactory@7ef7f696]" daemon [_thread_blocked, id=6635, stack(0x00007fc46b860000,0x00007fc46b961000)] 0x00000000405b7000 JavaThread "RMI TCP Connection(1078)-127.0.0.1" daemon [_thread_in_native, id=6634, stack(0x00007fc46aa52000,0x00007fc46ab53000)] 0x0000000041094000 JavaThread "RMI TCP Connection(1077)-127.0.0.1" daemon [_thread_in_native, id=6630, stack(0x00007fc46a54d000,0x00007fc46a64e000)] 0x0000000040359000 JavaThread "RMI TCP Connection(1076)-127.0.0.1" daemon [_thread_in_native, id=6627, stack(0x00007fc46ac54000,0x00007fc46ad55000)] 0x000000004048e800 JavaThread "RMI TCP Connection(1075)-127.0.0.1" daemon [_thread_in_native, id=6625, stack(0x00007fc46a74f000,0x00007fc46a850000)] 0x000000004029e800 JavaThread "RMI RenewClean-[localhost:56822,com.cisco.unicorn.server.LocalSocketFactory@62b3529b]" daemon [_thread_blocked, id=5983, stack(0x00007fc46a951000,0x00007fc46aa52000)] 0x000000004126d800 JavaThread "RMI RenewClean-[localhost:56822,com.cisco.unicorn.server.LocalSocketFactory@3448cbd6]" daemon [_thread_blocked, id=5207, stack(0x00007fc46a64e000,0x00007fc46a74f000)] 0x00000000402a0800 JavaThread "RMI RenewClean-[localhost:56822,com.cisco.unicorn.server.LocalSocketFactory@553a18dd]" daemon [_thread_blocked, id=4593, stack(0x00007fc46b35b000,0x00007fc46b45c000)] 0x000000004016c800 JavaThread "Dispatcher-2" daemon [_thread_blocked, id=14409, stack(0x00007fc46b25a000,0x00007fc46b35b000)] 0x0000000040bfc000 JavaThread "Dispatcher-1" daemon [_thread_blocked, id=12595, stack(0x00007fc46b159000,0x00007fc46b25a000)] 0x0000000040be3800 JavaThread "RpcWorker-1" daemon [_thread_blocked, id=12594, stack(0x00007fc46b55d000,0x00007fc46b65e000)] 0x0000000040117800 JavaThread "DestroyJavaVM" [_thread_blocked, id=12268, stack(0x00007fc478fb3000,0x00007fc4790b4000)] 0x0000000040bf2800 JavaThread "RMI RenewClean-[localhost:56822,com.cisco.unicorn.server.LocalSocketFactory@cde6570]" daemon [_thread_blocked, id=12452, stack(0x00007fc46b45c000,0x00007fc46b55d000)] 0x0000000040bdc800 JavaThread "RMI RenewClean-[localhost:56822,com.cisco.unicorn.server.LocalSocketFactory@5cca548b]" daemon [_thread_blocked, id=12449, stack(0x00007fc46b65e000,0x00007fc46b75f000)] 0x0000000040bda800 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=12448, stack(0x00007fc46b961000,0x00007fc46ba62000)] 0x0000000040bd9800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_vm, id=12444, stack(0x00007fc46ba62000,0x00007fc46bb63000)] 0x0000000040bbf000 JavaThread "GC Daemon" daemon [_thread_blocked, id=12439, stack(0x00007fc46bb63000,0x00007fc46bc64000)] 0x0000000040bca800 JavaThread "RMI Reaper" [_thread_blocked, id=12438, stack(0x00007fc46bc64000,0x00007fc46bd65000)] 0x0000000040bbc800 JavaThread "HeartBeat" daemon [_thread_blocked, id=12437, stack(0x00007fc46bd65000,0x00007fc46be66000)] 0x0000000040bba000 JavaThread "DbVacuum Agent" daemon [_thread_in_native, id=12436, stack(0x00007fc46be66000,0x00007fc46bf67000)] 0x0000000040c4a000 JavaThread "StatusNotifier" daemon [_thread_blocked, id=12435, stack(0x00007fc46c56d000,0x00007fc46c66e000)] 0x0000000040c4e800 JavaThread "ConsumerAdaptor Server" daemon [_thread_in_native, id=12434, stack(0x00007fc46c36b000,0x00007fc46c46c000)] 0x0000000040bb1000 JavaThread "Health Monitor" daemon [_thread_blocked, id=12433, stack(0x00007fc46bf67000,0x00007fc46c068000)] 0x0000000040bae800 JavaThread "StatsConsolidatorCe" daemon [_thread_blocked, id=12432, stack(0x00007fc46c068000,0x00007fc46c169000)] 0x0000000040bac800 JavaThread "StatsCollector" daemon [_thread_blocked, id=12431, stack(0x00007fc46c169000,0x00007fc46c26a000)] 0x0000000040c51000 JavaThread "StatsTransmitter" daemon [_thread_blocked, id=12430, stack(0x00007fc46c26a000,0x00007fc46c36b000)] 0x0000000040c4c800 JavaThread "Scheduler" daemon [_thread_blocked, id=12428, stack(0x00007fc46c46c000,0x00007fc46c56d000)] 0x0000000040c48800 JavaThread "PlaylistFileHandler" daemon [_thread_blocked, id=12426, stack(0x00007fc46c66e000,0x00007fc46c76f000)] 0x0000000040c3a000 JavaThread "FileSysHandler" daemon [_thread_blocked, id=12425, stack(0x00007fc46c76f000,0x00007fc46c870000)] 0x0000000040c37800 JavaThread "DataFeedPoll" daemon [_thread_blocked, id=12424, stack(0x00007fc46c870000,0x00007fc46c971000)] 0x0000000040c35800 JavaThread "MessengerSenderThread" daemon [_thread_blocked, id=12423, stack(0x00007fc46c971000,0x00007fc46ca72000)] 0x0000000040c33000 JavaThread "HttpsCertKeyDownloader" daemon [_thread_blocked, id=12422, stack(0x00007fc46ca72000,0x00007fc46cb73000)] 0x0000000040c32000 JavaThread "FileHandler" daemon [_thread_blocked, id=12421, stack(0x00007fc46cb73000,0x00007fc46cc74000)] 0x0000000040c2d000 JavaThread "SecurityManager" daemon [_thread_blocked, id=12420, stack(0x00007fc46cc74000,0x00007fc46cd75000)] 0x0000000040c59800 JavaThread "CmdProcessor" daemon [_thread_blocked, id=12419, stack(0x00007fc46cd75000,0x00007fc46ce76000)] 0x0000000040ba6800 JavaThread "Thread-2" daemon [_thread_in_native, id=12418, stack(0x00007fc46ce76000,0x00007fc46cf77000)] 0x0000000040c62000 JavaThread "HealthMonitor notification listener thread" daemon [_thread_blocked, id=12417, stack(0x00007fc47022a000,0x00007fc47032b000)] 0x0000000040adc800 JavaThread "SyslogListener" [_thread_blocked, id=12416, stack(0x00007fc46cf77000,0x00007fc46d078000)] 0x0000000040afb800 JavaThread "HealthProbeHandler" daemon [_thread_in_Java, id=12408, stack(0x00007fc46dad8000,0x00007fc46dbd9000)] 0x0000000040a36800 JavaThread "Consumer Notify Worker#0" daemon [_thread_blocked, id=12378, stack(0x00007fc47032b000,0x00007fc47042c000)] 0x00000000407d1000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=12351, stack(0x00007fc47042c000,0x00007fc47052d000)] 0x0000000040644000 JavaThread "RMI TCP Accept-1099" daemon [_thread_in_native, id=12324, stack(0x00007fc47056d000,0x00007fc47066e000)] 0x00000000401c5800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=12295, stack(0x00007fc472755000,0x00007fc472856000)] 0x00000000401c1800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=12294, stack(0x00007fc472856000,0x00007fc472957000)] 0x00000000401be000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=12293, stack(0x00007fc472957000,0x00007fc472a58000)] 0x00000000401bc000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=12292, stack(0x00007fc472a58000,0x00007fc472b59000)] 0x00000000401a0000 JavaThread "Finalizer" daemon [_thread_blocked, id=12291, stack(0x00007fc472b59000,0x00007fc472c5a000)] 0x0000000040198800 JavaThread "Reference Handler" daemon [_thread_blocked, id=12290, stack(0x00007fc472c5a000,0x00007fc472d5b000)] Other Threads: =>0x0000000040191800 VMThread [stack: 0x00007fc472d5b000,0x00007fc472e5c000] [id=12288] 0x00000000401d9000 WatcherThread [stack: 0x00007fc472654000,0x00007fc472755000] [id=12296] VM state:synchronizing (normal execution) VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event]) [0x00000000401152c0] Safepoint_lock - owner thread: 0x0000000040191800 [0x0000000040115350] Threads_lock - owner thread: 0x0000000040191800 Heap PSYoungGen total 11520K, used 5275K [0x00000000fd560000, 0x00000000fe1e0000, 0x0000000100000000) eden space 10368K, 42% used [0x00000000fd560000,0x00000000fd9aee80,0x00000000fdf80000) from space 1152K, 75% used [0x00000000fdf90000,0x00000000fe068000,0x00000000fe0b0000) to space 1216K, 0% used [0x00000000fe0b0000,0x00000000fe0b0000,0x00000000fe1e0000) PSOldGen total 87424K, used 4815K [0x00000000f8000000, 0x00000000fd560000, 0x00000000fd560000) object space 87424K, 5% used [0x00000000f8000000,0x00000000f84b3f98,0x00000000fd560000) PSPermGen total 47552K, used 36045K [0x00000000f2e00000, 0x00000000f5c70000, 0x00000000f8000000) object space 47552K, 75% used [0x00000000f2e00000,0x00000000f5133460,0x00000000f5c70000) Code Cache [0x00007fc475633000, 0x00007fc4758a3000, 0x00007fc478633000) total_blobs=759 nmethods=422 adapters=291 free_code_cache=48172928 largest_free_block=30848 Dynamic libraries: ................ ................. Memory: 4k page, physical 32841504k(26467244k free), swap 0k(0k free) vm_info: Java HotSpot(TM) 64-Bit Server VM (20.4-b02) for linux-amd64 JRE (1.6.0_29-b11), built on Oct 3 2011 01:19:20 by "java_re" with gcc 3.2.2 (SuSE Linux) time: Sun Jul 28 07:28:52 2013 elapsed time: 48895 seconds =========================================== Deadlock Detection: No deadlocks found. Thread 6636: (state = BLOCKED) Thread 6635: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run() @bci=95, line=516 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 6634: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Interpreted frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 (Interpreted frame) - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame) - java.io.BufferedInputStream.read() @bci=12, line=237 (Interpreted frame) - java.io.FilterInputStream.read() @bci=4, line=66 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, boolean) @bci=25, line=517 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685, line=790 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50, line=649 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 6630: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Interpreted frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 (Interpreted frame) - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame) - java.io.BufferedInputStream.read() @bci=12, line=237 (Interpreted frame) - java.io.FilterInputStream.read() @bci=4, line=66 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, boolean) @bci=25, line=517 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685, line=790 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50, line=649 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 6627: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Interpreted frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 (Interpreted frame) - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame) - java.io.BufferedInputStream.read() @bci=12, line=237 (Interpreted frame) - java.io.FilterInputStream.read() @bci=4, line=66 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, boolean) @bci=25, line=517 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685, line=790 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50, line=649 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 6625: (state = IN_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Interpreted frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 (Interpreted frame) - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame) - java.io.BufferedInputStream.read() @bci=12, line=237 (Interpreted frame) - java.io.FilterInputStream.read() @bci=4, line=66 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, boolean) @bci=25, line=517 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685, line=790 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50, line=649 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 5983: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run() @bci=95, line=516 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 5207: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run() @bci=95, line=516 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 4593: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run() @bci=95, line=516 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 14409: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.messaging.Dispatcher.run() @bci=36, line=160 (Interpreted frame) Thread 12595: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.messaging.Dispatcher.run() @bci=36, line=160 (Interpreted frame) Thread 12594: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.messaging.DistributionAdapterRpcImpl$WorkerThread.run() @bci=79, line=91 (Interpreted frame) Thread 12268: (state = BLOCKED) Thread 12452: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run() @bci=95, line=516 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12449: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run() @bci=95, line=516 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12448: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Interpreted frame) - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Interpreted frame) - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Interpreted frame) - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12444: (state = IN_VM) - java.net.SocketPermission.init(java.lang.String, int) @bci=375, line=439 (Interpreted frame) - java.net.SocketPermission.<init>(java.lang.String, java.lang.String) @bci=22, line=266 (Interpreted frame) - java.lang.SecurityManager.checkAccept(java.lang.String, int) @bci=88, line=1157 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=83, line=466 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() @bci=55, line=369 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12439: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - sun.misc.GC$Daemon.run() @bci=51, line=100 (Interpreted frame) Thread 12438: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame) - sun.rmi.transport.ObjectTable$Reaper.run() @bci=3, line=333 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12437: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.server.ADataFeedAgent$HeartBeat.run() @bci=156, line=1726 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12436: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame) - unicorn.RpcTcpServer.accept() @bci=4, line=46 (Interpreted frame) - com.cisco.unicorn.dbvacuum.DbVacuum.run() @bci=11, line=133 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12435: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame) - com.cisco.unicorn.vos.status.StatusNotifier.run() @bci=4, line=46 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12434: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame) - unicorn.RpcTcpServer.accept() @bci=4, line=46 (Interpreted frame) - com.cisco.unicorn.consumerAdaptor.ConsumerAdaptorImp.run() @bci=88, line=431 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12433: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.stats.HealthMonitor.runImpl() @bci=123, line=177 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12432: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.stats.AStatsConsolidator.runImpl() @bci=213, line=147 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12431: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.stats.StatsCollector.runImpl() @bci=191, line=303 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12430: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.stats.StatsTransmitter.runImpl() @bci=44, line=111 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12428: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.server.Scheduler.runImpl() @bci=76, line=242 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12426: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.director.playlistHandler.PlaylistFileHandler.runImpl() @bci=250, line=488 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12425: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.director.FileSysHandler.runImpl() @bci=95, line=150 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12424: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.server.ADataFeedAgent.runImpl() @bci=113, line=698 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12423: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.stats.Messenger.runImpl() @bci=44, line=208 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12422: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.pcm.HttpsCertKeyDownloader.runImpl() @bci=432, line=235 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12421: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - com.cisco.unicorn.server.FileHandler.runImpl() @bci=288, line=371 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12420: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.server.SecurityManager.runImpl() @bci=302, line=254 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12419: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.server.CmdProcessor.runImpl() @bci=17, line=106 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12418: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame) - unicorn.RpcTcpServer.accept() @bci=4, line=46 (Interpreted frame) - com.cisco.unicorn.messaging.DistributionAdapterRpcImpl.run() @bci=18, line=260 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12417: (state = BLOCKED) - com.cisco.unicorn.stats.HMHandle._listenForNotificationEvents(com.cisco.unicorn.stats.HMHandle, int, com.cisco.unicorn.stats.HMHandle$IEventListener) @bci=0 (Interpreted frame) - com.cisco.unicorn.stats.HMHandle.listen(com.cisco.unicorn.stats.HMHandle$IEventListener) @bci=27, line=211 (Interpreted frame) - com.cisco.unicorn.stats.HMHandle$eventNotifier.run() @bci=16, line=62 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12416: (state = BLOCKED) - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise) - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=220 (Compiled frame) - sun.nio.cs.StreamDecoder.readBytes() @bci=130, line=264 (Compiled frame) - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=306 (Compiled frame) - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=158 (Compiled frame) - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=167 (Compiled frame) - java.io.BufferedReader.fill() @bci=145, line=136 (Interpreted frame) - java.io.BufferedReader.readLine(boolean) @bci=44, line=299 (Compiled frame) - java.io.BufferedReader.readLine() @bci=2, line=362 (Compiled frame) - com.cisco.unicorn.stats.SyslogListener.run() @bci=33, line=108 (Compiled frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12408: (state = IN_JAVA) - com.cisco.unicorn.server.TimeoutSocket.getSocket(java.net.InetAddress, int, int, int) @bci=66, line=27 (Interpreted frame) - com.cisco.unicorn.server.TimeoutSocket.getSocket(java.lang.String, int, int, int) @bci=7, line=55 (Interpreted frame) - com.cisco.unicorn.server.LocalSocketFactory.createSocket(java.lang.String, int) @bci=8, line=39 (Interpreted frame) - sun.rmi.transport.tcp.TCPEndpoint.newSocket() @bci=62, line=595 (Interpreted frame) - sun.rmi.transport.tcp.TCPChannel.createConnection() @bci=22, line=198 (Interpreted frame) - sun.rmi.transport.tcp.TCPChannel.newConnection() @bci=101, line=184 (Interpreted frame) - sun.rmi.server.UnicastRef.newCall(java.rmi.server.RemoteObject, java.rmi.server.Operation[], int, long) @bci=18, line=322 (Interpreted frame) - sun.rmi.transport.DGCImpl_Stub.dirty(java.rmi.server.ObjID[], long, java.rmi.dgc.Lease) @bci=12 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(java.util.Set, long) @bci=64, line=342 (Interpreted frame) - sun.rmi.transport.DGCClient$EndpointEntry.registerRefs(java.util.List) @bci=238, line=285 (Interpreted frame) - sun.rmi.transport.DGCClient.registerRefs(sun.rmi.transport.Endpoint, java.util.List) @bci=7, line=121 (Interpreted frame) - sun.rmi.transport.ConnectionInputStream.registerRefs() @bci=74, line=80 (Interpreted frame) - sun.rmi.transport.StreamRemoteCall.releaseInputStream() @bci=22, line=138 (Interpreted frame) - sun.rmi.transport.StreamRemoteCall.done() @bci=1, line=292 (Interpreted frame) - sun.rmi.server.UnicastRef.done(java.rmi.server.RemoteCall) @bci=18, line=431 (Interpreted frame) - sun.rmi.registry.RegistryImpl_Stub.lookup(java.lang.String) @bci=128 (Interpreted frame) - java.rmi.Naming.lookup(java.lang.String) @bci=24, line=84 (Interpreted frame) - com.cisco.unicorn.server.AServer.rmiHealthCheck() @bci=41, line=885 (Interpreted frame) - com.cisco.unicorn.server.HealthProbeHandler.runImpl() @bci=96, line=64 (Interpreted frame) - com.cisco.unicorn.server.AModule.run() @bci=7, line=177 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12378: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - com.cisco.unicorn.consumerAdaptor.ConsumerNotifyQueue.loop() @bci=35, line=122 (Interpreted frame) - com.cisco.unicorn.consumerAdaptor.ConsumerNotifyQueue.run() @bci=1, line=98 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12351: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() @bci=55, line=369 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12324: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() @bci=55, line=369 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 12292: (state = BLOCKED) Thread 12291: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Compiled frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame) - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame) Thread 12290: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Compiled frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame) ======================================= using thread-local object allocation. Parallel GC with 18 thread(s) Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 134217728 (128.0MB) NewSize = 1310720 (1.25MB) MaxNewSize = 17592186044415 MB OldSize = 5439488 (5.1875MB) NewRatio = 2 SurvivorRatio = 8 PermSize = 21757952 (20.75MB) MaxPermSize = 85983232 (82.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 10616832 (10.125MB) used = 4517504 (4.3082275390625MB) free = 6099328 (5.8167724609375MB) 42.55039544753087% used From Space: capacity = 1179648 (1.125MB) used = 884736 (0.84375MB) free = 294912 (0.28125MB) 75.0% used To Space: capacity = 1245184 (1.1875MB) used = 0 (0.0MB) free = 1245184 (1.1875MB) 0.0% used PS Old Generation capacity = 89522176 (85.375MB) used = 4931480 (4.703025817871094MB) free = 84590696 (80.6719741821289MB) 5.508668600727489% used PS Perm Generation capacity = 48693248 (46.4375MB) used = 36910176 (35.200286865234375MB) free = 11783072 (11.237213134765625MB) 75.80142528179677% used |
 |
RednaxelaFX
2013-08-15
这看起来是Bug 7116986: Crash in SafepointSynchronize::begin()
现象跟bug report里描述的一模一样。不过看起来这bug还没“修”。 而真正的问题还是别的问题使得超时没进入safepoint 请问楼主给出的jstack输出是如何得到的？是在core dump上做jstack看到的还是对活进程做jstack看到的？ ============== 虽然这个bug看起来也有点像，但跟楼主遇到的肯定不是同一个根问题导致的： http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8014498 |
|
subchen
2013-08-15
谢谢你的回复，jstack 和 heap 都是 从 core dump 中导出来的。
由于问题是不定期发生的，近期出现了有好多次了，但是像重现的时候没法重现，分析了很多原因，一直没有找到问题。 不知道还有什么方法可以入手定位？ |
|
RednaxelaFX
2013-08-15
最有可能的问题成因可能是在需要进入safepoint时正好有Java线程正在执行一个被HotSpot Server Compiler (C2)编译的方法，而这个方法里有循环次数很大的一个counted loop；这个循环末尾没有safepoint检查的代码，导致这个线程一直在跑这个循环而迟迟无法进入safepoint，就跟这帖一样：http://hllvm.group.iteye.com/group/topic/38232，只不过您遇到的超时问题产生了更严重的现象而已。
如果真是这个问题，那相应的对策在那帖里也已经讲了。根本的解决办法是让C2提供一个开关，使它能够有选择的在counted loop末尾生成safepoint检查代码。但这个功能我之前在做还没完成，还有点bug所以还没发出来…抱歉 |
|
subchen
2013-09-18
最近看了一些 safepoint 相关的问题，回头又看了一下这个帖子。
不知道 JVM 等待所有线程进入safepoint的最大等待时间是多少？ 是不是超时之后，JVM就crach了？ 还有就是除了方法调用，内存分配操作会有safepoint检查，还有什么地方会有？ sleep, wait等特殊方法会有safe range的概念，类似于safepoint。 请RednaxelaFX在帮忙解释一下。谢谢！ |
