JVM hang住，所有线程BLOCKED（包括GC线程），Eden区100% - 讨论 - 高级语言虚拟机

群组首页 → 编程语言 → 高级语言虚拟机 → 论坛

发表回复

[讨论] JVM hang住，所有线程BLOCKED（包括GC线程），Eden区100%

diecui1202 2014-08-14

本人在做一个Java的热部署工具，hotcode，它本质是一个javaagent，通过修改了类的class结构，能够支持属性、方法等的增、删、改等操作；

在测试过程中，如果我新生代设置的比较小，在GC时，JVM会hang住：

1、JVM启动参数：

-Xms4096m -Xmx4096m -XX:PermSize=256m -XX:MaxPermSize=512m -Xmn128m -XX:+PrintGCDetails

我是故意将堆调大一些的，但是-Xmn设置稍等小一点的话，就会出现这个问题；

2、jstat输出：

jstat -gcutil 16063 1000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
  5.68  36.18 100.00   1.50  11.93     14    0.555     0    0.000    0.555
  5.68  36.18 100.00   1.50  11.93     14    0.555     0    0.000    0.555
  5.68  36.18 100.00   1.50  11.93     14    0.555     0    0.000    0.555
  5.68  36.18 100.00   1.50  11.93     14    0.555     0    0.000    0.555

3、jstack -F输出：

jstack -F 16063  
Attaching to process ID 16063, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b56
Deadlock Detection:

No deadlocks found.

Thread 16097: (state = BLOCKED)
 - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
 - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=45, line=792 (Interpreted frame)
 - java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) @bci=12, line=142 (Interpreted frame)
 - org.apache.catalina.loader.WebappClassLoader.findClassInternal(java.lang.String) @bci=408, line=2957 (Interpreted frame)
 - org.apache.catalina.loader.WebappClassLoader.findClass(java.lang.String) @bci=395, line=1210 (Interpreted frame)
 - org.apache.catalina.loader.WebappClassLoader.loadClass(java.lang.String, boolean) @bci=434, line=1690 (Interpreted frame)
 - com.taobao.tomcat.container.context.loader.AliWebappClassLoader.loadClass(java.lang.String, boolean) @bci=18, line=63 (Interpreted frame)
 - org.apache.catalina.loader.WebappClassLoader.loadClass(java.lang.String) @bci=3, line=1571 (Interpreted frame)
 - org.springframework.web.context.support.AbstractRefreshableWebApplicationContext.createEnvironment() @bci=0, line=156 (Interpreted frame)
 - org.springframework.context.support.AbstractApplicationContext.getEnvironment() @bci=9, line=288 (Interpreted frame)
 - org.springframework.web.context.support.AbstractRefreshableWebApplicationContext.initPropertySources() @bci=5, line=205 (Interpreted frame)
 - org.springframework.context.support.AbstractApplicationContext.prepareRefresh() @bci=70, line=514 (Interpreted frame)
 - org.springframework.context.support.AbstractApplicationContext.refresh() @bci=8, line=448 (Interpreted frame)
 - org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(org.springframework.web.context.ConfigurableWebApplicationContext, javax.servlet.ServletContext) @bci=164, line=389 (Interpreted frame)
 - org.springframework.web.context.ContextLoader.initWebApplicationContext(javax.servlet.ServletContext) @bci=133, line=294 (Interpreted frame)
 - com.alibaba.citrus.webx.context.WebxComponentsLoader.initWebApplicationContext(javax.servlet.ServletContext) @bci=11, line=117 (Interpreted frame)
 - org.springframework.web.context.ContextLoaderListener.contextInitialized(javax.servlet.ServletContextEvent) @bci=28, line=112 (Interpreted frame)
 - org.apache.catalina.core.StandardContext.listenerStart() @bci=650, line=4973 (Interpreted frame)
 - org.apache.catalina.core.StandardContext.startInternal() @bci=1204, line=5467 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.start() @bci=210, line=150 (Interpreted frame)
 - org.apache.catalina.core.ContainerBase.addChildInternal(org.apache.catalina.Container) @bci=170, line=901 (Interpreted frame)
 - org.apache.catalina.core.ContainerBase.addChild(org.apache.catalina.Container) @bci=26, line=877 (Interpreted frame)
 - org.apache.catalina.core.StandardHost.addChild(org.apache.catalina.Container) @bci=40, line=632 (Interpreted frame)
 - org.apache.catalina.startup.HostConfig.deployWAR(org.apache.catalina.util.ContextName, java.io.File) @bci=1671, line=1083 (Interpreted frame)
 - org.apache.catalina.startup.HostConfig$DeployWar.run() @bci=12, line=1880 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=262 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)


Thread 16096: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame)
 - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=425 (Interpreted frame)
 - java.util.concurrent.FutureTask.get() @bci=13, line=187 (Interpreted frame)
 - org.apache.catalina.startup.HostConfig.deployWARs(java.io.File, java.lang.String[]) @bci=399, line=818 (Interpreted frame)
 - com.taobao.tomcat.container.host.AliHostConfig.deployApps() @bci=203, line=61 (Interpreted frame)
 - org.apache.catalina.startup.HostConfig.start() @bci=313, line=1614 (Interpreted frame)
 - org.apache.catalina.startup.HostConfig.lifecycleEvent(org.apache.catalina.LifecycleEvent) @bci=142, line=330 (Interpreted frame)
 - org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(java.lang.String, java.lang.Object) @bci=37, line=117 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(java.lang.String, java.lang.Object) @bci=6, line=90 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.setStateInternal(org.apache.catalina.LifecycleState, java.lang.Object, boolean) @bci=140, line=402 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.setState(org.apache.catalina.LifecycleState) @bci=4, line=347 (Interpreted frame)
 - org.apache.catalina.core.ContainerBase.startInternal() @bci=322, line=1140 (Interpreted frame)
 - org.apache.catalina.core.StandardHost.startInternal() @bci=142, line=799 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.start() @bci=210, line=150 (Interpreted frame)
 - org.apache.catalina.core.ContainerBase$StartChild.call() @bci=4, line=1559 (Interpreted frame)
 - org.apache.catalina.core.ContainerBase$StartChild.call() @bci=1, line=1549 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=262 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)


Thread 16095: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)


Thread 16094: (state = BLOCKED)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=257 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
 - sun.net.httpserver.ServerImpl$Dispatcher.run() @bci=72, line=341 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)


Thread 16093: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame)
 - java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)


Thread 16092: (state = BLOCKED)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=257 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
 - org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23, line=240 (Interpreted frame)
 - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=44 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)


Thread 16091: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - com.taobao.pandora.framework.core.EventDispatcher$1.run() @bci=60, line=74 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)


Thread 16090: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - sun.misc.GC$Daemon.run() @bci=51, line=117 (Interpreted frame)


Thread 16076: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingDeque.takeFirst() @bci=22, line=489 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingDeque.take() @bci=1, line=678 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)


Thread 16072: (state = BLOCKED)


Thread 16071: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Compiled frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=189 (Compiled frame)


Thread 16070: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)


Thread 16064: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame)
 - java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=425 (Interpreted frame)
 - java.util.concurrent.FutureTask.get() @bci=13, line=187 (Interpreted frame)
 - org.apache.catalina.core.ContainerBase.startInternal() @bci=240, line=1123 (Interpreted frame)
 - org.apache.catalina.core.StandardEngine.startInternal() @bci=41, line=300 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.start() @bci=210, line=150 (Interpreted frame)
 - org.apache.catalina.core.StandardService.startInternal() @bci=63, line=443 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.start() @bci=210, line=150 (Interpreted frame)
 - org.apache.catalina.core.StandardServer.startInternal() @bci=61, line=751 (Interpreted frame)
 - org.apache.catalina.util.LifecycleBase.start() @bci=210, line=150 (Interpreted frame)
 - org.apache.catalina.startup.Catalina.start() @bci=37, line=699 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=57 (Interpreted frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=79, line=606 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.start() @bci=37, line=320 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=150, line=454 (Interpreted frame)

4、jstack -m输出：

jstack -m 16063
Attaching to process ID 16063, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b56
Deadlock Detection:

No deadlocks found.

----------------- 16064 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d8559645      Unsafe_Park + 0x125
0x00007ff0d04d2559      * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:186 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.FutureTask.awaitDone(boolean, long) bci:165 line:425 (Interpreted frame)
0x00007ff0d04c0254      * java.util.concurrent.FutureTask.get() bci:13 line:187 (Interpreted frame)
0x00007ff0d04c0961      * org.apache.catalina.core.ContainerBase.startInternal() bci:240 line:1123 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.core.StandardEngine.startInternal() bci:41 line:300 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.start() bci:210 line:150 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.core.StandardService.startInternal() bci:63 line:443 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.start() bci:210 line:150 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.core.StandardServer.startInternal() bci:61 line:751 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.start() bci:210 line:150 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.startup.Catalina.start() bci:37 line:699 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d847173f      _ZN10Reflection6invokeE19instanceKlassHandle12methodHandle6Handleb14objArrayHandle9BasicTypeS3_bP6Thread + 0x47f
0x00007ff0d84724f0      _ZN10Reflection13invoke_methodEP7oopDesc6Handle14objArrayHandleP6Thread + 0x160
0x00007ff0d8268314      JVM_InvokeMethod + 0x224
0x00007ff0d04d2559      * sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) bci:0 (Interpreted frame)
0x00007ff0d04c02b3      * sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:87 line:57 (Interpreted frame)
0x00007ff0d04c02b3      * sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:6 line:43 (Interpreted frame)
0x00007ff0d04c0961      * java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) bci:79 line:606 (Interpreted frame)
0x00007ff0d04c02b3      * org.apache.catalina.startup.Bootstrap.start() bci:37 line:320 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) bci:150 line:454 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d82188d9      _ZL17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread + 0x219
0x00007ff0d8221962      jni_CallStaticVoidMethod + 0x162
0x00007ff0d8fb16d9      JavaMain + 0x7e9
----------------- 16065 -----------------
0x00007ff0d83266f0              ????????
0x00007ff0d8466f77      _ZN9StealTask5do_itEP13GCTaskManagerj + 0x1b7
0x00007ff0d8139f0f      _ZN12GCTaskThread3runEv + 0x12f
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16066 -----------------
0x00007ff0d83266f9              ????????
0x00007ff0d8466f77      _ZN9StealTask5do_itEP13GCTaskManagerj + 0x1b7
0x00007ff0d8139f0f      _ZN12GCTaskThread3runEv + 0x12f
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16067 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      _ZN7Monitor5ILockEP6Thread + 0x1b8
0x00007ff0d83be9a7      _ZN7Monitor4lockEP6Thread + 0x127
0x00007ff0d841ce22      _ZN20ParallelScavengeHeap12mem_allocateEmPb + 0x72
0x00007ff0d81a55ae      _ZN13CollectedHeap24common_mem_allocate_initE11KlassHandlemP6Thread + 0x17e
0x00007ff0d81a0034      _ZN13instanceKlass17allocate_instanceEP6Thread + 0x74
0x00007ff0d81eb567      _ZN16java_lang_String12basic_createEibP6Thread + 0x87
0x00007ff0d81eb9d7      _ZN16java_lang_String15create_from_strEPKcP6Thread + 0x37
0x00007ff0d80f5433      _ZN10Exceptions13new_exceptionEP6ThreadP6SymbolPKc6HandleS6_S6_NS_22ExceptionMsgToUtf8ModeE + 0x153
0x00007ff0d80f5548      _ZN10Exceptions13new_exceptionEP6ThreadP6SymbolPKcNS_22ExceptionMsgToUtf8ModeE + 0x18
0x00007ff0d81506ed      _ZN14GenerateOopMap12verify_errorEPKcz + 0xad
0x00007ff0d8155d7d      _ZN14GenerateOopMap9interp_bbEP10BasicBlock + 0x18d
0x00007ff0d8156c3a      _ZN14GenerateOopMap17do_interpretationEv + 0x1aa
0x00007ff0d8156fba      _ZN14GenerateOopMap11compute_mapEP6Thread + 0x2aa
0x00007ff0d83f163a      _ZN19OopMapForCacheEntry11compute_mapEP6Thread + 0xca
0x00007ff0d83f26f0      _ZN16OopMapCacheEntry4fillE12methodHandlei + 0x110
0x00007ff0d83f2e1f      _ZN11OopMapCache6lookupE12methodHandleiP17InterpreterOopMap + 0x2bf
0x00007ff0d810382e      _ZN5frame19oops_interpreted_doEP10OopClosurePK11RegisterMapb + 0x29e
0x00007ff0d853048a      _ZN10JavaThread7oops_doEP10OopClosureP15CodeBlobClosure + 0x15a
0x00007ff0d8466aa5      _ZN15ThreadRootsTask5do_itEP13GCTaskManagerj + 0x75
0x00007ff0d8139f0f      _ZN12GCTaskThread3runEv + 0x12f
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16068 -----------------
0x00007ff0d8504fc8      _ZN22ParallelTaskTerminator17offer_terminationEP20TerminatorTerminator + 0xd8
0x00007ff0d8466f77      _ZN9StealTask5do_itEP13GCTaskManagerj + 0x1b7
0x00007ff0d8139f0f      _ZN12GCTaskThread3runEv + 0x12f
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16069 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83befaf      _ZN7Monitor5IWaitEP6Threadl + 0xef
0x00007ff0d83bf73e      _ZN7Monitor4waitEblb + 0x22e
0x00007ff0d81394f2      _ZN13GCTaskManager16execute_and_waitEP11GCTaskQueue + 0xd2
0x00007ff0d84656a9      _ZN10PSScavenge16invoke_no_policyEv + 0xa09
0x00007ff0d8466341      _ZN10PSScavenge6invokeEv + 0x41
0x00007ff0d841d440      _ZN20ParallelScavengeHeap19failed_mem_allocateEm + 0x70
0x00007ff0d857ccc7      _ZN29VM_ParallelGCFailedAllocation4doitEv + 0x97
0x00007ff0d85840f5      _ZN12VM_Operation8evaluateEv + 0x55
0x00007ff0d858252a      _ZN8VMThread18evaluate_operationEP12VM_Operation + 0xba
0x00007ff0d85828ae      _ZN8VMThread4loopEv + 0x1ce
0x00007ff0d8582d20      _ZN8VMThread3runEv + 0x70
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16070 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83ebe6d      _ZN13ObjectMonitor4waitElbP6Thread + 0x9bd
0x00007ff0d825d918      JVM_MonitorWait + 0x168
0x00007ff0d04d2559      * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007ff0d069b65c      * java.lang.Object.wait() bci:2 line:503 (Compiled frame)
* java.lang.ref.Reference$ReferenceHandler.run() bci:46 line:133 (Compiled frame)
----------------- 16071 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83ebe6d      _ZN13ObjectMonitor4waitElbP6Thread + 0x9bd
0x00007ff0d825d918      JVM_MonitorWait + 0x168
0x00007ff0d04d2559      * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007ff0d06b5f44      * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:135 (Compiled frame)
* java.lang.ref.ReferenceQueue.remove() bci:2 line:151 (Compiled frame)
----------------- 16072 -----------------
0x00007ff0d91d1d21      sem_wait + 0x41
0x00007ff0d83f5bc5      _ZL19signal_thread_entryP10JavaThreadP6Thread + 0x95
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16076 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d8559645      Unsafe_Park + 0x125
0x00007ff0d04d2559      * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:186 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() bci:42 line:2043 (Interpreted frame)
0x00007ff0d04c0786      * java.util.concurrent.LinkedBlockingDeque.takeFirst() bci:22 line:489 (Interpreted frame)
0x00007ff0d04c02b3      * java.util.concurrent.LinkedBlockingDeque.take() bci:1 line:678 (Interpreted frame)
0x00007ff0d04c0961      * java.util.concurrent.ThreadPoolExecutor.getTask() bci:156 line:1068 (Interpreted frame)
0x00007ff0d04c02b3      * java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) bci:26 line:1130 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:5 line:615 (Interpreted frame)
0x00007ff0d04c0786      * java.lang.Thread.run() bci:11 line:724 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16081 -----------------
0x00007ff0d91d2b9b      __libc_accept + 0x4b
0x00007ff0bdffcbb1      socketTransport_accept + 0x91
0x00007ff0d39313fa      acceptThread + 0x4a
0x00007ff0d82dcdd7      _ZN16JvmtiAgentThread19call_start_functionEv + 0x97
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16082 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d82e151d      _ZN15JvmtiRawMonitor10SimpleWaitEP6Threadl + 0x2cd
0x00007ff0d82e16c8      _ZN15JvmtiRawMonitor8raw_waitElbP6Thread + 0xa8
0x00007ff0d82bd727      _ZN8JvmtiEnv14RawMonitorWaitEP15JvmtiRawMonitorl + 0x67
0x00007ff0d393585c      debugMonitorWait + 0x2c
0x00007ff0d3926f0c      commandLoop + 0x6c
0x00007ff0d82dcdd7      _ZN16JvmtiAgentThread19call_start_functionEv + 0x97
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16083 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83befaf      _ZN7Monitor5IWaitEP6Threadl + 0xef
0x00007ff0d83bf7b6      _ZN7Monitor4waitEblb + 0x2a6
0x00007ff0d8031448      _ZN12CompileQueue3getEv + 0x128
0x00007ff0d803553a      _ZN13CompileBroker20compiler_thread_loopEv + 0x13a
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16084 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83befaf      _ZN7Monitor5IWaitEP6Threadl + 0xef
0x00007ff0d83bf7b6      _ZN7Monitor4waitEblb + 0x2a6
0x00007ff0d8031448      _ZN12CompileQueue3getEv + 0x128
0x00007ff0d803553a      _ZN13CompileBroker20compiler_thread_loopEv + 0x13a
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16085 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83befaf      _ZN7Monitor5IWaitEP6Threadl + 0xef
0x00007ff0d83bf73e      _ZN7Monitor4waitEblb + 0x22e
0x00007ff0d848cdb8      _ZN13ServiceThread20service_thread_entryEP10JavaThreadP6Thread + 0x198
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16086 -----------------
0x00007ff0d91d0150      __pthread_cond_timedwait + 0x120
0x00007ff0d83bf25e      _ZN7Monitor5IWaitEP6Threadl + 0x39e
0x00007ff0d83bf73e      _ZN7Monitor4waitEblb + 0x22e
0x00007ff0d8531d61      _ZNK13WatcherThread5sleepEv + 0x71
0x00007ff0d853206e      _ZN13WatcherThread3runEv + 0x5e
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16090 -----------------
0x00007ff0d91d0150      __pthread_cond_timedwait + 0x120
0x00007ff0d83eb70b      _ZN13ObjectMonitor4waitElbP6Thread + 0x25b
0x00007ff0d825d918      JVM_MonitorWait + 0x168
0x00007ff0d04d2559      * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007ff0d04c00d8      * sun.misc.GC$Daemon.run() bci:51 line:117 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16091 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      _ZN7Monitor5ILockEP6Thread + 0x1b8
0x00007ff0d83bea5f      _ZN7Monitor28lock_without_safepoint_checkEv + 0x2f
0x00007ff0d8489baf      _ZN20SafepointSynchronize5blockEP10JavaThread + 0x19f
0x00007ff0d83fd5f5      _ZN2os5sleepEP6Threadlb + 0x2e5
0x00007ff0d8265e32      JVM_Sleep + 0x342
0x00007ff0d04d2559      * java.lang.Thread.sleep(long) bci:0 (Interpreted frame)
0x00007ff0d04c00d8      * com.taobao.pandora.framework.core.EventDispatcher$1.run() bci:60 line:74 (Interpreted frame)
0x00007ff0d04c0786      * java.lang.Thread.run() bci:11 line:724 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16092 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      _ZN7Monitor5ILockEP6Thread + 0x1b8
0x00007ff0d83bea5f      _ZN7Monitor28lock_without_safepoint_checkEv + 0x2f
0x00007ff0d8489baf      _ZN20SafepointSynchronize5blockEP10JavaThread + 0x19f
0x00007ff0d85369cd      _ZN10JavaThread44check_safepoint_and_suspend_for_native_transEPS_ + 0x2ed
0x00007ff0d85370a1      _ZN10JavaThread40check_special_condition_for_native_transEPS_ + 0x11
0x00007ff0d04d25c2      * sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) bci:0 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.EPollArrayWrapper.poll(long) bci:18 line:257 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.EPollSelectorImpl.doSelect(long) bci:28 line:79 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.SelectorImpl.lockAndDoSelect(long) bci:37 line:87 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.SelectorImpl.select(long) bci:30 line:98 (Interpreted frame)
0x00007ff0d04c0254      * org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() bci:23 line:240 (Interpreted frame)
0x00007ff0d04c0786      * org.jboss.netty.util.ThreadRenamingRunnable.run() bci:55 line:108 (Interpreted frame)
0x00007ff0d04c0786      * org.jboss.netty.util.internal.DeadLockProofWorker$1.run() bci:14 line:44 (Interpreted frame)
0x00007ff0d04c0786      * java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) bci:95 line:1145 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:5 line:615 (Interpreted frame)
0x00007ff0d04c0786      * java.lang.Thread.run() bci:11 line:724 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16093 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      _ZN7Monitor5ILockEP6Thread + 0x1b8
0x00007ff0d83bea5f      _ZN7Monitor28lock_without_safepoint_checkEv + 0x2f
0x00007ff0d8489baf      _ZN20SafepointSynchronize5blockEP10JavaThread + 0x19f
0x00007ff0d83ebb0b      _ZN13ObjectMonitor4waitElbP6Thread + 0x65b
0x00007ff0d825d918      JVM_MonitorWait + 0x168
0x00007ff0d04d2559      * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.TimerThread.mainLoop() bci:201 line:552 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.TimerThread.run() bci:1 line:505 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16094 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      _ZN7Monitor5ILockEP6Thread + 0x1b8
0x00007ff0d83bea5f      _ZN7Monitor28lock_without_safepoint_checkEv + 0x2f
0x00007ff0d8489baf      _ZN20SafepointSynchronize5blockEP10JavaThread + 0x19f
0x00007ff0d85369cd      _ZN10JavaThread44check_safepoint_and_suspend_for_native_transEPS_ + 0x2ed
0x00007ff0d85370a1      _ZN10JavaThread40check_special_condition_for_native_transEPS_ + 0x11
0x00007ff0d04d25c2      * sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) bci:0 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.EPollArrayWrapper.poll(long) bci:18 line:257 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.EPollSelectorImpl.doSelect(long) bci:28 line:79 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.SelectorImpl.lockAndDoSelect(long) bci:37 line:87 (Interpreted frame)
0x00007ff0d04c0254      * sun.nio.ch.SelectorImpl.select(long) bci:30 line:98 (Interpreted frame)
0x00007ff0d04c0254      * sun.net.httpserver.ServerImpl$Dispatcher.run() bci:72 line:341 (Interpreted frame)
0x00007ff0d04c0786      * java.lang.Thread.run() bci:11 line:724 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16095 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      _ZN7Monitor5ILockEP6Thread + 0x1b8
0x00007ff0d83bea5f      _ZN7Monitor28lock_without_safepoint_checkEv + 0x2f
0x00007ff0d8489baf      _ZN20SafepointSynchronize5blockEP10JavaThread + 0x19f
0x00007ff0d84006a1      _ZN21ThreadStateTransition15trans_and_fenceE15JavaThreadStateS0_ + 0x51
0x00007ff0d83fc1cb      _ZN6Parker4parkEbl + 0x26b
0x00007ff0d8559645      Unsafe_Park + 0x125
0x00007ff0d04d2559      * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) bci:20 line:226 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) bci:68 line:2082 (Interpreted frame)
0x00007ff0d04c08a3      * java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() bci:122 line:1090 (Interpreted frame)
0x00007ff0d04c02b3      * java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() bci:1 line:807 (Interpreted frame)
0x00007ff0d04c0961      * java.util.concurrent.ThreadPoolExecutor.getTask() bci:156 line:1068 (Interpreted frame)
0x00007ff0d04c02b3      * java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) bci:26 line:1130 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:5 line:615 (Interpreted frame)
0x00007ff0d04c0786      * java.lang.Thread.run() bci:11 line:724 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16096 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d8559645      Unsafe_Park + 0x125
0x00007ff0d04d2559      * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:186 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.FutureTask.awaitDone(boolean, long) bci:165 line:425 (Interpreted frame)
0x00007ff0d04c0254      * java.util.concurrent.FutureTask.get() bci:13 line:187 (Interpreted frame)
0x00007ff0d04c0961      * org.apache.catalina.startup.HostConfig.deployWARs(java.io.File, java.lang.String[]) bci:399 line:818 (Interpreted frame)
0x00007ff0d04c00d8      * com.taobao.tomcat.container.host.AliHostConfig.deployApps() bci:203 line:61 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.startup.HostConfig.start() bci:313 line:1614 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.startup.HostConfig.lifecycleEvent(org.apache.catalina.LifecycleEvent) bci:142 line:330 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(java.lang.String, java.lang.Object) bci:37 line:117 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(java.lang.String, java.lang.Object) bci:6 line:90 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.setStateInternal(org.apache.catalina.LifecycleState, java.lang.Object, boolean) bci:140 line:402 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.setState(org.apache.catalina.LifecycleState) bci:4 line:347 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.core.ContainerBase.startInternal() bci:322 line:1140 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.core.StandardHost.startInternal() bci:142 line:799 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.start() bci:210 line:150 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.core.ContainerBase$StartChild.call() bci:4 line:1559 (Interpreted frame)
0x00007ff0d04c02b3      * org.apache.catalina.core.ContainerBase$StartChild.call() bci:1 line:1549 (Interpreted frame)
0x00007ff0d04c0961      * java.util.concurrent.FutureTask.run() bci:42 line:262 (Interpreted frame)
0x00007ff0d04c0786      * java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) bci:95 line:1145 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:5 line:615 (Interpreted frame)
0x00007ff0d04c0786      * java.lang.Thread.run() bci:11 line:724 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16097 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83befaf      _ZN7Monitor5IWaitEP6Threadl + 0xef
0x00007ff0d83bf7b6      _ZN7Monitor4waitEblb + 0x2a6
0x00007ff0d8582454      _ZN8VMThread7executeEP12VM_Operation + 0x324
0x00007ff0d841cf46      _ZN20ParallelScavengeHeap12mem_allocateEmPb + 0x196
0x00007ff0d854bbbf      _ZN14typeArrayKlass15allocate_commonEibP6Thread + 0x2ff
0x00007ff0d8235ae1      jni_NewByteArray + 0xc1
0x00007ff0d711efd8      transformClassFile + 0xd8
0x00007ff0d711be06      eventHandlerClassFileLoadHook + 0x96
0x00007ff0d82d404f      _ZN11JvmtiExport25post_class_file_load_hookEP6Symbol6HandleS2_PPhS4_S4_Pi + 0x63f
0x00007ff0d7fef0b1      _ZN15ClassFileParser14parseClassFileEP6Symbol6HandleS2_11KlassHandleP13GrowableArrayIS2_ER13TempNewSymbolbP6Thread + 0x1e1
0x00007ff0d84fed82      _ZN16SystemDictionary19resolve_from_streamEP6Symbol6HandleS2_P15ClassFileStreambP6Thread + 0x152
0x00007ff0d826758e      _ZL23jvm_define_class_commonP7JNIEnv_PKcP8_jobjectPKaiS4_S2_hP6Thread + 0x2ee
0x00007ff0d8267802      JVM_DefineClassWithSource + 0xc2
0x00007ff0d73321af      Java_java_lang_ClassLoader_defineClass1 + 0x25f
0x00007ff0d04d2559      * java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) bci:0 (Interpreted frame)
0x00007ff0d04c02b3      * java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) bci:45 line:792 (Interpreted frame)
0x00007ff0d04c02b3      * java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) bci:12 line:142 (Interpreted frame)
0x00007ff0d04c02b3      * org.apache.catalina.loader.WebappClassLoader.findClassInternal(java.lang.String) bci:408 line:2957 (Interpreted frame)
0x00007ff0d04c02b3      * org.apache.catalina.loader.WebappClassLoader.findClass(java.lang.String) bci:395 line:1210 (Interpreted frame)
0x00007ff0d04c02b3      * org.apache.catalina.loader.WebappClassLoader.loadClass(java.lang.String, boolean) bci:434 line:1690 (Interpreted frame)
0x00007ff0d04c02b3      * com.taobao.tomcat.container.context.loader.AliWebappClassLoader.loadClass(java.lang.String, boolean) bci:18 line:63 (Interpreted frame)
0x00007ff0d04c02b3      * org.apache.catalina.loader.WebappClassLoader.loadClass(java.lang.String) bci:3 line:1571 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0206      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_S2_P6Thread + 0x56
0x00007ff0d84ff499      _ZN16SystemDictionary19load_instance_classEP6Symbol6HandleP6Thread + 0x419
0x00007ff0d84ffc31      _ZN16SystemDictionary30resolve_instance_class_or_nullEP6Symbol6HandleS2_P6Thread + 0x781
0x00007ff0d8500094      _ZN16SystemDictionary15resolve_or_nullEP6Symbol6HandleS2_P6Thread + 0x54
0x00007ff0d8501174      _ZN16SystemDictionary15resolve_or_failEP6Symbol6HandleS2_bP6Thread + 0x34
0x00007ff0d82574ca      _Z28find_class_from_class_loaderP7JNIEnv_P6Symbolh6HandleS3_hP6Thread + 0x3a
0x00007ff0d8257b3b      JVM_FindClassFromClass + 0x29b
0x00007ff0d75537b2      load_class_global + 0x32
0x00007ff0d7553870      object_fullinfo_to_classclass + 0x40
0x00007ff0d75547bd      merge_fullinfo_types + 0xbd
0x00007ff0d7554cf5      isAssignableTo + 0x15
0x00007ff0d7556434      pop_stack + 0xbf4
0x00007ff0d75569c5      run_dataflow + 0x1c5
0x00007ff0d7558682      verify_method + 0x812
0x00007ff0d7559b05      VerifyClassForMajorVersion + 0x635
0x00007ff0d8560bb9      _ZN8Verifier16inference_verifyE19instanceKlassHandlePcmP6Thread + 0x169
0x00007ff0d856e2e1      _ZN8Verifier6verifyE19instanceKlassHandleNS_4ModeEbP6Thread + 0x561
0x00007ff0d819e56d      _ZN13instanceKlass15link_class_implE19instanceKlassHandlebP6Thread + 0x4fd
0x00007ff0d819e264      _ZN13instanceKlass15link_class_implE19instanceKlassHandlebP6Thread + 0x1f4
0x00007ff0d81a1ef8      _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread + 0x88
0x00007ff0d81a24fa      _ZN13instanceKlass10initializeEP6Thread + 0x6a
0x00007ff0d81d9593      _ZN18InterpreterRuntime4_newEP10JavaThreadP19constantPoolOopDesci + 0x153
0x00007ff0d04e3425      * org.springframework.web.context.support.AbstractRefreshableWebApplicationContext.createEnvironment() bci:0 line:156 (Interpreted frame)
0x00007ff0d04c02b3      * org.springframework.context.support.AbstractApplicationContext.getEnvironment() bci:9 line:288 (Interpreted frame)
0x00007ff0d04c02b3      * org.springframework.web.context.support.AbstractRefreshableWebApplicationContext.initPropertySources() bci:5 line:205 (Interpreted frame)
0x00007ff0d04c00d8      * org.springframework.context.support.AbstractApplicationContext.prepareRefresh() bci:70 line:514 (Interpreted frame)
0x00007ff0d04c00d8      * org.springframework.context.support.AbstractApplicationContext.refresh() bci:8 line:448 (Interpreted frame)
0x00007ff0d04c0786      * org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(org.springframework.web.context.ConfigurableWebApplicationContext, javax.servlet.ServletContext) bci:164 line:389 (Interpreted frame)
0x00007ff0d04c00d8      * org.springframework.web.context.ContextLoader.initWebApplicationContext(javax.servlet.ServletContext) bci:133 line:294 (Interpreted frame)
0x00007ff0d04c02b3      * com.alibaba.citrus.webx.context.WebxComponentsLoader.initWebApplicationContext(javax.servlet.ServletContext) bci:11 line:117 (Interpreted frame)
0x00007ff0d04c02b3      * org.springframework.web.context.ContextLoaderListener.contextInitialized(javax.servlet.ServletContextEvent) bci:28 line:112 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.core.StandardContext.listenerStart() bci:650 line:4973 (Interpreted frame)
0x00007ff0d04c03d0      * org.apache.catalina.core.StandardContext.startInternal() bci:1204 line:5467 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.util.LifecycleBase.start() bci:210 line:150 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.core.ContainerBase.addChildInternal(org.apache.catalina.Container) bci:170 line:901 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.core.ContainerBase.addChild(org.apache.catalina.Container) bci:26 line:877 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.core.StandardHost.addChild(org.apache.catalina.Container) bci:40 line:632 (Interpreted frame)
0x00007ff0d04c0786      * org.apache.catalina.startup.HostConfig.deployWAR(org.apache.catalina.util.ContextName, java.io.File) bci:1671 line:1083 (Interpreted frame)
0x00007ff0d04c00d8      * org.apache.catalina.startup.HostConfig$DeployWar.run() bci:12 line:1880 (Interpreted frame)
0x00007ff0d04c0786      * java.util.concurrent.Executors$RunnableAdapter.call() bci:4 line:471 (Interpreted frame)
0x00007ff0d04c0961      * java.util.concurrent.FutureTask.run() bci:42 line:262 (Interpreted frame)
0x00007ff0d04c0786      * java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) bci:95 line:1145 (Interpreted frame)
0x00007ff0d04c00d8      * java.util.concurrent.ThreadPoolExecutor$Worker.run() bci:5 line:615 (Interpreted frame)
0x00007ff0d04c0786      * java.lang.Thread.run() bci:11 line:724 (Interpreted frame)
0x00007ff0d04ba4e7      <StubRoutines>
0x00007ff0d81e1405      _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x365
0x00007ff0d81dfe68      _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x28
0x00007ff0d81e0137      _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x197
0x00007ff0d81e0257      _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x47
0x00007ff0d825b9e5      _ZL12thread_entryP10JavaThreadP6Thread + 0xe5
0x00007ff0d8535b2f      _ZN10JavaThread17thread_main_innerEv + 0xdf
0x00007ff0d8535c35      _ZN10JavaThread3runEv + 0xf5
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108
----------------- 16063 -----------------
0x00007ff0d91ccb35      pthread_join + 0x125

请各位大侠帮小弟分析一下，非常感谢！

diecui1202 2014-08-14

有同学帮忙看了下，说是GC线程执行时，执行到verify_error这个方法，然后blocked了：

----------------- 16067 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      _ZN7Monitor5ILockEP6Thread + 0x1b8
0x00007ff0d83be9a7      _ZN7Monitor4lockEP6Thread + 0x127
0x00007ff0d841ce22      _ZN20ParallelScavengeHeap12mem_allocateEmPb + 0x72
0x00007ff0d81a55ae      _ZN13CollectedHeap24common_mem_allocate_initE11KlassHandlemP6Thread + 0x17e
0x00007ff0d81a0034      _ZN13instanceKlass17allocate_instanceEP6Thread + 0x74
0x00007ff0d81eb567      _ZN16java_lang_String12basic_createEibP6Thread + 0x87
0x00007ff0d81eb9d7      _ZN16java_lang_String15create_from_strEPKcP6Thread + 0x37
0x00007ff0d80f5433      _ZN10Exceptions13new_exceptionEP6ThreadP6SymbolPKc6HandleS6_S6_NS_22ExceptionMsgToUtf8ModeE + 0x153
0x00007ff0d80f5548      _ZN10Exceptions13new_exceptionEP6ThreadP6SymbolPKcNS_22ExceptionMsgToUtf8ModeE + 0x18
0x00007ff0d81506ed      _ZN14GenerateOopMap12verify_errorEPKcz + 0xad
0x00007ff0d8155d7d      _ZN14GenerateOopMap9interp_bbEP10BasicBlock + 0x18d
0x00007ff0d8156c3a      _ZN14GenerateOopMap17do_interpretationEv + 0x1aa
0x00007ff0d8156fba      _ZN14GenerateOopMap11compute_mapEP6Thread + 0x2aa
0x00007ff0d83f163a      _ZN19OopMapForCacheEntry11compute_mapEP6Thread + 0xca
0x00007ff0d83f26f0      _ZN16OopMapCacheEntry4fillE12methodHandlei + 0x110
0x00007ff0d83f2e1f      _ZN11OopMapCache6lookupE12methodHandleiP17InterpreterOopMap + 0x2bf
0x00007ff0d810382e      _ZN5frame19oops_interpreted_doEP10OopClosurePK11RegisterMapb + 0x29e
0x00007ff0d853048a      _ZN10JavaThread7oops_doEP10OopClosureP15CodeBlobClosure + 0x15a
0x00007ff0d8466aa5      _ZN15ThreadRootsTask5do_itEP13GCTaskManagerj + 0x75
0x00007ff0d8139f0f      _ZN12GCTaskThread3runEv + 0x12f
0x00007ff0d83fda58      _ZL10java_startP6Thread + 0x108

但是还是不太明白GC为什么会执行到verify_error这里；

另外，JVM启动时，我没有设置 -noverify 参数；

RednaxelaFX 2014-08-14

diecui1202 写道

有同学帮忙看了下，说是GC线程执行时，执行到verify_error这个方法，然后blocked了：

...

但是还是不太明白GC为什么会执行到verify_error这里；

另外，JVM启动时，我没有设置 -noverify 参数；

那位同学眼睛挺尖的，在GC线程上要给新Java对象（那个VerifyError异常对象）分配内存，这是必然要死锁的。在这种情况下能出现死锁不得不说还是HotSpot VM不够完善，对错误不够包容。

----------------- 16067 -----------------
0x00007ff0d91cfee9      __pthread_cond_wait + 0xb9
0x00007ff0d83be828      Monitor::ILock(Thread*) + 0x1b8
0x00007ff0d83be9a7      Monitor::lock(Thread*) + 0x127
0x00007ff0d841ce22      ParallelScavengeHeap::mem_allocate(unsigned long, bool*) + 0x72
0x00007ff0d81a55ae      CollectedHeap::common_mem_allocate_init(KlassHandle, unsigned long, Thread*) + 0x17e
0x00007ff0d81a0034      instanceKlass::allocate_instance(Thread*) + 0x74
0x00007ff0d81eb567      java_lang_String::basic_create(int, bool, Thread*) + 0x87
0x00007ff0d81eb9d7      java_lang_String::create_from_str(char const*, Thread*) + 0x37
0x00007ff0d80f5433      Exceptions::new_exception(Thread*, Symbol*, char const*, Handle, Handle, Handle, Exceptions::ExceptionMsgToUtf8Mode) + 0x153
0x00007ff0d80f5548      Exceptions::new_exception(Thread*, Symbol*, char const*, Exceptions::ExceptionMsgToUtf8Mode) + 0x18
0x00007ff0d81506ed      GenerateOopMap::verify_error(char const*, ...) + 0xad
0x00007ff0d8155d7d      GenerateOopMap::interp_bb(BasicBlock*) + 0x18d
0x00007ff0d8156c3a      GenerateOopMap::do_interpretation() + 0x1aa
0x00007ff0d8156fba      GenerateOopMap::compute_map(Thread*) + 0x2aa
0x00007ff0d83f163a      OopMapForCacheEntry::compute_map(Thread*) + 0xca
0x00007ff0d83f26f0      OopMapCacheEntry::fill(methodHandle, int) + 0x110
0x00007ff0d83f2e1f      OopMapCache::lookup(methodHandle, int, InterpreterOopMap*) + 0x2bf
0x00007ff0d810382e      frame::oops_interpreted_do(OopClosure*, RegisterMap const*, bool) + 0x29e
0x00007ff0d853048a      JavaThread::oops_do(OopClosure*, CodeBlobClosure*) + 0x15a
0x00007ff0d8466aa5      ThreadRootsTask::do_it(GCTaskManager*, unsigned int) + 0x75
0x00007ff0d8139f0f      GCTaskThread::run() + 0x12f
0x00007ff0d83fda58      java_start(Thread*) + 0x108

楼主很明显在用product build的HotSpot。如果在用debug或fastdebug build的话，这三个assert的第一和第三个都应该会被触发，因为ParallelScavenge GC触发的时候VM肯定处于safepoint中，而且Heap_lock肯定被VMThread持有而不会被GCTaskThread所持有：

HeapWord* ParallelScavengeHeap::mem_allocate(
                                     size_t size,
                                     bool* gc_overhead_limit_was_exceeded) {
  assert(!SafepointSynchronize::is_at_safepoint(), "should not be at safepoint");
  assert(Thread::current() != (Thread*)VMThread::vm_thread(), "should not be in vm thread");
  assert(!Heap_lock->owned_by_self(), "this thread should not own the Heap_lock");

  ...
}

具体来说出问题的这个地方是：GC线程在扫描根集合时要扫描所有Java线程的栈的每个栈帧（stack frame）。当遇到一个以解释模式执行的Java方法的栈帧时，要通过GenerateOopMap类（的具体子类）来为该(method,bci)对生成一个OopMap来描述在该栈帧里什么地方有GC需要关心的指针（oop）。计算OopMap的过程会附带做一次字节码校验，如果通不过校验的话就会触发VerifyError。

但是楼主要想办法解决自己的问题的话还是从自己的代码入手比较好。

楼主的环境看起来是一个能用gdb的地方。如果楼主遇到的情况能稳定重现的话，在重现的时候用gdb attach上去，找到这个要抛异常的GC线程，然后到这俩frame的其中一个：

0x00007ff0d83f26f0      OopMapCacheEntry::fill(methodHandle, int) + 0x110
0x00007ff0d83f2e1f      OopMapCache::lookup(methodHandle, int, InterpreterOopMap*) + 0x2bf

去看那个methodHandle指向的methodOop是哪个方法。那个方法肯定被改写坏了（多半是楼主的字节码改写逻辑写得不尽正确）。

另外还有一个做法，那就是ASM有自带的字节码校验功能，可以单独在命令行上对现成的Class文件做校验，也可以在运行时作为库功能对生成的类做校验。请参考官网文档：
http://asm.ow2.org/doc/faq.html#Q4

引用

If your class is so "corrupted" that you cannot read it with a ClassReader, try to generate it by using a CheckClassAdapter in front of a ClassWriter:

  ClassWriter cw = new ClassWriter(true);
  ClassVisitor cv = new CheckClassAdapter(cw);
  // generate your class here:
  cv.visit(...);
  ...

You will probably get an exception which will indicate why your generated class is incorrect. For example, if you forget to call visit(...) (which can happen if you forget to call super.visit(...) in a class visitor), the generated class contains an invalid constant pool index, and ClassReader is unable to read it. If you generate your class by using a CheckClassAdapter, as above, you get an exception "Cannot visit member before visit has been called.", which shows that you forgot to call the visit method.

nijiaben 2014-08-14

RednaxelaFX 写道

这个一开始我也以为是VMThread持有Heap_lock，但是后面看了代码之后发现其实是触发gc的那个线程持有的，该线程会调用VMThread::execute，然后VMThread::execute调用VM_GC_Operation::doit_prologue，hotspot之所以这么做，是为了性能考虑，若没有持有该锁，那其他线程也可能因内存不够而触发GC，这样VMThread的操作队列充斥大量相同vmop，影响性能。

这应该是HotSpot的一个设计缺陷，不应该在GC(需要获取Heap_lock)处理过程中再进行堆内存分配

RednaxelaFX 2014-08-14

nijiaben 写道

RednaxelaFX 写道

嗯对对，标红的那句我确实是说错了。Heap_lock应该被触发GC的那个Java线程所持有，而不是被GCTaskThread或VMThread所持有。ParallelScavengeHeap::mem_allocate()只应该在Java线程上被调用，而此处出错的地方则是在GC线程上被调用了。

RednaxelaFX 2014-08-14

据说楼主已经发现是字节码改写错了，一个istore写成了astore。
坐等楼主来更新细节了

diecui1202 2014-08-15

仔细看了各位大牛的分析，受益非浅；

不过R大说出了问题的本质：

引用

但是楼主要想办法解决自己的问题的话还是从自己的代码入手比较好。

找公司专门做这个的同学一起帮忙定位，通过修改jvm的代码，在gc时加一些log，build后运行，发现每次hang住时，日志打印最后都是：

Gang worker#3 (Parallel GC Threads)--(9)--method.name == com.taobao.hotcode2.code.ClassTransformer.transformNewLoadClass(Ljava/lang/String;Ljava/lang/ClassLoader;[B)[B--
lim_bci == 18--bb[1]._bci == 18
Gang worker#3 (Parallel GC Threads)--(1)--method.name == java.lang.ClassLoader.defineClass(Ljava/lang/String;[BIILjava/security/ProtectionDomain;)Ljava/lang/Class;
aload_0
invokestatic
aload_1
aload_0
aload_2
invokestatic
astore_2
Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 0--is_address == 0--is_reference == 1
aload_2
arraylength
astore
Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 1--is_address == 0--is_reference == 0

既然这块有问题，我就检查了我修改的java/lang/ClassLoader的代码：

    @Override
    public MethodVisitor visitMethod(int access, String name, String desc, String signature, String[] exceptions) {
        MethodVisitor mv = super.visitMethod(access, name, desc, signature, exceptions);

        if (name.equals("defineClass")
            && desc.equals(Type.getMethodDescriptor(Type.getType(Class.class), Type.getType(String.class),
                                                    Type.getType(byte[].class), Type.INT_TYPE, Type.INT_TYPE,
                                                    Type.getType(ProtectionDomain.class)))) {
            return new MethodVisitor(Opcodes.ASM4, mv) {

                @Override
                public void visitCode() {
                    super.visitCode();
                    mv.visitVarInsn(Opcodes.ALOAD, 0);
                    mv.visitMethodInsn(Opcodes.INVOKESTATIC, Type.getInternalName(CRMManager.class),
                                       "registerClassLoader", "(Ljava/lang/ClassLoader;)V");
                    mv.visitVarInsn(Opcodes.ALOAD, 1);
                    mv.visitVarInsn(Opcodes.ALOAD, 0);
                    mv.visitVarInsn(Opcodes.ALOAD, 2);
                    mv.visitMethodInsn(Opcodes.INVOKESTATIC, Type.getInternalName(ClassTransformer.class),
                                       "transformNewLoadClass", "(Ljava/lang/String;Ljava/lang/ClassLoader;[B)[B");
                    mv.visitVarInsn(Opcodes.ASTORE, 2);
                    mv.visitVarInsn(Opcodes.ALOAD, 2);
                    mv.visitInsn(Opcodes.ARRAYLENGTH);
                    mv.visitVarInsn(Opcodes.ASTORE, 4);
                }
            };
        }

        return mv;
    }

果然，最后一行代码写错了，ARRAYLENGTH 返回的是 int，因此需要使用 ISTORE，正确的代码应该是：

mv.visitVarInsn(Opcodes.ISTORE, 4);

看来还是写代码时不够细心，一不小心，踩了这么大一个坑，让大家笑话了

后面我会把jvm那段代码贴上来，大家可以参考一下。

diecui1202 2014-08-15

在gc的过程中，会多线程处理每个java线程的frame，针对每个frame都会查这个frame对应的OopMap，以确定该
frame上oop的具体的偏移量，gc需要把oop作为root，来进行遍历，如果某个frame没有关联的OopMap，那么gc时会生成
OopMap，这就会对每个字节码进行遍历，校验，确定Oop的位置，由于hotCode改变了某些方法的字节码，需要重新生成OopMap，
下面是jvm看到的defineClass的字节码，针对astore的处理我加了一些log：

Gang worker#3 (Parallel GC Threads)--(1)--method.name == java.lang.ClassLoader.defineClass(Ljava/lan
g/String;[BIILjava/security/ProtectionDomain;)Ljava/lang/Class;
aload_0
invokestatic
aload_1
aload_0
aload_2
invokestatic
astore_2
Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 0--is_address == 0--is_r
eference == 1
aload_2
arraylength
istore（这个地方之前是astore，所以对要求是个）
aload_0
aload_1
aload
invokespecial
astore
Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 0--is_address == 0--is_r
eference == 1
aconst_null
astore
Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 0--is_address == 0--is_r
eference == 1
aload_0
aload
invokespecial
astore
Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 0--is_address == 0--is_r
eference == 1
aload_0
aload_1
aload_2
iload_3
iload
aload
aload
invokespecial
astore
Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 0--is_address == 0--is_r
eference == 1
_goto

18行的istore之前是astore，jvm输出的日志是：

Gang worker#3 (Parallel GC Threads)--do_astore--is_uninit == 0--is_value == 1--is_address == 0--is_r
eference == 0

这就表示，jvm内部存储的栈上的是一个primitive类型的数值，但是由于astore要求的是一个java对象的引用或者pc指针，
导致校验失败，jvm抛了异常verify_error，jvm内部的处理函数是下面这个，用于对jvm的astore进行校验。

void GenerateOopMap::do_astore(int idx) {
  CellTypeState r_or_p = pop();
  gclog_or_tty->print_cr("%s--do_astore--is_uninit == %d--is_value == %d--is_address == %d--is_refer
ence == %d", Thread::current()->name(), r_or_p.is_uninit(), r_or_p.is_value(), r_or_p.is_address() ,
r_or_p.is_reference());
  gclog_or_tty->flush();
  if (!r_or_p.is_address() && !r_or_p.is_reference()) {
    // We actually expected ref or pc, but we only report that we expected a ref. It does not
    // really matter (at least for now)
    verify_error("wrong type on stack (found: %c, expected: {pr})", r_or_p.to_char());
    return;
  } 
  set_var(idx, r_or_p); 
}

发表回复

>>返回群组首页

[讨论] JVM hang住，所有线程BLOCKED（包括GC线程），Eden区100%

相关讨论

相关资源推荐