[讨论] 这个jrockit dump怎么分析呀？
 |
haoweishow
2011-10-25
开发环境不定期的出现应用crash，取了其中一次dump文件，还请大家帮忙分析分析。 ===== BEGIN DUMP =============================================================
JRockit dump produced after 1 days, 04:04:57 on Sat Mar 27 19:05:04 2010
Additional information is available in:
/wls/wldomains/esb_provider/jrockit.12438.dump
No snapshot file (core dump) will be created because core dumps have been
disabled. To enable core dumping, try "ulimit -c unlimited"
before starting JRockit again.
If you see this dump, please open a support case with BEA and
supply as much information as you can on your system setup and
the program you were running. You can also search for solutions
to your problem at http://forums.bea.com in
the forum jrockit.developer.interest.general.
Error Message: Illegal memory access. [54]
Signal info : si_signo=11, si_code=2 si_addr=0x40
Version : BEA JRockit(R) R27.3.1-1_CR344434-89345-1.5.0_11-20070925-1628-linux-ia32
GC Strategy : Mode: throughput. Currently using strategy: genparpar
GC Status : OC is not running. Last finished OC was OC#3010.
: YC is currently running. This is YC#959.
OC History : Strategy singleparpar was used for OC#2993 to OC#2997.
: Strategy genparpar was used for OC#2998 to OC#2999.
: Strategy singleparpar was used for OC#3000 to OC#3003.
: Strategy genparpar was used for OC#3004 to OC#3006.
: Strategy singleparpar was used for OC#3007 to OC#3010.
YC History : Ran 0 YCs before OC#3006.
: Ran 0 YCs before OC#3007.
: Ran 0 YCs before OC#3008.
: Ran 0 YCs before OC#3009.
: Ran 0 YCs before OC#3010.
: Started 2 YCs since last OC.
Heap : 0x8100000 - 0x18100000
Compaction : 0x15100000 - 0x16100000
NurseryList : 0xdf01150 - 0x12edbd50
KeepArea : 0x107161f0 - 0x11af53a8
NurseryMarker: [ 0x107161f0, 0x11af53a8 ]
CompRefs : References are 32-bit.
CPU : Intel Core SSE SSE2 SSE3 SSSE3 Core EM64T
Number CPUs : 1
Tot Phys Mem : 3988893696 (3804 MB)
OS version : Red Hat Enterprise Linux ES release 4 (Nahant Update 7)
Linux version 2.6.9-78.ELsmp (brewbuilder@hs20-bc2-3.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-10)) #1 SMP Wed Jul 9 15:39:47 EDT 2008 (i686)
Thread System: NPTL
State : JVM is running
Command Line : -Xms256m -Xmx512m -Xverify:none -da -Dplatform.home=/wls/wls81/bea10/wlserver_10.0 -Dwls.home=/wls/wls81/bea10/wlserver_10.0/server -Dweblogic.home=/wls/wls81/bea10/wlserver_10.0/server -Dwli.home=/wls/wls81/bea10/wlserver_10.0/integration -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/wls/wls81/bea10/patch_wls1001/profiles/default/sysext_manifest_classpath -Dweblogic.Name=AdminServer -Djava.security.policy=/wls/wls81/bea10/wlserver_10.0/server/lib/weblogic.policy -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=27005,server=y,suspend=n -Dfile.encoding=GBK -Dsun.java.launcher=SUN_STANDARD weblogic.Server
java.home : /wls/wls81/bea10/jrockit_150_11/jre
j.class.path : :/wls/wls81/bea10/patch_wls1001/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/wls/wls81/bea10/jrockit_150_11/lib/tools.jar:/wls/wls81/bea10/wlserver_10.0/server/lib/weblogic_sp.jar:/wls/wls81/bea10/wlserver_10.0/server/lib/weblogic.jar:/wls/wls81/bea10/modules/features/weblogic.server.modules_10.0.1.0.jar:/wls/wls81/bea10/modules/features/com.bea.cie.common-plugin.launch_2.1.2.0.jar:/wls/wls81/bea10/wlserver_10.0/server/lib/webservices.jar:/wls/wls81/bea10/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/wls/wls81/bea10/modules/net.sf.antcontrib_1.0b2.0/lib/ant-contrib.jar::/wls/wls81/bea10/wlserver_10.0/common/eval/pointbase/lib/pbclient51.jar:/wls/wls81/bea10/wlserver_10.0/server/lib/xqrl.jar::
j.lib.path : /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/jrockit:/wls/wls81/bea10/jrockit_150_11/jre/lib/i386:/wls/wls81/bea10/jrockit_150_11/jre/../lib/i386::/opt/CA/SharedComponents/lib::/wls/wls81/bea10/wlserver_10.0/server/native/linux/i686:/wls/wls81/bea10/wlserver_10.0/server/native/linux/i686/oci920_8
JAVA_HOME : /wls/wls81/bea10/jrockit_150_11
_JAVA_OPTIONS: <not set>
LD_LIBRARY_PATH: /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/jrockit:/wls/wls81/bea10/jrockit_150_11/jre/lib/i386:/wls/wls81/bea10/jrockit_150_11/jre/../lib/i386::/opt/CA/SharedComponents/lib::/wls/wls81/bea10/wlserver_10.0/server/native/linux/i686:/wls/wls81/bea10/wlserver_10.0/server/native/linux/i686/oci920_8
LD_ASSUME_KERNEL: <not set>
C Heap : Good; no memory allocations have failed
StackOverFlow: 0 StackOverFlowErrors have occured
OutOfMemory : 0 OutOfMemoryErrors have occured
Registers (from ThreadContext: 0xaec5deac / OS context: 0xaec5dfa8):
eax = 0001a6d8 ecx = 0004bd08 edx = 00000000 ebx = 00000000
esp = aec5e298 ebp = aec5e2d0 esi = 12e90048 edi = 131e3658
es = 0000007b cs = 00000073 ss = 0000007b ds = 0000007b
fs = 00000000 gs = 00000033
eip = b7ed89f3 eflags = 00000283
Stack:
(* marks the word pointed to by the stack pointer)
aec5e298: 00000000* 00cf1830 aef13c68 00000020 017587ca 00000001
aec5e2b0: 01010101 00000000 00cefff4 aef13c68 12e90048 12e90048
aec5e2c8: 12e90048 131e3658 aec5e300 b7ed8f87 00000000 131e3638
aec5e2e0: aec5e300 b7ed6ea3 00000000 00000000 00000000 131e3664
Code:
(* marks the word pointed to by the instruction pointer)
b7ed89c0: 00000344 f8ba24a1 008b65b7 00022805 ec458900 fb1b28a1
b7ed89d8: 0fc085b7 0001e685 f05d8b00 fe1f3ca1 31138bb7 04503bdb
b7ed89f0: 83c3940f* 0f03407a 0001de84 44428b00 89f04d8b 41b60fc2
b7ed8a08: 74c0a807 08c28303 00e845c7 8b000000 f770840d 85d789b7
Loaded modules:
(* denotes the module causing the exception)
08048000-08056fd3 /wls/wls81/bea10/jrockit_150_11/bin/java
00d33000-00d4085b /lib/tls/libpthread.so.0
00cfc000-00d1cc8f /lib/tls/libm.so.6
00cf6000-00cf7967 /lib/libdl.so.2
00bc5000-00ced590 /lib/tls/libc.so.6
00bab000-00bc044b /lib/ld-linux.so.2
b7cc0000-b7f73447 */wls/wls81/bea10/jrockit_150_11/jre/lib/i386/jrockit/libjvm.so
00d82000-00d89a76 /lib/tls/librt.so.1
b7ca4000-b7caca57 /lib/libnss_files.so.2
b7c0f000-b7c40cd9 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libjdwp.so
b7c03000-b7c0d7db /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libverify.so
b7be0000-b7c00217 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libjava.so
00b8b000-00b9d66f /lib/libnsl.so.1
b7a79000-b7a7ef13 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/native_threads/libhpi.so
b75fa000-b76084c4 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libzip.so
b752d000-b7530404 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libdt_socket.so
b6f92000-b6fa3103 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libnet.so
b6f43000-b6f46313 /lib/libnss_dns.so.2
00d47000-00d55fef /lib/libresolv.so.2
b6909000-b690f1e7 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libnio.so
b60c7000-b60cc246 /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libmanagement.so
b609b000-b60a3fcb /wls/wls81/bea10/jrockit_150_11/jre/lib/i386/libjmapi.so
b5f54000-b5f55de4 /wls/wls81/bea10/wlserver_10.0/server/native/linux/i686/libwlfileio2.so
Scan Dump Helper:
Processing references from a Card Table Part workchunk, with data: 0x263.
No objIter registered, this is probably a root reference.
"pool-5-thread-95" id=212 idx=0x32c tid=18920 lastJavaFrame=0xaec5e670
Stack 0: start=0xaec3e000, end=0xaec60000, guards=0xaec43000 (ok), forbidden=0xaec41000
Thread Stack Trace:
at ycProcessNormalObject+67()@0xb7ed89f3
at ycProcessReference+119()@0xb7ed8f87
at ycProcessWorkChunk+936()@0xb7ed99e8
at ycWorkerProcessRoots+202()@0xb7eda07a
at ycGc+1055()@0xb7eda4bf
at mmGetTLAMemoryFromNurseryDefault+344()@0xb7e4e348
at mmGetMoreTLAMemoryLoop+103()@0xb7ebde77
at mmAllocateThreadLocalAreaMemory+428()@0xb7ebe72c
at RJNI_jrockit_vm_Allocator_nativeGetNewTLA+83()@0xb7e87e63
-- Java stack --
at jrockit/vm/Allocator.nativeGetNewTLA()V(Native Method)
at jrockit/vm/Allocator.getNewTLA(Allocator.java:435)
at jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:463)
at jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:567)
at jrockit/vm/Allocator.allocObjectOrArray(Allocator.java:600)
at jrockit/vm/Allocator.allocArray(Allocator.java:392)
at java/lang/AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
at java/lang/AbstractStringBuilder.append(AbstractStringBuilder.java:393)
at java/lang/StringBuilder.append(StringBuilder.java:120)
at java/lang/StackTraceElement.toString(StackTraceElement.java:157)
at java/lang/String.valueOf(String.java:2615)
at java/lang/StringBuilder.append(StringBuilder.java:116)
at java/lang/Throwable.printStackTrace(Throwable.java:513)
^-- Holding lock: java/io/PrintWriter@0x92e2a88[thin lock]
at org/apache/log4j/spi/LocationInfo.<init>(LocationInfo.java:105)
^-- Holding lock: java/io/StringWriter@0x92e2a70[thin lock]
at org/apache/log4j/spi/LoggingEvent.getLocationInformation(LoggingEvent.java:191)
at org/apache/log4j/helpers/PatternParser$LocationPatternConverter.convert(PatternParser.java:483)
at org/apache/log4j/helpers/PatternConverter.format(PatternConverter.java:64)
at org/apache/log4j/PatternLayout.format(PatternLayout.java:503)
at org/apache/log4j/WriterAppender.subAppend(WriterAppender.java:301)
at org/apache/log4j/RollingFileAppender.subAppend(RollingFileAppender.java:234)
at org/apache/log4j/WriterAppender.append(WriterAppender.java:159)
at org/apache/log4j/AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
^-- Holding lock: org/apache/log4j/RollingFileAppender@0xd42d780[thin lock]
at org/apache/log4j/helpers/AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
at org/apache/log4j/Category.callAppenders(Category.java:203)
^-- Holding lock: org/apache/log4j/spi/RootLogger@0x97177e8[thin lock]
at org/apache/log4j/Category.forcedLog(Category.java:388)
at org/apache/log4j/Category.log(Category.java:853)
at org/apache/commons/logging/impl/Log4JLogger.debug(Log4JLogger.java:171)
at com/xx/bkesb/provider/adapter/EsbAs400Adapter.readMessage(EsbAs400Adapter.java:113)
at com/xx/bkesb/provider/adapter/EsbAs400Adapter.connect(EsbAs400Adapter.java:98)
at com/xx/bkesb/provider/adapter/EsbAs400Adapter.getPort(EsbAs400Adapter.java:60)
at com/xx/bkesb/provider/adapter/EsbAs400Adapter.doComm(EsbAs400Adapter.java:218)
at com/xx/bkesb/provider/service/EsbStandardService.doComm(EsbStandardService.java:609)
at com/xx/bkesb/provider/service/EsbStandardService.execute(EsbStandardService.java:307)
at com/xx/bkesb/provider/service/SimpleServiceImplx.execute(SimpleServiceImplx.java:239)
at com/tt/integrator/channel/platform/service/impl/ServiceExecutorImpl.execute(ServiceExecutorImpl.java:78)
at com/xx/bkesb/provider/dispatcher/ServiceDispatcherImpl.doService(ServiceDispatcherImpl.java:390)
at com/xx/bkesb/provider/dispatcher/ServiceDispatcherImpl.dispatch(ServiceDispatcherImpl.java:213)
at com/xx/bkesb/provider/dispatcher/ProviderExtendService.execute(ProviderExtendService.java:73)
at com/xx/bkesb/service/impl/EsbBusinessServiceImpl.doService(EsbBusinessServiceImpl.java:49)
at sun/reflect/GeneratedMethodAccessor233.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java/lang/reflect/Method.invoke(Method.java:585)
at org/apache/tuscany/sca/implementation/java/invocation/JavaImplementationInvoker.invoke(JavaImplementationInvoker.java:132)
at org/apache/tuscany/sca/binding/sca/impl/SCABindingInvoker.invoke(SCABindingInvoker.java:61)
at org/apache/tuscany/sca/core/invocation/JDKInvocationHandler.invoke(JDKInvocationHandler.java:287)
at org/apache/tuscany/sca/core/invocation/JDKInvocationHandler.invoke(JDKInvocationHandler.java:154)
at org/apache/tuscany/sca/core/invocation/CglibProxyFactory$CglibMethodInterceptor.intercept(CglibProxyFactory.java:145)
at com/xx/bkesb/service/impl/EsbBusinessServiceImpl$$EnhancerByCGLIB$$87aaed2c.doService(Lcom/xx/bkesb/provider/conv/sdo/EsbSdoForCd;)Lcom/xx/bkesb/provider/conv/sdo/EsbSdoForCd;(<generated>)
at sun/reflect/GeneratedMethodAccessor849.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java/lang/reflect/Method.invoke(Method.java:585)
at com/dx/esb/service/impl/BusinessServiceImpl.invokeBusinessService(BusinessServiceImpl.java:133)
at com/dx/esb/service/impl/BusinessServiceImpl.invoke(BusinessServiceImpl.java:44)
at sun/reflect/GeneratedMethodAccessor199.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java/lang/reflect/Method.invoke(Method.java:585)
at org/apache/tuscany/sca/implementation/java/invocation/JavaImplementationInvoker.invoke(JavaImplementationInvoker.java:132)
at org/apache/tuscany/sca/binding/sca/impl/SCABindingInvoker.invoke(SCABindingInvoker.java:61)
at org/apache/tuscany/sca/core/invocation/JDKInvocationHandler.invoke(JDKInvocationHandler.java:287)
at org/apache/tuscany/sca/core/invocation/JDKInvocationHandler.invoke(JDKInvocationHandler.java:154)
at $Proxy58.invoke(Lcom/dx/esb/nm/NormalizedMessage;)Lcom/dx/esb/nm/NormalizedMessage;(Unknown Source)
at com/dx/esb/service/impl/DefaultInvokeService.invokeBusinessService(DefaultInvokeService.java:164)
at com/dx/esb/service/impl/DefaultInvokeService.invokeService(DefaultInvokeService.java:76)
at com/dx/esb/adapter/data/DataOutAdapterImpl.process(DataOutAdapterImpl.java:117)
at com/dx/esb/adapter/data/DataOutAdapterImpl.execute(DataOutAdapterImpl.java:61)
at com/xx/bkesb/provider/dispatcher/EsbProtocalAdapter.invokeDataOutAdapter(EsbProtocalAdapter.java:175)
at com/xx/bkesb/provider/dispatcher/EsbProtocalAdapter.dispatch(EsbProtocalAdapter.java:91)
at com/tt/impls/esb/client/ESBClient.reply(ESBClient.java:705)
at com/tt/impls/esb/client/ESBClient.access$500(ESBClient.java:66)
at com/tt/impls/esb/client/ESBClient$2.run(ESBClient.java:628)
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
Additional information is available in:
/wls/wldomains/esb_provider/jrockit.12438.dump
No snapshot file (core dump) will be created because core dumps have been
disabled. To enable core dumping, try "ulimit -c unlimited"
before starting JRockit again.
If you see this dump, please open a support case with BEA and
supply as much information as you can on your system setup and
the program you were running. You can also search for solutions
to your problem at http://forums.bea.com in
the forum jrockit.developer.interest.general.
Extended, platform specific info:
libc release: 2.3.4-stable
Elf headers:
libc ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00bd9ee0 PHOFF: 00000034 SHOFF: 00177d7c EF: 0x0 HS: 52 PS: 32 PHN; 10 SS: 40 SHN: 67 STIDX: 66
libpthread ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00d37850 PHOFF: 00000034 SHOFF: 00019f00 EF: 0x0 HS: 52 PS: 32 PHN; 9 SS: 40 SHN: 39 STIDX: 38
libjvm ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 0005a080 PHOFF: 00000034 SHOFF: 002f599c EF: 0x0 HS: 52 PS: 32 PHN; 4 SS: 40 SHN: 19 STIDX: 16
===== END DUMP ===============================================================
其中这部分内容可能是倒是crash的原因，但不确定，也没有理论根据。 是不是在扩展内存的时候出现错误了啊？但是也没有出现oom的异常。 -- Java stack --
at jrockit/vm/Allocator.nativeGetNewTLA()V(Native Method)
at jrockit/vm/Allocator.getNewTLA(Allocator.java:435)
at jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:463)
at jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:567)
at jrockit/vm/Allocator.allocObjectOrArray(Allocator.java:600)
at jrockit/vm/Allocator.allocArray(Allocator.java:392)
at java/lang/AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
|
 |
RednaxelaFX
2011-10-26
好老的JRockit和Linux…还有好老的日志
这个恐怕到Oracle的论坛去问会更有用些。 这个crash log里能看到的有用信息并不多。可以知道的是，事发时VM正在ycProcessNormalObject()函数内，在做young GC，而且发生了空指针异常。事发地的指令是这样的： 0xb7ed89f3: 837a4003 cmp dword [edx+0x40], 0x3 而当时edx寄存器的值是0x00000000（也就是空指针），所以这条指令访问到了0x00000040地址，引发了访问异常。 光看这个日志看不出来空指针是什么地方造成的，不知道源头是哪里。所以说可能去Oracle那边的专门论坛问会更有用。在那边很有可能得到的建议是升级你的JRockit… |
|
haoweishow
2011-10-26
谢谢 RednaxelaFX 的回复。
jrockit和linux是比较老，客户的开发环境。 日志不老，是因为系统日期切换了。 请问 RednaxelaFX 下面这个指令是怎么得出来的呢？ 0xb7ed89f3: 837a4003 cmp dword [edx+0x40], 0x3 |
|
RednaxelaFX
2011-10-26
首先有
eip = b7ed89f3 知道了指令指针的值是0xb7ed89f3。 然后是 Code: (* marks the word pointed to by the instruction pointer) b7ed89c0: 00000344 f8ba24a1 008b65b7 00022805 ec458900 fb1b28a1 b7ed89d8: 0fc085b7 0001e685 f05d8b00 fe1f3ca1 31138bb7 04503bdb b7ed89f0: 83c3940f* 0f03407a 0001de84 44428b00 89f04d8b 41b60fc2 b7ed8a08: 74c0a807 08c28303 00e845c7 8b000000 f770840d 85d789b7 这个是eip所指向的内存区域前后一段的内存dump。它是把4字节为单位的数据当作整型以16进制的方式显示出来的。这样显示比较紧凑。因为x86是little-endian的，所以这里数据每4个字节为单元内部的顺序跟实际在内存里的顺序是反的。 既然我们关注的是指令，实际上我们需要的是单个单个字节按内存里原本的顺序排布。所以得把上面的数据调整到适合的形式再分析。例如说从b7ed89f0那行开始看，实际从0xb7ed89f0地址开始的8个字节的数据在内存里应该是： 0f 94 c3 83 7a 40 03 0f 可以看到这8个字节是如何从83c3940f 0f03407a转换而来的么？4字节内部顺序颠倒就好了。 那么根据eip的值就知道事发地的指令是下面这串字节开始的： 83 7a 40 03 0f ... 然后把这串字节随便扔给一个x86的反汇编器就能得到结果了。 |
 |
ttttstttt
2011-10-27
com/xx/bkesb/provider/adapter/EsbAs400Adapter.readMessage( )
函数里有个log4j的日志，检查下这个日志的内容、长度是不是有问题？ |
