在我们的hadoop集群上，jvm的版本是6u32，内核版本也是32的内核，64位的。当我们malloc大内存之后，再调用libhdfs的hdfsConnect（初始化JNI环境，调用hdfs的Java方法），会出现概率性的core掉，且可以稳定复现。

复现的场景比较奇怪，当申请1G、4G、8G空间的时候没有问题，申请2G、3G、7G、10G内存的时候，就会core掉。  物理内存是64G的，我们怀疑是jvm的内存管理和内核的内存管理有冲突，因为不了解jvm的内部实现，有点无从下手，请大牛提供点思路，谢谢~~

是在task worker上出的问题么？
请问core的时候有HotSpot VM的crash log不？默认是形如 hs_err_pid<pid>.log 的文件名的。有的话最好能找个地方发出来看看。
没的话看看是不是worker失败后对应的目录被删了，是的话请在JVM的启动参数里加上 -XX:ErrorFile=<path> 来指定一个具体路径放crash log，这个path可以是不包含空格的字符串，里面可以含有%p，会被替换为实际的pid。
如果这样还是没HotSpot VM的crash log的话那就…从别的地方着手看看。

P.S. 请问是阿里系的同事么？是的话可以试试在旺旺上加同事“坤谷”询问，有个专门解答这类问题的JVM讨论群的。

crash log有的，我来找一下~~~不是阿里系，百度的。我度没有对jvm有深入研究的人，只能来请教大神了 

呵呵，欢迎多交流～
等crash log有了再分析一下看看。

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f941cfda865, pid=12107, tid=140284911437568
#
# JRE version: 6.0_32-b05
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.7-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J  java.lang.String.hashCode()I
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000000509800):  JavaThread "main" [_thread_in_Java, id=12107, stack(0x00007fff23031000,0x00007fff23131000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00007f93a1338000

Registers:
RAX=0x00000000c7575903, RBX=0x000000040c2b9b60, RCX=0x000000000000001a, RDX=0x00000000824d23e4
RSP=0x00007fff2312e6f8, RBP=0x00007fff2312e750, RSI=0x000000000000006e, RDI=0x000000000000001a
R8 =0x0000000000000069, R9 =0x0000000000000000, R10=0x0000000000000067, R11=0x000000040c2bc888
R12=0x0000000000000000, R13=0x00007fff2312e708, R14=0x0000000000000000, R15=0x0000000000509800
RIP=0x00007f941cfda865, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007fff2312e6f8)
0x00007fff2312e6f8:   00007f941cf8096e 00007f941cf8096e
0x00007fff2312e708:   000000040c2bc888 00007fff2312e710
0x00007fff2312e718:   00000007faebe7f2 00007fff2312e780
0x00007fff2312e728:   00000007faec0ee0 0000000000000000
0x00007fff2312e738:   00000007faebe848 00007fff2312e708
0x00007fff2312e748:   00007fff2312e778 00007fff2312e7d8
0x00007fff2312e758:   00007f941cf80929 0000000000000000
0x00007fff2312e768:   0000000000000000 0000000000000000
0x00007fff2312e778:   000000040c2bc888 000000040c23fc20
0x00007fff2312e788:   0000000000000001 000000040c23fc20
0x00007fff2312e798:   00007fff2312e788 00000007fae2a6b4
0x00007fff2312e7a8:   00007fff2312e810 00000007faeaac50
0x00007fff2312e7b8:   0000000000000000 00000007fae2a728
0x00007fff2312e7c8:   00007fff2312e778 00007fff2312e808
0x00007fff2312e7d8:   00007fff2312e858 00007f941cf80929
0x00007fff2312e7e8:   0000000000000000 0000000000000000
0x00007fff2312e7f8:   000000040c23fc20 0000000000000000
0x00007fff2312e808:   000000040c2bc888 000000040c23fa80
0x00007fff2312e818:   00007fff2312e818 00000007fafce160
0x00007fff2312e828:   00007fff2312e8c8 00000007fafd14d0
0x00007fff2312e838:   0000000000000000 00000007fafce2e8
0x00007fff2312e848:   00007fff2312e808 00007fff2312e8b0
0x00007fff2312e858:   00007fff2312e910 00007f941cf80929
0x00007fff2312e868:   0000000000000000 0000000000000000
0x00007fff2312e878:   0000000000000000 0000000000000000
0x00007fff2312e888:   0000000000000000 0000000000000000
0x00007fff2312e898:   000000040c2bc888 000000040c230900
0x00007fff2312e8a8:   000000000000001a 0000000000000001
0x00007fff2312e8b8:   000000040c2bc868 000000040c2b9b40
0x00007fff2312e8c8:   000000040c23fa80 00007fff2312e8d0
0x00007fff2312e8d8:   00000007fafced04 00007fff2312e938
0x00007fff2312e8e8:   00000007fafd14d0 0000000000000000

Instructions: (pc=0x00007f941cfda865)
0x00007f941cfda845:   d2 c1 e2 05 41 2b d2 03 d6 3b c8 7c 9e 3b cf 0f
0x00007f941cfda855:   8c 76 ff ff ff 8b c2 41 89 43 18 48 83 c4 20 5d
0x00007f941cfda865:   85 05 95 d7 35 84 c3 33 c0 eb f0 be 86 ff ff ff
0x00007f941cfda875:   4c 89 1c 24 44 89 4c 24 08 89 7c 24 0c 90 e8 18

Register to memory mapping:

RAX=0x00000000c7575903 is an unknown value
RBX=0x000000040c2b9b60 is an oop
[C
- klass: {type array char}
- length: 37
RCX=0x000000000000001a is an unknown value
RDX=0x00000000824d23e4 is an unknown value
RSP=0x00007fff2312e6f8 is pointing into the stack for thread: 0x0000000000509800
RBP=0x00007fff2312e750 is pointing into the stack for thread: 0x0000000000509800
RSI=0x000000000000006e is an unknown value
RDI=0x000000000000001a is an unknown value
R8 =0x0000000000000069 is an unknown value
R9 =0x0000000000000000 is an unknown value
R10=0x0000000000000067 is an unknown value
R11=0x000000040c2bc888 is an oop
java.lang.String
- klass: 'java/lang/String'
R12=0x0000000000000000 is an unknown value
R13=0x00007fff2312e708 is pointing into the stack for thread: 0x0000000000509800
R14=0x0000000000000000 is an unknown value
R15=0x0000000000509800 is a thread


Stack: [0x00007fff23031000,0x00007fff23131000],  sp=0x00007fff2312e6f8,  free space=1013k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J  java.lang.String.hashCode()I
j  java.lang.ClassLoader.getPackage(Ljava/lang/String;)Ljava/lang/Package;+12
j  java.net.URLClassLoader.defineClass(Ljava/lang/String;Lsun/misc/Resource;Z)Ljava/lang/Class;+32
j  java.net.URLClassLoader.access$000(Ljava/net/URLClassLoader;Ljava/lang/String;Lsun/misc/Resource;Z)Ljava/lang/Class;+4
j  java.net.URLClassLoader$1.run()Ljava/lang/Object;+44
v  ~StubRoutines::call_stub
V  [libjvm.so+0x4dff90]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1e0
V  [libjvm.so+0x710259]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x19
V  [libjvm.so+0x4dfda5]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x25
V  [libjvm.so+0x551d09]  JVM_DoPrivileged+0x439
C  [libjava.so+0x10ebf]  Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2Ljava_security_AccessControlContext_2+0xf
j  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+47
j  sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+41
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
v  ~StubRoutines::call_stub
V  [libjvm.so+0x4dff90]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1e0
V  [libjvm.so+0x710259]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x19
V  [libjvm.so+0x4df8d6]  JavaCalls::call_virtual(JavaValue*, KlassHandle, symbolHandle, symbolHandle, JavaCallArguments*, Thread*)+0x126
V  [libjvm.so+0x4df9e3]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, symbolHandle, symbolHandle, Handle, Thread*)+0x53
V  [libjvm.so+0x7ec73a]  SystemDictionary::load_instance_class(symbolHandle, Handle, Thread*)+0x38a
V  [libjvm.so+0x7ea485]  SystemDictionary::resolve_instance_class_or_null(symbolHandle, Handle, Handle, Thread*)+0x575
V  [libjvm.so+0x7e9211]  SystemDictionary::resolve_or_null(symbolHandle, Handle, Handle, Thread*)+0x81
V  [libjvm.so+0x7e8e24]  SystemDictionary::resolve_or_fail(symbolHandle, Handle, Handle, bool, Thread*)+0x34
V  [libjvm.so+0x3dd725]  constantPoolOopDesc::klass_at_impl(constantPoolHandle, int, Thread*)+0x275
V  [libjvm.so+0x3dedde]  constantPoolOopDesc::klass_ref_at(int, Thread*)+0x4e
V  [libjvm.so+0x651cab]  LinkResolver::resolve_klass(KlassHandle&, constantPoolHandle, int, Thread*)+0x2b
V  [libjvm.so+0x6562a5]  LinkResolver::resolve_pool(KlassHandle&, symbolHandle&, symbolHandle&, KlassHandle&, constantPoolHandle, int, Thread*)+0x35
V  [libjvm.so+0x6564ac]  LinkResolver::resolve_invokestatic(CallInfo&, constantPoolHandle, int, Thread*)+0x5c
V  [libjvm.so+0x65622b]  LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle, int, Bytecodes::Code, Thread*)+0x3b
V  [libjvm.so+0x4d84c2]  InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code)+0x202
j  org.apache.hadoop.conf.Configuration.<clinit>()V+3
v  ~StubRoutines::call_stub
V  [libjvm.so+0x4dff90]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1e0
V  [libjvm.so+0x710259]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x19
V  [libjvm.so+0x4dfda5]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x25
V  [libjvm.so+0x4af9a5]  instanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0xd5
V  [libjvm.so+0x4adf28]  instanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x2f8
V  [libjvm.so+0x4ad2ea]  instanceKlass::initialize(Thread*)+0x6a
V  [libjvm.so+0x56b279]  find_class_from_class_loader(JNIEnv_*, symbolHandle, unsigned char, Handle, Handle, unsigned char, Thread*)+0xc9
V  [libjvm.so+0x4ef2ac]  unsigned+0x35c

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f9416803800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=12145, stack(0x0000000040f44000,0x0000000041045000)]
  0x00007f9416801000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=12144, stack(0x0000000040e43000,0x0000000040f44000)]
  0x000000000058b800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=12143, stack(0x0000000040a15000,0x0000000040b16000)]
  0x0000000000589800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=12142, stack(0x00000000400c5000,0x00000000401c6000)]
  0x000000000056d000 JavaThread "Finalizer" daemon [_thread_blocked, id=12141, stack(0x00000000413a3000,0x00000000414a4000)]
  0x0000000000565800 JavaThread "Reference Handler" daemon [_thread_blocked, id=12140, stack(0x0000000041fba000,0x00000000420bb000)]
=>0x0000000000509800 JavaThread "main" [_thread_in_Java, id=12107, stack(0x00007fff23031000,0x00007fff23131000)]

Other Threads:
  0x000000000055f000 VMThread [stack: 0x0000000041ac9000,0x0000000041bca000] [id=12139]
  0x00007f9416817800 WatcherThread [stack: 0x00000000411de000,0x00000000412df000] [id=12146]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation   total 309312K, used 5500K [0x000000040c200000, 0x0000000421190000, 0x000000055bb50000)
  eden space 275008K,   2% used [0x000000040c200000, 0x000000040c75f0b0, 0x000000041ce90000)
  from space 34304K,   0% used [0x000000041ce90000, 0x000000041ce90000, 0x000000041f010000)
  to   space 34304K,   0% used [0x000000041f010000, 0x000000041f010000, 0x0000000421190000)
tenured generation   total 687232K, used 0K [0x000000055bb50000, 0x0000000585a70000, 0x00000007fae00000)
   the space 687232K,   0% used [0x000000055bb50000, 0x000000055bb50000, 0x000000055bb50200, 0x0000000585a70000)
compacting perm gen  total 21248K, used 2986K [0x00000007fae00000, 0x00000007fc2c0000, 0x0000000800000000)
   the space 21248K,  14% used [0x00000007fae00000, 0x00000007fb0ea878, 0x00000007fb0eaa00, 0x00000007fc2c0000)
No shared spaces configured.

Code Cache  [0x00007f941cf7b000, 0x00007f941d1eb000, 0x00007f941ff7b000)
total_blobs=174 nmethods=3 adapters=126 free_code_cache=49941248 largest_free_block=13184

Dynamic libraries:
00400000-00401000 r-xp 00000000 08:03 43958282                           /home/work/baiming/cm-real-test/test_libhdfs
00500000-00501000 rw-p 00000000 08:03 43958282                           /home/work/baiming/cm-real-test/test_libhdfs
00501000-00715000 rw-p 00000000 00:00 0                                  [heap]
400c5000-400c8000 ---p 00000000 00:00 0
400c8000-401c6000 rwxp 00000000 00:00 0
40a15000-40a18000 ---p 00000000 00:00 0
40a18000-40b16000 rwxp 00000000 00:00 0
40e43000-40e46000 ---p 00000000 00:00 0
40e46000-40f44000 rwxp 00000000 00:00 0
40f44000-40f47000 ---p 00000000 00:00 0
40f47000-41045000 rwxp 00000000 00:00 0
411de000-411df000 ---p 00000000 00:00 0
411df000-412df000 rwxp 00000000 00:00 0
413a3000-413a6000 ---p 00000000 00:00 0
413a6000-414a4000 rwxp 00000000 00:00 0
41ac9000-41aca000 ---p 00000000 00:00 0
41aca000-41bca000 rwxp 00000000 00:00 0
41fba000-41fbd000 ---p 00000000 00:00 0
41fbd000-420bb000 rwxp 00000000 00:00 0
40c200000-421190000 rw-p 00000000 00:00 0
421190000-55bb50000 rw-p 00000000 00:00 0
55bb50000-585a70000 rw-p 00000000 00:00 0
585a70000-7fae00000 rw-p 00000000 00:00 0
7fae00000-7fc2c0000 rw-p 00000000 00:00 0
7fc2c0000-800000000 rw-p 00000000 00:00 0
302ad00000-302ad15000 r-xp 00000000 08:02 146321                         /lib64/ld-2.3.4.so
302ae14000-302ae16000 rw-p 00014000 08:02 146321                         /lib64/ld-2.3.4.so
302af00000-302b02b000 r-xp 00000000 08:02 146311                         /lib64/tls/libc-2.3.4.so
302b02b000-302b12a000 ---p 0012b000 08:02 146311                         /lib64/tls/libc-2.3.4.so
302b12a000-302b12d000 r--p 0012a000 08:02 146311                         /lib64/tls/libc-2.3.4.so
302b12d000-302b130000 rw-p 0012d000 08:02 146311                         /lib64/tls/libc-2.3.4.so
302b130000-302b134000 rw-p 00000000 00:00 0
302b200000-302b202000 r-xp 00000000 08:02 146336                         /lib64/libdl-2.3.4.so
302b202000-302b301000 ---p 00002000 08:02 146336                         /lib64/libdl-2.3.4.so
302b301000-302b303000 rw-p 00001000 08:02 146336                         /lib64/libdl-2.3.4.so
302b400000-302b485000 r-xp 00000000 08:02 146313                         /lib64/tls/libm-2.3.4.so
302b485000-302b584000 ---p 00085000 08:02 146313                         /lib64/tls/libm-2.3.4.so
302b584000-302b586000 rw-p 00084000 08:02 146313                         /lib64/tls/libm-2.3.4.so
302b600000-302b613000 r-xp 00000000 08:02 589366                         /usr/lib64/libz.so.1.2.1.2
302b613000-302b712000 ---p 00013000 08:02 589366                         /usr/lib64/libz.so.1.2.1.2
302b712000-302b713000 rw-p 00012000 08:02 589366                         /usr/lib64/libz.so.1.2.1.2
302b800000-302b810000 r-xp 00000000 08:02 146310                         /lib64/tls/libpthread-2.3.4.so
302b810000-302b90f000 ---p 00010000 08:02 146310                         /lib64/tls/libpthread-2.3.4.so
302b90f000-302b911000 rw-p 0000f000 08:02 146310                         /lib64/tls/libpthread-2.3.4.so
302b911000-302b915000 rw-p 00000000 00:00 0
302ba00000-302ba11000 r-xp 00000000 08:02 146356                         /lib64/libresolv-2.3.4.so
302ba11000-302bb11000 ---p 00011000 08:02 146356                         /lib64/libresolv-2.3.4.so
302bb11000-302bb13000 rw-p 00011000 08:02 146356                         /lib64/libresolv-2.3.4.so
302bb13000-302bb15000 rw-p 00000000 00:00 0
302bc00000-302bc09000 r-xp 00000000 08:02 146317                         /lib64/tls/librt-2.3.4.so
302bc09000-302bd08000 ---p 00009000 08:02 146317                         /lib64/tls/librt-2.3.4.so
302bd08000-302bd0a000 rw-p 00008000 08:02 146317                         /lib64/tls/librt-2.3.4.so
302bd0a000-302bd1a000 rw-p 00000000 00:00 0
302c000000-302c022000 r-xp 00000000 08:02 589379                         /usr/lib64/libk5crypto.so.3.0
302c022000-302c121000 ---p 00022000 08:02 589379                         /usr/lib64/libk5crypto.so.3.0
302c121000-302c123000 rw-p 00021000 08:02 589379                         /usr/lib64/libk5crypto.so.3.0
302c200000-302c215000 r-xp 00000000 08:02 589363                         /usr/lib64/libgssapi_krb5.so.2.2
302c215000-302c315000 ---p 00015000 08:02 589363                         /usr/lib64/libgssapi_krb5.so.2.2
302c315000-302c316000 rw-p 00015000 08:02 589363                         /usr/lib64/libgssapi_krb5.so.2.2
302c400000-302c46d000 r-xp 00000000 08:02 589389                         /usr/lib64/libkrb5.so.3.2
302c46d000-302c56d000 ---p 0006d000 08:02 589389                         /usr/lib64/libkrb5.so.3.2
302c56d000-302c571000 rw-p 0006d000 08:02 589389                         /usr/lib64/libkrb5.so.3.2
302c800000-302c80b000 r-xp 00000000 08:02 146306                         /lib64/libgcc_s-3.4.5-20051201.so.1
302c80b000-302c90a000 ---p 0000b000 08:02 146306                         /lib64/libgcc_s-3.4.5-20051201.so.1
302c90a000-302c90b000 rw-p 0000a000 08:02 146306                         /lib64/libgcc_s-3.4.5-20051201.so.1
302cc00000-302cd0e000 r-xp 00000000 08:02 146450                         /lib64/libcrypto.so.0.9.7a
302cd0e000-302ce0e000 ---p 0010e000 08:02 146450                         /lib64/libcrypto.so.0.9.7a
302ce0e000-302ce2c000 rw-p 0010e000 08:02 146450                         /lib64/libcrypto.so.0.9.7a
302ce2c000-302ce30000 rw-p 00000000 00:00 0
302d300000-302d3d6000 r-xp 00000000 08:02 589313                         /usr/lib64/libstdc++.so.6.0.3
302d3d6000-302d4d5000 ---p 000d6000 08:02 589313                         /usr/lib64/libstdc++.so.6.0.3
302d4d5000-302d4de000 rw-p 000d5000 08:02 589313                         /usr/lib64/libstdc++.so.6.0.3
302d4de000-302d4f0000 rw-p 00000000 00:00 0
7f94166ec000-7f9416800000 r--s 012ae000 08:03 123387907                  /home/hpc/soft/hadoop-client/hadoop/hadoop-2-core.jar
7f9416800000-7f9416895000 rw-p 00000000 00:00 0
7f9416895000-7f9416900000 ---p 00000000 00:00 0
7f9416940000-7f9416963000 rw-p 00000000 00:00 0
7f9416963000-7f94169c2000 r--s 00bac000 08:03 129597478                  /home/hpc/soft/hadoop-client/java6/lib/tools.jar
7f94169c2000-7f9419804000 r--p 00000000 08:02 522793                     /usr/lib/locale/locale-archive
7f9419804000-7f9419838000 rw-p 00000000 00:00 0
7f9419838000-7f94199cf000 r--s 03079000 08:03 182026277                  /home/hpc/soft/hadoop-client/java6/jre/lib/rt.jar
7f94199cf000-7f9419a02000 rw-p 00000000 00:00 0
7f9419a02000-7f9419a21000 rw-p 00000000 00:00 0
7f9419a21000-7f9419b71000 rw-p 00000000 00:00 0
7f9419b71000-7f941af1b000 rw-p 00000000 00:00 0
7f941af1b000-7f941afc3000 rw-p 00000000 00:00 0
7f941afc3000-7f941b997000 rw-p 00000000 00:00 0
7f941b997000-7f941bae8000 rw-p 00000000 00:00 0
7f941bae8000-7f941ce91000 rw-p 00000000 00:00 0
7f941ce91000-7f941ce9c000 rw-p 00000000 00:00 0
7f941ce9c000-7f941ceba000 rw-p 00000000 00:00 0
7f941ceba000-7f941cec5000 rw-p 00000000 00:00 0
7f941cec5000-7f941cf7b000 rw-p 00000000 00:00 0
7f941cf7b000-7f941d1eb000 rwxp 00000000 00:00 0
7f941d1eb000-7f941ff7b000 rw-p 00000000 00:00 0
7f941ff7b000-7f941ff89000 r-xp 00000000 08:03 128944277                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libzip.so
7f941ff89000-7f942008b000 ---p 0000e000 08:03 128944277                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libzip.so
7f942008b000-7f942008e000 rw-p 00010000 08:03 128944277                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libzip.so
7f942008e000-7f942008f000 rw-p 00000000 00:00 0
7f942008f000-7f9420099000 r-xp 00000000 08:02 146346                     /lib64/libnss_files-2.3.4.so
7f9420099000-7f9420199000 ---p 0000a000 08:02 146346                     /lib64/libnss_files-2.3.4.so
7f9420199000-7f942019b000 rw-p 0000a000 08:02 146346                     /lib64/libnss_files-2.3.4.so
7f94201ae000-7f94201c2000 r-xp 00000000 08:02 146340                     /lib64/libnsl-2.3.4.so
7f94201c2000-7f94202c1000 ---p 00014000 08:02 146340                     /lib64/libnsl-2.3.4.so
7f94202c1000-7f94202c3000 rw-p 00013000 08:02 146340                     /lib64/libnsl-2.3.4.so
7f94202c3000-7f94202c5000 rw-p 00000000 00:00 0
7f94202c5000-7f94202ee000 r-xp 00000000 08:03 128944274                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libjava.so
7f94202ee000-7f94203ed000 ---p 00029000 08:03 128944274                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libjava.so
7f94203ed000-7f94203f4000 rw-p 00028000 08:03 128944274                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libjava.so
7f94203f4000-7f9420401000 r-xp 00000000 08:03 128944263                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libverify.so
7f9420401000-7f9420500000 ---p 0000d000 08:03 128944263                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libverify.so
7f9420500000-7f9420503000 rw-p 0000c000 08:03 128944263                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/libverify.so
7f9420503000-7f96a0508000 rw-p 00000000 00:00 0
7f96a0508000-7f96a050b000 r-xp 00000000 08:02 146368                     /lib64/libcom_err.so.2.1
7f96a050b000-7f96a060a000 ---p 00003000 08:02 146368                     /lib64/libcom_err.so.2.1
7f96a060a000-7f96a060b000 rw-p 00002000 08:02 146368                     /lib64/libcom_err.so.2.1
7f96a060b000-7f96a060d000 rw-p 00000000 00:00 0
7f96a060d000-7f96a0f28000 r-xp 00000000 08:03 128944281                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/server/libjvm.so
7f96a0f28000-7f96a1029000 ---p 0091b000 08:03 128944281                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/server/libjvm.so
7f96a1029000-7f96a11de000 rw-p 0091c000 08:03 128944281                  /home/hpc/soft/hadoop-client/java6/jre/lib/amd64/server/libjvm.so
7f96a11de000-7f96a1219000 rw-p 00000000 00:00 0
7f96a1219000-7f96a1225000 r-xp 00000000 08:03 149798915                  /home/hpc/soft/hadoop-client/hadoop/libhdfs/libhdfs.so.1
7f96a1225000-7f96a1324000 ---p 0000c000 08:03 149798915                  /home/hpc/soft/hadoop-client/hadoop/libhdfs/libhdfs.so.1
7f96a1324000-7f96a1325000 rw-p 0000b000 08:03 149798915                  /home/hpc/soft/hadoop-client/hadoop/libhdfs/libhdfs.so.1
7f96a1325000-7f96a1326000 rw-p 00000000 00:00 0
7f96a132f000-7f96a1337000 rw-s 00000000 08:02 1126553                    /tmp/hsperfdata_work/12107
7f96a1337000-7f96a1338000 rw-p 00000000 00:00 0
7f96a1338000-7f96a1339000 r--p 00000000 00:00 0
7f96a1339000-7f96a133a000 rw-p 00000000 00:00 0
7fff23031000-7fff23034000 ---p 00000000 00:00 0
7fff23035000-7fff23131000 rwxp 00000000 00:00 0                          [stack]
7fff23131000-7fff23133000 rw-p 00000000 00:00 0
7fff231ff000-7fff23200000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
java_command: <unknown>
Launcher Type: generic

Environment Variables:
JAVA_HOME=:/home/hpc/soft/hadoop-client/hadoop/../java6:/home/hpc/soft/hadoop-client/hadoop/../java6
PATH=/home/work/valgrind/bin:/home/hpc/soft/openmpi/bin:/home/hpc/soft/torque/bin/:/home/hpc/soft/torque/sbin/:/home/hpc/soft/hadoop-client/hadoop/bin/:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/share/baidu/bin:/home/work/bin
LD_LIBRARY_PATH=/home/hpc/soft/openmpi/lib:/home/hpc/soft/torque/lib:/home/hpc/soft/hadoop-client/hadoop/lib:/home/hpc/soft/hadoop-client/hadoop/libhce/lib:/home/hpc/soft/hadoop-client/hadoop/libhdfs:/home/hpc/soft/hadoop-client/java6/jre/lib/amd64/server::/home/hpc/soft/hadoop-client/hadoop/libhdfs::/home/hpc/soft/hadoop-client/hadoop/../java6/jre/lib/amd64::/home/hpc/soft/hadoop-client/hadoop/../java6/jre/lib/amd64/native_threads::/home/hpc/soft/hadoop-client/hadoop/../java6/jre/lib/amd64/server:/home/hpc/soft/hadoop-client/hadoop/libhdfs::/home/hpc/soft/hadoop-client/hadoop/../java6:/home/hpc/soft/hadoop-client/hadoop/../java6/jre/lib/amd64::/home/hpc/soft/hadoop-client/hadoop/../java6:/home/hpc/soft/hadoop-client/hadoop/../java6/jre/lib/amd64/native_threads::/home/hpc/soft/hadoop-client/hadoop/../java6:/home/hpc/soft/hadoop-client/hadoop/../java6/jre/lib/amd64/server
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x8602a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x8602a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x70ef10], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x70ef10], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x70ef10], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x70ef10], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x711d40], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x711940], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x711940], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x711940], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x711940], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


---------------  S Y S T E M  ---------------

OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 3)

uname:Linux 2.6.32_1-3-3-2 #1 SMP Tue Dec 27 16:20:01 CST 2011 x86_64
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE infinity, NPROC 515386, NOFILE 1024, AS infinity
load average:0.14 0.08 0.31

/proc/meminfo:
MemTotal:       65970868 kB
MemFree:         1178688 kB
Buffers:          789044 kB
Cached:         50354528 kB
SwapCached:            0 kB
Active:         32179848 kB
Inactive:       29782568 kB
Active(anon):   10574072 kB
Inactive(anon):   293320 kB
Active(file):   21605776 kB
Inactive(file): 29489248 kB
Unevictable:        1892 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               160 kB
Readahead         0 kB
Writeback:             0 kB
AnonPages:      10820880 kB
Mapped:            34612 kB
Shmem:             46656 kB
Slab:            1518828 kB
SReclaimable:    1502504 kB
SUnreclaim:        16324 kB
KernelStack:        2416 kB
PageTables:        33060 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    32985432 kB
Committed_AS:   15239964 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      407088 kB
VmallocChunk:   34359323655 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8192 kB
DirectMap2M:    67092480 kB


CPU:total 8 (4 cores per cpu, 2 threads per core) family 6 model 44 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht

/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.21
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.84
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 0
siblings : 4
core id : 9
cpu cores : 4
apicid : 18
initial apicid : 18
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.93
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 0
siblings : 4
core id : 10
cpu cores : 4
apicid : 20
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.84
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 32
initial apicid : 32
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.88
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 1
siblings : 4
core id : 1
cpu cores : 4
apicid : 34
initial apicid : 34
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.89
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 1
siblings : 4
core id : 9
cpu cores : 4
apicid : 50
initial apicid : 50
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.90
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping : 2
cpu MHz : 2399.609
cache size : 12288 KB
physical id : 1
siblings : 4
core id : 10
cpu cores : 4
apicid : 52
initial apicid : 52
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips : 4799.88
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 65970868k(1178688k free), swap 0k(0k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (20.7-b02) for linux-amd64 JRE (1.6.0_32-b05), built on Apr  5 2012 14:09:59 by "java_re" with gcc 3.2.2 (SuSE Linux)

time: Thu Jan 10 13:35:12 2013
elapsed time: 0 seconds

我们的测试代码就是这么一段简单的：
int main(int argc, char **argv) {
    if (argc != 2) {
        fprintf(stderr, "Usage: hdfs_write <mem>\n");
        exit(-1);
    }
    tSize m = strtoul(argv[1], NULL, 10);

    fprintf(stderr, "allocate %ld bytes\n", m * 1024 * 1024 * 1024L);
    char* memmem = (char*)malloc(m * 1024*1024*1024L);
    fprintf(stderr, "memmem %p \n", memmem);
    fprintf(stderr, "before\n");

    hdfsFS fs = hdfsConnect("default", 0);
    if (!fs) {
        fprintf(stderr, "Oops! Failed to connect to hdfs!\n");
        exit(-1);
    }

    fprintf(stderr, "after\n");

    return 0;
}

嗯，这是个JVM bug，正好在你们用的JDK6u32上还没修。是这个：JDK-6964776: c2 should ensure the polling page is reachable on 64 bit
解决办法是升级到JDK7（现在的话建议升级到JDK7u11），或者手工打patch（代码的链接在上面bug report链接的最下面有）。

顺带广告一下TaobaoJDK的新版因为是基于JDK7所以包含有这个bugfix的：

（图来自阿里核心系统部专用计算组老大 @王王争 的微博，欢迎关注）
之前淘宝也遇到过同一个bug所以我们正好知道，呵呵

多谢多谢~~~我刚用java7试了一下，那个简单的测试case过去了。我们准备用线上用户的真实case跑一下。   还是阿里系对jvm研究深入，真心赞一个