2.6.25.4 and segfault

Installation, configuration and troubleshooting of the Cisco VPN Client on Linux systems

2.6.25.4 and segfault

Postby Pjot » Tue Jul 01, 2008 8:57 am

Hi,

I downloaded the VPN client package 'vpnclient-linux-x86_64-4.8.01.0640-k9.tar.gz' and followed your instructions. I am on a 32-bit Linux installation.

peter[installation]$ tar -xzf vpnclient-linux-x86_64-4.8.01.0640-k9.tar.gz
peter[installation]$ cd vpnclient
peter[vpnclient]$ patch <../vpnclient-linux-2.6.24-final.diff
patching file GenDefs.h
patching file interceptor.c
peter[vpnclient]$ su
Password:
root[vpnclient]# ./vpn_install
Cisco Systems VPN Client Version 4.8.01 (0640) Linux Installer
Copyright (C) 1998-2006 Cisco Systems, Inc. All Rights Reserved.

By installing this product you agree that you have read the
license.txt file (The VPN Client license) and will comply with
its terms.


Directory where binaries will be installed [/usr/local/bin]

Automatically start the VPN service at boot time [no]

In order to build the VPN kernel module, you must have the
kernel headers for the version of the kernel you are running.


Directory containing linux kernel source code [/lib/modules/2.6.25.4/build]

* Binaries will be installed in "/usr/local/bin".
* Modules will be installed in "/lib/modules/2.6.25.4/CiscoVPN".
* The VPN service will *NOT* be started automatically at boot time.
* Kernel source from "/lib/modules/2.6.25.4/build" will be used to build the module.

Is the above correct [y]

Making module
make -C /lib/modules/2.6.25.4/build SUBDIRS=/home/peter/installation/vpnclient modules
make[1]: Map '/usr/src/linux-2.6.25.4' wordt binnengegaan
CC [M] /home/peter/installation/vpnclient/linuxcniapi.o
CC [M] /home/peter/installation/vpnclient/frag.o
CC [M] /home/peter/installation/vpnclient/IPSecDrvOS_linux.o
CC [M] /home/peter/installation/vpnclient/interceptor.o
CC [M] /home/peter/installation/vpnclient/linuxkernelapi.o
LD [M] /home/peter/installation/vpnclient/cisco_ipsec.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/peter/installation/vpnclient/cisco_ipsec.mod.o
LD [M] /home/peter/installation/vpnclient/cisco_ipsec.ko
make[1]: Map '/usr/src/linux-2.6.25.4' wordt verlaten
Create module directory "/lib/modules/2.6.25.4/CiscoVPN".
Copying module to directory "/lib/modules/2.6.25.4/CiscoVPN".
Already have group 'bin'

Creating start/stop script "/etc/rc.d/vpnclient_init".
/etc/rc.d/vpnclient_init
Creating global config /etc/opt/cisco-vpnclient

Installing license.txt (VPN Client license) in "/opt/cisco-vpnclient/":
/opt/cisco-vpnclient/license.txt

Installing bundled user profiles in "/etc/opt/cisco-vpnclient/Profiles/":
* New Profiles : sample

Copying binaries to directory "/opt/cisco-vpnclient/bin".
Adding symlinks to "/usr/local/bin".
/opt/cisco-vpnclient/bin/vpnclient
/opt/cisco-vpnclient/bin/cisco_cert_mgr
/opt/cisco-vpnclient/bin/ipseclog
Copying setuid binaries to directory "/opt/cisco-vpnclient/bin".
/opt/cisco-vpnclient/bin/cvpnd
Copying libraries to directory "/opt/cisco-vpnclient/lib".
/opt/cisco-vpnclient/lib/libvpnapi.so
Copying header files to directory "/opt/cisco-vpnclient/include".
/opt/cisco-vpnclient/include/vpnapi.h

Setting permissions.
/opt/cisco-vpnclient/bin/cvpnd (setuid root)
/opt/cisco-vpnclient (group bin readable)
/etc/opt/cisco-vpnclient (group bin readable)
/etc/opt/cisco-vpnclient/Profiles (group bin readable)
/etc/opt/cisco-vpnclient/Certificates (group bin readable)
* You may wish to change these permissions to restrict access to root.
* You must run "/etc/rc.d/vpnclient_init start" before using the client.
* You will need to run this script every time you reboot your computer.
root[vpnclient]# ls -l /opt/cisco-vpnclient/bin/cvpnd
---s--x--x 1 root bin 2181944 2008-07-01 08:36 /opt/cisco-vpnclient/bin/cvpnd
root[vpnclient]# /etc/rc.d/vpnclient_init start
Starting /opt/cisco-vpnclient/bin/vpnclient: Done
root[vpnclient]# dmesg | tail -3
hda-intel: Invalid position buffer, using LPIB read method instead.
Real Time Clock Driver v1.12ac
Cisco Systems VPN Client Version 4.8.01 (0640) kernel module loaded


So far so good! I installed my profiles and try to run the client.

peter[~]$ vpnclient connect company
Cisco Systems VPN Client Version 4.8.01 (0640)
Copyright (C) 1998-2007 Cisco Systems, Inc. All Rights Reserved.
Client Type(s): Linux
Running on: Linux 2.6.25.4 #1 SMP PREEMPT Fri May 16 14:10:46 CEST 2008 i686
Config file directory: /etc/opt/cisco-vpnclient

Segfault
peter[~]$ vpnclient connect company
Cisco Systems VPN Client Version 4.8.01 (0640)
Copyright (C) 1998-2007 Cisco Systems, Inc. All Rights Reserved.
Client Type(s): Linux
Running on: Linux 2.6.25.4 #1 SMP PREEMPT Fri May 16 14:10:46 CEST 2008 i686
Config file directory: /etc/opt/cisco-vpnclient

Segfault
peter[~]$ dmesg | tail -4
Real Time Clock Driver v1.12ac
Cisco Systems VPN Client Version 4.8.01 (0640) kernel module loaded
vpnclient[3409]: segfault at 80834db ip 08070bdf sp bf83cfbc error 7 in vpnclient[8048000+9d000]
vpnclient[3412]: segfault at 80834db ip 08070bdf sp bfa2499c error 7 in vpnclient[8048000+9d000]


Any idea what may have gone wrong...?

Thanks
Peter
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby Pjot » Thu Jul 03, 2008 8:55 am

The newly released package "vpnclient-linux-x86_64-4.8.02.0030-k9.tar.gz" from july 1 compiles well without any patching, but also segfaults...

peter[installation]$ tar -xzf vpnclient-linux-x86_64-4.8.02.0030-k9.tar.gz
peter[installation]$ cd vpnclient
peter[vpnclient]$ su
Password:
root[vpnclient]# ./vpn_install
Cisco Systems VPN Client Version 4.8.02 (0030) Linux Installer
Copyright (C) 1998-2006 Cisco Systems, Inc. All Rights Reserved.

By installing this product you agree that you have read the
license.txt file (The VPN Client license) and will comply with
its terms.


Directory where binaries will be installed [/usr/local/bin]

Automatically start the VPN service at boot time [no]

In order to build the VPN kernel module, you must have the
kernel headers for the version of the kernel you are running.


Directory containing linux kernel source code [/lib/modules/2.6.25.4/build]

* Binaries will be installed in "/usr/local/bin".
* Modules will be installed in "/lib/modules/2.6.25.4/CiscoVPN".
* The VPN service will *NOT* be started automatically at boot time.
* Kernel source from "/lib/modules/2.6.25.4/build" will be used to build the module.

Is the above correct [y]

Making module
make -C /lib/modules/2.6.25.4/build SUBDIRS=/home/peter/installation/vpnclient modules
make[1]: Map '/usr/src/linux-2.6.25.4' wordt binnengegaan
CC [M] /home/peter/installation/vpnclient/linuxcniapi.o
CC [M] /home/peter/installation/vpnclient/frag.o
CC [M] /home/peter/installation/vpnclient/IPSecDrvOS_linux.o
CC [M] /home/peter/installation/vpnclient/interceptor.o
CC [M] /home/peter/installation/vpnclient/linuxkernelapi.o
LD [M] /home/peter/installation/vpnclient/cisco_ipsec.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/peter/installation/vpnclient/cisco_ipsec.mod.o
LD [M] /home/peter/installation/vpnclient/cisco_ipsec.ko
make[1]: Map '/usr/src/linux-2.6.25.4' wordt verlaten
Copying module to directory "/lib/modules/2.6.25.4/CiscoVPN".
Already have group 'bin'

Creating start/stop script "/etc/rc.d/vpnclient_init".
/etc/rc.d/vpnclient_init

Installing license.txt (VPN Client license) in "/opt/cisco-vpnclient/":
/opt/cisco-vpnclient/license.txt

Installing bundled user profiles in "/etc/opt/cisco-vpnclient/Profiles/":
* Replaced Profiles: sample

Copying binaries to directory "/opt/cisco-vpnclient/bin".
Adding symlinks to "/usr/local/bin".
/opt/cisco-vpnclient/bin/vpnclient
/opt/cisco-vpnclient/bin/cisco_cert_mgr
/opt/cisco-vpnclient/bin/ipseclog
Copying setuid binaries to directory "/opt/cisco-vpnclient/bin".
/opt/cisco-vpnclient/bin/cvpnd
Copying libraries to directory "/opt/cisco-vpnclient/lib".
/opt/cisco-vpnclient/lib/libvpnapi.so
Copying header files to directory "/opt/cisco-vpnclient/include".
/opt/cisco-vpnclient/include/vpnapi.h

Setting permissions.
/opt/cisco-vpnclient/bin/cvpnd (setuid root)
/opt/cisco-vpnclient (group bin readable)
/etc/opt/cisco-vpnclient (permissions not changed)
* You may wish to change these permissions to restrict access to root.
* You must run "/etc/rc.d/vpnclient_init start" before using the client.
* You will need to run this script every time you reboot your computer.
root[vpnclient]# dmesg | tail -1
Cisco Systems VPN Client Version 4.8.02 (0030) kernel module loaded
root[vpnclient]# exit
exit
peter[vpnclient]$
peter[vpnclient]$
peter[vpnclient]$ vpnclient connect company
Cisco Systems VPN Client Version 4.8.02 (0030)
Copyright (C) 1998-2007 Cisco Systems, Inc. All Rights Reserved.
Client Type(s): Linux
Running on: Linux 2.6.25.4 #1 SMP PREEMPT Fri May 16 14:10:46 CEST 2008 i686
Config file directory: /etc/opt/cisco-vpnclient

Segmentation error
peter[vpnclient]$ dmesg| tail -2
Cisco Systems VPN Client Version 4.8.02 (0030) kernel module loaded
vpnclient[2998]: segfault at 80834db ip 08070bdf sp bff836fc error 7 in vpnclient[8048000+9d000]


How annoying! I guess it's something in the way the kernel was compiled (using a stock kernel, Zenwalk Linux 5.2).

Greetings
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby Pjot » Thu Jul 03, 2008 9:30 am

Even switching to one CPU delivers the segfault. Tried to figure out error code '7' in /usr/src/linux-2.6.25.4/arch/x86/mm/fault.c:

/*
* Page fault error code bits
* bit 0 == 0 means no page found, 1 means protection fault
* bit 1 == 0 means read, 1 means write
* bit 2 == 0 means kernel, 1 means user-mode
* bit 3 == 1 means use of reserved bit detected
* bit 4 == 1 means fault was an instruction fetch
*/


So bits 0, 1 and 2 are set, meaning a protection fault during a write in user-mode... hmmm... doesn't help me much. I'll try to see if I can compile the VPN binaries with '-g' and then run it with Valgrind, see what comes up.

Peter
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby Pjot » Thu Jul 03, 2008 5:07 pm

Compiling with '-g' and running with valgrind did not help a lot, the segfault occurs even before Valgrind starts tracing.

It appears that when editing a profile to the default layout (similar to the 'sample.pcf' file) at least the segfault does not occur. The vpnclient starts a connection but then remains silent; the 'cvpnd' jumps to 100% CPU load. This also occurs when disabling the other cores in the CPU (so only with 1 core enabled):
[quote]
peter[~]$ vpnclient connect company
Cisco Systems VPN Client Version 4.8.02 (0030)
Copyright (C) 1998-2007 Cisco Systems, Inc. All Rights Reserved.
Client Type(s): Linux
Running on: Linux 2.6.25.4 #1 SMP PREEMPT Fri May 16 14:10:46 CEST 2008 i686
Config file directory: /etc/opt/cisco-vpnclient

Enter a group password:
Initializing the VPN connection.

[quote]

After the message 'Initializing the VPN connection.' nothing happens, the cvpnd daemon consumes all system resources...
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby euiseong » Thu Jul 03, 2008 7:49 pm

I have the same issue. I have used vpclient well on my x86_64 with Ubuntu 8.04.
Recently I got a Phenom processor based workstation. So I did the same compilation procedure. However, I got a segfault when I run vpnclient. The profile was the same one. But, it did not make any error in my Athlon based computer (also Ubuntu 8.04 was installed)

[ 1310.289281] vpnclient[10300]: segfault at 80834db rip 8070bdf rsp ffa3224c error 7

when I use gdb for vpnclient, I got this message

Program received signal SIGSEGV, Segmentation fault.
0x08070bdf in SHATransformI32_3 ()

Actually since I don't have source code for vpnclient I could not find what SHATransforml32_3() is doing. The source codes in distribution is for the kernel module, not the user-level part.
euiseong
Private
Private
 
Posts: 5
Joined: Thu Jul 03, 2008 7:42 pm

Re: 2.6.25.4 and segfault

Postby euiseong » Fri Jul 04, 2008 5:30 am

Now I found some hints related to my problem.

On Athlon 64 X2 6000+, GDB shows that SHATransformAMD_k7 function is called.

Code: Select all
#0  0x0806ae69 in SHATransformAMD_k7 ()
#1  0x0806a9aa in A_SHAUpdateCommon ()
#2  0x0806ab15 in A_SHAFinalCommon ()
#3  0x0806a90e in A_SHAFinal ()
#4  0x080695ab in ?? ()
#5  0x080763ef in AHChooseDigestFinal ()
#6  0x08074d01 in B_AlgorithmDigestFinal ()
#7  0x0806a3c9 in B_DigestFinal ()
#8  0x08068865 in CHashSha1::hash ()
#9  0x0806875b in CHash::Hash ()
#10 0x08065e9d in ConfigFile::Decrypt ()
#11 0x0805eb92 in GI_ReadProfile ()
#12 0x080536d3 in ?? ()
#13 0x08052e84 in ?? ()
#14 0x08056941 in ?? ()
#15 0xf7d51450 in __libc_start_main () from /lib32/libc.so.6
#16 0x080528a1 in ?? ()


In the exactly same environment Phenom processor choose "SHATransformI32_3()" instead of SHATransformAMD_k7 and the SHATransformI32_3 function causes the stack corruption (and bang! the segfault occurs here!)

Code: Select all
#0  0x0806f9e0 in SHATransformI32_3 ()
#1  0x0806a9aa in A_SHAUpdateCommon ()
#2  0x0806ab15 in A_SHAFinalCommon ()
#3  0x0806a90e in A_SHAFinal ()
#4  0x080695ab in ?? ()
#5  0x080763ef in AHChooseDigestFinal ()
#6  0x08074d01 in B_AlgorithmDigestFinal ()
#7  0x0806a3c9 in B_DigestFinal ()
#8  0x08068865 in CHashSha1::hash ()
#9  0x0806875b in CHash::Hash ()
#10 0x08065e9d in ConfigFile::Decrypt ()
#11 0x0805eb92 in GI_ReadProfile ()
#12 0x080536d3 in ?? ()
#13 0x08052e84 in ?? ()
#14 0x08056941 in ?? ()
#15 0xf7d51450 in __libc_start_main () from /lib32/libc.so.6
#16 0x080528a1 in ?? ()


If I could make vpnclient use SHATransformAMD_k7 it would go without any problem.

New hardware sometimes makes trouble.
euiseong
Private
Private
 
Posts: 5
Joined: Thu Jul 03, 2008 7:42 pm

Re: 2.6.25.4 and segfault

Postby Pjot » Fri Jul 04, 2008 1:46 pm

Thanks for your post, and actually, I have a AMD Phenom too, a Phenom X4 9550, Box, 4 MB, AM2+, 64 bit, 2200 MHz.

Would it be processor-related then?
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby euiseong » Fri Jul 04, 2008 3:51 pm

Yes, it seems certain.
In my guess, vpnclient uses a function pointer which points different functions depending on the processor architecture.
The candidates for that function pointer are "SHATransformI32_3()", "SHATransformI32_5()", "SHATransformI32_II()", and "SHATransformAMD_k7()". SHATransformI32_3 looks like the function implementation for 80386 architecture.
But I could not find how vpnclient identifies the processor architecture. If I know that I can make vpnclient use "SHATransformAMD_k7()".
Changing the processor family value to "15" (which was "16") in proc/cpuinfo to masquerade did not work.
euiseong
Private
Private
 
Posts: 5
Joined: Thu Jul 03, 2008 7:42 pm

Re: 2.6.25.4 and segfault

Postby Pjot » Fri Jul 04, 2008 5:12 pm

Thanks, I understand. :D Maybe there is a way to fiddle around with '/sys/devices/system/cpu/' and subdirs? I can check this also later today.
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby Pjot » Sun Jul 06, 2008 10:54 am

It doesn't work unfortunately. I guess there are two ways to go now:

1) Hack the VPNclient binaries to see where the CPU decision is made and change the value of the CPU ID to 16.
2) Install some VM with Linux and setup a VPN in there.
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby euiseong » Sun Jul 06, 2008 6:13 pm

I disassembled and looked around it. Found that CPU checking was not done by reading system files. It would be useless to try masquerade the system info files. Actually I found where the problematic function is called and how I can evade the function by replacing the "call" instruction so that it calls AMD_k7 function. But the problem is that I don't know how can I replace the instruction and write to the file in GDB.

If one of these two things could be done, I could make the patch.

1) Replace an instruction and write the change into the binary file in GDB.
2) Rebuild or rearrange the symbol table in ELF shared library.

Unfortunately I could not find any solution for these.
euiseong
Private
Private
 
Posts: 5
Joined: Thu Jul 03, 2008 7:42 pm

Re: 2.6.25.4 and segfault

Postby Pjot » Tue Jul 08, 2008 9:09 am

Well, I am afraid these thing are not possible with GDB.

But there are tools like 'readelf' and 'objcopy' to check and change the binary file. Also I ran into 'livepatch', which is able to change a running process in memory: http://ukai.jp/Software/livepatch/
Pjot
Private
Private
 
Posts: 9
Joined: Tue Jul 01, 2008 8:49 am

Re: 2.6.25.4 and segfault

Postby dtrucken » Tue Jul 29, 2008 5:43 am

I too am getting this message with my new AMD machine:

vendor_id : AuthenticAMD
cpu family : 16
model : 2
model name : AMD Phenom(tm) 9600 Quad-Core Processor

Is there any hope?
dtrucken
Private
Private
 
Posts: 3
Joined: Tue Jul 29, 2008 5:40 am

Re: 2.6.25.4 and segfault

Postby raison » Mon Aug 25, 2008 8:06 am

I, too, am having this issue.

vendor_id : AuthenticAMD
cpu family : 16
model : 2
model name : AMD Phenom(tm) 9550 Quad-Core Processor
raison
Private
Private
 
Posts: 1
Joined: Mon Aug 25, 2008 8:04 am

Re: 2.6.25.4 and segfault

Postby tuxx-home.at » Mon Aug 25, 2008 9:46 am

Thanks for your detailed debugging information. I don't have an idea how to fix that, so all I can provide for now is the offer to forward this detailed bugreport to Cisco directly.

As I don't have an idea how to do this, I forwarded it to our Cisco consultant at work and asked him if he could take care of that.
The mail is out, when I get an answer, I'll update you all on this issue.

Glad I don't have a Phenom processor by now ;)

One very wild idea would be to run the vpnclient in a chroot()ed environment with a modified /proc filesystem faking the CPU model and/or architecture, but that will only work if the client makes use of the /proc filesystem for gathering that kind of information.
User avatar
tuxx-home.at
Supreme Commander
Supreme Commander
 
Posts: 2199
Joined: Mon Jan 01, 2007 12:51 pm
Location: Vassach - Austria - Europe

Next

Return to Cisco VPN Client on Linux systems

Who is online

Users browsing this forum: No registered users and 1 guest

cron