Cisco VPN client freezes computer when using wireless wlan0

[jordi]

As I mentioned in the previous posts, I have a 64-bit processor, but after I apply the patch "cisco_skbuff_offset.patch", I install the software and I get some warnings related to 64-bit issues. I will just copy all the console messages during the installation of vpnclient to see if somebody sees any problem or any warning that could be related to this freezing problem.
Jordi

--------------------------------------------------
make
make -C /lib/modules/2.6.22.17-0.1-default/build SUBDIRS=/home/user/Desktop/vpnclient modules
make[1]: Entering directory `/usr/src/linux-2.6.22.17-0.1-obj/x86_64/default'
make -C ../../../linux-2.6.22.17-0.1 O=../linux-2.6.22.17-0.1-obj/x86_64/default modules
CC [M] /home/user/Desktop/vpnclient/linuxcniapi.o
In file included from /home/user/Desktop/vpnclient/Cniapi.h:15,
from /home/user/Desktop/vpnclient/linuxcniapi.c:31:
/home/user/Desktop/vpnclient/GenDefs.h:110:2: warning: #warning 64 bit
CC [M] /home/user/Desktop/vpnclient/frag.o
In file included from /home/user/Desktop/vpnclient/Cniapi.h:15,
from /home/user/Desktop/vpnclient/frag.c:20:
/home/user/Desktop/vpnclient/GenDefs.h:110:2: warning: #warning 64 bit
CC [M] /home/user/Desktop/vpnclient/IPSecDrvOS_linux.o
In file included from /home/user/Desktop/vpnclient/IPSecDrvOS_linux.c:24:
/home/user/Desktop/vpnclient/GenDefs.h:110:2: warning: #warning 64 bit
CC [M] /home/user/Desktop/vpnclient/interceptor.o
In file included from /home/user/Desktop/vpnclient/Cniapi.h:15,
from /home/user/Desktop/vpnclient/interceptor.c:34:
/home/user/Desktop/vpnclient/GenDefs.h:110:2: warning: #warning 64 bit
CC [M] /home/user/Desktop/vpnclient/linuxkernelapi.o
LD [M] /home/user/Desktop/vpnclient/cisco_ipsec.o
Building modules, stage 2.
MODPOST 1 modules
WARNING: could not find /home/user/Desktop/vpnclient/.libdriver64.so.cmd for /home/user/Desktop/vpnclient/libdriver64.so
CC /home/user/Desktop/vpnclient/cisco_ipsec.mod.o
LD [M] /home/user/Desktop/vpnclient/cisco_ipsec.ko
make[1]: Leaving directory `/usr/src/linux-2.6.22.17-0.1-obj/x86_64/default'



./vpn_install
Cisco Systems VPN Client Version 4.8.01 (0640) Linux Installer
Copyright (C) 1998-2006 Cisco Systems, Inc. All Rights Reserved.

By installing this product you agree that you have read the
license.txt file (The VPN Client license) and will comply with
its terms.


Directory where binaries will be installed [/usr/local/bin]

Automatically start the VPN service at boot time [yes]no

In order to build the VPN kernel module, you must have the
kernel headers for the version of the kernel you are running.


Directory containing linux kernel source code [/lib/modules/2.6.22.17-0.1-default/build]

* Binaries will be installed in "/usr/local/bin".
* Modules will be installed in "/lib/modules/2.6.22.17-0.1-default/CiscoVPN".
* The VPN service will *NOT* be started automatically at boot time.
* Kernel source from "/lib/modules/2.6.22.17-0.1-default/build" will be used to build the module.

Is the above correct [y]

Making module
make -C /lib/modules/2.6.22.17-0.1-default/build SUBDIRS=/home/user/Desktop/vpnclient modules
make[1]: Entering directory `/usr/src/linux-2.6.22.17-0.1-obj/x86_64/default'
make -C ../../../linux-2.6.22.17-0.1 O=../linux-2.6.22.17-0.1-obj/x86_64/default modules
Building modules, stage 2.
MODPOST 1 modules
WARNING: could not find /home/user/Desktop/vpnclient/.libdriver64.so.cmd for /home/user/Desktop/vpnclient/libdriver64.so
make[1]: Leaving directory `/usr/src/linux-2.6.22.17-0.1-obj/x86_64/default'
Copying module to directory "/lib/modules/2.6.22.17-0.1-default/CiscoVPN".
Already have group 'bin'

Creating start/stop script "/etc/init.d/vpnclient_init".
/etc/init.d/vpnclient_init
vpnclient_init 0:off 1:off 2:off 3:off 4:off 5:off 6:off
vpnclient_init 0:off 1:off 2:off 3:on 4:off 5:on 6:off
Creating global config /etc/opt/cisco-vpnclient

Installing license.txt (VPN Client license) in "/opt/cisco-vpnclient/":
/opt/cisco-vpnclient/license.txt

Installing bundled user profiles in "/etc/opt/cisco-vpnclient/Profiles/":
* New Profiles : sample

Copying binaries to directory "/opt/cisco-vpnclient/bin".
Adding symlinks to "/usr/local/bin".
/opt/cisco-vpnclient/bin/vpnclient
/opt/cisco-vpnclient/bin/cisco_cert_mgr
/opt/cisco-vpnclient/bin/ipseclog
Copying setuid binaries to directory "/opt/cisco-vpnclient/bin".
/opt/cisco-vpnclient/bin/cvpnd
Copying libraries to directory "/opt/cisco-vpnclient/lib".
/opt/cisco-vpnclient/lib/libvpnapi.so
Copying header files to directory "/opt/cisco-vpnclient/include".
/opt/cisco-vpnclient/include/vpnapi.h

Setting permissions.
/opt/cisco-vpnclient/bin/cvpnd (setuid root)
/opt/cisco-vpnclient (group bin readable)
/etc/opt/cisco-vpnclient (group bin readable)
/etc/opt/cisco-vpnclient/Profiles (group bin readable)
/etc/opt/cisco-vpnclient/Certificates (group bin readable)
* You may wish to change these permissions to restrict access to root.
* You must run "/etc/init.d/vpnclient_init start" before using the client.
* You will need to run this script every time you reboot your computer.

[alesstimec]

Hello hello...

Sorry for a bit of delay..

I, again, confirm that i am running 32bit version of Ubuntu 7.10 on a Toshiba Satellite Pro with Intel Core2Duo. I compiled the 2.6.24 kernel and it still doesn't work!

Vpnclient in combination with Intel's wireless (iwl4965 module) freezes.. the funny thing is that i can ssh for as long as i want when connected via vpn.. But as soon as i try http access to a machine on the vpn network the computer freezes (kernel panic).. The same happens if i try VNC...

Any solutions out there?

Cheers,
Ales

[jordi]

Hi all,
I just wanted to confirm what happens to "alesstimec" from my end: my vpnclient freezes my computer immediately when I visit a HTML page that uses the "vpnclient tunnel". Moreover, when I run intensive simulations via ssh on the konsole with vpnclient, my computer freezes after 1-2 minutes (not immediately as when I access an HTML website). Therefore, I think there might be some sort of threshold in the amount of data that goes through the vpnclient tunnel, and if you exceed this threshold, the computer unexplicably freezes.

So far (considering only the people that reported to this forum about this problem), this freezing problem seems to be independent of the wireless driver and the bits of the processor:
alesstimec: 32-bit and iwl4965.
jordi: 64-bit and iwlwifi 1.2.25.
mio: ??-bit and iwl3945
clark: ??-bit and iwl????

Does anybody have a hint on what's going on?
Jordi

[srankin1826]

I can confirm that the kernel panic happens using 64 bit ubuntu 7.10 with cisco_skbuff_offset.patch on vpnclient-linux-x86_64-4.8.01.0640-k9.

In my case it always happens a few seconds after opening up a vnc session. I've been able to get vpnc running via kvpnc.

Hardware info: T61 laptop

lspci:

00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c)
00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c)
00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c)
00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Contoller #4 (rev 03)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 03)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 03)
00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 03)
00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 3 (rev 03)
00:1c.3 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 4 (rev 03)
00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 03)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f3)
00:1f.0 ISA bridge: Intel Corporation 82801HBM (ICH8M-E) LPC Interface Controller (rev 03)
00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE Controller (rev 03)
00:1f.2 SATA controller: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) SATA AHCI Controller (rev 03)
00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 03)
03:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN Network Connection (rev 61)
15:00.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev b6)

[dmalone]

I too am looking for a fix to this. To add the the information posted I have...
Dell D830
64-bit Centrino Dual Core
32-bit Ubuntu 7.10 Gutsy
Kernel 2.6.22-14
wireless wlan0 using iwl4965 driver

Just wanted to post in case it helps to narrow down the issue.

[chb421]

Same thing with different wireless drivers.

Dell Precision M65
2.6.22-14-generic #1 SMP Tue Feb 12 02:46:46 UTC 2008 x86_64 GNU/Linux
wireless eth1 using bcm43xx driver

Tried with both the 0490 and 0640 builds, no compile errors. Can connect to the VPN session, doesn't lock up during VNC or HTTP, but will eventually lock up. Nothing written to log, keyboard completely unresponsive.

[chb421]

Correction. It also locks up when connected through 10/100 on eth0. Not just on wireless.

[chb421]

Could these be SMP rather than the x64 kernel?

I opened a Cisco TAC case to see what I could get from Cisco, below is the correspondence.

--

After looking at your service request, it looks like you are trying to run the Cisco VPN client on an Ubuntu x64 Linux system, which isn’t explicitly supported as you can see from our documentation here:
http://www.cisco.com/en/US/docs/securit ... #wp1024664

Sometimes compiling may work on your specific kernel, but it isn’t officially supported. However, I see from your case notes that you’re using an SMP kernel, which is not supported at this time at all, which is most likely why you’re experiencing problems (see link above). If you were using a single processor kernel, you might find that the client compiles, but there is no guarantee.

If you have any other questions about the Linux client, please let me know and I’ll be happy to answer your questions.

--

Are there any plans to support SMP and x64 in the future? As far as I know, this is the most common configuration of the new kernels of linux.

--

I agree with you on this one, but I can’t say what our development team has in store down the road, they keep that well guarded unfortunately. I can only imagine that there must be a large demand for this, but with our focus shifting from the standalone vpn client (IPSEC) to the AnyConnect client (SSL) I don’t know how much we can expect resources to be devoted to further development on the standalone client. What is more likely is that you’ll see support in the future for SMP kernels in our AnyConnect client. Does that answer your question?

--

Does the VPN3005 support the anyconnect client?

--

No, the AnyConnect client will only work on one of the 5500 series Adaptive Security Appliances. If a hardware upgrade is in your future, you may want to consider that if you’re going to be supporting Linux clients using VPN.

--

I see a place in the VPN3005 that shows install SSL VPN client. It’s under the WebVPN settings area. Would you happen to know what that is for?

--

The SSL VPN client is supported on the VPN3005, that is true, however the SSL VPN Client will not support Linux users unfortunately. This can be a little confusing, but basically the SSL VPN client is what came before AnyConnect. They both operate over SSL, however the AnyConnect client is the only one of the two clients that supports Mac and Linux users, and is only available on the ASA devices, and IOS routers running 12.4(15)T or later. Hopefully that clears things up a little.

--

That does. We have the VPN3005, several PIXs, and many 18xx/28xx series routers. I’ll just set it up on one of the 2800s for now.

--

Sounds good; that should work for you in the interim, depending on how many users you have. Please note: you have 10 free user licenses on the 28xx series routers, beyond that it looks like you’ll need to purchase additional licenses. You can find that information here:
http://cisco.com/en/US/prod/collateral/ ... Sheet.html

--

EOF

[tuxx-home.at]

That's very interesting news.
Could you guys please try to disable SMP in your linux kernel for testing purposes?

There are several approaches for doing so, e.g.

* Disable Hyperthreading/multi-core technology in your BIOS
* Boot a non-smp kernel
* Pass the option "nosmp" or "maxcpus=1" to your kernel commandline (the grub bootloader supports editing the kernel command line on the fly, otherwise
you might need to modify your bootloaders configuration file (/etc/lilo.conf or /boot/grub/menu.lst))
* If you have CPU hotplugging enabled in your kernel, you should also be able to shutdown all but one CPUs at runtime but I don't know if that convinces the VPN client that it runs on a non-smp machine (echo 0 > /sys/devices/system/cpu/cpuX/online)

To see if your modifications worked, you could have a look at

* the dmesg output
* the `cat /proc/cpuinfo` output
* the `uname -a` output


Please tell me/us your experiences.

[jordi]

Dear all,
Good news!

In my Dell Latitude D830, the VPN client does not freeze the computer anymore if:
(1) I disable the multi-core option from the BIOS.

or

(2) I use the maxcpus=1 on the fly at the bootloader (Personally, I prefer this option...).

However, my question is: why the VPN client does not freeze my computer even with the BIOS multi-core option enabled or without the maxcpus=1 option? Only the wireless connection seems to be affected by this dual core issue...

In any case, that is an awesome workaround until Cisco supports dual core... in 1000 years from now, maybe??
Thanks for the fix! I would have not been able to come up w/ this workaround!!
Jordi

[tuxx-home.at]

Thanks for reporting success, I had tons of mail conversations, debugging trial- and errors with various people trying to find a fix for this. Unfortunately, I don't know if this problem is related to the wlan drivers or not, as some people are saying that the problem also occurs on wired connections.

It might still be possible that only the existance of the ipw*/iwl* modules in the kernel might cause this problem, so f.ex. if the WLAN modules are loaded into kernel space and you're using the wired connection, the Cisco VPN client interferes with all available network interfaces and therefore the WLAN modules might cause the problem although they're not actively used. I definetly need someone to confirm this

If you need HT for your daily work you could even switch it off manually (as mentioned above) just before you establish the VPN connection and bring the second CPU back online after the VPN connection has been closed. I think of a small wrapper script like this one (untested):

Code: Select all

#!/bin/sh

# path to the real vpnclient binary
VPNCLIENTBIN=/usr/local/bin/vpnclient

# path to the CPU-Hotplug facility for the second cpu. If you have a quadcore system, you
# should set this value to "1 2 3" or something like this.
SMP_CPUS="1"

# if a connection is to be established, switch off all other CPUs
if [ "$1" = "connect" ]; then
  for cpu in $SMP_CPUS; do
    echo 0 > /sys/devices/system/cpu/cpu$cpu/online
  done
fi

# start the VPN client with all given arguments
$VPNCLIENTBIN $*

# after the connection has been closed (either by hitting CTRL-C or by using a separate
# `vpnclient disconnect` invocation, turn the CPUs back on
for cpu in $SMP_CPUs; do
  echo 1 > /sys/devices/system/cpu/cpu$cpu/online
done


[jordi]

Hi,
Setting a either a 0 or 1 to the file "online" of the second CPU works great. There is no need to restart the machine, and the cisco VPN does not freeze the computer at all. This is definitely a much better option!

I have a question though: when I bring the second CPU offline while connected via the VPN client, do I lose 1/2 of my CPU computational power? Or do I have the same computational power as if I had both CPUs online, but the computer only sees one CPU logical unit instead of 2?

Jordi

[tuxx-home.at]

Setting a either a 0 or 1 to the file "online" of the second CPU works great.

Good, that sounds like the best solution.

when I bring the second CPU offline while connected via the VPN client, do I lose 1/2 of my CPU computational power?

The Micky Mouse answer to this question would be "yes", although it isn't absolutely correct. If you're not running multiple CPU intensive tasks at the same time during you're connected with the VPN client, you won't see a big difference.
But I think your question was if the CPU hotplug does only fake turning the second CPU off and the answer to this question is definetly: No.
If you turn a CPU off with the commands mentioned above you can even replace it with a newer CPU physically during runtime (if your hardware supports CPU hotplugging too).

[smootcm]

I can confirm that turning off the 2nd CPU does result in the Cisco VPN client working on 64 bit systems. I am running Ubuntu 7.10 on a duel core ThinkPad T61. I got consistent freezes after a few minutes when running both cores. Turning one core off in the BIOS eliminated the freezes.

I will also point out I am using the Atheros wireless driver and not the Intel drivers. So unless the Atheros wireless has the same issue as the Intel wireless drivers, I suspect wireless does not play a part in the kernel freezes.

[jch]

For what it's worth, I have a Dell D630 with 64 bit Fedora 8 installed and currently running 2.6.24.3-34.fc8. vpnclient, with the two needed patches, works just fine when I'm using the wired network connection but the machines locks up within a few minutes if I use wlan0. I noticed it was actually panicking and got this from a serial console (long lines appear to have been truncated by minicom, sorry):

Code: Select all

Unable to handle kernel NULL pointer dereferenc
[<ffffffff8860e474>] :cisco_ipsec:CBCDecryptUpdate+0xb4/0x160
PGD 50422067 PUD 50503067 PMD 0
Oops: 0000 [1] SMP
CPU 1
Modules linked in: aes_generic aes_x86_64 i915 drm cisco_ipsec(P)(U) rfcomm l2cd
Pid: 0, comm: swapper Tainted: P        2.6.24.3-34.fc8 #1
RIP: 0010:[<ffffffff8860e474>]  [<ffffffff8860e474>] :cisco_ipsec:CBCDecryptUpd0
RSP: 0018:ffff81007f36f9b0  EFLAGS: 00010203
RAX: 0000000000000000 RBX: ffff810045c99cd0 RCX: 000000000000345c
RDX: 0000000000008848 RSI: 0000000000000000 RDI: 0000000000000001
RBP: 0000000000000008 R08: 0000000000000000 R09: 00000000a02280c0
R10: ffff81005a402550 R11: a754293900000000 R12: ffffffff8864e38c
R13: ffffffff8864e38c R14: 0000000000000002 R15: 00000000000003f0
FS:  0000000000000000(0000) GS:ffff81007f001700(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000068105000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff81007f36a000, task ffff81007f35f160)
Stack:  ffff810045c99cd0 ffff81005a402550 0000000000000008 000001607b94ad48
ffff81007f36fa74 ffff810076cc6410 ffffffff8863f880 0000000000000000
0000000000000008 ffff81007b94ad10 0000000000000550 ffffffff8864e22c
Call Trace:
<IRQ>  [<ffffffff88615213>] :cisco_ipsec:AHChooseFeedbackUpdate+0x23/0x40
[<ffffffff88614a03>] :cisco_ipsec:AHFeedbackCipherDecryptUpdate+0x133/0x240
[<ffffffff88615d04>] :cisco_ipsec:B_AlgorithmDecryptUpdate+0x94/0xa0
[<ffffffff885fe668>] :cisco_ipsec:BSafeDecrypt+0x108/0x1a0
[<ffffffff885fea1d>] :cisco_ipsec:BSafeESPDecrypt+0xfd/0x1e0
[<ffffffff885fd8e9>] :cisco_ipsec:CniMemFree+0x79/0x80
[<ffffffff88607ea9>] :cisco_ipsec:ESPIn+0x69/0x210
[<ffffffff88607bc8>] :cisco_ipsec:IPSecOverUDPIn+0xe8/0x100
[<ffffffff88607612>] :cisco_ipsec:IPIn+0xd2/0x100
[<ffffffff88607267>] :cisco_ipsec:TransformIn+0x77/0xf0
[<ffffffff8860250a>] :cisco_ipsec:CNIReceiveComplete+0x23a/0xb50
[<ffffffff88601e59>] :cisco_ipsec:CNIReceiveIndication+0x599/0x660
[<ffffffff885fd188>] :cisco_ipsec:recv_ip_packet_handler+0x201/0x2d6
[<ffffffff811f9bea>] netif_receive_skb+0x3a7/0x421
[<ffffffff811fc2bb>] process_backlog+0x73/0xbb
[<ffffffff811fbd0b>] net_rx_action+0x9e/0x1c4
[<ffffffff8103c3f9>] __do_softirq+0x55/0xc3
[<ffffffff8100cf8c>] call_softirq+0x1c/0x28
[<ffffffff8100e3c0>] do_softirq+0x2c/0x85
[<ffffffff8103c35f>] irq_exit+0x3f/0x84
[<ffffffff8100e67c>] do_IRQ+0x13e/0x162
[<ffffffff8100c3a1>] ret_from_intr+0x0/0xa
<EOI>  [<ffffffff811709af>] acpi_idle_enter_bm+0x265/0x2d1
[<ffffffff811709a5>] acpi_idle_enter_bm+0x25b/0x2d1
[<ffffffff811de957>] cpuidle_idle_call+0x74/0xa9
[<ffffffff811de8e3>] cpuidle_idle_call+0x0/0xa9
[<ffffffff8100b12d>] cpu_idle+0x94/0xbc


Code: 41 8b 04 b0 31 04 b3 44 39 f7 72 f0 31 f6 31 ff 39 ee 73 4b
RIP  [<ffffffff8860e474>] :cisco_ipsec:CBCDecryptUpdate+0xb4/0x160
RSP <ffff81007f36f9b0>
CR2: 0000000000000000
---[ end trace b49f57c7b6417b0f ]---
Kernel panic - not syncing: Aiee, killing interrupt handler!