Compile a (CentOS) Kernel And IPTables With TPROXY Support

A default (CentOS) kernel doesn't have TPROXY support, which is needed if you want to it to behave as a transparant proxy. This is useful if you have a cluster set-up with one or more loadbalancers, but you still want each underlying node to see the original source IP from the request.

Without transparant proxy (TPROXY), all request would appear to come from the load balancer's IP address, instead of the actual client (which can mess up your logging, scripts, ...).

This article is derived from the "How To Compile a Kernel -- The CentOS way" pages at howtoforge.com, and the guides at LoadBalancer.org.

We will assume you have a running CentOS at this point (version 5.5 or later). First up, download the kernel source. Since the TPROXY patch isn't compatible with all Kernel versions, we're stuck with the 2.6.25 kernel. Please browse the Kernel Source Index, and pick the latest linux-2.6.25.* kernel work from. (if the version is no longer available for download, here's a local copy).

cd /usr/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.20.tar.gz

Now, unpack the source code, and symlink the resulting directory to 'linux' (will make things easier later on).

tar xzf linux-2.6.25.20.tar.gz
ln -s linux-2.6.25.20 linux

Next step is to download the TPROXY patch. (if the version is no longer available for download, here's a local copy)

wget http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2
tar xjf tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

Now to apply the patch mentioned.

cd linux/
cat ../tproxy-2.6.25-20080519/00* | patch -p1 --dry-run
cat ../tproxy-2.6.25-20080519/00* | patch -p1

To start compiling the kernel, we'll first clean any leftovers that may exist.

make clean && make mrproper

And copy our current Kernel configuration to our local .config file. This will help in making sure our currently installed applications will continue working, and we don't modify the kernel too heavily.

cp /boot/config-`uname -r` ./.config

Make sure you have all the necessary developer-tools, to compile the kernel.

yum install make rpm-build gcc gcc-c++ ncurses-devel elfutils elfutils-libs libstdc++-devel

Let's pop up the kernel configuration menu, which will allow us to easily change configs.

make menuconfig

Since we saved our currently running Kernel's configuration in the .config file, we will choose the "Load an alternative Configuration File" option, and enter .config as the filename.

menuconfig: load config

Let's enable TPROXY support. Navigate in the menu to

  • Networking
  • > Networking support
  • > Networking options
  • > Network packet filtering framework (Netfilter)
  • > Core Netfilter Configuration

and highlight:

Transparent proxying support (EXPERIMENTAL)
Netfilter Xtables support (required for ip_tables)
Netfilter Connection tracking
> Connection tracking flow accounting
> Connection mark tracking support
"TPROXY" target support (EXPERIMENTAL)
"socket" match support (EXPERIMENTAL)

Beware that you can check these options in 2 ways:

[*]: Built-in
[M]: Module

Try to select the options listed above as [M], so they are modules.

Then hit <ESC><ESC> a few times, to get back to the main menu, and navigate to General Setup > () Local version -- append to kernel release. Add a custom suffix there, to identify this kernel. I've choosen "-tproxy" as suffix (so I know it has tproxy support). It's important you add a version number if you try to reinstall the kernel again afterwards, or you'll end up with "this package is already installed" messages.

Once you've applied the above changes, exit the menu and confirm you want to save your changes.

Now start compiling the source and create the RPM install file.

make rpm

This _will_ take a long time. If you're running this inside a virtual machine, consider adding more (virtual) CPU's to speed up this process. It's safe to assume this will run for at least 1 hour, probably more.

After it's been created, you will find the resulting Source RPM file in /usr/src/redhat/SRPMS/.

lb02.lab.mojah.be linux $ ls -al /usr/src/redhat/SRPMS/
total 62124
drwxr-xr-x 2 root root     4096 Aug 21 17:31 .
drwxr-xr-x 7 root root     4096 Aug 21 16:20 ..
-rw-r--r-- 1 root root 63536285 Aug 21 17:27 kernel-2.6.25.20tproxy-1.src.rpm

And the binary RPM file in /usr/src/redhat/RPMS/i386/ (or x86_64 if you're running 64 bit).

lb02.lab.mojah.be linux $ ls -al /usr/src/redhat/RPMS/i386/
total 120256
drwxr-xr-x 2 root root      4096 Aug 21 17:32 .
drwxr-xr-x 9 root root      4096 Aug 21 16:20 ..
-rw-r--r-- 1 root root 123002989 Aug 21 17:31 kernel-2.6.25.20tproxy-1.i386.rpm

Now it's time to install our custom kernel.

rpm -ivh --nodeps /usr/src/redhat/RPMS/i386/kernel-2.6.25.20tproxy-1.i386.rpm

And create the ramdisk for our system.

mkinitrd /boot/initrd-2.6.25.20-tproxy.img 2.6.25.20-tproxy

Let's see what files were made in the /boot partition. We'll need these filenames to edit the grub config later on.

lb02.lab.mojah.be linux $ ls -alh /boot/ | grep -i tproxy
-rw-r--r--  1 root root   73K Aug 21 17:27 config-2.6.25.20-tproxy
-rw-------  1 root root  3.2M Aug 21 17:33 initrd-2.6.25.20-tproxy.img
-rw-r--r--  1 root root 1015K Aug 21 17:27 System.map-2.6.25.20-tproxy
-rw-r--r--  1 root root  1.9M Aug 21 17:27 vmlinuz-2.6.25.20-tproxy

And edit the menu-file.

vi /boot/grub/menu.lst

And add the following snippet below the "hiddenmenu" line, and right above the first kernel declaration. This consists of copying an already existing boot-item, and modify the vmlinuz and initrd locations.

title CentOS-Tproxy (2.6.25.20-tproxy)
root (hd0,0)
kernel /vmlinuz-2.6.25.20-tproxy ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.25.20-tproxy.img

The /vmlinuz and /initrd should point to the filenames you discovered earlier. Please don't directly copy/paste the example above, but copy an entry from your file, and modify it (as to preserve the hard disk order and volume names).

Now reboot into your newly created kernel. Your boot screen would look a bit like this now, with a notice to the newly named kernel.

tproxy kernel

You can verify this once the server's booted up.

lb02.lab.mojah.be ~ $ uname -a
Linux lb02.lab.mojah.be 2.6.25.20-tproxy #1 SMP Sat Aug 21 17:22:41 CEST 2010 i686 i686 i386 GNU/Linux

Now we have our kernel with TPROXY support running, time to compile and patch our iptables to make use of it. To get started, download 1.4.0 iptables source. It's import you take the 1.4.0 version, newer versions won't work. (if the version is no longer available for download, here's a local copy).

cd /usr/src/
wget http://www.netfilter.org/projects/iptables/files/iptables-1.4.0.tar.bz2
tar xjf iptables-1.4.0.tar.bz2

Now also download the tproxy iptables patch. (if the version is no longer available for download, here's a local copy)

wget http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-20080521-113954-1211362794.patch

Apply the tproxy path.

cd iptables-1.4.0/
cat ../tproxy-iptables-1.4*.patch | patch -p1
make && make install

Now you've installed both your kernel, and iptables, with the tproxy patch.

Downloads

iptables 1.4.0.tar.bz: full iptables source code
linux-2.6.25.20.tar.gz: full kernel 2.6.25.20 source code
tproxy-iptables-1.4.0-20080521.patch: tproxy patch for iptables
tproxy-kernel-2.6.25-20080519.tar.bz2: tproxy patch for 2.6.25 kernel

.config: the .config file I used to compile my kernel (with all necessary modules checked)
kernel-2.6.25.20tproxy.i386.rpm: rpm install file for a 32-bit (i386) kernel with tproxy support. Follow the "mkinitrd" steps above to install & use this kernel. The kernel is named "2.6.25.20-tproxyfourteen".

Troubleshooting: unknown match socket

If you've done the above steps, and still get the UNKNOWN match `socket' message in your iptables, you've probably skipped a kernel module required for this to work.

Troubleshooting: bad exit status during kernel compile

You could run into something similar to the following when compiling your kernel.

... [snip]
LD [M]  drivers/scsi/scsi_mod.o
LD      drivers/built-in.o
error: Bad exit status from /var/tmp/rpm-tmp.48540 (%build)

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.48540 (%build)
make[1]: *** [rpm] Error 1
make: *** [rpm] Error 2

That will prevent you kernel from successfully compiling. It usually means you selected a kernel option with a certain dependency that wasn't checked. So it's dependent on an option, that's not being compiled.

Tricky part here is to track down which one, and I'm afraid to say I don't know how. Also, it's beyond the scope of this document. If you want to retry the compilation again, make sure to run:

cp .config .config_backup
make clean && make mrproper

to reset your current attempt, clear created files and reset the config back to zero. Your "broken" config can then still be found in the .config_backup file. In my experience, it's better to just start all over ...

You could also consider deleting the generated files in /usr/src/redhat/BUILD/kernel-* as they are obsolete now.

Hi! This is a blog where I write down solutions to some of the problems I've faced when working as a sysadmin/dev. I hope you find the information shown here useful to you. Please use the comments on this blog to give feedback about the content!. I'm @mattiasgeniar on Twitter.

Tagged with:
Posted in Devops
18 comments on “Compile a (CentOS) Kernel And IPTables With TPROXY Support
  1. Reese says:

    Awesome tutorial, thanx! Just wish all howto’s was this simple…..

  2. Matt says:

    third time?

    I have followed your steps with no error until i get to this line:

    cat ../tproxy-2.6.25-20080519/00* | patch -p1 –dry-run

    After which, I am met with the folloing error:

    1 out 7 hunks FAILED — saving rejects to file to net/ipv4/netfilter/nf_conntrack_l3proto_ipv4.c.re

    This is my second time through. I didnt notice that error the first time. However, when got the step “make rpm” it errored out. It wasnt until I tried just plain “make && make modules_install” that I was able to find that it couldnt link to the file listed above.

    I have used all of the exact downloads you provided in your tutorial on the most recent version (as of this post) of CentOS 5.5.

    Any insight is appreciated.

    Thanks,
    Matt

  3. Matti says:

    Hi Matt,

    First things to doublecheck:
    - Kernel version you downloaded
    - Delete what you already did, and download a fresh copy (without previous ‘make’ history)

    You could also try to just download the very latest kernel, it should have (or so I’m told, but have not yet verified) TPROXY support by default, without having to go through the patch.

    Also, the patch only needs to be applied once. If you’re doing this a second time, on the same files, don’t re-apply the patch.

    Matti

  4. Matt says:

    yes sir….

    as stated in the post, i downloaded exactly what was posted in this tutorial by using the links and verified them, where possible, against the md5 files. the only possibility is if the links are not pointing to the appropriate versions.

    only applied patch once. second time i tried was from a brand new fresh install of OS.

  5. Matti says:

    Matt, have you tried just downloading the very latest Linux kernel? It should allow you to get by without applying the patch, as it has TPROXY built-in (the rest of the steps, to active it, still apply).

    I don’t immediately have an idea on where to search for your error, if you’ve followed the rest of the steps.

    Regards,
    M.

  6. Christian says:

    Excellent Tutorial. Followed all steps and I now have a new kernel. Thank you very much.

  7. Alan says:

    Brilliant tutoral. First time I’ve ever dabbled in installing a new kernel. Followed the steps and all went without a hitch!

  8. ibnu says:

    [root@CACHE ~]# mkinitrd /boot/initrd-2.6.25-tproxy.img 2.6.25-tproxy
    No modules available for kernel “2.6.25-tproxy”.

    why ?

  9. santosh says:

    ERROR: “tcp_splice_match” [net/netfilter/xt_socket.ko] undefined!
    ERROR: “tcp_splice_init” [net/netfilter/xt_socket.ko] undefined!
    ERROR: “tcp_splice_cleanup” [net/netfilter/xt_socket.ko] undefined!

    this error is coming while “make rpm”

  10. Ravi says:

    Hello,

    I am following your instructions and i get the following error while operating the command: make clean && make mrproper..

    Makefile:1523: *** mixed implicit and normal rules. Stop.

    Please kindly advise,,

  11. Ahmd says:

    dear All ,
    just simple question .
    now after performing the above steps .
    do i need to compile the squid with tproxy enable ?

    or

    i can just use yum install squid without any new steps ???

    with my best regards

  12. This answer sucks, but it depends: if the packaged yum version is up-to-date and compiled with tproxy support, it will work. If not, you’ll have to compile it yourself.

  13. Ahmd says:

    thnaks mattias

    but i want to ask

    what nest steps i need to do after the steps above

    i mean do i need to configure iptables or any thing ??
    plz support me with guide with step by step to operate tproxy with squid

    regards

    • This isn’t easy. In fact, if you rely on a step-by-step guide, I suggest to seek professional help. You’ll need to configure both iptables and your internal network to send all traffic referring to your webserver back to your transaparant proxy. If this sounds like chinese to you, I kindly suggest to talk to your hosting provider for help. Explaining this in a step-by-step blogpost would require several days of work, a luxury I don’t have at this moment.

      Good luck.

  14. Ahmd says:

    hi ,
    i understand you .
    im a good linux user but not professional in squid.
    now agian ,
    as a brief steps what i need to do next ?
    ================
    what will happen if i use ubuntu 12.4 last version instaed of centos .

    do i need compile kernel ?? or it is compiled by default?

    regards

  15. Ahmd says:

    hi matt ,
    i have somethig wrong in iptables :
    “”ip_conntrack_netbios_ns [FAILED]“”

    i tried steps aove on centos 5.9 32 bits !!

    ===========================

    [root@virus ~]# iptables -V
    iptables v1.4.0
    [root@virus ~]# serive iptables status
    -bash: serive: command not found
    [root@virus ~]# service iptables status
    Table: filter
    Chain INPUT (policy ACCEPT)
    num target prot opt source destination

    Chain FORWARD (policy ACCEPT)
    num target prot opt source destination

    Chain OUTPUT (policy ACCEPT)
    num target prot opt source destination

    [root@virus ~]# service iptables stop
    Flushing firewall rules: [ OK ]
    Setting chains to policy ACCEPT: filter [ OK ]
    Unloading iptables modules: [ OK ]
    [root@virus ~]# service iptables start
    Applying iptables firewall rules: [ OK ]
    Loading additional iptables modules: ip_conntrack_netbios_ns [FAILED]
    [root@virus ~]#

    any help ?

  16. Ahmd says:

    hi matt , i solved it :
    1-edit the “/etc/sysconfig/iptables-config” file
    2-Change the following two options to look like the following:
    =================
    Code:
    IPTABLES_MODULES=””
    IPTABLES_MODULES_UNLOAD=”no”
    =========================

    what squid version do u recommedn to use on centos 5.9 after patching ??
    regards

2 Pings/Trackbacks for "Compile a (CentOS) Kernel And IPTables With TPROXY Support"
  1. [...] squid , and it will work with tproxy ? regards plz have a look here about modifying the kernel http://mattiasgeniar.be/2010/09/01/c…proxy-support/ [...]

  2. Squid TPROXY says:

    [...] with those options, do I need to replace it? There is another article about TPROXY kernel support (http://mattiasgeniar.be/2010/09/01/c…proxy-support/) that suggests patching the kernel source code before building it, do I need this patch? If you [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

*

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>