http://proj.sunet.se/E2E/tcptune.html


 

TCP Tuning parameters for different OS:es
NOTE! The information on this page is for informational purposes only.
SUNET is not responsible for problems occuring when using this information.


The operating systems of today has good support for auto configuration of TCP. Good performance can be acheived just by changing the TCP window size. All operating systems we tested had much too low TCP window size as default to utilize a gigabit etehernet card, even on fairly short distances. We have also found out that only very late releases of the operating systems have good TCP stacks and drivers for the network cards, so upgrading to a fairly rescent OS is vital.

Below is a short description of what we did on the different operating systems we tested. In the table are some values shown with dark grey background - those values are usually the default, and thus not nesseceary to change.

We have used the following conventions:

[wmax] - maximum size (in bytes) of TCP window. 
[wstd] - standard size (in bytes) of TCP window. 
[wmin] - minimum size (in bytes) of TCP window.
[sbmax] - maximum allowed size of the system socket buffer 
The values of those parameters varies according to transmission speed and round-trip-time (RTT). Don't blindly set theese values too large, as they consume kernel memory, and too large values may cause degradation in performance.

The [sbmax] parameter should be set to a size larger than the maximum TCP window size.

The theoretical value (in bytes) for [wstd] is bps / 8 * rtt, where bps is bandwidth in bits/second and rtt=round-trip-time. You can use this Perl program that will determine the RTT and do the calculations for You..


--------------------------------------------------------------------------------

NetBSD
Only the NetBSD 2.0 (and later) release support really fast connections, as there has been much work done on the "zero-copy" TCP features and removal of  TCP stack linear searches.

Install NetBSD 2.0 
Enter the line "options NMBCLUSTERS=16384" in your kernel config file 
Recompile and install your new kernel. 
Do not try to set the values below too high until you have increased NMBCLUSTERS - risk of "hanging" the network part of the OS! 
Command
 Description
 
sysctl -w net.inet.tcp.rfc1323=1 Activate window scaling and timestamp options according to RFC 1323. 
sysctl -w kern.somaxkva=[sbmax] Set maximum size for all socket buffers together in the system 
sysctl -w kern.sbmax=[sbmax] Set maximum size of socket buffer for one TCP flow 
sysctl -w net.inet.tcp.recvspace=[wstd] Set default size of TCP receive window. 
sysctl -w net.inet.tcp.sendspace=[wstd] Set default size of TCP transmit window. 
sysctl kern.mbuf.nmbclusters View maximum number of mbuf clusters. Used for storage of data packets to/from the network interface. Can only be set by recompiling Your kernel - see above! 

You can enter the values to /etc/sysctl.conf to have them set at boot-time, as in this example.


--------------------------------------------------------------------------------

FreeBSD
We haven't been able to get as good results with FreeBSD as with NetBSD, which is mostly due to linear searching in buffer chains in the transmit/receive path. FreeBSD 5.2 though gives significantly better performance that the FreeBSD 4 releases.

Install FreeBSD 4.10-RELEASE or FreeBSD 5.2.1-RELEASE 
Apply this "netlion" patch. 
Enter the following line to /boot/loader.conf:
kern.ipc.nmbclusters="16384" # Set the number of mbuf clusters

Reboot your system. Do not try to set the values below too high until you have increased nmbclusters - risk of "hanging" the network part of the OS! 
Command
 Description
 
sysctl net.inet.tcp.rfc1323=1 Activate window scaling and timestamp options according to RFC 1323. 
sysctl ipc.maxsockbuf=[sbmax] Set maximum size of TCP window. 
sysctl net.inet.tcp.recvspace=[wstd] Set default size of TCP receive window. 
sysctl net.inet.tcp.sendspace=[wstd] Set default size of TCP transmit window. 
sysctl kern.ipc.nmbclusters View maximum number of mbuf clusters. Used for storage of data packets to/from the network interface. Can only be set att boot time - see above. 
sysctl net.inet.tcp.liondmask=7 Used to enable the "netlion" patch above. 

You can enter the values to /etc/sysctl.conf to have them set at boot-time, as in this example.



--------------------------------------------------------------------------------

Linux
The table below shows which parameters that may need to be changed. These are true for both 2.4 and 2.6 kernels. With theese changes, it is possible to get results in the same order as our NetBSD tests, with the exception that Linux will do a data copy in the transmit path so the transmitting machine will be more loaded.

Command
 Description
 
echo "1" > /proc/sys/net/ipv4/tcp_window_scaling Activate window scaling according to RFC 1323 
echo "1" > /proc/sys/net/ipv4/tcp_timestamps Activate timestamps according to RFC 1323 
echo [wmax] > /proc/sys/net/core/rmem_max Set maximum size of TCP receive window. 
echo [wmax] > /proc/sys/net/core/wmem_max Set maximum size of TCP transmit window. 
echo [wmax] > /proc/sys/net/core/rmem_default Set default size of TCP receive window.  
echo [wmax] > /proc/sys/net/core/wmem_default Set default size of TCP transmit window.  
echo "[wmin] [wstd] [wmax]" > /proc/sys/net/ipv4/tcp_rmem Set min, default, max receive window. Used by the autotuning function. 
echo "[wmin] [wstd] [wmax]" > /proc/sys/net/ipv4/tcp_wmem Set min, default, max transmit window. Used by the autotuning function. 
echo "bmin bdef bmax" > /proc/sys/net/ipv4/tcp_mem Set maximum total TCP buffer-space allocatable. Used by the autotuning function. 
ifconfig eth? txqueuelen 1000 Define length of transmit queue. Replace "?" with actual interface number. 

--------------------------------------------------------------------------------

Windows 2000/XP/2003
Tests performed with Windows 2003 give results similar to the NetBSD results. To achieve this we had to change some variables in the registry.

Registry entry
 Description
 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Tcp1323Opts=1 Turn on window scaling option 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize=[wmax]  Set maximum size of TCP window 

Further reading on Windows TCP tuning:

Windows tweaks 
Börje Josefsson 2004-06-07


========================================

#!/usr/bin/perl

# Name:
#       Tweak
#       Tweak parameters to tune tcp buffers
#
# Author:
#       Börje Josefsson   2003-03-04
#
# Function:
#       Calculate RTT, and compute buffer size needed for TCP
#
# Revision history:
#       Date            Who             Comment
#       2003-03-04      bj@sunet.se     Initial version
#
# Parameters:
#       -v      Verbose
#       -s      Set values locally
#       -b      Bandwidth (in Mbit/sec)
#       -d      Duration of ttcp test (in seconds)

##########
# Initializion values
##########

$numpings = 4;                     # Number of pings per probing
$buflen   = 61440;                 # 15 * pagesize on FreeBSD

require "getopts.pl";
&Getopts('vsb:d:') || die "Illegal option(s), see manual!\n";

###############
# Main program
###############

$server   = $ARGV[0];
$verbose  = $opt_v;
$bitrate  = $opt_b;             # in Mbit/sec
$duration = $opt_d;

$duration = 10 if ($duration < 10);
$duration = 60 if ($duration > 60);
$bitrate  = 10 if ($bitrate < 10);
$bitrate  = 2000 if ($bitrate > 2000);

&ping;

$buf=($RTTmax/1e+3) * $bitrate * 1e+6/8;    # bitrate -> bytes/sec * RTT
$mbuf=$buf/(1024*1024);

printf "RTTmax = %5.2f ms. Buffer size needed = %d bytes (%5.2f Mbytes)\n\n", $RTTmax, $buf, $mbuf;

$buf = int($buf * 1.25);      # Add 25% just in case...

print "The following settings can be used on the remote system:\n";
print "\tSyntax for NetBSD/FreeBSD, Your mileage may vary...\n\n";
print "sysctl -w net.inet.tcp.sendspace=$buf\n";
print "sysctl -w net.inet.tcp.recvspace=$buf\n";
print "\n\n";

if ($opt_s) {
  system "sysctl net.inet.tcp.sendspace=$buf";
  system "sysctl net.inet.tcp.recvspace=$buf";
}

$bytes_to_send = (($bitrate * 1e+6) / 8) * $duration;
$numbufs = int ($bytes_to_send / $buflen);

print "\tTTCP parameters, calculated for $duration seconds on $bitrate Mbit/sec:\n";
print "Sender:\n"; 
print "ttcp -s -t -f m -l $buflen -n $numbufs $server\n";
print "Receiver:\n";
print "ttcp -s -r -f m -l $buflen -n $numbufs\n";

###############
# Subroutines
###############

sub ping {
# Run 'ping' to calculate RTT
  print "RTT ping start\n" if $verbose;
  foreach (`/sbin/ping -n -q -c $numpings $server`) {
# Note - command syntax for FreeBSD, needs to be tweaked for other OS:es!!
    if (/^round-trip.*= (.*)\/(.*)\/(.*)\/.* ms/) {
      $RTTmin    = $1;
      $RTTavg    = $2;
      $RTTmax    = $3;
      $RTT    = sprintf "%5.2f/%5.2f/%5.2f", $RTTmin, $RTTavg, $RTTmax;
    }
  }
  print "RTT ping finished = $RTT\n" if $verbose;
}

===================================================
# Examples of values to add in /etc/sysctl.conf on FreeBSD
# Note that the actual values may vary according to Your setup.
#
# The following setting can only be set when the kernel is loaded
# and must be set in /boot/loader.conf - it cannot be set here!
#
## kern.ipc.nmbclusters="16384"        # Set the number of mbuf clusters
#
# IMPORTANT NOTE - that must be done BEFORE setting the values below,
# otherwise You will run out of mbufs!

kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendspace=3217968
net.inet.tcp.recvspace=3217968
net.inet.tcp.rfc1323=1
====================================================