DOC HOME SITE MAP MAN PAGES GNU INFO SEARCH PRINT BOOK
 
Setting Up and Using Secure IP (IPsec)

Setting Up and Using Secure IP (IPsec)

The IPsec (secure IP) protocol suite and associated tools provides the ability to encrypt and authenticate IP packets transmitted between cooperating hosts or subnets.

When IPsec is configured for a given communication path between hosts or subnets, most of the IP header and the entire data portion of each packet sent over the network is encrypted by the sending host, and decrypted by the receiving host. This is in contrast to non-IPsec packets, which are not encrypted.

In addition to encrypting IP packets, IPsec can authenticate each packet using the information in the expanded header supported by this protocol. Authentication can also be provided using private keys and signed certificates.

In this way, each host can not only be assured that each packet has been encrypted for delivery, but also validate that the packet received has originated with the expected host and that no third party has tampered with or had access to the data in the packets during transmission.

One of the uses of IPsec is to implement a Virtual Private Network (VPN). In a VPN, a non-secure communication path (such as an internet connection) is used for the transmission of encrypted and authenticated packets between hosts that have been set up to use that path and only provide IPsec packets over the path.

A VPN is really a set of security associations established on each host that requires secure IP communications, along with a security policy established for each "subnet" in the VPN. Thus, a corporate VPN might be defined by a gateway router that allows a number of remote systems (or other gateways) to connect over public transmission facilities (phone lines, cable modem, wireless), and access the corporate network. A properly configured IPsec facility on the gateways and the various remote systems prevents the kind of security threats inherent in public transmission systems, such as spoofing, masquerading, denial of service, and others.

For an overview of IPsec, see Security Architecture for the Internet Protocol(rfc2401).

Prerequisites

IPsec requires the openssl package, OpenSSL version 0.9.7 or later.

The nd and nics packages include an updated version of the tcpdump command, used to confirm that packets are being encrypted as you intend between hosts.

Enabling IPsec

IPsec is disabled by default. Note that it is important to configure IPsec as soon as it is enabled. If IPsec is enabled without being setup (or if it is setup incorrectly), overall system throughput will most likely be reduced due to the extra processing in the kernel required for IPsec.

To enable IPsec, do the following:

  1. Edit the file /etc/conf/pack.d/inet/space.c.

  2. Change the line reading:
       ipsec_enable = 0
    
    to read as follows:
       ipsec_enable = 1
    

    Save the file.

  3. Rebuild the inet driver:
       idbuild -M inet
    

  4. Reboot the system:
       shutdown -i6 -g0 -y
    

Once the system comes back up, log in as root to configure IPsec.

Configuring IPsec

IPsec is configured using the /usr/sbin/setkey and /usr/sbin/racoon commands. Note that ipseckey can be used as an alias for /usr/sbin/setkey, to avoid confusion with the /bin/setkey command (used to configure terminal function keys). In the examples below, we use ipseckey.

setkey/ipseckey configures the SAD and SPD databases used by IPsec
racoon configures automatic negotiation of encryption keys and certificates used by hosts over IPsec; also configures SAD entries appropriate for auto-negotiation
racoon.conf configuration file for racoon

Basically, you use setkey (and its many internal commands) to tell the kernel which packets must be sent via IPsec, and to configure IPsec options for each host, range of hosts, subnet, etc., to which you want secure communication.

The racoon command is used to provide automatic encryption key and certificate management between hosts. Using racoon is optional, yet recommended. By automating the process of periodically changing and synchronizing keys and certificates within the set of hosts you define for IPsec, you not only reduce maintenance, but also reduce the likelihood that a key will be compromised.

The following procedure outlines the steps to follow to configure basic IPsec between two systems. The steps are illustrated with examples used to configure IPsec between two systems on the same subnet. This is a simple example, and touches only the basics of IPsec usage, but will help you understand how IPsec works and what general steps you need to follow to configure IPsec in any environment. A bit of information on debugging IPsec is included.

Also see the references in the section ``Additional IPsec Documentation'' for more help configuring IPsec.

  1. If your setup includes one or more routers on either or both ends of the communication path, be sure that routing is set up correctly on each end. This is usually the case with VPN configurations. Router configuration is documented in the online documentation set under /Administering TCP/IP and Internet services.

    The example we use in the steps that follow is a simple IPsec tunnel between two hosts (relent.iii.com and elmer.iii.com), so routing setup is not necessary (in fact, routing is disabled on these systems).

  2. The ESP (Encapsulating Security Payload) packets used in secure IP connections are not the same as the packets that firewalls are typically configured by default to pass through (e.g., TCP, UDP, ICMP, and other packets). As defined in rfc2406, the protocol header (IPv4, IPv6, or Extension) immediately preceding the ESP header contains the value 50 in its Protocol (IPv4) or Next Header (IPv6, Extension) field. If an IPsec connection passes through a firewall at either end, each firewall needs to be configured to let ESP (IP Protocol 50) packets pass through to the hosts you are configuring for IPsec. The configuration necessary is particular to your firewall setup.

  3. Start tcpdump in a desktop window so you can watch network packets as you work on your configuration. Do this on each system that you want to configure for IPsec. Enter:
       tcpdump
    

    The command should attach to the first network interface it finds (/dev/net0); if your network interface is on another device, specify that device with the -i option of tcpdump. See tcpdump(ADM). Try logging in to the other machine via telnet or ssh, and observe the tcpdump output. It should look something like this:

       00:05:06.620839 relent.iii.com.32805 > elmer.iii.com.22: P 22147699:22147747(48) ack 16115521 win 25038 <nop,nop,timestamp 2088362 2549559> [tos 0x10]
       00:05:06.623762 relent.iii.com.32805 > elmer.iii.com.22: P 0:48(48) ack 1 win 25038 <nop,nop,timestamp 2088362 2549559> [tos 0x10]
       00:05:06.623847 elmer.iii.com.22 > relent.iii.com.32805: P 1:49(48) ack 48 win 24820 <nop,nop,timestamp 2562612 2088362> [tos 0x10]
       00:05:06.820200 relent.iii.com.32805 > elmer.iii.com.22: . ack 49 win 25038 <nop,nop,timestamp 2088382 2562612> [tos 0x10]
       00:05:06.820315 relent.iii.com.32805 > elmer.iii.com.22: . ack 49 win 25038 <nop,nop,timestamp 2088382 2562612> [tos 0x10]
       00:05:09.220973 relent.iii.com.32805 > elmer.iii.com.22: P 48:96(48) ack 49 win 25038 <nop,nop,timestamp 2088622 2562612> [tos 0x10]
       00:05:09.221106 relent.iii.com.32805 > elmer.iii.com.22: P 48:96(48) ack 49 win 25038 <nop,nop,timestamp 2088622 2562612> [tos 0x10]
       00:05:09.227721 elmer.iii.com.22 > relent.iii.com.32805: P 49:97(48) ack 96 win 24820 <nop,nop,timestamp 2562872 2088622> [tos 0x10]
       00:05:09.420154 relent.iii.com.32805 > elmer.iii.com.22: . ack 97 win 25038 <nop,nop,timestamp 2088642 2562872> [tos 0x10]
       00:05:09.426195 relent.iii.com.32805 > elmer.iii.com.22: . ack 97 win 25038 <nop,nop,timestamp 2088642 2562872> [tos 0x10]
       00:05:36.990958 relent.iii.com.32805 > elmer.iii.com.22: P 96:144(48) ack 97 win 25038 <nop,nop,timestamp 2091399 2562872> [tos 0x10]
       00:05:37.010744 relent.iii.com.32805 > elmer.iii.com.22: P 96:144(48) ack 97 win 25038 <nop,nop,timestamp 2091399 2562872> [tos 0x10]
       00:05:37.019121 elmer.iii.com.22 > relent.iii.com.32805: P 97:1009(912) ack 144 win 24820 <nop,nop,timestamp 2565648 2091399> [tos 0x10]
       00:05:37.190131 relent.iii.com.32805 > elmer.iii.com.22: . ack 1009 win 25038 <nop,nop,timestamp 2091419 2565648> [tos 0x10]
       00:05:37.190216 relent.iii.com.32805 > elmer.iii.com.22: . ack 1009 win 25038 <nop,nop,timestamp 2091419 2565648> [tos 0x10]
    

  4. Configure IPsec on one of the hosts; in this example, we start with the system elmer.

    [Note: If you are configuring any hosts via remote login (ssh, telnet, etc.), log into all remote systems before configuring IPsec. Then, configure the remote hosts first; configure the local host last.]

    In this example, we are configuring the host relent (10.0.0.100) to use IPsec on all TCP packets exchanged with the host elmer (10.0.0.110) on a local network. On the system elmer, do the following:

    1. Create a file named /etc/inet/elmer.ipsec.cf with these contents:
         # associate esp with the two systems -- add SAD entries
         

      add 10.0.0.100 10.0.0.110 esp 0x10001 -m tunnel -E 3des-cbc "thescogp12341234thescogp" ; add 10.0.0.110 10.0.0.100 esp 0x10002 -m tunnel -E 3des-cbc "thescogp43214321thescogp" ;

      # associate ah with the two systems -- add SAD entries

      add 10.1.1.100 10.1.1.110 ah 0x10003 -m tunnel -A hmac-md5 "scogroupscogroup";

      add 10.1.1.110 10.1.1.100 ah 0x10004 -m tunnel -A hmac-md5 "groupscogroupsco";

      # set the security policy for the systems -- add SPD entries # must reverse "in" and "out" on these lines on the other host

      spdadd 10.0.0.100[any] 10.0.0.110[any] tcp -P in ipsec esp/tunnel/10.0.0.100-10.0.0.110/use ah/tunnel/10.0.0.100-10.0.0.110/use ; spdadd 10.0.0.110[any] 10.0.0.100[any] tcp -P out ipsec esp/tunnel/10.0.0.110-10.0.0.100/use ah/tunnel/10.0.0.110-10.0.0.100/use ;

    2. Enter:
         ipseckey -f /etc/inet/relent.ipsec.cf
      

    The commands in /etc/inet/relent.ipsec.cf define the SAD and SDP entries that the kernel uses to define the address ranges for which IPsec is configured, and to set the encryption and authentication policies for those addresses.


    addresses
    See the setkey(ADM) manual page for an explanation of address notation used in these commands.

    encryption - ESP
    There are two choices: esp which enables ESP based on rfc2405; and, esp-old which enables ESP based on rfc1827.

    authentication - AH
    There are two choices: ah which enables AH based on rfc2402; and, ah-old which enables AH based on rfc1826.

    mode
    This connection is set up to use tunnel mode, as indicated in the command lines with the -m tunnel option. The default mode is transport mode. The modes differ in the structure of the IP packet used.

    Transport mode uses an unencrypted IP header on each packet, and encrypts the packet data (sometimes called the payload). It is normally used when setting up IPsec in direct peer-to-peer connections (as in this example; we use tunnel mode in the example only to illustrate how to set it up).

    Tunnel mode encrypts both IP header and data, and adds a new (unencrypted) IP header to the packet. It is normally used in situations where a gateway system is forwarding packets over a public transmission line. The unencrypted portion of the packet reveals only the gateway system's address, not the address of the originating system, which is contained in the encrypted portion of the packet.


    encryption algorithms
    The available ESP algorithms (3des-cbc in the example) and AH algorithms (hmac-md5 in the example) are listed on the setkey(ADM) manual page.

    keys
    This example uses manually set encryption keys in the SAD entries. These are set in the add commands for ESP and AH. Manually set keys need to be changed periodically to reduce the likelihood that someone can guess or otherwise determine the keys being used. Keys must be changed at the same time on each system that uses the manually set keys. See ``Changing Manually Set IPsec Keys''.

    The preferred method of managing keys is to use racoon to periodically change keys on all hosts. This procedure does not use racoon, for simplicity. For examples of using racoon to manage packet authentication, see the sections ``Configuring racoon for Automatic Shared Public Key Management'' and ``Configuring racoon for Private Keys and Certificates''.

    The spdadd command specifies the security policy to be enforced on incoming and outgoing packets, for particular address ranges. In our example, the spdadd commands specify that incoming and outgoing TCP packets between 10.0.0.100 and 10.0.0.101 will have an ESP (encruption) header, followed by an AH (authentication) header, in tunnel mode (see above).

    The spdadd command insludes the protocol or protocols for which IPsec processing is enabled, tcp in our example. Any of the protocols specified in the file /etc/protocols can be used. Use the keyword all in place of tcp on the spdadd commands above to enable IPsec processing on all packets; note that this may significantly slow network traffic.

    The use keyword indicates that a Security Association (setup with the add command, above) will be used on packets matching the Security Policy (setup with spdadd) if one exists; otherwise, normal packet processing will be performed. A Security Association is required if you use the require keyword, as shown in the other examples below.

  5. Attempt to log in to the system relent from the system elmer. The login attempt should fail. The tcpdump output from these attempts should look like this:
       00:16:29.650189 relent.iii.com.32805 > elmer.iii.com.22: P 2208:2256(48) ack 3105 win 25038 <nop,nop,timestamp 2156665 2628439> [tos 0x10]
       00:16:29.650308 relent.iii.com.32805 > elmer.iii.com.22: P 2208:2256(48) ack 3105 win 25038 <nop,nop,timestamp 2156665 2628439> [tos 0x10]
       00:16:29.670680 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x14) [tos 0x10]
       00:16:31.570241 relent.iii.com.32805 > elmer.iii.com.22: P 2208:2256(48) ack 3105 win 25038 <nop,nop,timestamp 2156857 2628439> [tos 0x10]
       00:16:31.590436 relent.iii.com.32805 > elmer.iii.com.22: P 2208:2256(48) ack 3105 win 25038 <nop,nop,timestamp 2156857 2628439> [tos 0x10]
       00:16:31.610063 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x15) [tos 0x10]
       00:16:33.490198 relent.iii.com.32805 > elmer.iii.com.22: P 2208:2256(48) ack 3105 win 25038 <nop,nop,timestamp 2157049 2628439> [tos 0x10]
       00:16:33.490316 relent.iii.com.32805 > elmer.iii.com.22: P 2208:2256(48) ack 3105 win 25038 <nop,nop,timestamp 2157049 2628439> [tos 0x10]
       00:16:33.510561 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x16) [tos 0x10]
    

    Note that ESP (encrypted) packets are being sent from elmer to relent, but unencrypted packets continue to be sent from relent to elmer.

  6. Configure the other system (relent).

    1. Create a file named /etc/inet/relent.ipsec.cf with these contents:
         # associate esp with the two systems -- add SAD entries
         

      add 10.0.0.100 10.0.0.110 esp 0x10001 -m tunnel -E 3des-cbc "thescogp12341234thescogp" ; add 10.0.0.110 10.0.0.100 esp 0x10002 -m tunnel -E 3des-cbc "thescogp43214321thescogp" ;

      # associate ah with the two systems -- add SAD entries

      add 10.1.1.100 10.1.1.110 ah 0x10003 # -m tunnel -A hmac-md5 "scogroupscogroup";

      add 10.1.1.110 10.1.1.100 ah 0x10004 # -m tunnel -A hmac-md5 "groupscogroupsco";

      # set the security policy for the systems -- add SPD entries # must reverse "in" and "out" on these lines on the other host

      spdadd 10.0.0.100[any] 10.0.0.110[any] tcp -P out ipsec esp/tunnel/10.0.0.100-10.0.0.110/use ah/tunnel/10.0.0.100-10.0.0.110/use ; spdadd 10.0.0.110[any] 10.0.0.100[any] tcp -P in ipsec esp/tunnel/10.0.0.110-10.0.0.100/use ah/tunnel/10.0.0.110-10.0.0.100/use ;

      Note that the only difference between relent.ipsec.cf and elmer.ipsec.cf is the spdadd entries, which have the keywords "out" and "in" reversed. Otherwise, the files are exactly the same.

    2. Enter:
         ipseckey -f /etc/inet/relent.ipsec.cf
      

  7. Try netscape, ssh, etc., between the two machines to generate some traffic. The tcpdump output should now look something like this:
       00:29:18.649114 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x13b)
       00:29:18.650545 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x13c)
       00:29:18.652723 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x14)
       00:29:18.652786 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x14)
       00:29:18.652843 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x13d)
       00:29:18.652899 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x13e)
       00:29:18.652949 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x15)
       00:29:18.652999 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x15)
       00:29:18.653053 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x13f)
       00:29:18.654697 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x140)
       00:29:18.654761 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x16)
       00:29:18.654810 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x16)
       00:29:18.657405 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x141)
       00:29:18.724816 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x17)
       00:29:18.724920 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x17)
       00:29:18.733677 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x142)
       00:29:18.735721 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x143)
       00:29:18.740320 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x18)
       00:29:18.740409 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x18)
       00:29:18.877759 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x19)
       00:29:18.877863 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x19)
       00:29:18.893063 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x144)
       00:29:18.894040 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x145)
       00:29:18.894101 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x1a)
       00:29:18.894151 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x1a)
       00:29:18.894206 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x146)
       00:29:18.896189 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x147)
       00:29:18.896259 elmer.iii.com > relent.iii.com: ESP(spi=0x00010002,seq=0x148)
       00:29:18.896310 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x1b)
       00:29:18.896359 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x1b)
       00:29:19.092980 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x1c)
       00:29:19.094512 relent.iii.com > elmer.iii.com: ESP(spi=0x00010001,seq=0x1c)
    

    No details about the packets are displayed, since they are encrypted.

  8. Enter:
       netstat -nsp ipsec
    

    on either machine, to check IPsec protocol statistics. The output should look something like the following:

       ipsec:
               406 inbound packets processed successfully
               0 inbound packets violated process security policy
               351 inbound packets with no SA available
               0 invalid inbound packets
               0 inbound packets failed due to insufficient memory
               0 inbound packets failed getting SPI
               0 inbound packets failed on AH replay check
               0 inbound packets failed on ESP replay check
               0 inbound packets considered authentic
               0 inbound packets failed on authentication
               ESP input histogram:
                       3des-cbc: 406
               582 outbound packets processed successfully
               0 outbound packets violated process security policy
               0 outbound packets with no SA available
               0 invalid outbound packets
               0 outbound packets failed due to insufficient memory
               0 outbound packets with no route
               ESP output histogram:
                       3des-cbc: 582
    

    For more information on netsat, see the netstat(ADM) manual page.

  9. Once you have your setup working, you will want to change the debug line in /etc/inet/racoon.conf, to reduce the amount of messages sent to /var/adm/syslog (and /var/adm/log/osmlog). We recommend you use:
       log notify ;
    

Changing Manually Set IPsec Keys

To change IPsec keys set manually in SAD entries, you must:

  1. Remove the SAD entries containing the keys from all systems. See ``Disabling or Reconfiguring IPsec'' for how to remove SAD entries.

  2. Add new SAD entries containing the new keys. Follow the instructions in the section ``Configuring IPsec''.

Remember that the above must be done on each host that uses the given SAD entry keys.

Configuring racoon for Automatic Shared Public Key Management

This configuration uses simple shared public keys kept in a plain text file (/etc/inet/psk.txt) and racoon to provide packet encryption and authentication. Once the intital key is exchanged, racoon manages key updates automatically between systems.

The protocol used by racoon is known as the Internet Key Exchange (IKE) protocol, a hybrid protocol that uses elements of two other protocols (ISAKMP and Oakley) to achieve secure key exchange between hosts.

At a minimum, IKE includes the following attributes:

The above are defined during Phase 1 of IKE negotiation, as seen in the example below. Phase 1 establishes the secure communication channel. During Phase 2, attributes are defined for the various services to be supported over the channel.

Multiple channels and services can be configured in racooon.conf. See the racoon.conf(M) and RFC2409 for more information on the syntax of racoon.conf and IKE.

  1. On the system elmer (10.0.0.110), set the initial public shared key in /etc/inet/psk.txt, as in this example:
       10.0.0.100      mekmitasdigoat
    

  2. Create the file /etc/inet/elmer.ipsec.conf, with the following contents:
       spdadd 10.0.0.100[any] 10.0.0.110[any] tcp -P in ipsec
               esp/tunnel/10.0.0.100-10.0.0.110/require ;
       spdadd 10.0.0.110[any] 10.0.0.100[any] tcp -P out ipsec
               esp/tunnel/10.0.0.110-10.0.0.100/require;
    

    In this example, no add commands are needed to establish Security Associations in the ipsec.conf files on either system, because we are going to use racoon to manage encryption and authentication keys. The Security Associations for configurations that use racoon appear in the racoon configuration file (see the next step).

    Note the use of the keyword require, rather than the keyword use (as in the previous example). The require keyword indicates that when this Security Policy matches a packet, a Security Association will be required to process the packet.

  3. Create /etc/inet/racoon.conf, with these contents:
       path pre_shared_key "/etc/inet/psk.txt" ;
       log debug;
       

    remote anonymous { exchange_mode aggressive ; my_identifier address 10.0.0.110 ; lifetime time 1 hour ; # sec,min,hour

    # phase 1 proposal (for ISAKMP SA) proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method pre_shared_key ; dh_group 2 ; }

    # the following makes racoon (as a responder) obey the # initiator's lifetime and PFS group proposal. # this makes testing easier. proposal_check obey; }

    sainfo anonymous { pfs_group 2; lifetime time 10 hour ; encryption_algorithm 3des, blowfish; authentication_algorithm hmac_sha1, hmac_md5 ; compression_algorithm deflate ; }

    Basically, the remote statement Phase 1 SA and key negotiation; the anonymous keyword specifies that this specification applies to all packets (or, if there are multiple remote statements, anonymous matches any packets that don't match any other remote statement). The sainfo statement defines Phase 2 negotiation.

    See the racoon.conf(M) manual page for an explanation of the file's syntax.

  4. Enter:
       tail -f /var/adm/syslog
    

  5. Start racoon:
       /usr/sbin/racoon &
    

    Check the syslog output from the previous step. Correct errors as necessary if racoon fails to start. To restart racoon, it may be necessary to delete the file /tmp/.racoon, as shown:

       rm -f /tmp/.racoon
    

    Successful startup is indicated by messages like the following in /var/adm/syslog:

       Nov 12 12:46:45 relent racoon: INFO: main.c:176: @(#)racoon 20001216 20001216 sakane@kame.net
       Nov 12 12:46:45 relent racoon: INFO: main.c:177: @(#)This product linked OpenSSL 0.9.7b 10 Apr 2003 (http://www.openssl.org/)
       Nov 12 12:46:45 relent racoon: DEBUG: algorithm.c:612: hmac(modp1024)
       Nov 12 12:46:45 relent racoon: DEBUG: pfkey.c:2247: compression algorithm can not be checked because sadb message doesn't support it.
       Nov 12 12:46:45 relent racoon: DEBUG: admin.c:478: open /tmp/.racoon (fd 15) as racoon management.
       Nov 12 12:46:45 relent racoon: DEBUG: grabmyaddr.c:595: socket(PF_ROUTE) suceeded: fd 16
       Nov 12 12:46:45 relent racoon: DEBUG: grabmyaddr.c:324: my interface: 127.0.0.1 (lo0)
       Nov 12 12:46:45 relent racoon: DEBUG: grabmyaddr.c:324: my interface: 10.0.0.100 (net0)
       Nov 12 12:46:45 relent racoon: DEBUG: grabmyaddr.c:476: configuring default isakmp port.
       Nov 12 12:46:45 relent racoon: DEBUG: grabmyaddr.c:498: 2 addrs are configured successfully
       Nov 12 12:46:45 relent racoon: INFO: sockmisc.c:635: setsockopt_bypass: in bypass
       Nov 12 12:46:45 relent racoon: INFO: sockmisc.c:660: setsockopt_bypass: out bypass
       Nov 12 12:46:45 relent racoon: INFO: isakmp.c:1368: 10.0.0.100[500] used as isakmp port (fd=17)
       Nov 12 12:46:45 relent racoon: INFO: sockmisc.c:635: setsockopt_bypass: in bypass
       Nov 12 12:46:45 relent racoon: INFO: sockmisc.c:660: setsockopt_bypass: out bypass
       Nov 12 12:46:45 relent racoon: INFO: isakmp.c:1368: 127.0.0.1[500] used as isakmp port (fd=18)
       Nov 12 12:46:45 relent racoon: DEBUG: pfkey.c:195: get pfkey X_SPDDUMP message
       Nov 12 12:46:45 relent racoon: DEBUG: pfkey.c:195: get pfkey X_SPDDUMP message
       Nov 12 12:46:45 relent racoon: DEBUG: policy.c:184: sub:8047028: 10.0.0.100/32[0] 10.0.0.110/32[0] proto=tcp dir=out
       Nov 12 12:46:45 relent racoon: DEBUG: policy.c:185: db :809a680: 10.0.0.110/32[0] 10.0.0.100/32[0] proto=tcp dir=in
    

  6. Once racoon has successfully started, do the above steps on the other machine; in this example, relent (10.0.0.100).

    The file /etc/inet/psk.txt on relent should look like this:

       10.0.0.110      mekmitasdigoat
    

    The file /etc/inet/relent.ipsec.conf should look like this:

       spdadd 10.0.0.100[any] 10.0.0.110[any] tcp -P out ipsec
               esp/tunnel/10.0.0.100-10.0.0.110/require ;
       spdadd 10.0.0.110[any] 10.0.0.100[any] tcp -P in ipsec
               esp/tunnel/10.0.0.110-10.0.0.100/require;
    

    The file /etc/inet/relent.racoon.conf should look like this:

       

    path pre_shared_key "/etc/inet/psk.txt" ;

    log debug;

    remote anonymous { exchange_mode aggressive ; my_identifier address 10.0.0.100 ; lifetime time 1 hour ; # sec,min,hour

    # phase 1 proposal (for ISAKMP SA) proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method pre_shared_key ; dh_group 2 ; }

    # the configuration makes racoon (as a responder) to obey the # initiator's lifetime and PFS group proposal. # this makes testing so much easier. proposal_check obey; }

    sainfo anonymous { pfs_group 2; lifetime time 10 hour ; # lifetime byte 50 MB ; encryption_algorithm 3des, blowfish; authentication_algorithm hmac_sha1, hmac_md5 ; compression_algorithm deflate ; }

  7. Once racoon is running on both systems, generate some traffic between the systems using ssh, telnet, netscape, etc. Then, on both systems, check the output in /var/adm/syslog. Whenever a new key is generated and shared by racoon on the two systems, messages like the following appear:
       Nov 12 13:43:42 relent racoon: INFO: isakmp.c:1610: phase2 sa expired (invalid)-(invalid)
       Nov 12 13:43:43 relent racoon: INFO: isakmp.c:1641: phase2 sa deleted (invalid)-(invalid)
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:195: get pfkey ACQUIRE message
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:1526: suitable outbound SP found: 10.0.0.100/32[0] 10.0.0.110/32[0] proto=tcp dir=out.
       Nov 12 13:45:31 relent racoon: DEBUG: policy.c:184: sub:8046ffc: 10.0.0.110/32[0] 10.0.0.100/32[0] proto=tcp dir=in
       Nov 12 13:45:31 relent racoon: DEBUG: policy.c:185: db :809a680: 10.0.0.110/32[0] 10.0.0.100/32[0] proto=tcp dir=in
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:1542: suitable inbound SP found: 10.0.0.110/32[0] 10.0.0.100/32[0] proto=tcp dir=in.
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:1581: new acquire 10.0.0.100/32[0] 10.0.0.110/32[0] proto=tcp dir=out
       Nov 12 13:45:31 relent racoon: DEBUG: sainfo.c:100: anonymous sainfo selected.
       Nov 12 13:45:31 relent racoon: DEBUG: proposal.c:825:  (proto_id=ESP spisize=4 spi=00000000 spi_p=00000000 encmode=Tunnel reqid=0:0)
       Nov 12 13:45:31 relent racoon: DEBUG: proposal.c:859:   (trns_id=3DES encklen=0 authtype=2)
       Nov 12 13:45:31 relent racoon: DEBUG: proposal.c:859:   (trns_id=3DES encklen=0 authtype=1)
       Nov 12 13:45:31 relent racoon: DEBUG: proposal.c:859:   (trns_id=BLOWFISH encklen=128 authtype=2)
       Nov 12 13:45:31 relent racoon: DEBUG: proposal.c:859:   (trns_id=BLOWFISH encklen=128 authtype=1)
       Nov 12 13:45:31 relent racoon: DEBUG: remoteconf.c:129: anonymous configuration selected for 10.0.0.110.
       Nov 12 13:45:31 relent racoon: INFO: handler.c:163: ph1 local addr 10.0.0.100[500]
       Nov 12 13:45:31 relent racoon: INFO: handler.c:165: arg local addr 10.0.0.100[0]
       Nov 12 13:45:31 relent racoon: INFO: handler.c:167: ph1 remote addr 10.0.0.110[500]
       Nov 12 13:45:31 relent racoon: INFO: handler.c:169: arg remote addr 10.0.0.110[0]
       Nov 12 13:45:31 relent racoon: DEBUG: isakmp.c:1724: begin QUICK mode.
       Nov 12 13:45:31 relent racoon: DEBUG: isakmp.c:947: ===
       Nov 12 13:45:31 relent racoon: DEBUG: isakmp.c:948: begin QUICK mode.
       Nov 12 13:45:31 relent racoon: INFO: isakmp.c:952: initiate new phase 2 negotiation: 10.0.0.100[0]<=>10.0.0.110[0]
       Nov 12 13:45:31 relent racoon: DEBUG: oakley.c:2550: compute IV for phase2
       Nov 12 13:45:31 relent racoon: DEBUG: oakley.c:2551: phase1 last IV:
       Nov 12 13:45:31 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:32 relent Nov 12 13:45:3120f905a0 58df31e4 a3b13ba3
       Nov 12 13:45:31 relent racoon: DEBUG: algorithm.c:254: hash(sha1)
       Nov 12 13:45:31 relent racoon: DEBUG: algorithm.c:384: encription(3des)
       Nov 12 13:45:31 relent racoon: DEBUG: oakley.c:2583: phase2 IV computed:
       Nov 12 13:45:31 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:32 relent Nov 12 13:45:31be26f490 86a556eb
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:792: call pfkey_send_getspi
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:805: pfkey GETSPI sent: ESP/Tunnel 10.0.0.110[0]->10.0.0.100[0]
       Nov 12 13:45:31 relent racoon: DEBUG: isakmp_quick.c:131: pfkey getspi sent.
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:195: get pfkey GETSPI message
       Nov 12 13:45:31 relent racoon: DEBUG: pfkey.c:876: pfkey GETSPI succeeded: ESP/Tunnel 10.0.0.110[0]->10.0.0.100[0] spi=230166657(0xdb81081)
       Nov 12 13:45:31 relent racoon: DEBUG: algorithm.c:612: hmac(modp1024)
       Nov 12 13:45:31 relent last message repeated 8 times
       Nov 12 13:45:31 relent racoon: DEBUG: oakley.c:256: compute DH's private.
       Nov 12 13:45:31 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:33 relent Nov 12 13:45:3159be842f f5290870 c44b6c87 eda44992 6da70434 6d6a585d 89771721 dfa39b64
       Nov 12 13:45:33 relent Nov 12 13:45:3196e9394f ca1ee345 19f2af3b 9c52768b 0853a38e 3177879d 6e83c717 39268af6
       Nov 12 13:45:33 relent Nov 12 13:45:31aeeaaf01 e57713cc 5b5ac68b 73763d17 6d246bd8 c108656c 82fcec64 0bf46c7c
       Nov 12 13:45:33 relent Nov 12 13:45:31414ad3cb 520c032b b3a2d699 802fe007 e011e2da f8d83dfb 061c6256 99c6eb5d
       Nov 12 13:45:31 relent racoon: DEBUG: oakley.c:258: compute DH's public.
       Nov 12 13:45:31 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:33 relent Nov 12 13:45:315ee7b9c9 c97c8d43 97dcd2fd 69b50179 98c02cda 2f3e9be7 9d00f2b9 8056078f
       Nov 12 13:45:33 relent Nov 12 13:45:310eba690b d5b3a394 08e340ea 37decc57 73e575b4 c5f1bf28 3c76dc87 a4a0dc16
       Nov 12 13:45:33 relent Nov 12 13:45:31cb549551 8c51e521 ef66c9a4 094fc3f2 8ab048ef e414b0c2 abd03017 c68440c5
       Nov 12 13:45:33 relent Nov 12 13:45:317117b7c3 e0f2a13f 6df8120c 031634a2 cb47fc90 757cd9ea bb7ad251 219eff9a
       Nov 12 13:45:32 relent racoon: DEBUG: proposal.c:859:   (trns_id=BLOWFISH encklen=128 authtype=2)
       Nov 12 13:45:32 relent racoon: DEBUG: proposal.c:859:   (trns_id=BLOWFISH encklen=128 authtype=1)
       Nov 12 13:45:32 relent racoon: DEBUG: ipsec_doi.c:1012: matched
       Nov 12 13:45:32 relent racoon: DEBUG: isakmp.c:746: ===
       Nov 12 13:45:32 relent racoon: DEBUG: isakmp_quick.c:564: HASH(3) generate
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:692: HASH with:
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:33 relent Nov 12 13:45:3200a3b13b a3b0aa39 deb02e5d 747c5136 6049d1e6 988069b2 71dac5bb 56c78812
       Nov 12 13:45:33 relent Nov 12 13:45:32df96c750 e8
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:702: HASH computed:
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:33 relent Nov 12 13:45:3208b5e7c3 46ed883b a27b045d c83b87ae 0a84ebe0
       Nov 12 13:45:32 relent racoon: DEBUG: isakmp.c:2135: add payload of len 20, next type 0
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:2749: begin encryption.
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:384: encription(3des)
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:2765: pad length = 8
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:33 relent Nov 12 13:45:3200000018 08b5e7c3 46ed883b a27b045d c83b87ae 0a84ebe0 e832dbec d5331307
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:384: encription(3des)
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:2800: with key:
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:34 relent Nov 12 13:45:32d0be6b9f 2746a633 188064b1 be976e84 57620259 c1a9a4f4
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:2808: encrypted payload by IV:
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:34 relent Nov 12 13:45:325036c720 4678ee8b
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:2815: save IV for next:
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:34 relent Nov 12 13:45:325036c720 4678ee8b
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:2832: encrypted.
       Nov 12 13:45:32 relent racoon: DEBUG: sockmisc.c:430: sockname 10.0.0.100[500]
       Nov 12 13:45:32 relent racoon: DEBUG: sockmisc.c:432: send packet from 10.0.0.100[500]
       Nov 12 13:45:32 relent racoon: DEBUG: sockmisc.c:434: send packet to 10.0.0.110[500]
       Nov 12 13:45:32 relent racoon: DEBUG: sockmisc.c:579: 1 times of 60 bytes message will be sent to 10.0.0.110[500]
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:34 relent Nov 12 13:45:3246743924 b0525ac7 4219398c 554990da 08102001 a3b13ba3 0000003c 56fc30f6
       Nov 12 13:45:34 relent Nov 12 13:45:329a753f67 684cf693 2fcf5adb 137947ba d8e34306 5036c720 4678ee8b
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:210: compute DH's shared.
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:34 relent Nov 12 13:45:3226d92ccd 8b57e551 ee87c4b4 c77a881e 3169b74d 75b0320a 3a3c30ea 2dc6a220
       Nov 12 13:45:34 relent Nov 12 13:45:329c28bef6 8d9bc774 2a1be830 2b027f2b cc7eff40 9c32ddb0 32dfb8bc b5cd3a59
       Nov 12 13:45:34 relent Nov 12 13:45:320f542a7f f9ecfa51 bfa92c84 ca832935 d4a927f4 4c39a4c1 55114714 530f1638
       Nov 12 13:45:34 relent Nov 12 13:45:325ee7ab90 294bbcc6 f7aca5d8 335fedd4 93824ab2 3d32bb89 e14e0210 05be7ab5
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:464: KEYMAT compute with
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:34 relent Nov 12 13:45:3226d92ccd 8b57e551 ee87c4b4 c77a881e 3169b74d 75b0320a 3a3c30ea 2dc6a220
       Nov 12 13:45:34 relent Nov 12 13:45:329c28bef6 8d9bc774 2a1be830 2b027f2b cc7eff40 9c32ddb0 32dfb8bc b5cd3a59
       Nov 12 13:45:34 relent Nov 12 13:45:320f542a7f f9ecfa51 bfa92c84 ca832935 d4a927f4 4c39a4c1 55114714 530f1638
       Nov 12 13:45:34 relent Nov 12 13:45:325ee7ab90 294bbcc6 f7aca5d8 335fedd4 93824ab2 3d32bb89 e14e0210 05be7ab5
       Nov 12 13:45:34 relent Nov 12 13:45:32030db810 81b0aa39 deb02e5d 747c5136 6049d1e6 988069b2 71dac5bb 56c78812
       Nov 12 13:45:34 relent Nov 12 13:45:32df96c750 e8
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:497: encklen=192 authklen=160
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:504: generating 640 bits of key (dupkeymat=4)
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:522: generating K1...K4 for KEYMAT.
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 12 13:45:32 relent last message repeated 2 times
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:35 relent Nov 12 13:45:32b32fb81a f3193fa8 db2cd160 4adbf6fd 5ac999b7 f0b66a67 a8a7119b 73c873d8
       Nov 12 13:45:35 relent Nov 12 13:45:32847a06ce e64e4327 b366680e 0e16f144 6a460208 e478a96b 1c9269dd af6a6af1
       Nov 12 13:45:35 relent Nov 12 13:45:32661fb5b7 e8c6a39e 5658a95d 9d9de1e8
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:464: KEYMAT compute with
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:35 relent Nov 12 13:45:3226d92ccd 8b57e551 ee87c4b4 c77a881e 3169b74d 75b0320a 3a3c30ea 2dc6a220
       Nov 12 13:45:35 relent Nov 12 13:45:329c28bef6 8d9bc774 2a1be830 2b027f2b cc7eff40 9c32ddb0 32dfb8bc b5cd3a59
       Nov 12 13:45:35 relent Nov 12 13:45:320f542a7f f9ecfa51 bfa92c84 ca832935 d4a927f4 4c39a4c1 55114714 530f1638
       Nov 12 13:45:35 relent Nov 12 13:45:325ee7ab90 294bbcc6 f7aca5d8 335fedd4 93824ab2 3d32bb89 e14e0210 05be7ab5
       Nov 12 13:45:35 relent Nov 12 13:45:320304c13e deb0aa39 deb02e5d 747c5136 6049d1e6 988069b2 71dac5bb 56c78812
       Nov 12 13:45:35 relent Nov 12 13:45:32df96c750 e8
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:497: encklen=192 authklen=160
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:504: generating 640 bits of key (dupkeymat=4)
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:522: generating K1...K4 for KEYMAT.
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 12 13:45:32 relent last message repeated 2 times
       Nov 12 13:45:32 relent racoon: DEBUG: plog.c:195:
       Nov 12 13:45:35 relent Nov 12 13:45:324ca2a2ba 2558d9e7 9f45c003 a1cf2644 26ce0508 d6ba625a 98dc891f 62262a4b
       Nov 12 13:45:35 relent Nov 12 13:45:3261670d04 ef7bfe89 f75d3f29 322add42 5e4f1c9e 86237b4e f227eac9 b8c2b199
       Nov 12 13:45:35 relent Nov 12 13:45:32d3f92672 fa82126f 43340b6e 9c03e40a
       Nov 12 13:45:32 relent racoon: DEBUG: oakley.c:392: KEYMAT computed.
       Nov 12 13:45:32 relent racoon: DEBUG: isakmp_quick.c:649: call pk_sendupdate
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 12 13:45:32 relent racoon: DEBUG: pfkey.c:974: call pfkey_send_update
       Nov 12 13:45:32 relent racoon: DEBUG: isakmp_quick.c:654: pfkey update sent.
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 12 13:45:32 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 12 13:45:32 relent racoon: DEBUG: pfkey.c:1215: call pfkey_send_add
       Nov 12 13:45:32 relent racoon: DEBUG: isakmp_quick.c:661: pfkey add sent.
       Nov 12 13:45:32 relent racoon: DEBUG: pfkey.c:195: get pfkey UPDATE message
       Nov 12 13:45:32 relent racoon: DEBUG: pfkey.c:1103: pfkey UPDATE succeeded: ESP/Tunnel 10.0.0.110[0]->10.0.0.100[0] spi=230166657(0xdb81081)
       Nov 12 13:45:32 relent racoon: INFO: pfkey.c:1110: IPsec-SA established: ESP/Tunnel 10.0.0.110[0]->10.0.0.100[0] spi=230166657(0xdb81081)
       Nov 12 13:45:32 relent racoon: DEBUG: pfkey.c:1148: ===
       Nov 12 13:45:32 relent racoon: DEBUG: pfkey.c:195: get pfkey ADD message
       Nov 12 13:45:32 relent racoon: INFO: pfkey.c:1322: IPsec-SA established: ESP/Tunnel 10.0.0.100[0]->10.0.0.110[0] spi=79773406(0x4c13ede)
       Nov 12 13:45:32 relent racoon: DEBUG: pfkey.c:1327: ===
    

  8. Once you have your setup working, you will want to change the debug line in /etc/inet/racoon.conf, to reduce the amount of messages sent to /var/adm/syslog (and /var/adm/log/osmlog). We recommend you use:
       log notify ;
    

Configuring racoon for Private Keys and Certificates

In this example, we improve security between the two systems in the previous example by using private keys and signed certificates to provide authentication between two systems, instead of the human readable text files used by the pre_shared_key facility.

  1. On the system elmer (10.0.0.110), create the key and certificate files required:

    1. Enter the following commands, as root:
         # mkdir -p /usr/local/ssl/certs
         # cd /usr/local/ssl/certs
         # openssl req -new -nodes -newkey rsa:1024 -sha1 -keyform PEM \
         > -keyout privkey.pem -out request.pem
         # openssl x509 -req -in request.pem -signkey privkey.pem -out elmer_cert.pem
         # ln -s elmer_cert.pem `openssl x509 -noout -hash -in elmer_cert.pem`.0
      

      The default number of days for which a certificate is valid is 30 days. Use the -days option to the openssl req and openssl x509 commands to specify an alternate number of days.

    2. Copy the file elmer_cert.pem to /usr/local/ssl/certs on relent.

    3. Create the file /etc/inet/elmer.ipsec.conf, with the following contents:
         spdadd 10.0.0.100[any] 10.0.0.110[any] tcp -P in ipsec
                 esp/tunnel/10.0.0.100-10.0.0.110/require ;
         spdadd 10.0.0.110[any] 10.0.0.100[any] tcp -P out ipsec
                 esp/tunnel/10.0.0.110-10.0.0.100/require;
      

    4. Create /etc/inet/racoon.conf, with these contents:
         

      path certificate "/usr/local/ssl/certs" ;

      log debug;

      remote 10.0.0.100 { exchange_mode aggressive ;

      my_identifier asn1dn ; peers_identifier asn1dn ;

      certificate_type x509 "elmer_cert.pem" "privkey.pem"; peers_certfile "relent_cert.pem";

      lifetime time 1 hour ; # sec,min,hour

      # phase 1 proposal (for ISAKMP SA) proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method rsasig ; dh_group 2 ; }

      # the configuration makes racoon (as a responder) to obey the # initiator's lifetime and PFS group proposal. # this makes testing so much easier. proposal_check obey; }

      # phase 2 proposal (for IPsec SA). # actual phase 2 proposal will obey the following items: # - kernel IPsec policy configuration (like "esp/transport//use) # - permutation of the crypto/hash/compression algorithms presented below sainfo anonymous { pfs_group 2; lifetime time 10 hour ; # lifetime byte 50 MB ; encryption_algorithm 3des, blowfish; authentication_algorithm hmac_sha1, hmac_md5 ; compression_algorithm deflate ; }

    5. Enter:
         tail -f /var/adm/syslog
      

    6. In another shell, start IPsec and racoon:
         ipseckey -f /etc/inet/elmer.ipsec.conf
         /usr/sbin/racoon &
      

      Check the syslog output from the previous step. Correct errors as necessary if racoon fails to start. To restart racoon, it may be necessary to delete the file /tmp/.racoon, as shown:

         rm -f /tmp/.racoon
      

      Successful startup is indicated by messages like the following in /var/adm/syslog:

         Nov 17 17:37:18 elmer racoon: INFO: main.c:176: @(#)racoon 20001216 20001216 sakane@kame.net
         Nov 17 17:37:18 elmer racoon: INFO: main.c:177: @(#)This product linked OpenSSL 0.9.7b 10 Apr 2003 (http://www.openssl.org/)
         Nov 17 17:37:18 elmer racoon: DEBUG: algorithm.c:612: hmac(modp1024)
         Nov 17 17:37:18 elmer racoon: DEBUG: pfkey.c:2247: compression algorithm can not be checked because sadb message doesn't support it.
         Nov 17 17:37:18 elmer racoon: DEBUG: admin.c:478: open /tmp/.racoon (fd 15) as racoon management.
         Nov 17 17:37:18 elmer racoon: DEBUG: grabmyaddr.c:595: socket(PF_ROUTE) suceeded: fd 16
         Nov 17 17:37:18 elmer racoon: DEBUG: grabmyaddr.c:324: my interface: 127.0.0.1 (lo0)
         Nov 17 17:37:18 elmer racoon: DEBUG: grabmyaddr.c:324: my interface: 10.0.0.100 (net0)
         Nov 17 17:37:18 elmer racoon: DEBUG: grabmyaddr.c:476: configuring default isak mp port.
         Nov 17 17:37:18 elmer racoon: DEBUG: grabmyaddr.c:498: 2 addrs are configured successfully
         Nov 17 17:37:18 elmer racoon: INFO: sockmisc.c:635: setsockopt_bypass: in bypass
         Nov 17 17:37:18 elmer racoon: INFO: sockmisc.c:660: setsockopt_bypass: out bypass
         Nov 17 17:37:18 elmer racoon: INFO: isakmp.c:1368: 10.0.0.100[500] used as isakmp port (fd=17)
         Nov 17 17:37:18 elmer racoon: INFO: sockmisc.c:635: setsockopt_bypass: in bypass
         Nov 17 17:37:18 elmer racoon: INFO: sockmisc.c:660: setsockopt_bypass: out bypass
         Nov 17 17:37:18 elmer racoon: INFO: isakmp.c:1368: 127.0.0.1[500] used as isakmp port (fd=18)
         Nov 17 17:37:18 elmer racoon: DEBUG: pfkey.c:195: get pfkey X_SPDDUMP message
         Nov 17 17:37:18 elmer racoon: DEBUG: pfkey.c:195: get pfkey X_SPDDUMP message
         Nov 17 17:37:18 elmer racoon: DEBUG: policy.c:184: sub:8047030: 10.0.0.100/32[0] 10.0.0.110/32[0] proto=tcp dir=out
         Nov 17 17:37:18 elmer racoon: DEBUG: policy.c:185: db :809aa58: 10.0.0.110/32[0] 10.0.0.100/32[0] proto=tcp dir=in
      

  2. Once racoon has successfully started on elmer, do all the above steps on the other machine; in this example, relent (10.0.0.100).

    1. Use the following commands to create the keys and certificates on relent:
         # mkdir -p /usr/local/ssl/certs
         # cd /usr/local/ssl/certs
         # openssl req -new -nodes -newkey rsa:1024 -sha1 -keyform PEM \
         > -keyout privkey.pem -out request.pem
         # openssl x509 -req -in request.pem -signkey privkey.pem -out relent_cert.pem
         # ln -s relent_cert.pem `openssl x509 -noout -hash -in relent_cert.pem`.0
      

      The default number of days for which a certificate is valid is 30 days. Use the -days option to the openssl req and openssl x509 commands to specify an alternate number of days.

    2. Copy the file relent_cert.pem to /usr/local/ssl/certs on elmer.

    3. The file /etc/inet/relent.ipsec.conf should look like this:
         spdadd 10.0.0.100[any] 10.0.0.110[any] tcp -P out ipsec
                 esp/tunnel/10.0.0.100-10.0.0.110/require ;
         spdadd 10.0.0.110[any] 10.0.0.100[any] tcp -P in ipsec
                 esp/tunnel/10.0.0.110-10.0.0.100/require;
      

    4. The file /etc/inet/relent.racoon.conf should look like this:
         

      path certificate "/usr/local/ssl/certs" ;

      log debug;

      remote 10.0.0.110 { exchange_mode aggressive ;

      my_identifier asn1dn ; peers_identifier asn1dn ;

      certificate_type x509 "relent_cert.pem" "privkey.pem"; peers_certfile "elmer_cert.pem";

      lifetime time 1 hour ; # sec,min,hour

      # phase 1 proposal (for ISAKMP SA) proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method rsasig ; dh_group 2 ; }

      # the configuration makes racoon (as a responder) to obey the # initiator's lifetime and PFS group proposal. # this makes testing so much easier. proposal_check obey; }

      # phase 2 proposal (for IPsec SA). # actual phase 2 proposal will obey the following items: # - kernel IPsec policy configuration (like "esp/transport//use) # - permutation of the crypto/hash/compression algorithms presented below sainfo anonymous { pfs_group 2; lifetime time 10 hour ; # lifetime byte 50 MB ; encryption_algorithm 3des, blowfish; authentication_algorithm hmac_sha1, hmac_md5 ; compression_algorithm deflate ; }

    5. Start IPsec and racoon using the commands shown in Steps 4 and 5, above.

  3. Once racoon is running on both systems, generate some traffic between the systems using ssh, telnet, netscape, etc. Then, on both systems, check the output in /var/adm/syslog. Whenever a new key is generated and shared by racoon on the two systems, messages like the following appear:
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:231: ===
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:232: 319 bytes message received from 10.0.0.110[500]
       Nov 17 17:37:25 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:25 relent Nov 17 17:37:2599ab2e4d c4999d1f 00000000 00000000 01100400 00000000 0000013f 04000034
       Nov 17 17:37:25 relent Nov 17 17:37:2500000001 00000001 00000028 01010001 00000020 01010000 800b0001 800c0e10
       Nov 17 17:37:25 relent Nov 17 17:37:2580010005 80030003 80020002 80040002 0a000084 92ab31a7 30d93225 c297312a
       Nov 17 17:37:25 relent Nov 17 17:37:25ca52e2c9 205ead9e 756436ce b896187d 345ce5ba 4b415681 405e0072 401fc40c
       Nov 17 17:37:25 relent Nov 17 17:37:259b8d4d18 c6e339a8 1fda5790 8486b352 e5d2d09b 2cbad154 70d9b95e c5b3fcee
       Nov 17 17:37:25 relent Nov 17 17:37:25bd771050 983c54f6 eab3c101 0264400c a292ad9d 1a2cd9d7 fa56312c 14937066
       Nov 17 17:37:25 relent Nov 17 17:37:25d78d2bcc 4f968f55 59d2ed46 8d0e9527 e01da5d1 05000014 b51b076f 6eecca3b
       Nov 17 17:37:25 relent Nov 17 17:37:25c70cb1ff 0b2d32c9 00000057 09000000 304d310b 30090603 55040613 02555331
       Nov 17 17:37:25 relent Nov 17 17:37:250b300906 03550408 13024e59 310e300c 06035504 07130542 45524e45 3121301f
       Nov 17 17:37:25 relent Nov 17 17:37:2506035504 0a131849 6e746572 6e657420 57696467 69747320 50747920 4c7464
       Nov 17 17:37:25 relent racoon: DEBUG: remoteconf.c:118: configuration found for 10.0.0.110[500].
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:899: ===
       Nov 17 17:37:25 relent racoon: INFO: isakmp.c:904: respond new phase 1 negotiation: 10.0.0.100[500]<=>10.0.0.110[500]
       Nov 17 17:37:25 relent racoon: INFO: isakmp.c:909: begin Aggressive mode.
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp_agg.c:621: agg_r1recv: begin
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1122: begin.
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1149: seen nptype=1(sa)
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1149: seen nptype=4(ke)
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1149: seen nptype=10(nonce)
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1149: seen nptype=5(id)
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1188: succeed.
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp_agg.c:653: received payload of type ke
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp_agg.c:653: received payload of type nonce
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp_agg.c:653: received payload of type id
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1117: total SA len=48
       Nov 17 17:37:25 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:26 relent Nov 17 17:37:2500000001 00000001 00000028 01010001 00000020 01010000 800b0001 800c0e10
       Nov 17 17:37:26 relent Nov 17 17:37:2580010005 80030003 80020002 80040002
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1122: begin.
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1149: seen nptype=2(prop)
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1188: succeed.
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1170: proposal #1 len=40
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1122: begin.
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1149: seen nptype=3(trns)
       Nov 17 17:37:25 relent racoon: DEBUG: isakmp.c:1188: succeed.
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1311: transform #1 len=32
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1870: type=Life Type, flag=0x8000, lorv=seconds
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1870: type=Life Duration, flag=0x8000, lorv=3600
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1870: type=Encryption Algorithm, flag=0x8000, lorv=3DES-CBC
       Nov 17 17:37:25 relent racoon: DEBUG: algorithm.c:384: encription(3des)
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1870: type=Authentication Method, flag=0x8000, lorv=RSA signatures
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1870: type=Hash Algorithm, flag=0x8000, lorv=SHA
       Nov 17 17:37:25 relent racoon: DEBUG: algorithm.c:254: hash(sha1)
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1870: type=Group Description, flag=0x8000, lorv=1024-bit MODP group
       Nov 17 17:37:25 relent racoon: DEBUG: algorithm.c:612: hmac(modp1024)
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1213: pair 1:
       Nov 17 17:37:25 relent racoon: DEBUG: proposal.c:892:  80a0fa8: next=0 tnext=0
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:1248: proposal #1: 1 transform
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:322: prop#=1, prot-id=ISAKMP, spi-size=0, #trns=1
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:327: trns#=1, trns-id=IKE
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:491: type=Life Type, flag=0x8000, lorv=seconds
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:491: type=Life Duration, flag=0x8000, lorv=3600
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:491: type=Encryption Algorithm, flag=0x8000, lorv=3DES-CBC
       Nov 17 17:37:25 relent racoon: DEBUG: ipsec_doi.c:491: type=Authentication Method, flag=0x8000, lorv=RSA signatures
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:27 relent Nov 17 17:37:2699ab2e4d c4999d1f dd6ec225 894f8636 08102001 7d8467eb 0000003c cc73d415
       Nov 17 17:37:27 relent Nov 17 17:37:2667e3c374 919d41eb aee767dc a3d40b09 03fdebec f31c4f13 32777f7b
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2626: begin decryption.
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:384: encription(3des)
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2640: IV was saved for next processing:
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:27 relent Nov 17 17:37:26f31c4f13 32777f7b
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:384: encription(3des)
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2665: with key:
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:27 relent Nov 17 17:37:262eb69857 3bc11fde 90f07315 fb8121bc cfb70f69 2e289b64
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2673: decrypted payload by IV:
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:27 relent Nov 17 17:37:26f31c4f13 32777f7b
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2676: decrypted payload, but not trimed.
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:27 relent Nov 17 17:37:2600000018 e84dcf9e a7831750 3b62090c 14c0dd74 18378b23 e9c56349 0c7ac907
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2685: padding len=8
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2699: skip to trim padding.
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:2714: decrypted.
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:27 relent Nov 17 17:37:2699ab2e4d c4999d1f dd6ec225 894f8636 08102001 7d8467eb 0000003c 00000018
       Nov 17 17:37:27 relent Nov 17 17:37:26e84dcf9e a7831750 3b62090c 14c0dd74 18378b23 e9c56349 0c7ac907
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp.c:1122: begin.
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp.c:1149: seen nptype=8(hash)
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp.c:1188: succeed.
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp_quick.c:1430: HASH(3) validate:
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:27 relent Nov 17 17:37:26e84dcf9e a7831750 3b62090c 14c0dd74 18378b23
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:692: HASH with:
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:28 relent Nov 17 17:37:26007d8467 ebf61ace 15f04966 80d59719 045c9049 8f9a83bd b6069c8f ace0697d
       Nov 17 17:37:28 relent Nov 17 17:37:265b26dbfc 07
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:702: HASH computed:
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:28 relent Nov 17 17:37:26e84dcf9e a7831750 3b62090c 14c0dd74 18378b23
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp.c:746: ===
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:210: compute DH's shared.
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:28 relent Nov 17 17:37:26f055d09f b2a6195d 1a7bbed2 a03c86a9 98d8a49f bacd008f 9de6cf63 6ee532fe
       Nov 17 17:37:28 relent Nov 17 17:37:26bb8beaad cc63c60e 5c851f5b 5bc047f0 f66e0816 1efd60bc 42204868 243b040c
       Nov 17 17:37:28 relent Nov 17 17:37:26b775d457 5200a886 51f4f914 60670010 5590b1dd bb455673 b19e2094 ace3a496
       Nov 17 17:37:28 relent Nov 17 17:37:26134ce1b1 2a54a6ae 52083878 434ea944 1c90806a 770f8953 ea8fe89e a13ff8bd
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:464: KEYMAT compute with
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:28 relent Nov 17 17:37:26f055d09f b2a6195d 1a7bbed2 a03c86a9 98d8a49f bacd008f 9de6cf63 6ee532fe
       Nov 17 17:37:28 relent Nov 17 17:37:26bb8beaad cc63c60e 5c851f5b 5bc047f0 f66e0816 1efd60bc 42204868 243b040c
       Nov 17 17:37:28 relent Nov 17 17:37:26b775d457 5200a886 51f4f914 60670010 5590b1dd bb455673 b19e2094 ace3a496
       Nov 17 17:37:28 relent Nov 17 17:37:26134ce1b1 2a54a6ae 52083878 434ea944 1c90806a 770f8953 ea8fe89e a13ff8bd
       Nov 17 17:37:28 relent Nov 17 17:37:26030fe664 d6f61ace 15f04966 80d59719 045c9049 8f9a83bd b6069c8f ace0697d
       Nov 17 17:37:28 relent Nov 17 17:37:265b26dbfc 07
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:497: encklen=192 authklen=160
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:504: generating 640 bits of key (dupkeymat=4)
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:522: generating K1...K4 for KEYMAT.
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 17 17:37:26 relent last message repeated 2 times
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:28 relent Nov 17 17:37:26bd103196 069c2547 4496d6bc 2a9a75b6 f13fa832 280b2a72 2247100f 645318e4
       Nov 17 17:37:28 relent Nov 17 17:37:26ff31409e ff953e25 a8d82feb da8a39fc 6bf8234f 34c799e7 8a0fcaf9 e9a75bc1
       Nov 17 17:37:28 relent Nov 17 17:37:26ed0c2f8d bd2bb80f 9d925eb7 5e4e65f2
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:464: KEYMAT compute with
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:28 relent Nov 17 17:37:26f055d09f b2a6195d 1a7bbed2 a03c86a9 98d8a49f bacd008f 9de6cf63 6ee532fe
       Nov 17 17:37:28 relent Nov 17 17:37:26bb8beaad cc63c60e 5c851f5b 5bc047f0 f66e0816 1efd60bc 42204868 243b040c
       Nov 17 17:37:29 relent Nov 17 17:37:26b775d457 5200a886 51f4f914 60670010 5590b1dd bb455673 b19e2094 ace3a496
       Nov 17 17:37:29 relent Nov 17 17:37:26134ce1b1 2a54a6ae 52083878 434ea944 1c90806a 770f8953 ea8fe89e a13ff8bd
       Nov 17 17:37:29 relent Nov 17 17:37:2603098fd6 4af61ace 15f04966 80d59719 045c9049 8f9a83bd b6069c8f ace0697d
       Nov 17 17:37:29 relent Nov 17 17:37:265b26dbfc 07
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:497: encklen=192 authklen=160
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:504: generating 640 bits of key (dupkeymat=4)
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:522: generating K1...K4 for KEYMAT.
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:324: hmac(hmac_sha1)
       Nov 17 17:37:26 relent last message repeated 2 times
       Nov 17 17:37:26 relent racoon: DEBUG: plog.c:195:
       Nov 17 17:37:29 relent Nov 17 17:37:26b5474a43 c8e1a82b 7eb74f66 cc07e0ca 9ce2f248 a19703c0 9c355b30 df664a58
       Nov 17 17:37:29 relent Nov 17 17:37:2605c6b27f 50ec1ce0 ce1c364f 39c49258 96c8ebca 162e8f35 0d1f6757 f97c2fca
       Nov 17 17:37:29 relent Nov 17 17:37:26e4c17f21 7e1bff84 6f23d2c9 e33f085e
       Nov 17 17:37:26 relent racoon: DEBUG: oakley.c:392: KEYMAT computed.
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp_quick.c:1613: call pk_sendupdate
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 17 17:37:26 relent racoon: DEBUG: pfkey.c:974: call pfkey_send_update
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp_quick.c:1618: pfkey update sent.
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:511: encription(3des)
       Nov 17 17:37:26 relent racoon: DEBUG: algorithm.c:554: hmac(hmac_sha1)
       Nov 17 17:37:26 relent racoon: DEBUG: pfkey.c:1215: call pfkey_send_add
       Nov 17 17:37:26 relent racoon: DEBUG: isakmp_quick.c:1625: pfkey add sent.
       Nov 17 17:37:26 relent racoon: DEBUG: pfkey.c:195: get pfkey UPDATE message
       Nov 17 17:37:26 relent racoon: DEBUG: pfkey.c:1103: pfkey UPDATE succeeded: ESP/Tunnel 10.0.0.110[0]->10.0.0.100[0] spi=266757334(0xfe664d6)
       Nov 17 17:37:26 relent racoon: INFO: pfkey.c:1110: IPsec-SA established: ESP/Tunnel 10.0.0.110[0]->10.0.0.100[0] spi=266757334(0xfe664d6)
       Nov 17 17:37:26 relent racoon: DEBUG: pfkey.c:1148: ===
       Nov 17 17:37:26 relent racoon: DEBUG: pfkey.c:195: get pfkey ADD message
       Nov 17 17:37:26 relent racoon: INFO: pfkey.c:1322: IPsec-SA established: ESP/Tunnel 10.0.0.100[0]->10.0.0.110[0] spi=160421450(0x98fd64a)
       Nov 17 17:37:26 relent racoon: DEBUG: pfkey.c:1327: ===
       Nov 17 17:38:03 relent in.telnetd[10637]: connect from elmer.iii.com
       Nov 17 17:39:18 relent in.telnetd[10660]: connect from elmer.iii.com
    

  4. Once you have your setup working, you will want to change the debug line in /etc/inet/racoon.conf, to reduce the amount of messages sent to /var/adm/syslog (and /var/adm/log/osmlog). We recommend you use:
       log notify ;
    

Processing of Multiple Security Associations

If multiple security associations exist in the SAD database for the same IP addresses and ports, by default the newest entry is selected as the policy for that connection. To change this default so that the oldest entry in the SAD database is selected, use the scoadmin system tuner or the inconfig(ADMN) command to set the tuneable parameter key_preferred_oldsa to 1. Then, rebuild the kernel and reboot, as in this command line example:

   # inconfig key_preferred_oldsa 1
   # idbuild -B
   # shutdown -i6 -g0 -y

Mixing IPsec and non-IPsec Traffic Between Hosts

In many configurations, traffic on some ports will not be required to support IPsec. DNS traffic on port 53 is one example. Since the entries in the SAD and SPD databases are parsed in order (and not on best match, for example), entries for non-IPsec connections to particular ports should be specified before entries for secure IP connections. For example, entries like the following would need to appear in the /etc/inet/ipsec.conf file before other entries for the same connections:

   spdadd hostA[any] DNSserver[53] udp -P in none ;
   spdadd DNSserver[53] hostA[any] udp -P out none ;

Configuring IPsec for IP Compression (IPcomp)

Compressing encrypted data has been shown to be largely ineffective in most cases. Enabling IP payload compression (IPcomp) means that the packets exchanged over the secure communication channel will be compressed before they are encrypted using ESP. Enabling IPcomp generally means better performance as the packet size is reduced to a minimum before encryption. The following attribute in an sainfo section of racoon.conf enables IP compression:

   compression_algorithm deflate

IP compression is enabled in all the samples used in this document.

Configuring IPsec for Dynamic IP

In general, setting the following attributes in Phase 1 IKE negotiation (the remote statement in /etc/inet/racoon.conf) configures the secure channel for a remote client that does not have a fixed IP address (dynamic IP allocation):

   my_identifier asn1dn;
   peers_identifier asn1dn;
   passive on;
   generate_policy on;

The identifier used on the channel must be asn1dn; so that the remote system's identifier is taken from the Subject field of the remote system's certificate. The passive attribute must be on so that communication on the channel is initiated by the remote system. The generate_policy attribute must be on so that Security Policy Database (SPD) entries can be constructed at the clients request using identifier and policy information obtained from the client.

Note that it is important that the remote client's SPD entries (contained in the sainfo statements in the rmeote client's racoon.conf file) for this channel match a security policy defined on the other system, or future negotiations for services will be denied.

Disabling or Reconfiguring IPsec

To disable IPsec completely, follow the instructions in the section ``Enabling IPsec'' and set ipsec_enable to 0 to disable IPsec processing.

To reconfigure or selectively disable IPsec, do the following:

  1. You may want to disable the network interface being reconfigured to preserve system security during the reconfiguration.

  2. Remove the appropriate SAD and SDP entries on all systems. IPsec SAD and SDP entries can be removed completely or selectively:

    1. Completely remove the current configuration:
         ipseckey -F
         ipseckey -FP
      

    2. Selectively remove SAD and SDP entries using the setkey delete and sdpdelete commands (see setkey(ADM)).

    [Note: If you are doing this on some systems via remote login (ssh, telnet, etc.), log into all remote systems before you disable the local system's IPsec; then disable IPsec on all remote systems before disabling IPsec on the local system.]

  3. If you are using racoon, use the ps and kill commands as shown in the example below to stop the currently running racoon process:
       # ps -eaf | grep racoon
           root  4267  3191   TS  80  0 17:27:12 pts/18   0:00 racoon
       # kill -9 4267
    

  4. Add new SAD/SDP entries, either using the setkey add and sdpadd commands or by modifying the confguration file (/etc/inet/systemname.ipsec.conf in our example). See setkey(ADM).

  5. If you are using racoon, modify the /etc/inet/racoon.conf configuration file as required for your new configuration. Then, start racoon:
       # racoon &
    

  6. Re-enable the network interface, if disabled in Step 1.

Starting IPsec Automatically on Reboot

If you want to configure IPsec to start automatically on every reboot, add the setkey commands you used to manually start IPsec to a file under /etc/rc2.d to configure on every reboot on all hosts. For the system elmer, for example, we could add a file named /etc/rc2.d/S99ipsec.elmer with the following command:

   ipseckey -f /etc/inet/elmer.ipsec.conf

IPsec Interoperability with Windows 2000 and XP

IPsec hosts and Windows 2000 and XP systems can be configured to exchange IPsec packets, with the Windows system acting as a client.

To configure IPsec on the Windows client:

  1. Open Start > Settings > Control Panel > Administrative Tools on Windows, and click on Local Security Policy. Right-click the mouse and select Create IPsecurity Policy; give the policy an appropriate name and ensure that all defaults automatically filled in by Windows match your configuration, and update them as appropriate.

  2. Click on Add in the Policy Property Window to add a new rule to the policy you just created.

  3. See the following Windows articles on the web to help you fill out the security policy rule information for your configuration:

Note that contrary to some information in the above articles, it is not necessary when defining an IPsec VPN to define a static route on Windows between the two machines. This is done by the Phase 1 (remote) definitions in the Security Policy rules on Windows and in racoon.conf on the IPsec host.

More articles as well as IPsec-related Service Packs for Windows 2000 and XP can be found by searching the Windows Support Web Site.

Also see the ``IPsec Road Warrior Configuration'' section.

libipsec Manual Pages

The library libipsec includes several functions for manipulating secure IP policy information. See the ipsec_set_policy(S) and ipsec_strerror(S) manual pages.

Additional IPsec Documentation

The following web sites provide a great deal of information about IPsec.

www.netbsd.org netBSD home page
www.freebsd.org freeBSD home page
www.kame.net KAME Project home page (developers/owners of IPsec)


© 2007 The Santa Cruz Operation, Inc. All rights reserved
OpenServer 6 and UnixWare (SVR5) HDK - 05 June 2007