One-way network links for small businesses: Part 1

Traditionally, Industrial Control Systems (ICS) have used self-contained networks, not connected to the office network or to the Internet, in order to protect from hacking or malware. This is generally referred to as “air-gapping”.

But as businesses demand more information about what is going on in the plant in order to better plan, the ICS networks often get connected to the office networks, so management can have up-to-the-minute reports. This makes the ICS network vulnerable to insider attacks from the office side, or attacks from the outside if the office network is breached.

One might think the problem could be handled by setting the rules appropriately in the router between the ICS network and the office network. But routers can be bypassed by if they become infected by malware, like the recent VPNFilter malware. And setting up router rules is not trivial, especially for those who don’t do it every day, so routers may be mis-configured.

What is needed is a hardware-based solution that requires minimal configuration.

Hardware goals

A hardware solution should satisfy these goals:

  • Low cost: While large corporations can spend as much as necessary to secure their network (though they often do not), you may have a small ICS network, and the solution should not break the bank. You may not even have an ICS network, but an internal network that needs to be isolated from the Internet-facing network.
  • Standard components: You should be able to easily replace components that fail, or stockpile replacements. One-of-a-kind or custom-built components put you at risk if there is a failure and you cannot get a replacement.
  • No special cables: Some proposed one-way Ethernet solutions involve making custom Ethernet cables. You should not have to break out the soldering iron when you have a failure. If you move on to another job, your successor may not have a soldering iron, or know how to make special cables.
  • Maintainable by ordinary network engineers: It should be possible to document the setup in such a way that any competent network engineer can cable it up, without scratching his or her head and saying “Huh?”
  • Error proof: In line with the poka-yoke philosophy of Lean manufacturing, it should be difficult to cable the setup improperly, and it should be easy to detect improper cabling.

Special cables

Since 10BASE-T and 100BASE-T Ethernet use separate pairs for transmit and receive, one might think ensuring one-way connectivity would be as easy as snipping the transmit wires in the cable. Unfortunately, it isn’t that simple.

Ethernet interfaces transmit pulses, called Normal Link Pulses (NLP) in 10BASE-T, or Fast Link Pulses (FLP) in 100BASE-T, that determine whether the interface is connected. In 100BASE-T, they are also part of the auto-negotiation process. NLP and FLP are not often mentioned in the literature. Since I had to do some searching to find out how they work, I have included references at the end of this post.

If the interface does not receive the NLP or FTP, it will not come up. A couple of kludges have been suggested for dealing with this problem:

Both require custom hand-made cables, which we would like to avoid. The first one counts on having the right size capacitor to let the NLP/FLP through, but mangle the Ethernet frames. I am a lot more comfortable if the frames do not even try to get through, so I don’t have to worry about them not getting mangled enough, and do not have to worry about the additional overhead the switch has rejecting the bad frames. The second suggestion takes up an additional port on the switch, and is non-standard, which is confusing to maintainers. 

Ethernet TAPs

Luckily, there is an answer, in the form of the Ethernet TAP (said to stand for Test Access Port). These are used by intrusion detection systems to monitor all the traffic coming into a site. In order to remain non-intrusive, they have no facility to inject traffic into the network. Here is a line diagram of how an Ethernet TAP works:

Traffic goes in the A port and out the B port. The traffic going from A to B also comes out the TAP A port, and the traffic going from B to A comes out the TAP B port. The transmit pairs of the TAP A and TAP B ports provide NLP/FLP, but are otherwise unconnected.

The reason for having two TAP ports is this: With a 100BASE-T full-duplex network, you can have 100 Mbs going in each direction, so if you are capturing both directions, that adds up to 200 Mbs, which exceeds the capacity of a 100BASE-T network card. So intrusion detection systems have two NICs, one for each direction, and join them (called bonding or bridging) so they appear to the operating system as one 200 Mbs NIC.

Ethernet TAPs are built to be reliable (some even have dual power supplies), since organizations put them in front of their main Internet connections. Some, like the Finisar Shadow Taps, can lose power without even briefly interrupting the Ethernet traffic through them. For this reason, they tend to be very expensive, with new ones costing over a thousand dollars.

However, with a lot of organizations going to gigabit Ethernet connections, new and used 10/100BASE-T taps are now available on eBay for less than a hundred dollars.

Data Diodes

There is also a purpose-built one-way device called a data diode. Like a regular Ethernet TAP, it has A and B, and TAP A and TAP B ports. It can be configured one of two ways.

In the first configuration, the transmit signals from the A port are sent to the receive side of the TAP A and TAP B ports, allowing two monitoring devices to be attached. The B port is not used in this configuration.

In the second configuration, the transmit signals from the A port are sent to the receive side of the TAP A port, and the transmit signals from the B port are sent to the receive side of the TAP B port, allowing two networks to be monitored.

Here is an example of a data diode from Garland Technology. Since these are specialized devices, you are unlikely to find them on the surplus market.

(In any case, if you have the budget, you should buy new devices from the manufacturers. If your network is depending on a device, it is better to have a new, supported device. Also, it helps keep the manufacturers in business, so they keep making them.)

Most of what follows applies whether you are using a regular Ethernet TAP or a data diode. Exceptions are noted below.

Gigabit Ethernet

Things are a little different if you are using Gigabit Ethernet (1000BASE-T), since it uses all four cable pairs in both directions. The concepts are the same, but the actual implementation is more complicated than shown in the line drawings above. And kludges like cutting wire pairs or putting capacitors in series with them would never work.

If you set up your one-way link using a 10/100 Ethernet TAP, then decide you need to upgrade to Gigabit Ethernet, all you have to do is replace your TAP with one that supports Gigabit Ethernet. You do not have to redesign your solution, like you would if you were using special cables.

Hardware setup

Here is how we use the Ethernet TAP. The sending machine, inside the ICS perimeter, has two NICs, in addition to whatever NICs it needs to talk to the ICS network. Let’s call them NIC6 and NIC7.

Ideally, the ICS and office networks are in separate rooms, and the Ethernet TAP is a locked cabinet. This helps prevent someone from cabling directly from one network to the other because they do not understand the setup (poka-yoke).

However, even if they do we can still detect it. Each message is sent from NIC6 to NIC7, and then from NIC7 to NIC6. The received messages are compared with the original, to make sure they were transmitted successfully. If the Ethernet tap is mis-cabled, for example with NIC6 connected to NIC2, and NIC7 connected to NIC3, the sending machine will not receive the message it sent, and will know that something is wrong.

We could actually get by with sending the messages from NIC6 to NIC7 (or the other way), and avoiding the reverse trip, and using only one of the TAP ports. Transmitting it in both directions gives us two copies of the message, one on each cable. This gives us redundancy. If the receiving machine stops receiving messages on one of the cables, it can notify someone, but continue working until the bad cable is fixed.

If you are using a data diode, you can connect NIC6 to the A port, and NIC7 to the B port. You will still get the redundancy of two message streams, but you will not get the protection from mis-cabling.

Routing

Routing between two NICs on the sending machine is a little tricky. Here’s how you do it on Windows:

First we set the NICs up with RFC 1918 IPv4 addresses, such as 10.10.10.6 and 10.10.10.7, with a subnet mask of 255.255.255.0. (If both addresses do not fall under the same masked subnet, they will not see packets from each other.)

  • Settings / Network & Internet / Change adapter options.
  • Right-click adapter, select Properties.
  • Select Internet Protocol Version 4 (TCP/IPv4) and click Properties.
  • Set the IP address and Subnet mask. Leave Default gateway blank. Be sure to uncheck “Validate settings upon exit”.


Next we have to set up the routing between the two NICs. First, open an administrative command window, and issue an ipconfig /all command.

Look for the two NICs that you set to addresses 10.10.10.6 and 10.10.10.7:

Ethernet adapter Ethernet 6:
 
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 PT Dual Port Server Adapter
   Physical Address. . . . . . . . . : 00-15-17-3A-54-62
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.10.10.6(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet 7:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 PT Dual Port Server Adapter #2
   Physical Address. . . . . . . . . : 00-15-17-3A-54-63
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.10.10.7(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   NetBIOS over Tcpip. . . . . . . . : Enabled

Now issue a route print -4 command to show the routing table. At the beginning of the output is the interface list:

===========================================================================
Interface List
 19...f6 15 85 26 ce 54 ......Hyper-V Virtual Ethernet Adapter
 10...f6 15 30 0f 51 0d ......Hyper-V Virtual Ethernet Adapter #2
 22...d0 50 99 8a 9d ad ......Realtek PCIe GBE Family Controller
 15...00 15 17 3a 54 62 ......Intel(R) PRO/1000 PT Dual Port Server Adapter
 27...00 15 17 3a 54 63 ......Intel(R) PRO/1000 PT Dual Port Server Adapter #2
  1...........................Software Loopback Interface 1
 25...4e 15 d4 92 42 e2 ......Hyper-V Virtual Ethernet Adapter #3
===========================================================================

If you compare the Physical Address from the ipconfig /all command with the MAC addresses in the interface list, you can see that the NIC with IP address 10.10.10.6 has an interface number of 15, and the NIC with IP address 10.10.10.7 has an interface number of 27. We will need this to set up the routes.

Now issue the following route commands:

  route -p add 10.10.10.7 mask 255.255.255.255 10.10.10.6 if 15
  route -p add 10.10.10.6 mask 255.255.255.255 10.10.10.7 if 27

The -p indicates that the routes should be persistent across reboots.

Note that the subnet mask is 255.255.255.255, since this is a route for a specific NIC. 10.10.10.7 is routed to 10.10.10.6 on interface 15, and 10.10.10.6 is routed to 10.10.10.7 on interface 27.

Now if you ping either address, the ping should be successful.

IPv6 Setup

You can also do this using IPv6, using link-local addresses.

For each of the two NICs, go to

  • Settings / Network & Internet / Change adapter options.
  • Right-click adapter, select Properties.

This will display the following pop-up:

Make sure Internet Protocol Version 4 (TCP/IPv4) is unchecked, and Internet Protocol Version 6 (TCP/IPv6) is checked. Highlight Internet Protocol Version 6 (TCP/IPv6) and select Properties.

That will display this pop-up:

Make sure that obtaining IPv6 address and DNS server address automatically is selected, and “Validate settings upon exit” is not selected. Click “OK” until you get to the “Change Adapter Settings” page, then  repeat the process for the other NIC.

Now open an administrative command window and issue the ipconfig command. Look for the two NICs in the output:

Ethernet adapter Ethernet 6:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::3169:b3bf:8e9f:922f%15
   Default Gateway . . . . . . . . . :

Ethernet adapter Ethernet 7:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::15d9:523b:709c:b43f%27
   Default Gateway . . . . . . . . . :

Note the link-local IPv6 address and interface (the number after the percent sign) for the two NICs, in this case fe80::3169:b3bf:8e9f:922f and 15 for NIC6, and fe80::15d9:523b:709c:b43f and 27 for NIC7. (Link-local IPv6 addresses can be identified by the FE80 prefix, per RFC 4291.)

Issue the following routing commands:
route -p add fe80::3169:b3bf:8e9f:922f/128 fe80::15d9:523b:709c:b43f if 27
route -p add fe80::15d9:523b:709c:b43f/128 fe80::3169:b3bf:8e9f:922f if 15 

Similar to the IPv4 configuration, each of the NICs is reached via the other NIC’s IPv6 address and interface. And like before, the -p keyword indicates a permanent route.

If you are using a data diode, the routing will be similar, but instead of routing from one NIC to the other, you will use two bogus IP addresses within the same subnet. (Since we will be using UDP to send our packets, the fact that the addresses do not exist will not be a problem.)

Ensuring reliable transmission

TCP (Transmission Control Protocol) is the standard protocol for reliable delivery over Ethernet. It makes sure that each packet transmitted is acknowledged, and if the packet is fragmented, makes sure the packets are all received and put in the right order. But TCP requires two-way communication, both for the setup of the connection (the three-way handshake) and for acknowledging receipt of packets. This will not work with our one-way Ethernet connection.

Instead, we must use the simpler UDP (User Datagram Protocol) protocol. While UDP can have a checksum to protect us from data corruption, it is optional, and not terribly strong. Also, we are on our own regarding packet loss. We must do these things to ensure reliable transmission:

  • Include a strong and mandatory checksum in the packet.
  • Include a sequence number in each message. If the receiving machine gets a sequence number that is more than one greater than the previous one, it knows a packet has been lost.
  • Have a watchdog timer. The sending machine sends a message every few seconds, whether or not it has new information to send. If the timer expires before a message is received, the receiving machine knows a packet has been lost, the cable has been interrupted, or the sending machine is down.
  • Compare the packets received on both NICs. If they are not the same, there is some sort of transmission error. If it receives a packet on only one of the NICs, it can continue, but one of the cables has probably been interrupted.

In this case, the receiving machine should signal the ICS room. It should not use Ethernet to do so, because that could introduces a two-way connection if the patch panel was mis-cabled. Instead, it might use something like one of these USB relays from Numato that triggers an alarm bell like this in the ICS room. When the bell rings, the operator in the ICS room calls the office-network room to find out what the receiving system is complaining about.

Or we could use a BetaBrite scrolling LED sign, also available on eBay, which is programmed via a USB or RS-232 connection, to display a message in the ICS room indicating what the problem is. There is more information about communicating with BetaBrite signs here and here.

The idea is that the return path to the ICS room should not be an Ethernet cable that can accidentally be plugged into someplace where it can be an inbound data path to the ICS systems.

Also, because we have a one-way connection, the receiving system cannot ask for the particular report it wants; we have to send the data for all reports. This can be done as an XML or JSON file, with formatting added at the receiving end. If XML is used, XML stylesheets, a.k.a. XSLT or an XQuery  program can be used to transform the data into the desired report. If there is a lot of data to send, we might consider EXI, a compressed form of XML, or EXI4JSON, a compressed form of JSON.

An alternative approach

SANS has an excellent article on their website about sending syslog messages over a one-way link. They use a different approach. As each syslog message is sent, it is also stored locally in a container. Every hour a new container is started. A hash of the contents of the previous container is sent to a different port on the receiving machine.. The receiving machine calculates the hash on the messages it received over the last hour. If it does match the received hash, the receiver knows one or more messages were lost. In that case, the container is retrieved from the sending machine by physically visiting the machine. (Be careful with this; the STUXNET virus was able to infect the air-gapped systems of Iran’s uranium processing facility by traveling on infected USB sticks brought in by contractors.)

Sending software

Here is a sample Python 3 program to send out UDP packets on the two interfaces, and verify that they are received correctly. In real life, the program would be sending your data, rather than dummy XML data. The program adds a CRC-32 to the data before sending it, and verifies that the two received copies have the same CRC. The XML data is wrapped in a <packet> XML element, with an attribute that is the sequence number of the packet.

from PyCRC.CRC32 import CRC32
import socket
import time

# Author: Lynn Grant
# Date: 2019-02-16
# License: To the extent possible under law, Lynn Grant has waived all copyright and
# related or neighboring rights to this program. This work is published from: Belize. 

# socket.AF_INET for IPv4, socket.AF_INET6 for IPv6 
IP_ADDRESS_TYPE = socket.AF_INET 
UDP_IP_A = "10.10.10.6"
#UDP_IP_A = "fe80::3169:b3bf:8e9f:922f"
UDP_IP_B = "10.10.10.7"
#UDP_IP_B = "fe80::15d9:523b:709c:b43f"
UDP_PORT = 5005
 
# XML <packet> tags wrapping the data. Note that the sequence number must
# immediately follow the XML_PREFIX1 string.
XML_PREFIX1 = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><packet seq=\""
XML_PREFIX2 = "\">"
XML_SUFFIX = "</packet>"

# Dummy report data
DATA = \
   "<data>" \
      "<report1>" \
         "Data for report 1" \
      "</report1>" \
      "<report2>" \
         "Data for report 2" \
      "</report2>" \
   "</data>"


# calcCrc: Calculate the CRC for a message
#
# message is the message for which the CRC is to be calculated.
#
# The CRC is returned.
 
def calcCrc(message): 
   return CRC32().calculate(message).to_bytes(4, 'big') 


# getSequence: Get sequence number from message
#
# We extract the text between the end of the 
# XML_PREFIX1, and the first double quote we find.
#
# Note: I generally do not kludgey manual parsing
#       of XML; there are are libraries to do this.
#       But this is a single field, so it does not
#       seem worth the overhead of a proper parser. 
#
# message is the message to be processed.

def getSequence(message): 
   nopref = message[len(XML_PREFIX1):]   # Message without XML_PREFIX1
   j = nopref.index("\"")                # Index of trailing double quote on seq field
   sequence = nopref[:j]                 # Isolate the sequence number
   return sequence


# makeMsg: Compose a message for transmission, encoding it and adding a crc
#
# msgData is the data to be transmitted
# sequence is the message sequence number
# 
# The CRC is computed over the composed and UTF-8-encoded message.
#
# The message, with CRC appended, is returned, as well as the CRC itself, 
# which can be used for later validations.

def makeMsg(msgData, sequence): 

   # Assemble the message
   message = XML_PREFIX1 + str(sequence) + XML_PREFIX2 + msgData + XML_SUFFIX
   
   # Encode the message
   messageEncode = message.encode('utf-8')

   # Calculate a CRC on the encoded message
   crc = CRC32().calculate(messageEncode).to_bytes(4,'big')
  
   # Append the CRC to the message  
   messageSend = messageEncode + crc

   return messageSend, crc


# processMsg: Send the message on the two interfaces, and make sure it is properly received on both
#
# message is the message to be processed
#
# count is the message count

def processMsg(message, count):
 
   # Build the message
   messageBuilt, crc0 = makeMsg(message, count)

   # Send the message from A to B and from B to A
   sockA.sendto( messageBuilt, (UDP_IP_B, UDP_PORT))
   sockB.sendto( messageBuilt, (UDP_IP_A, UDP_PORT))
   print( "\nSent message " + str(count) + " on " + sockA.getsockname()[0] + " and " + sockB.getsockname()[0] )

   # Receive message on interface A
   rc = recvMsg(sockA, crc0)

   # Receive message on interface B
   rc = recvMsg(sockB, crc0)


# recvMsg: Receive a message
#
# sock is the socket to receive the message on
#
# crc is the expected CRC
# 
# The return code is: 0 - message was successful
#                     1 - message timed out or CRC was invalid
 
def recvMsg(sock, crc):

   retcode = 0
   sockName = sock.getsockname()[0]   # Socket name for messages
   try: 
      data = sock.recv(1024) # buffer size is 1024 bytes
   except socket.timeout:
      print("Recieve on " + sockName + " timed out")
      retcode = 1
   else: 
      messageRcvd, crcRcvd = splitMsg(data)
      messageDecode = messageRcvd.decode('UTF-8')
      seqRcvd = getSequence(messageDecode)
      print( "Received message on " + sockName )
      if (crcRcvd != crc):
         retcode = 1
         print( "CRC of message received on " + sockName + " is incorrect" )
   return retcode


# splitMsg: Split a received message into message data and CRC
#
# message is the received message
# 
# Message data and CRC are returned. 
 
def splitMsg(message):
   dataLen = len(message) - 4
   msgData = message[0:dataLen]
   crc = message[dataLen:]
   return msgData, crc


#
# Mainline program
#

# Set up the sockets
sockA = socket.socket(IP_ADDRESS_TYPE, # Internet
                     socket.SOCK_DGRAM) # UDP

sockB = socket.socket(IP_ADDRESS_TYPE, # Internet
                     socket.SOCK_DGRAM) # UDP

# Bind the sockets to their addresses
sockA.bind((UDP_IP_A, UDP_PORT))
sockA.settimeout(1)
 
sockB.bind((UDP_IP_B, UDP_PORT)) 
sockB.settimeout(1)

# Generate endless test messages
count = 0
while( True ):

   # Process the message
   processMsg(DATA, count)
    
   # Bump the count
   count = count + 1

   # Wait a bit
   time.sleep(1)

The receiving end

On the receiving machine, the NICs (NIC2 and NIC3 in this example) must be set to promiscuous mode, meaning that they see all packets, not just the ones addressed to them.

At the Ethernet level (Layer 2 in the OSI model), packets are addressed by MAC address, rather than IP address. So when NIC6 wants to send packet to NIC7 (10.10.10.7), it will first send an ARP (Address Resolution Protocol) request that says “Who has 10.10.10.7?”. NIC7 will send back a response giving its MAC address, and NIC6 will use that address in the Ethernet frame.

So even if we set up NIC2 and NIC3 to have the same IP addresses as NIC6 (10.10.10.6) and NIC7 (10.10.10.7), they would not see the packets addressed to NIC6 and NIC7, because they have different MAC addresses. With the NICs set to promiscuous mode, they can receive whatever packets come down the wire.

In Unix-like systems, you set the NICs to promiscuous mode like this:
ifconfig nic2 promisc
ifconfig nic3 promisc

It is more difficult in Windows, and appears to depend on the specific driver for your NIC. A solution that will work for both Unix-like systems and Windows systems is libpcap/WinPcap, which is used by packet-capture tools like Wireshark and tcpdump. There are several bindings that let you use these libraries in the language of your choice. For Python, this includes PyPCAP, winpcapy, PyPl, PcapPy, and pylibpcap.

In Part 2, we will look into how to use these libraries to receive the packets from the Ethernet TAP.

References for Normal Link Pulses and Fast Link Pulses


Comments

Leave a Reply

Your email address will not be published. Required fields are marked *