You will all have heard of TCP/IP and know that it is somehow used by the Internet, but what it is really and how does it work? Today’s communications equipment is sufficiently advanced that it is able to determine automatically the best parameters to use when communicating over the Internet, and connections to remote systems are seamless, but when you appreciate the sheer amount of activity that happens in the background, it is amazing that the system works at all, let alone as well as it does!
I thought a blog post might make some of you a little more “reverential”!
This is a vast subject, and this post is by no means intended to be a comprehensive analysis of the technology. I hope, instead, to give a broad summary of how the protocol works in simple terms. This subject is, however, inherently technical in nature.
What is it?
TCP/IP is two distinct protocols: TCP and IP. They are always lumped together, but each is a protocol in its own right and they do not necessarily have to be used together.
TCP is the Transmission Control Protocol. Simply put, it is this protocol that breaks down the data you wish to send to another device into small “packets” and reassembles them all again at the other end. It was developed during the 1970s by the US Defence Department and was designed to allow dissimilar systems (such as Windows and Unix) to communicate with each other: it is completely hardware and software independent. Its main function is to translate the data created by different applications, be it email, FTP, HTTP or whatever, into a common form. Once created, the data is then transmitted,monitored, and any errors which occur are signalled and corrected.
IP is the Internet Protocol. It is this protocol that is responsible for the routing of data across the Internet, ensuring that it is delivered via the shortest route, and managing the eventuality of one or more nodes on the Internet being damaged. Again, this technology was developed by the US military and was intended to be able to successfully withstand a military installation being destroyed in the event of a mainland conflict. Prior to 1969, the entire defence computer system was held in a central location. This was deemed an unsatisfactory risk and so millions was spent on developing the DARPAnet (Defence Advanced Research Projects Agency network) – which spanned several states and was the largest Wide Area Network in the world, allowing for data to be replicated between sites. DARPAnet was the precursor to the Internet as we know it today, before it was connected up to JANet (the network shared by the academic community), and finally other enterprises were allowed access to it.
TCP
The Transmission Control Protocol is actually a suite of protocols rather than just one, and for this reason it is sometimes referred as a “stack”. At the top of the stack is the application which interacts with the user and receives the data to be transmitted, the application may be a web browser, an email client, an instant messaging program or whatever. At the bottom of the stack is the physical network medium over which the assembled “packets” are to be sent. The network medium can be a copper-based cable, light (in the case of fiber optic or infrared), radio waves (in the case of 3G or cellular connections) or whatever. TCP operates between the application and the network hardware, and does not need to be aware of the specifics of either: it receives the data, breaks it down into small units and sends them on their way to their destination.
TCP is what is known as a “Transport Layer” protocol. To help standardise the TCP implementations used by the different hardware and software manufacturers around the world, the ISO (International Standards Organisation), developed a theoretical construct, called the ISO 7-layer model, which breaks down the stack between the application and the networking hardware into seven “layers”:
This is a model only, and used to help explain the different functions within the TCP protocol suite. In reality, the functions of some of the layers are combined.
I will now look at the function of each layer in turn, beginning at the top of the stack, with the Application layer. Before I do, however, it is perhaps necessary to look at how machines communicate with each over a network.
All networks use a form of addressing to transfer data from one machine to another. Using the analogy of a letter, a network address is the equivalent of a mailing address in that it contains a name, an address and a route. In networking terms, the name could be a user, an application or a machine. The address is the machine’s location on the network. Each device on a network that communicates with others has a unique physical address (also known as the hardware address, or MAC address). MAC stands for Media Access Control and the MAC address is a unique address burnt into the physical networking hardware. Legally no two network interfaces can have the same MAC address. The route tells the systems how to send data so that it gets to the address.
The TCP stack
Each of the layers in the stack communicate with each other. There may be several protocols operating at each layer of the stack, and it is important that as the data passes between the layers, it is delivered to the correct protocol. To achieve this, as the data passes down through the stack, each layer adds its own information to the data, in the form of a “header” at the beginning of the data fragment. As the data passes up through the stack, this header is read by the layer above and then removed by the appropriate protocol. The Data Link layer also adds a “footer” at the end of the packet prior to the packet being sent on its way over the network.
The Application Layer
The Application layer is the point at which applications can access the TCP stack and the underlying network resources. The protocols operating at the Application layer are the protocols that programs use to access the network. These include the protocols you will have heard of already, such as SMTP, FTP, HTTP, etc.
The Presentation Layer
This layer is responsible for ensuring that the data being submitted to and received from applications is done so on the correct “port”: the TCP stack is divided into different ports, with each application protocol having its own individual port number (SMTP uses port 25, for example, and HTTP port 80). As data is passed down through the stack, the Presentation layer adds a header to the data including information on the port number that the data was received via. On the receiving system, as the data passes up through the stack, the Presentation layer reads this same header, removes it and then submits the data to the correct port to the application waiting on the layer above.
Normally, there is no need for the user to specify port information within an application, as it is “assumed” that if you are using a web browser then the information you are requesting should be requested over port 80, and if you are using an FTP program, for example, then you wish to connect to the FTP server on port 21. If you wish to use non-standard ports, then these can be specified by placing a colon after the address of the server, like this:
http://www.myserver.com:81
The Session Layer
The Session layer is responsible for coordinating the exchange of data between the upper layers and the lower layers. The layers in the lower section of the stack are concerned solely with the transmission of the data.
In reality, these three layers tend to be combined, so that there is a single application layer which delivers to the Transport Layer, which in turn delivers to the underlying network layers.
The Transport Layer
The Transport layer provides such services as packet “numbering” and error correction. When large amounts of data are sent over a network link, it is vital that the packetized data be reassembled in the correct order. Therefore all packets in a single “data stream” are numbered, so that should not all packets arrive at the destination in the correct order, the receiving system can order them correctly before passing them up to the next layer.TCP operates on this Layer, but is not the only Transport Layer protocol. I will look at TCP in more detail later.
The Network Layer
The protocols operating on the Network layer are known as the end-to-end protocols: it is the Network layer protocols that communicate with the destination machine to ensure that the data is received successfully.
The most common of the Network layer protocols is IP, the Internet Protocol.
The Network layer adds addressing information to the packet when it adds its own header. The addressing information in this header contains the address of the destination machine. In the case of IP, it is the IP address that is used. I look at the IP address in more detail later.
The Data Link Layer
This is the interface between the networking hardware and the software operating in the network layer above. At the Data Link layer, the data passed down from the network layer has a header and footer appended to it, which prepares it for transmission over the Physical layer. Using the analogy of a letter, the header and footer added at the Data Link layer are the equivalent of the envelope you use to send a letter (or data): it contains the address of the sending system and the destination system.
It is important to note that the Data Link layer protocols are only able to communicate with the protocols operating at the Data Link layer on other systems: that is to say that the Data Link layer on an Ethernet network is only able to communicate with other machines on the local network, even if the final destination of the packet is not on that local network. The addressing scheme used by the Network layer and the Data Link layer are completely different: an IP address identifies the specific machine AND the network it is on, whereas the address in the header added by the Data Link layer only identifies a machine on that local network, or more specifically the physical address of the network interface of that machine (ie the MAC address of the machine’s hardware).
The Data Link layer is also responsible for determining which protocol on the Network layer created the packet (Ethernet, IR, etc), and for ensuring that the packet gets delivered to the correct protocol on the target system.
Error correction is also performed at this layer. In the footer added to the packet by the Data Link layer on the sending system, a CRC value is included. CRC stands for Cyclic Redundancy Check. Essentially what this does is to add the values of all of the “1” values in the data “payload” and stores that value in the footer. The receiving system compares the CRC value with the payload it has received, and if they don’t match, the packet is considered to be corrupt and is discarded, and a signal is sent to the sending system to re-send that specific packet. The CRC value is also known as the “Checksum”.
The Physical Layer
This is the hardware element of the computer’s connection to the network: be it an Ethernet card, infrared port, analogue modem or cellular device, such as a 3G datacard.
It is important to understand that ALL data arriving at the Physical layer is passed up to the Data Link layer and then to the Network layer. The network layer will examine the data it receives and then determine whether or not it is actually intended for that machine and, if not, resubmit it back to the Data Link layer to be re-transmitted on its way.
Therefore on a packet’s journey, the information contained in the Data Link header may change several times as the physical medium over which the packet travels changes, but the Network layer header will remain intact throughout.
The protocols operating on the Network layer “know” whether or not the packet is intended for that system based on the packet’s “address”.
Therefore, it is the Network layer that is responsible for “routing” of data. The individual, interconnected networks that make up the Internet are connected by Routers:it is the function of a router to examine each packet that passes through it and determine whether or not it is destined for a machine on the local network that it is connected to and if not, forward it to the next router:
Internet Protocol
As we saw earlier, networks all use a form of addressing. Ethernet is the most commonly-used form of “fixed line” networking. The addressing system used by Ethernet can only be used to send data to other Ethernet devices, however, and as such is a Data Link layer protocol. Only Network layer protocols can address packets to devices lying in other networks. The Internet Protocol is the most commonly-used Network layer protocol.
It is important to note that the Internet Protocol is so called because it is network-agnostic: it can route data between two incompatible physical networks; it is an inter-network protocol. It is not, therefore, used solely by the Internet that we know today – it can be used on dedicated networks that have no connection to the Internet at all.
IP was developed by the engineers working on the DARPAnet, the most famous of which was a man called Vint Cerf. IP is responsible for routing the data that it presented to it by the TCP stack to its destination.
IP operates at the Network layer. As data is passed to it from the layer above, address information is added to the data in the form of a header. This header will contain several pieces of information, among them the address of both the sending machine and the destination machine. This header will not be altered until the data packet reaches the destination machine, at which point the header will be removed and the data passed to the higher layers. The IP packet may, on its journey, be “encapsulated” within a larger packet, with its own IP header, as it is sent over different networks on the route to its destination, but the IP header itself will not be altered. Using the analogy of a letter, the letter itself may be placed into different mailbags and transported by various trucks and planes during the course of its journey, but the envelope remains sealed. Only the addressee is permitted to open the contents of the envelope.
The IP Header
The IP Header is added to the data packet at the Network layer. Currently we are at version 6 of the protocol, but this has largely been unimplemented, version 4 still being the most commonly used. I will therefore only be looking at IPv4 in this article.
The IP header itself is 20-bytes of data and contains several pieces of information:
Version – the version of IP that created the header
Length – details the “length” of the IP header in bits
Type Of Service – this field is not always used, but can be used to “prioritise” traffic – a value of 1 in this field would cause a router to forward this packet in preference to a packet with a value of, say, 4 in this field
Total Length – details the length of the entire data packet
Identification – details the packets “number” if it is part of a longer stream of data
Time To Live – (TTL) this field specifies the number of networks that the packet should be allowed to travel through on the way to its destination before it should be deleted. As the packet passes through a router, the router will reduce the value of this field by 1. If this practice was not implemented, packets could in theory roam the Internet forever.
Header Checksum – a CRC value for the data contained in the IP header so far, enabling the receiving system to perform error correction on the received packet.
Source IP Address – the address of the sending system
Destination IP Address – the address of the target system
Options – this field can be used to include up to 16 different optional values, if supported by the equipment through which the packet is passing. One of these options is “timestamping”.
This is the information contained in the IP header. The rest of the information in the IP packet is the data, or “payload”.
Fragmentation
Different networks support different maximum packet sizes. Sometimes a packet will need to cross a network that supports a smaller packet size than the network which created it. In this situation, the packet will need to be “fragmented” – divided into smaller packets to be routed across the network, before being re-joined when a network is reached that supports a larger packet size.
When this happens, the IP header added to the smaller “sub-packet” will contain further information on the fragmentation process, including numbering, length, etc (whilst still maintaining the information on the “whole” packet).
The TCP Header
TCP also uses a Header which contains several pieces of information. The TCP header is added to the data packet at the Transport Layer before being passed to the Network Layer. Therefore the IP packet information is said to “encapsulate” the TCP packet.
A TCP header can contain the following information:
Source Port – the port number of the application that submitted the data
Destination Port – the port number of the target application on the receiving system
Sequence Number – the number of the packet segment within the whole data stream (note the initial packet is not always number 1, to prevent delayed data from an old connection being incorporated into a new connection)
Acknowledgement Sequence Number – this field is used by the receiving system when communicating with the sending system, to let the sender know that a certain sequence packet has been received. This type of TCP packet is known as an “ACK”.
Next come six fields which are all 1-bit in length, either 0 or 1 to indicate off or on status:
URG – used to indicate whether a packet is urgent or not
ACK – used to indicate that the packet is an ACK or not
PSH – used to indicate whether the sending system wants to “push” data to the receiving system
RST – used to reset the connection
SYN – (for synchronisation) used to establish the connection
FIN – used to close down the connection
Window – this field can be used to hold “padding data” (should any of the previous fields not have been used) to ensure that the Header has a uniform size for the Checksum
Checksum – this field is used to create a CRC value of the preceding fields which is verified by the receiving system.
Next come a number of optional fields:
Maximum Transmission Unit (MTU) – this field is used when the connection is being established, to inform the remote machine of the maximum packet size it is capable of receiving.
Window Scale – this field is used to address the issue of latency (the delay between the transmission and reception of data across the network). This option sets the amount of “payload” data that the packet holds. In conditions of high latency, it may be preferable to send lots of small packets rather than fewer, larger packets.
SACK-permitted – this field can be used to reduce the amount of traffic being sent over the network: provided that both systems support this feature, instead of acknowledging all received packets, the receiving system only requests those packets it doesn’t receive (if they don’t arrive within a preset interval).
All of this information adds a significant amount of data to the size of the final packet, resulting a large amount of data “overhead”. When sending data over a 1 Mbps link, only a fraction of this connection speed is used to transmit the user data, or payload, the rest is taken up with error correction and sequencing information and acknowledgement. Should communications not require such a “reliable” connection protocol, enabling a higher rate of data throughput, others are available. UDP is the most commonly-used “unreliable” protocol used.
Establishing a TCP connection
When two systems need to communicate via TCP. The sending system sends the remote system an initial sequence number using a SYN packet (telling the remote system what number its stream will begin with)
The remote system responds with aACK packet to acknowledge receipt of the SYN packet, and also with a SYN packet of its own (to let the sending system know that number its own stream will begin with)
The sending system then responds with an ACK of this packet.
The connection is now open. TCP will now endeavour to exchange data with the remote system, establishing where possible the peak efficiency between speed and data loss. Both sending and receiving systems will have a receive “buffer” where packets are stored prior to being processed. This is why when starting a download, normally the connection will begin very quickly and then settle down at a slower rate. Managing this rate is very important: increased data transfer leads to network congestion which leads to packet loss, which requires more data retransmission which leads to further network congestion which leads to a declining spiral.
During the establishment of the connection, the receiving system will advertise the size of its receiving buffer, and the sending system will ensure that the rate it sends at will not exceed this value. The sending system must also keep a copy of all data it sends in its send buffer until it receives a positive ACK from the receiving system that the data has been successfully received and can be cleared from the send buffer (in case it needs to be re-sent). This buffer cannot be exceeded either. Both machines will calculate, based on the size of the sending and receiving buffers, what is known as a “congestion window”, and is the optimum amount of data that can be placed on the network at any given time. The point at which packet loss occurs is known as the “threshold”. Therefore the value of the threshold should always be greater than the value of the congestion window. Should packet loss occur, both systems will lower the rate of transmission until the rate of packet loss falls, and then slowly increase the transmission rate again. This is why TCP connections vary in the rate of connection speed.
IP Addressing
IP addresses contain information on both the network, and the system’s location on that network. IPv4 addresses are 32 bits in length. They are written using four decimal numbers ranging from 0 to 255, separated by periods, as in 192.168.1.12. This is known as decimal dotted notation. This provides a maximum of some 4,294,967,296 unique addresses.
Unlike MAC addresses, which are “hard-coded” into the network interface hardware, IP addresses must be assigned to machines. On local networks, this is done by the network administrator. On the Internet, this is done by the Internet Assigned Numbers Authority (IANA). On either type of network, it is essential that no two machines on the same network share the same IP address.
IP addressing is complicated. Each IP address contains the machine’s “host” identifier and its “network” identifier. The division between the two is not always the same: there are different “classes” of IP address. Five different classes of address: A, B, C, D and E.
Class A addresses are used by large networks that have many machines. 24 bits of the address are used for the local address, and 7 bits are used for the network address.
Class B addresses are used by intermediate networks. 16-bits are used for the local address and 14 bits are used for the network address.
Class C addresses use 8 bits for the local address (limiting them to only 256 machines), and 21 bits for the network address.
Class D addresses are used for “multicasting” when a general broadcast to more than one device is necessary.
Class E addresses are, as yet, unused.
To Summarise
So far you have seen how a packet is constructed by TCP, and then routed to its destination by IP.
When an application needs to send data over a network connection, a data packet is constructed. A checksum is made of the data and stored in the packet header. The IP address of the next “hop” on the network (the network “gateway”, if the packet is not destined for a machine on the local network) is determined and entered in the IP header of the packet. A checksum for the IP header itself is calculated and added. The packet is then passed to the physical connection to the network.
As the packet passes along the network, each gateway stops the packet and examines the outer IP header. The checksum is verified to ensure the validity of the packet. If the check fails, the packet is discarded and a message sent to the originating system to regenerate the packet. Provided that the check is successful, the TTL value is then decreased. If decreasing the value of the TTL field returns a value of 0, then the packet is discarded and an error message is sent to the originating system. Otherwise, the TTL value is decreased by 1, the address of the next hop on the network is identified and the IP header rebuilt, with a new checksum value.
If fragmentation is necessary, the packet is divided, and new, smaller, packets are assembled with the correct header information and passed back to the network layer.
When the packet is finally received by the destination device, a checksum is performed. Provided that the check is successful, the device checks to see if the packet is a fragment or a complete packet. If it is a fragment it will wait for the remaining fragments to also be delivered, checking the numbering of each fragment as it arrives. Finally, the IP header is removed and the data is passed up the TCP layers. If a response was required, an acknowledgement message is returned to the sending system.
It is a miracle that it works at all!
At this stage, I’m afraid, things are going to get a bit technical.
DHCP (Dynamic Host Configuration Protocol)
DHCP was developed by Microsoft to address the need for machines (or “hosts”) on an IP network to all have unique IP addresses. You may have seen the option on a PC within the networking settings to “obtain an IP address automatically”, selecting this option activates the DHCP client on the PC and causes it to look for a DHCP server on the local network. Should a DHCP server have been set up on the network, it will receive the request from the client and assign it an IP address. The DHCP server itself will be allocated a “pool” of addresses it can assign to clients, and each address, when assigned, will only be valid for a certain length of time after which the client will need to re-request an address. Should the client not re-request an address, that address will then be available to be assigned to another client.
PPP (Point to Point Protocol)
PPP is a Data Link layer protocol. It is used to establish communications between two machines on a dedicated connection. Unlike a LAN, where the connection medium is shared and MAC addresses are required to ensure that the data is routed correctly, PPP connections operate on media that are not shared, such as a dial-up connection to a modem, or cellular or DSL connections.
The function of PPP is to agree on the networking parameters to be used once the physical link between nodes on the network has been established (ie, after TCP has completed its own handshaking procedure). The procedure is commenced by the transmission of an LCP (Link Control Protocol) Request message. This message will contain a list of options which the receiving system can either acknowledge or deny and propose a list of its own until the two systems agree. Typical options will include the method of compression to be used, among others.
NAT (Network Address Translation)
NAT addresses the issue of the shortage of available IP addresses. Although the 32-bit IP addresses allows for 4,294,967,296 IP addresses, in reality there are fewer available, due to the reservation of some addresses for special uses and the way in which the different “Classes” of address were allocated. The explosion of the Internet has meant that the available addresses space is not sufficient for demand.
The answer is to redesign the address, a solution that has been provided in the form of IPv6, but the implementation of this new technology will take several years as it will require upgrading the infrastructure of the entire Internet. In the interim, NAT allows for a private network to present a single IP address to the Internet, and have many more “private” IP addresses “behind” the “public” IP address.
Here is an example of how NAT works:
1 An internal network has been set up with what are called “non-routable” IP addresses (addresses that were not assigned to it by the IANA). Non-routable, or private, IP addresses are usually in the form 192.168.x.x, 10.x.x.x or 172.16.x.x
2 A NAT-capable router is installed, with one network interface connected to the local network, with a non-routable address assigned to it, and one interface connected to the Internet, with the public, or “routable”, IP address allocated to it that was assigned to the company by the IANA.
3 A machine on the internal network requests a web page from a web server on the Internet. The machine creates an HTTP request and submits it to the network, which sends it to the router.
4 The router receives the request and sees that it is destined for a machine not on the local network. The router saves the machine’s non-routable IP address to an address translation table. It then re-writes the IP header, replacing the source IP address with its own public IP address and sends the request out across the Internet.
5 When the response comes back from the web server, the router checks the address translation table, rewrites the IP header of the incoming data, changing the destination address from its own to the address of the machine on the internal network, and forwards it on.
Subnetting
Subnetting is another method of increasing the number of machines that can be “squeezed” into the available IP address space.
If a network administrator wants to create a new network that is accessible from the Internet, NAT can be used, but the network still requires one public IP address which needs to be requested from the IANA. Subnetting would allow the existing IP address to be used, with both internal networks sitting behind it. Subnetting effectively adds another level to the IP “hierarchy”.
We saw earlier that IP addresses contain both a network address and the address of the machine on that network. The “length” of the network address depends on the Class of the IP address (A,B,C,D or E). Subnetting adds another level to the address: the network address is left intact, but the machine address is divided, resulting in a network address, subnet address and machine address. Because the network address is the same, the route to the network is the same, but when the packet arrives at the external router, the router will then determine which subnet on the local network the packet is destined for. As far as the Internet is concerned, it doesn’t need to know about what happens behind the public address of the router.
The length of the subnet address can vary, depending on the number of subnets the administrator wishes to create. Because a larger number of subnets will require a larger address, more data will need to be “borrowed” from the machine address, meaning that each subnet will only be able to have a smaller number of machines. The administrator will need to take all of this into account when designing the network.
The division between the subnet address and the machine address is identified by the Subnet Mask.
Let’s have a look at an example:
A network has been assigned a public IP address of 193.1.1.0, a Class C address.
The administrator needs to divide the network into 6 subnets, the largest of which will need to support 25 machines.
Because only x number of individual bits can be borrowed from the machine address to provide the subnet address, subnets can only be created in powers of 2. Therefore, to create 6 subnets, the administrator will actually need to create 8 (and just not use 2 of them).
8 is 2 to the power of 3. Therefore 3 bits will need to be used to define the subnet address.
A Class C IP address has 24 bits assigned to the network address. Therefore the new “extended” address will be 27-bits long.
A byte is made up of 8 bits, and can represent any value between 0 and 255:
1 1 1 1 1 1 1 1
128 64 32 16 8 4 2 0
Borrowing 3 bits from the address means that values 16, 32 and 128 will not be available. 16+32+176 = 224
Therefore, the subnet mask is written as 255.255.255.224
This leaves 5 bits for the machine address. 2 to the power of 5 is 32, meaning that a maximum of 32 machines can be defined on each of the 8 subnets – more than enough for the 25 the administrator needs to support.
The 8 subnets now need to be defined. They are numbered 0 to 7. In binary notation, these numbers are written as follows:
0 00000000
1 00000001
2 00000010
3 00000011
4 00000100
5 00000101
6 00000110
7 00000111
The public IP address 193.1.1.0, in binary notation, is written as follows:
11000001.00000001.00000001.00000000
The 3-bit identifier of the subnet needs to be added to the network address to identify it, therefore:
Subnet 1 11000001.00000001.00000001.00000000 193.1.1.0 / 255.255.255.224
Subnet 2 11000001.00000001.00000001.00100000 193.1.1.32 / 255.255.255.224
Subnet 3 11000001.00000001.00000001.01000000 193.1.1.64 / 255.255.255.224
Subnet 4 11000001.00000001.00000001.01100000 193.1.1.96 / 255.255.255.224
Subnet 5 11000001.00000001.00000001.10000000 193.1.1.128 / 255.255.255.224
Subnet 6 11000001.00000001.00000001.10100000 193.1.1.160 / 255.255.255.224
Subnet 7 11000001.00000001.00000001.11000000 193.1.1.192 / 255.255.255.224
Subnet 8 11000001.00000001.00000001.11100000 193.1.1.224 / 255.255.255.224
The remaining 5 bits can be used to identify the machine.
NOTE – certain subnet masks cannot be used. A mask containing all 1s or all 0s are normally reserved for “broadcasting”: sending a message to all machines on a network.
Bootnote
I hope that this post has enlightened you to the sheer complexity that is involved in sending and receiving data over the Internet, and will give you pause for thought in the future should not everything go according to plan or your download be slower than you’d like!