How to calculate the usable bandwidth on a Gigabit Ethernet network.
In this article we will look at how much throughput of actual data we have on a Gigabit Ethernet based network and if this will increase by using Jumbo Frames.
How much of this is user data and how much is overhead?
The bandwidth on a Gigabit Ethernet network is defined that a node could send 1 000 000 000 bits each second, that is one billion 1 or 0s every second. Bits are most often combined into bytes and since eight bits make up one byte this will give the possibility to transfer 125 000 000 bytes per second (1000000000 / 8).
Unfortunately not all of these 125000000 bytes/second can be used to send data as we have multiple layers of overhead. As you may be aware of, the data transferred over a Ethernet based network must be divided into “frames”. The size of these frames regulates the maximum number of bytes to send together. The maximum frame size for Ethernet has been 1518 byte for the last 25 years or more.
Each frame will cause some overhead, both inside the frame but less known also on the “outside”. Before each frame is sent there is certain combination of bits that must be transmitted, called the Preamble, which basically signals to the receiver that a frame is coming right behind it. The preamble is 8 bytes and is sent just before each and every frame.
When the main body of the frame (1518 byte) has been transferred we might want to send another one. Since we are not using the old CSMA/CD access method (used only for half duplex) we do not have to “sense the cable” to see if it is free – which would cost time, but the Ethernet standard defines that for full duplex transmissions there has to be a certain amount of idle bytes before next frame is sent onto the wire.
This is called the Interframe Gap and is 12 bytes long. So between all frames we have to leave at least 12 bytes “empty” to give the receiver side the time needed to prepare for the next incoming frame.
This will mean that each frame actually uses:
12 empty bytes of Interframe Gap + 1518 bytes of frame data + 8 bytes of preamble = 1538
This makes that each frame actually consumes 1538 bytes of bandwidth and if we remember that we had “time slots” for sending 125000000 bytes each second this will allow space for 81274 frames per second. (125000000 / 1538)
So on default Gigabit Ethernet we can transmit over 81000 full size frames each second, a quite impressive number. Since we are running full duplex we could at the same time receive 81000 frames too!
We shall continue to study the overhead for this. So for each frame, we lose 12 + 8 bytes used for Interframe Gap and Preamble, which could be considered to be “outside” of the frame, but could we use the rest to send our actual data? No, there is some more overhead that will be going on.
The first 14 byte of the frame will be used for the Ethernet header and the last 4 bytes will contain a checksum trying to detect transfer errors. This uses the CRC32 checksum algorithm and is called the Frame Check Sequence (FCS).
This means that we lose a total of 18 bytes in overhead for the Ethernet header in the beginning and the checksum at the end. (The blue parts above could be seen as something like a “frame” around the data carried inside.) The number of bytes left is called the Maximum Transmission Unit (MTU) and will be 1500 bytes on default Ethernet. MTU is the payload that could be carried inside an Ethernet frame, see picture above. It is a common misunderstanding that MTU is the frame size, but really is the data inside the frame only.
Just behind the Ethernet header we will most likely find the IP header. If using ordinary IPv4 this header will be 20 bytes long. And behind the IP header we will also most likely find the TCP header, which have the same length of 20 bytes. The amount of data that could be transferred in each TCP segment is called the Maximum Segment Size (MSS) and is typically 1460 bytes.
So the Ethernet header and checksum plus the IP and TCP headers will together add 58 bytes to the overhead. Adding the Interframe Gap and the Preamble gives 20 more. So for each 1460 bytes of data sent we have a minimum of 78 extra bytes handling the transfer at different layers. All of these are very important, but does cause an overhead at the same time.
As noted in the beginning of this article we had the possibility to send 125000000 bytes/second on Gigabit Ethernet. When each frame consumes 1538 byte of bandwidth that gave us 81274 frames/second (125000000 / 1538). If each frame carries a maximum of 1460 bytes of user data this means that we could transfer 118660598 data bytes per second (81274 frames x 1460 byte of data), i.e. around 118 MB/s.
This means that when using default Ethernet frame size of 1518 byte (MTU = 1500) we have an efficiency of around 94% (118660598 / 125000000), meaning that the other 6% is used for the protocols at various layer, which we could call overhead.
If enabling so called Jumbo Frames on all equipment, we could have a potential increase in the actual bandwidth used for our data. Let us look at that.
A commonly used MTU value for Jumbo Frames is 9000. First we would have to add the overhead for Ethernet (14+4 bytes), Preamble (8 bytes) and Interframe Gap (12 bytes). This makes will make the frame consume 9038 bytes of bandwidth and from the total amount of 125000000 bytes available to send each second we will have a total of 13830 jumbo frames (125000000 / 9038). So a lot less frames than the 81000 normal sized frames, but we will be able to carry more data inside each of the frames and by that reduce the network overhead.
(There are also other types of overhead, like CPU time in hosts and the work done at network interface cards, switches and routers, but in this article we will only look at the bandwidth usage.)
If we remove the overhead for Interframe Gap, Ethernet CRC, TCP, IP, Ethernet header and the Preamble we would end up with 8960 bytes of data inside each TCP segment. This means that the Maximum Segment Size, the MSS, is 8960 byte and is a lot larger than default 1460 byte. A MSS of 8960 multiplied with 13830 (number of frames) gives 123916800 bytes for user data.
This will give us a really great efficiency, of 99% (123916800 / 125000000). So by increasing the frame size we would have almost five percent more bandwidth available for data, compared to about 94% for default frame size.
Conclusion: Default Gigabit Ethernet has an impressive number of frames (about 81000 per second) possible and a high throughput for actual data (about 118 MB/s). By increasing the MTU to 9000 we could deliver even more data on the same bandwidth, up to 123 MB/s, thanks to the decreased amount of overhead due to a lower number of frames. Jumbo Frames could use the whole of 99% of Gigabit Ethernet bandwidth to carry our data.