Network Byte Order

There are probably perfectly legitimate reasons for the world being this way but I don’t know what it is. In a pretty substantial chunk of the world, when we write numbers, we write them from left to right meaning the portion of the number with the largest value is on the left hand side and typically, we would write from left to right. What this means is that if I write the number 6785, what I mean is six thousand, seven hundred eighty-five. When we are talking about digital communications, however, everything is in the form of a byte. Rather than dealing with the all of the individual bits that a byte would normally be represented as, let’s shorthand it to hexadecimal. One hexadecimal digit pair is how we would represent a single byte. The reason for that is simple. Four bits gives me the values of 0-15 since 2^0 + 2^1 + 2^2 + 2^3 = 1 + 2 + 4 + 8 = 15 as the maximum value for a 4 bit number. Since a byte is 2 pairs of 4 bits and a single hexadecimal digit (values 0-F or 0-15) is 4 bits, 2 hexadecimal digits is a whole byte. Simple, right? 

Let’s move on to writing values, knowing that we are going to be talking about writing out bytes for now and we are going to represent them as hexadecimal. We are going to write out the word hello and it doesn’t much matter where we write this out because we can run into the same problem, no matter what we are doing. The title of this suggests we are talking about writing out to a network interface but we have the same problem on hard disks and in memory. No matter where we have to write bits and bytes, we have to decide how we are going to write it. When we write character values, we have to have a way of converting them to a number. As a result, we use a table lookup. The common table to lookup characters to get a numeric representation is the ASCII table. After doing the lookup, we get the following: 68 65 6C 6C 6F. Again, without getting into the bit level, we have to decide what order we are going to send these in. Do you send the h first or the o first and then follow with the rest of the characters?

Thinking about numbers where the result is more catastrophic if you get it wrong, let’s take a look at a 16-bit value. The value 1348 is 0x0544 in hexadecimal. This is two bytes. If I send the 44 followed by the 05, how does the receiving party interpret that. If I send the 05 before I send the 44, I am sending in big-endian form. The reason for that is that I am sending the most significant data first — the data that has the largest value or is the biggest. If I send the 44 first, I am sending in little-endian form. If the receiving end is used to doing things a different way, I could go from sending the value 1348 but on the receiving end getting 17413. This is a very big difference. The reason is that if I send 05 then 44, which is big-endian, but the other end assumes little-endian, it would view what I sent as 44 05. 

So, which is the right way? Neither, actually. But since little-endian systems need to talk to big-endian systems, there had to be some consensus. As a result, there are two ordering schemes. There is host-order, which is whatever order your particular system architecture uses (Intel uses little-endian, by the way) and then there is network order. Network byte order is a synonym for big-endian, since historically more hardware architectures used the big-endian form of storing data. Of course, these days, far more systems on the network use little-endian simply because of the ubiquity of systems with Intel processors. 

When you are storing data on your own system, it doesn’t much matter how it’s represented because the operating system has to take care of writing and reading so you get the real value at a programmatic layer. When you are trying to interface with values on disk at a raw level, as you might in the case of forensics, you have to be aware of multi-byte values and what architecture the data was written on. If you have a multi-byte value that was written from a little-endian system, you need to remember to reverse the order of the bytes. But only within that value. 

If you are talking to another system, something has to handle the translation from host to network form. Languages that are capable of talking to the network, generally have those functions available. As an example, we can see how the process works in Python, below. 

kilroy@opus:~$ python3

Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 16 2013, 23:39:35) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import socket

>>> socket.ntohl(45)

754974720

>>> socket.htonl(45)

754974720

>>> 

 

The socket class has a number of conversion functions including the two above. In the first example, I am converting from network byte order to a host long. In this case, that means I am converting to a 32-bit value that is little-endian. In the second example, I am converting from a little-endian number to a network long. Again, a long data type is 32 bytes in this case. As a result, you take the value of 45 in bits and then just turn all the bits around and re-calculate back to decimal. You can see the result we get is significantly larger than the value we have put in. 

 

 

More Net Neutrality

Our friends at the Federal Candy Company, specifically in the person of Tom Wheeler are likely to release new guidance on a concept called Net Neutrality, sometimes called The Open Internet. The FCC’s current stance is that no traffic should be blocked unless it is illegal. However, that may well soon change. Not surprisingly, this has caused some amount of anguish on the part of Internet activists and anyone who has gotten used to the idea that their traffic flows freely (it doesn’t really, but more on that later) across the Internet. Considering that Tom Wheeler comes from a background of companies he is now responsible for regulating, it may not be terribly surprising that under his guidance, the FCC may soon back down from their previous stance that carriers should not discriminate regarding the type of traffic they carry. 

Why are we in this position? Well, in 2010, the FCC released the Open Internet Order, which is the current stance of the FCC. Make note, by the way, that the FCC, for what say it does have, only has say over Internet service providers in the United States. The rest of the world is free to act however they damn well please. Verizon took the FCC to court to challenge the Open Internet Order and earlier this year, a court indicated that the FCC couldn’t make a such a rule. As a result, the FCC was sent back to its room to redo its homework. It is about to turn in its homework, which is why there is such a ruckus. 

Why can’t the FCC make such rules and hold the Internet service providers in the United States to them? The problem, in part anyway, is that the FCC designated the Internet and the service providers responsible for it as an information service. The District of Columbia court of appeals ruled that by designating the Internet as an information service, it couldn’t make rules like the Open Internet Order. What’s the way out of this mess? Well, one of the ways out is to designate all Internet service providers as a common carrier. A former FCC commissioner, Michael Copps, has made that very suggestion. What is a common carrier? The telephone companies like Fairpoint, Verizon, AT&T and others are all common carriers. A common carrier is an entity that provides a service for the “public convenience and necessity” meaning that what you are getting is a utility that you rely on in your day to day life. Common carriers have certain obligations that fall under Title II of the Communications Act of 1934. At the moment, Internet service providers do not fall under Title II, though the FCC could easily designate them under Title II and life would be very different. 

One of the biggest concerns around the Net Neutrality discussion is the impact on consumers and average businesses. Why? Well, another way out of this kerfuffle is to codify what the ISPs want to be able to do and that is to charge what are being called premiums to companies to carry their traffic. We covered this previously. Let’s break this down to a simple example. Take a look at the diagram below. You can see a nice little neighborhood of Alice, Bob and me. 

 

Let’s say that Alice and Bob have packages they want to exchange with one another. It would make some sense that when Bob has packages for Alice, she should come and get them. The same holds true if Alice has packages for Bob. She should let him know so he can come get them. We can assume something similar with me and Bob. This all makes sense and works out nicely. What happens if Alice suddenly has packages for me? When Bob comes to get his packages, Alice is throwing in packages for me into the mix, meaning that Bob now has to come and get packages for me. Maybe the same is true in the case of packages to Alice from me. Suddenly, Bob has become something of a pack mule shuffling packages between me and Alice. Bob has entered into these neighborly arrangements in good faith, assuming that he was getting something out of it. In this case, he gets to send packages to me and Alice and get packages in return. If suddenly, though, he is being asked to carry packages from me to Alice and vice versa, he gets nothing out of the deal. As a result, he may want to change his agreement with both me and Alice so we pay him to carry packages back and forth. Now it’s equitable. 

The same is true for Internet service providers. Let’s say that instead of the names Alice, Bob and Ric in those clouds, the names are YouTube (Google), Level 3 and Comcast. Picture me in the Comcast cloud, trying to get to YouTube. If Comcast doesn’t have a direct connection (peering arrangement) with YouTube (Google), it would need to carry that traffic across Level 3’s network. Level 3 has peering arrangements with both YouTube (Google) and Comcast because it makes sense for Level 3’s customers to have that peering arrangement, meaning that Level 3 expects to send roughly the same amount of packages to the others as it gets from them. This is an equitable deal. If it happens that suddenly Level 3 is receiving a lot of traffic between YouTube and Comcast without any benefit for itself, it may want to make a different arrangement with these other companies, shifting the relationship from one of peering to one of transit, meaning that the company, say Comcast, is now paying Level 3 to ship packages to other parts of the Internet on its behalf. This is not without lengthy precedent, including a highly charged and publicized case from nearly a decade ago involving Level 3 and Cogent. 

This all seems like good business practice, right? The problem we have is that the Telecommunications Act of 1996 made a lot of changes to the way the world of communications works and as a result, we have seen a lot of consolidation in the telecom space. Now we have companies like Comcast providing the vast majority of consumer broadband where at one point phone companies had a foot in the space as well. For the most part, phone companies have either pulled out or simply can’t compete when it comes to speed, though they sometimes have an advantage when it comes to reach. Why is this potentially troubling? Because Comcast sells Internet services and consumers are moving more and more to the Internet for their entertainment, which is Comcast’s biggest money maker. As its Internet customers begin moving away from entertainment services like cable television, Comcast will want to make up that money somewhere. What it may do is require that companies like Netflix, YouTube, VuDu, Hulu and so on pay for transit in order to get access to the eyeballs on the Comcast network.

What this means is that the biggest companies will end up winning because they will be the ones with the money to pay for access to the end user. One reason for this need to get access to the end user is because in many cases the end user is the product. YouTube (Google) makes money by selling ads to businesses that will be viewed by you, the end user. The same is true for several other companies. They make money by selling their users in some regard. Companies like Netflix offer low rates to you, the end user, because they may not be paying much for their Internet connection as compared to them having to pay surcharges to a number of Internet service providers just to  make sure their service is fast enough that end users will continue to stay with them.

Another risk is that Comcast, with its extensive reach into the desktop (end user) space could simply decide to choke a business off if it felt that there was too much competition coming from that new business. It would do this by slowing down the speed that packages from that business arrive at the end user, potentially making the service utterly unusable. 

Make no mistake. This is happening today in many different ways. You get the amount of bandwidth you pay for. If you can’t afford bandwidth for your business, particularly if it consumes a lot of data, you are going to be a little out of luck. Also, service providers like Comcast and Time Warner have a long history of crippling services. While their argument is commonly that the services are illegal, that’s not always the case. Certainly, Gnutella, LimeWire and various other peer to peer file sharing services often carried information that violated intellectual property rights, not all of the files shared fell into that category and yet all of it was either slowed substantially or outright blocked. The same is true for BitTorrent streams. Yes, there are files that are shared illegally but not all files being shared are illegal. Does a company like Comcast or Time Warner or Verizon have the right to block all traffic simply because they are concerned that some of it may be illegal? Whether they have the legal right or not, it is happening.

Is your head spinning yet? It is a very complicated issue, this whole Net Neutrality/Open Internet mess. We haven’t even touched on how all of this is handled in other countries. That’s a whole different ball of wax and one for another time.