Truckloads of hard disks

Imagine you put a portable hard disk of 500 gigabytes in your backpack and start walking.
In which cases are you faster than your internet connection?

Illustrations by Diego Marmolejo

Before the arrival of the internet, computer files were exchanged via storage media, similar to what happened with music cassettes or videotapes. Floppy disks were sent by post or delivered by foot, bike, car or train.

After the appearance of the internet, computer freaks invented a term for this: the sneakernet. Now that the internet is established, and our connections have become faster, the sneakernet sounds outmoded. Nevertheless, the opposite is true when somewhat larger files are considered. Because storage media evolve much faster than internet connections, it becomes ever more interesting to choose the route of physical transport over the internet.

Information density

Portable hard disks now come with a capacity of up to 500 gigabytes, while the maximum capacity of a single hard disk in a desktop computer is around 1,000 gigabytes (or 1 terabyte). That is twice as much as one year ago.

Since 1990, the information density of hard disks has increased by 100 percent per year. The capacity of internet connections evolves only by 30 to 40 percent per year. By comparison: the calculation speed of computers – described by Moore’s Law – increases by 60 percent per year. Storage media are thus by far the fastest evolving information technology. No matter how large the capacity of a hard disk is, it never seems to be a problem to use it all. It is not even necessary to store more files. For example, digital cameras can take pictures with ever higher resolutions, so that every picture takes up much more storage space. But, since internet connections evolve 3 times slower than storage media, the information on our hard disks becomes less and less mobile.

“Are truckloads of hard disks crossing the country another dent in the environmentally friendly image of the 21st century information society?”

The relatively slow evolution of internet connections is not a technological issue. There exist much faster connections than the ones that most of us are using now. However, the necessary improvements of the network infrastructure (for laying fibre connections straight into every home, for instance) demand a lot of time and money, which has to be provided by the telecommunication companies.

New storage media, on the other hand, are applied much faster because this investment is made by the end users, in small steps. We are making the sneakernet more relevant every day by continuously buying hard disks and other storage media with ever higher capacities.

Whether or not the sneakernet is faster than the internet depends on four factors; the size of the digital file, the speed of the internet connection (both from sender and receiver), the speed of the sneakernet (by foot, truck, ship, carrier pigeon, etcetera) and the distance between sender and receiver.

1. Parcel service versus the internet

Some parcel services offer the possibility to send whatever package to whatever place in the world within 24 hours. If we assume this to be the speed of the sneakernet, when does it become interesting to send a file by parcel service instead of the internet?

A dial-up connection achieves a speed of 56 kbits per second, which corresponds to a transfer of 7,000 bytes of information per second. (Storage capacity is measured in bytes, internet speed is measured in bits, and one byte equals 8 bits.) This means that 420 kilobytes can be sent or received every minute. That is 25.2 megabytes per hour, and 612 megabytes per day.

Thus, compared to a dial-up connection, it already becomes interesting to use the sneakernet with a file size of 600 megabytes. That is about the storage capacity of a cd-rom. In fact, it is interesting with only 300 megabytes, because the receiver also needs to download the file. If both sender and receiver have only a dial-up connection at their disposal, the transfer of a 300 megabyte file will take about 24 hours.

Broadband connections are obtainable with many different speeds, but to express well the difference with a dial-up connection, let’s choose a fast DSL-connection with a download speed of 8 Mbits per second. This equals to 60 megabytes per minute, 3.6 gigabytes per hour and 86.4 gigabytes per day. A dial-up connection would need more than 150 days to send 86.4 gigabytes. Yet, even this fast DSL-connection has to surrender surprisingly fast when compared to the sneakernet.

That is because the upload speed of a DSL-connection is about 8 times lower than the download speed: only 1 Mbits per second in this case, or 8 megabytes per minute, 480 megabytes per hour and 11.5 gigabytes per day. This means that a file which is larger than 11.5 gigabytes (only a small pie from an average hard disk) is faster to send by parcel service than by a fast broadband connection – and in this case we do not even consider the receiving time (which is 8 times faster).

“Sending 500 gigabytes goes faster by foot than by DSL from the moment the distance is less than 1,225 kilometres (761 miles)”

Increasing the speed of broadband connections has its limits and is largely dependent on the distance from the telephone switchboard (see the article “faster internet is impossible”). Fibre connections do not have these limitations. In most countries, these connections are reserved for large companies, universities and the government, but in Scandinavian countries they are already established.

To express the difference well with the broadband connection, let’s choose a fibre connection with an upload and download speed of 100 Mbps (the fastest connection available for private persons in Scandinavia). It can send 12.5 megabytes per second, 750 megabytes per minute, 45 gigabytes per hour and 1 terabyte per day.

In this case, the sneakernet becomes interesting only when the file is larger than 500 gigabytes or 0.5 terabytes (upload + download). This compares to a half or a full hard disk of a computer. However, by the time that fibre connections are established in all countries, one terabyte will not mean a thing anymore.

Let’s assume it takes another ten years for all of us to receive a fibre connection in our home. By that time, if storage media keep evolving the way they do today, the average computer has a hard disk of 1,024 terabytes (or 1 petabyte). Sending a full hard disk will then take considerably more time than it does today, in spite of the faster connection.

2. Hiker versus the internet

The sneakernet does not need the impressive delivery speed of UPS or FedEx to beat the internet. Imagine you put a portable hard disk of 500 gigabytes in your backpack and start walking (10 hours per day, 5 km/h). In which cases are you faster than your internet connection?

A dial-up connection needs about 2.5 years to send 500 gigabytes, and if the receiver also has a dial-up connection, it takes 2.5 years more.

The result is clear: a hiker is always faster than a dial-up connection transferring 500 gigabytes, even if he walks until the other end of the world (which would take him – in theory at least – 400 days there and 400 days back).

A DSL-connection can send 11.5 gigabytes per day and thus needs 43 days to send 500 gigabytes. Our hiker covers 2,150 kilometres (1,336 miles) in 43 days. If the receiver also has a DSL-connection, the download is much faster. The file can be downloaded in just less than 6 days, while our hiker needs 43 days to come back. Conclusion: sending 500 gigabytes goes faster by foot than by internet from the moment the distance is less than 1,225 kilometres (761 miles).

Driving

But, if our hiker gets into a Citroen 2CV from 1974 and drives to his destination at the car’s top speed of 100 kilometres an hour (62 miles per hour), then he is faster than the broadband connection as long as the distance is less than 24,500 kilometres (15,223 miles). That is 49,000 kilometres (30,447 miles) divided by two, because the driver has to come back.

In this case we arrive at the same conclusion as before: a guy in an antique car is always faster than the internet when transferring 500 gigabytes of information, even if he drives to the other end of the world. The Earth has a circumference of 40,000 kilometres (25,000 miles) so 20,000 kilometres (12,500 miles) of driving takes you to the other side – in theory, that is.

“Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway”, Andrew Tanenbaum 

 

Of course a fibre connection is hard to beat – or is it? Fibre transfers 500 gigabytes in just 24 hours (12 hours of uploading, 12 hours of downloading). Since our hiker only walks ten hours a day, he is only faster than the fibre connection if the distance is less than 25 kilometres or 15.5 miles (50 kilometres or 30 miles divided by 2). But, also in this case it is not hard to find a transport service that is faster than the internet.

If our hiker boards a plane and flies with a speed of 1,000 kilometres an hour (620 miles per hour) to his destination, then he is faster than the fibre connection as long as the distance is less than 10,000 kilometres or 6,200 miles. (He loses 4 hours waiting in the airport). This means that, finally, the internet connection scores better than any other means of transport but only when the data has to be sent to the other side of the world.

Two hard disks

Unfortunately, this advantage disappears when our guy puts two 500 gigabyte hard disks in his backpack. The fibre connection needs twice as much time to send this amount of data (48 hours), while our guy maintains his speed.

The larger the file is, the more interesting the sneakernet becomes compared to the internet, because the amount of data has no influence on the speed of the sneakernet (except in the case of the hiker, who can take a few extra hard disks without lowering his speed or endurance, but not 20 of them). If you fill up a freight train, a cargo ship or a large plane with hard disks, then of course even the fastest internet connection crumbles to nothing compared to the sneakernet, whatever the distance.

3. Carrier pigeon versus the internet

There is another way to send digital data: via carrier pigeon. This may sound ridiculous (and it has been a popular joke for many years), but thanks to shrinking storage media, the speed and capacity of the pigeon internet promises to become quite amazing.

Pigeons were the fastest means of communication for thousands of years, from early Antiquity to the arrival of the electrical telegraph in the 19th century. Pigeons were used by private persons, governments, the military, press agencies and stockbrokers.

A well trained contemporary carrier pigeon can maintain a speed of 50 kilometres an hour (31 mph) over a distance of 600 kilometres (373 miles), and carry a weight of 1 gram. More ‘freight’ is possible, but then the performance goes down.

One gram does not seem to be much, but thanks to the increasing information density of storage media, this weight can already contain quite some data. For instance, the Transcend Micro SD card (meant for use in mobile telephones) weighs 1 gram and has a capacity of 2 gigabytes. Then how does the carrier pigeon resemble the internet?

Compared to a fibre connection, the pigeon has to surrender quite fast. This internet connection only needs 2.6 minutes to send 2 gigabytes. A carrier pigeon only flies 2 kilometres (1.2 miles) far in that time. A carrier pigeon is thus faster than a fibre connection when the distance is shorter than 2 kilometres or 1.2 miles. A broadband connection needs 4 hours to send 2 gigabytes, while the pigeon can reach a distance of 200 kilometres (125 miles) in 4 hours.

This means that sending 2 gigabytes of information from Amsterdam to Brussels goes faster by carrier pigeon than by a fast broadband connection. (The download goes faster, so if you also consider that the pigeon has to fly back, its advantage becomes smaller). A dial-up connection needs 3.3 days to send 2 gigabytes, so in that case, the pigeon (flying 600 km or 370 miles per day) is faster than the internet up to a distance of about 2,000 kilometres (1,250 miles).

“Ten years in the future, a carrier pigeon will be able to transfer 2 terabytes of information”

The bandwidth of a carrier pigeon increases faster than the bandwidth of the internet. Ten years in the future, the information density of storage media will have multiplied by a thousand, while the speed of the internet will only have multiplied by 350. This means that a pigeon will be able to carry 2 terabytes (around 2,000 gigabytes) while our fibre connection will need 8.5 minutes for sending the same amount of data. The carrier pigeon is then faster than a fibre connection if the distance is less than 7 kilometres – compared to 2 kilometres today.

Compared to today’s internet connections, a pigeon carrying 2 terabytes beats both the dial-up and the DSL-line, even if it has to fly to the other end of the world. While the pigeon needs 33 days to reach the other end of the world (flying 600 km or 370 miles per day), a dial-up needs 10 years to send 2 terabytes, a DSL connection 172 days. A fibre connection needs 2 days to send 2 terabytes, so in ten years a pigeon will be faster than today’s fibre connections up to a distance of 1,200 kilometres (745 miles).

Send the complete computer

The above calculations are a bit too simplistic. There is another factor to consider: if you choose to send digital data via postal service, a hiker or a pigeon, you first have to transfer it from the computer to an external storage medium. If you use the internet, this is not necessary. The now obsolete USB 1.1 has a transfer speed of 12 Mbits per second (which is still faster than our broadband connection). USB 2.0, which most of us use now, reaches 480 Mbps – almost 5 times faster than a fibre connection.

Transferring 500 gigabytes from your computer to an external hard disk thus takes an additional 2 hours. The receiver has to transfer the information from the external hard disk to his or her computer, so this takes another two hours. The sneakernet thus needs an extra 4 hours to send 500 gigabytes – unless you choose to send the complete computer (without the monitor, of course). This sounds ridiculous, but it happens more and more.

For most consumers today, a fibre connection of 100 Mbps would be more than enough. But for many companies and institutions, this is not the case. For instance, in the financial world, the movie industry and in scientific research, data files with a size of many terabytes are no exceptions. If they have to be sent, the sneakernet is the preferred option. Microsoft (in 2002) as well as Google (in 2007) developed logistic services for sending data files up to 120 terabytes.

“The capacity of storage media is evolving 3 times faster than the speed of internet connections”

There is another reason why the sneakernet is preferred in those cases: if you decide to send 500 gigabytes via the internet, your fibre connection is occupied for 12 hours. You could use it for other things at the same time, but only at the expense of an even longer upload or download speed. This advantage also plays on a much smaller scale; it is faster and more practical to transfer your music collection from one music player to another, instead of making a detour via the internet. Privacy is another advantage here.

Is the internet obsolete?

The fact that it is many times faster and more practical to send the contents of a hard disk via courier does not mean that the internet is obsolete, or that we all have to start training carrier pigeons. The advantage of the sneakernet only applies to files above a certain size. This threshold lowers every year, but most internet applications require only a very limited transfer of data.

Downloading an email or a webpage rarely takes more than a minute, and the sneakernet will never be able to compete with that. (Some nerds did transfer a webpage with pigeons, but that was another joke).

The sneakernet is being used for transferring emails and websites, but that happens only in places where there are computers but no internet connection. Also, streaming of movies and music would be impossible with the sneakernet, because you could only start to watch or listen to the file once it arrived completely. An application like videoconferencing would be impossible, too.

Digital cargo

However, the good performance of the sneakernet does raise a couple of questions. For example: is the internet suited for the large scale distribution of high resolution movies or television programs? Many people see this as the future, but it seems not so plausible. It might turn out that it will always be faster, cheaper and more practical to send high-res movies by postal service than by internet - companies like Netflix are already doing this.

The good performance of the sneakernet for sending large files is also another dent in the environmentally friendly image of the information society. Energy-guzzling datacenters, toxic production methods and mountains of e-waste have already stained this perception. Are truckloads of hard disks crossing the country another part of the 21st century information society? The virtual world seems to become more real every day.

Note: Whether or not the sneakernet is cheaper than the internet is much harder to calculate, but some good efforts were made.

© Kris De Decker (edited by Matt Hill). Illustrations: Diego Marmolejo 

Related: 

• A world without trucks: underground freight networks
• Faster internet is impossible: we need ever faster connections to surf the web at the same speed
• Computing without electricity: complicated calculations are a product of fossil fuels
• The optical telegraph: email in the 18th century
• Cargo cyclists replace truck drivers on European city streets
• Wind powered factories: the history (and future?) of industrial windmills
• The monster footprint of digital technology
• Human powered cranes & lifting devices

Comments

[Alexander Gieg]

(1)

There are two small problems with this article.

First is the notion that you must upload a file to an intermediary and then download from it. This actually isn't necessary. One can upload directly to another person. Provided the download speed from the receiving end is equal or faster than the upload speed from the sender, the file will be received at maximum upload speed.

Second is the notion about pigeons "flying back". As far as I know, this doesn't happen. A pigeon only flies in one direction: towards its home, the place it was born. Thus, if you want to use carrier pigeons to receive something, you must have some raised with you, the sender must take them with him for when they're needed, and once he wants to send you something, he frees the pigeon. Two way communication is only possible as long as both sides have a "livestock" of mutually raised pigeons. As soon as both are depleted, the only way to replenish is for both parties to have their pigeons shipped again to each other.

Correcting both things would fix a few results. But overall the conclusions wouldn't change much.

[nf]

(2)

The article also neglects to acknowledge multicast. That is, one can distribute a large amount of data to a large number of recipients in a locality simultaneously (eg an ISP maintaining a local FTP mirror of popular content), whilst Sneakernet is very much a 1-to-1 distribution method.

[Kris De Decker]

(3)

@ Alexander: "One can upload directly to another person"

Agreed, by using p2p. But in most cases (http, ftp, email), this is not the case. You upload to an intermediary server, and I download from it. P2P has disadvantages, and not everybody knows how to use it.

[Peter Murray]

(4)

A couple of years ago I attempted calculations for the bandwidth of a Boeing 747 filled with BluRay discs traveling between New York and Los Angeles. 74Tb/s. (http://dltj.org/article/internet2-hopi-network/#comment-2142)

[Kees Huyser]

(5)

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. pp. 83. ISBN 0-13-349945-6.

The original version of this quotation came much earlier; the very first problem in Tanenbaum's 1981 textbook Computer Networks asks the student to calculate the throughput of a St. Bernard carrying floppy disks (which are said to hold 250 kilobytes of data). The first USENET citation is July 16, 1985, and it was widely considered a chestnut already, possibly dating from the 1970s[citation needed]. Other alleged speakers included Tom Reidel, Warren Jackson, or Bob Sutterfield. The station wagon transporting magnetic tapes is the canonical version, but variants using trucks or Boeing 747s and later storage technologies such as CD-ROMs have frequently appeared.

Source: http://en.wikipedia.org/wiki/Sneakernet

[Drew]

(6)

This is a very interesting arguement but you have to be moving some very large files to make it worthwhile. How many 500 GB files are there in the world that the average user would need to send to someone.

Also, I can't take the time from my job to walk one mile (each way) let alone 25.

I am thinking of massive databases, but when I do a pull from a database (such as the Census database) I am not need the entire database, but simply a small subset that I can download in few seconds.

[Leslie Viljoen]

(7)

About pigeons "flying back", the wikipedia article says that they can be trained to do that by placing their home at one end and food on the other: http://en.wikipedia.org/wiki/Carrier_pigeon

[Jon]

(8)

You wrote "After the appearance of the internet, computer freaks invented a term for this: the sneakernet."

Sneakernets were in use long before the internet. In 1986 we used the term "sneakernet" to describe carrying 5.25" DSDD floppies up and down the hall for file sharing. We learned the term at a conference held by then-fledgling and soon-to-be-giant Novell.

Jon P. in Florida

[Paul]

(9)

They did that carrier pigeon with a memory stick a few years ago in South Africa, and the pigeon bet the upload by an hour.

Also, why did you assume that there was a 4hr wait at the airport to depart? Maybe he has access to a private jet and isn't flying commerical. Some of those private jets can fly at transonic speeds M0.9+

[Someone]

(10)

To correct the middleman requirement for FTP: Of course one can start an FTP server on the very same machine that is the source or the destination of the data. Then just download from or upload to that FTP server and there is a file transfer between two machines.
Same procedure works using SSH and is the standard for transfering data from or to remote controlled servers in the internet now. The masses fear everything that is called server software. But its there and it is used by them who do not fear it.

[Peter]

(11)

Interesting thought. However, you rarely need to send such an amount of data at once. Now is fibernet pretty common in the world. I don't know about Asia or Africa though, but most countries that had no network before have it actually faster than the developed world, because they start with fibernet and not already obsolete technologies.

Sneakernet is now basically used only if the computer to receive data is very close to you / in the same building. So for example it is faster to upload a movie on a USB stick and then give it to your friend to download it. But even then, it might be faster to upload it via Wifi, which I believe now everybody has, through the router directly to the other computer.

And, in the end, you should count the time one spends when sending a medium, when you are uploading, you just need to spend few minutes and the process goes automatically. You do not need to go to the post of contact delivery service, or go by foot.

[Dan Howell]

(12)

Well, it's been over ten years, and we don't have even close to petabyte hard drives in our computers--even high end computers usually come with no more than 2TB standard, and the biggest hard drives you can buy are 16TB, 18TB or maybe 20TB. And fiber to every home is not a reality yet, but we are getting there.

But one prediction may have came fully true: For fast data transfer by carrier pigeon, we can probably strap a couple of 1TB microSD cards to a pigeon to carry 2TB of data!