Introduction to Distributed Computing
Distributed computing is the concept of using a group of networked computer systems to solve a common problem or achieve a single goal. It works on the premise that a large network of computers will effectively be able to harness more processing power than any single machine normally could in a given period of time. Most large Organisations simply buy their own high specification servers, but the cost of this can be incredibly high, putting it out of reach of smaller (and less well funded) causes.
The concept involves taking a complex task and breaking it into smaller chunks, known as work units. Work units are distributed to computers involved in the project, and are returned upon completion. Each work unit will require a minimum amount of processing time, with complexity varying in nature.
Many Universities and Organisations have adoped Distributed Computing systems as a way of performing research and solving mathematical calculations. Examples include:
Seti@Home - Searching for Extraterrestrial Intelligence
Einstein@Home - Astrophysics Calculations
Folding@Home - Cancer research by Protein folding
LHC@Home - Research for the Large Haydron Collider (LHC)/CERN
A more comprehensive list is available here:
Wikipedia: Distributed Projects
 |
| Corporate Servers: Cirprian Popescu |
Taking Part
Taking part in distributed computing projects can directly help to benefit science and perhaps even humanity as a whole (when considering the big picture). Projects such as folding@home are used to aid Cancer research which may even help to save lives one day.
The good news is that taking part does not require any special equipment, hardware or additional expense. Anyone with a PC, laptop or server can take part, with most projects offering Windows based installer files with simple instructions. The link above will help you get an idea of what projects are available, and links to the projects themselves.
Some projects run on whats known as the Berkeley Open Infrastructure for Networked Computing platform (BOINC for short). This is a small program which allows you to run multiple projects at the same time, multitasking and load balancing between them. To verify compatibility and understand more, check it out here:
 |
| BOINC |
BOINC for Distributed Platforms
How does it work?
Each project has a slightly different mechanism for working, but effectively the overall process is pretty much the same for each:
1. You are assigned or asked to choose a username, which forms the basis of your "account". This can be set up either on the project website or in the application itself. E.g. Seti@Home will give the option to sign up when installed through BOINC.
2. You download an install the project software, and log into that software using the pre-assigned username.
3. Your Computer will request work units from the project servers.
4. The project servers will issue your computer work units, with a deadline. The type of workunits you receive will be hardware dependent. Many
projects now support processing on a graphics card (GPU) which has the
advantage of being up to ten times quicker than your CPU. E.g. Seti at
home and Folding@Home both support Nvidia CUDA architecture, allowing owners of Nvidia graphics cards to process even more work units.
Specific details for BOINC based projects can be found here:
Boinc Wiki
General Configuration
Each project will have its own set of configurations available. These range from specifying the type of work units to process to choosing when processing can take place. For full details, please refer to the website of your projects of interest.
Most projects will present the user with two possible options: continuous running (at low priority), or screensaver mode. Screensaver mode is generally the preferred option for most users as it does not require any processor time whilst the computer is actually in use. BOINC in particular is good at sensing when a user is active, and when the computer is in an idle state. As a bonus, many projects project a colourful visualisation dipicting the current status of your project work. Constant running mode will use the CPU all of the time, pausing processing only for other higher-priority processes.
It is important to note that project workunit calculations are very CPU hungry, causing your computer to produce more heat. Before running projects for a long period of time, ensure that your computer has adequate cooling and ventilation. This is particuarly important if you intend to use a laptop, since many laptops have poor heat dissipation.
Teams and Credit
Projects provide the user with credit for each work unit completed by them, with the general intention of creating a spirit of competition. Whilst credits have no monetary/physical world value, they act as an indication of how much effort and time a user has given to projects (in a "bigger number is better" fashion). Many websites are able to track a users progress, giving day to day statistics, graphs and various other metrics of users performance. My personal favourite site is BoincStats, which can track a users progress across a multitude of projects at once. My stats are accessible below:
BoincStats - Galacticminor
Additionally it is possible to join a team. By joining a team, your statistics are added to that of a team and the competition can turn international. In 2012 I joined the UK Boinc Team, quickly ascending to position 42 of 2402 (at the time of writing). Most teams provide their users with forums and support pages.
http://forum.ukboincteam.com/
 |
| The UK BOINC Team Logo |
The Project
The "Distributed Project" Big bang
I first joined Seti@home in 2002, inspired by TV documentaries about Space, the Universe and the Drake equation. At the time I owned a basic Dell Dimension pc with a Pentium 4 1.3Ghz (Antiquated by todays standards) and ran the software for several years before having a break whilst studying for my degree.
Jumping to 2012, my interest was reignited by a friend -Chris Armstrong- who turned out to be an avid fan of the Folding@Home project. He had a collection of his own servers/hardware that were churning out work units for FAH, so we decided to put our knowledge together and collectively combine our resources.
 |
| The Rack! |
Chris's server rack (Pictured left, populated) had been acquired a few months earlier via Ebay, and had plenty of room for expansion. It was located in his land-ladys garage, keeping it cool in the summer and the garage suitably heated in the winter!
Within a few months, we had put a few desktop pcs into the rack and experimented with remote desktop applications. At this point, we decided to use Ubuntu Linux as the core operating system. The key points to this decision were:
-Ease of use - Ubuntu is widely regarded as one of the more user-friendly and forgiving Linux distros.
-Scalability - Linux scales incredibly well, whether on a 200mhz processor or a multi CPU server.
-Compatibility - Ubuntu already has native drivers written for it by the leading brands such as Intel, Atheros and Nvidia.
-Support - Ubuntu has a large support network on the internet, ranging from forums to official "How-To" guides.
-Cost - As a totally free operating system, there are no hidden costs or licensing woes to worry about.
-Security - Linux is open source with updates published practically daily. Secure network connections/links are available by using components such as Open-SSH.
Additional hardware such as a US Robotics switch were also sourced online, along with a KVM switch and several metres of network CAT5 cabling.
The number of computers in the collection soon started to grow, and I took to a naming convention of naming my computers after famous Satellites. My desktop PC was already called "Arianne" (not quite a Satellite!), so the following computers would be named "Voyager, Voyager II, Cassini and Stardust.
Servers - A First Look
It didn't take long to realise that desktop pcs aren't the most economical shape for a server rack, and neither are they the most electrically efficient. After pulling nearly 3 kilowatts of power from the mains (total rack consumption) I decided to research into servers. Having never actually owned a server before, I decided to look online.
I found a seller on ebay offering reconditioned servers for just £50. I took the plunge, and bought a Dell Poweredge SC1425:
 |
| Dell SC1425 |
The specs:
-1U (Single Rack Unit) Dell Server, Ex-Corporate
-Dual Intel Xeon Processor (Single Core) @ 2.8Ghz
-4GB DDR RAM
-No OS/DVD Drive
 |
| Dell SC1425 Interior. |
Ultimately, this unit (named Voyager) turned out to be an ideal "starter server". It allowed me to to thoroughly understand how to turn this ex-corporate server into a dedicated distributed computing project number cruncher. Obstacles included:
-Installing Ubuntu from a USB drive
-Making the system run with a headless setup (No monitor)
-Accessing the server remotely in a secure manner (using Open SSH)
-Understanding the future upgrade paths available
Unfortunately, the end performance of this particular machine proved rather disappointing. Despite being able to crunch two work units at a time (Being dual CPU) the server had a low output, and consumed 300watts at the mains (as high as my Quad core desktop pc!). It was initially produced around the 2004 year mark, so it should have been no surprise really. In addition, it had no power-saving features and the fans were always on at a high RPM. I have since kept this server, but use it as a test guinea-pig for software and tool deployments.
Servers - Quad Core
A few days of research into Intel Xeon CPUs revealed that great strides had been made since the old single core chips. Newer servers were up to 8 times faster than the previous server I had purchased, and used less than half the power. Brand new servers were simply out of the question - top end Intel Xeon based systems cost several thousands of pounds - money I simply couldn't afford. However, Intel L5420 based systems were available from system refurbishers online for as little as £150 and less than two years old in many cases! The L5420 is a Xeon quad core chip, with plenty of bang for buck and a 60watt thermal design profile. I decided to stick with Dell (as a brand I know, and a good reputation of just working) and made the jump to buy a CS24-series server.
 |
| Dell CS24 Series |
The server ("Cassini") still only had a 1U size profile, but had the following specs:
-1U (Single Rack Unit) Dell Server, Ex-Corporate
-Dual Intel Xeon Processor (Quad Core @ 2.4Ghz)
-8GB DDR2 RAM
-No OS/DVD Drive
-PCI-Express Slot for GPU
-Hot-swappable HD bays
 |
| Dell SC24 Series Interior. Note the PCI-E Slot |
From the very start it became clear that these were great units. Under full load it used just 170watts of power, and was able to crunch eight work units at once.
 |
| Servers in the rack |
Compared to the older computers, the 1U servers slotted comfortably into the rack and showed there was plenty of room for further expansion.
Unexpected Hitch
Due to work commitments Chris had to move away to Edinburgh. For the next few months, the project was stalled as Chris moved out and my servers were relocated back to my place. We were determined to not let this have a long term affect on our scientific activies, and continue to still discuss and propose ideas for development.
I adjusted my network at home to cope with the addition of the servers, installing a small Netgear switch and Ethernet over mains adaptors. This allowed me to place them in the shed, keeping the noise and heat out of the main house. Unlike the older generation servers, the CS24 series have several energy saving features including automatic fan speed regulation and automatic CPU voltage regulation, resulting in a more quiet operation.
Servers - GPU Upgrades
The next few months saw further server expansion. I was so impressed with the performance of the CS24 series that I purchased another two of them (Stardust and Voyager II) sending my week by week scores skyrocketing on Boinc-stats. In the space of 3 months my points practically doubled, breaking the 1 million mark.
I had been aware for some time that adding GPU's to computer systems had generally been regarded as the best way to improve performance with mathmatical calculation work. Publications in magazines such as the IET magazine gave examples of supercomputers being assembled in the USA and China, made up almost entirely of banks of Nvidia GPU cards such as the TESLA. A little research revealed that even basic Nvidia graphics cards could now be used with impressive results.
Early 2013 saw me buy two Nvidia GT630s and a GT640. I chose these cards specifically because they didn't require any additional power outside of the PCI-Express slot, and were both priced at less than £60. One of the snags of using a server case was that a graphics card would only fit if it were one slot wide, there simply was no room for a double width card (which are incredibly commonplace on most desktop PCs these days).
 |
| Asus Nvidia GT640 in server |
Another hitch that I soon discovered was that Seti@Home do not currently supply GPU applications for Linux users. After some research and googling I was able to overcome this by downloading a custom application written by some dedicated coders at a coding site known as "Lunatics". It was a reasonably straightforward process, a case of copying/pasting the new files into the correct directory within the Seti@Home file system. The files are available here:
After some configuration file editing and sensor monitoring, I was off and the work units started crunching on the GPU. However, I found that the GPUs were starting to overheat. The solution was to install more fans into the case to improve the airflow. Thanks to a kind ebayer, I was able to source an additional three fans at a reasonable price:
 |
| Additionally installed San Ace 40 Cooling Fan |
Predictably, the performance of my servers improved substantially and I am now well on my way to 3 million + points.
 |
| Recent Stats - June 2011 until Feb 2013 |
Additional useful tools
Ubuntu is an incredibly rich Operating System, providing a whole range of useful tools, features and addons. I could probably fill a whole blog with just setting up/deploying Ubuntu. By following some guides online, I have found that it is possible to use OpenSSH to provide secure remote access to my servers via the internet. Programs such as HTOP and LmSensors are invaluable, especially on a server deployment (see below).
 |
| Htop, as seen via PuTTY in Windows |
Power Budget
Throughout the project electricity level monitoring and usage were important factors. There is no point in having arrays of servers and GPUs if you cannot afford to pay the electric bill at the end of the month! I invested in an electrical meter designed to monitor the power consumption of individual devices; capable of displaying the time used, voltage levels and watts used (see below).
 |
| Power Metre |
-Choose to use your machines part time , maybe only a few hours a day
-Choose to use low power machines with small power supplies
-Research and find CPU/GPU combinations that are low power (Wikipedia and the manufacturers website will define the power requirements for any given CPU or GPU).
More powerful machines do generally use more power at the mains than lower power equivalents, meaning it can be a balancing act. Careful research and planning can ensure that you don't run up high bills.
Hardware Budget
In this project I have used a variety of servers and desktop pcs. As with any project, the more money you can afford to spend the more points and performance you will recieve. I chose to use server systems as they tend to have a lower power consumption and are much cheaper to buy online (Most being ex-corporate equipment and refurbished). Many people use normal ATX style desktop PCs to do their workunits, and the end choice of hardware is entirely up to you; there is no "wrong" way to do it. Some of the highest performing computers are simply low specification machines equipped with an array of expensive graphics cards such as the Nvidia Tesla series. Ultimately the deicison is yours and down to personal preference.
The Future
At this moment in time it is difficult to see where to take the project next. The servers are currently upgraded as far as the hardware will allow, and I have hit a comfortable electrical budget. For the next year or so it would seem that the servers will simply be left to further increase my Seti@Home statistics, and potentially find alien life. In the not too distant future I may decide to switch projects over to Folding at home, and help towards the fight against Cancer. Until then...ET: "You can run...but you can't hide"!
