Monday, February 18, 2013

Building a Raspberry Pi Cluster - Part 2

The Grand Finale

In a previous post on Building a Raspberry Pi Cluster, I wrote about how we built a cluster  using several Raspberry Pis. This cluster was used for hosting the WSO2Con App. In this post we will take a look at the finishing touches & some interesting information from WSO2Con 2013.

The project was unveiled at WSO2Con 2013, London & was an instant hit. The team received many kudos from participants.

Power Supply

Like I mentioned in my previous post, we had to build a custom power supply that can cater to the voltage & current requirements of the Raspberry Pi cluster. Some readers also requested for the schematic diagram of this power supply. This power supply is built around the LM338 voltage regulator. Each power supply is capable of powering up to 16 Raspberry Pis in the cluster. To be on the safe side, we used one power supply per row in the rack, to avoid the power supply being the single point of failure. Each row in the rack consisted of 8 Pis.

Raspberry Pi cluster - power supply


Raspberry Pi cluster - Power Supply Schematic

USB Power Board

Shown below is the USB power distribution board which could power up to 8 Raspberry Pis, or a single row in the rack. Each USB power line contains an on/off (single pole - single throw) switch, connected in parallel with an indicator LED which was connected in series with a current limiting resistor.

Raspberry Pi - Power distribution board
Raspberry Pi Cluster - Power Distribution Board Schematic


The +5V & GND lines in the board were connected to the power supply.

--

Raspberry Pi Cluster Rack - the finished product

The rack had to be very lightweight, and being able to easily disassemble it was a prerequisites since we had to carry the setup from Colombo, Sri Lanka to London, UK. So we built it with Aluminium bars, and light plastic sheets. The setup could easily be assembled using nuts & bolts.

The rows containing the Pis were slanted at about 15 degrees for aesthetic reasons.

The final product, a 24 Raspberry Pi cluster, is shown below.

Two views of the finished Aluminium & Plastic Raspberry Pi rack

An Ammeter was added in series to the power line, and a Voltmeter was added in parallel. Each row in the rack consisted of such an Ammeter & Voltmeter pair. This allowed us to individually monitor the rows in the rack to get an idea of the power consumption, and get an indication if something was wrong. 

Voltmeter & Ammeter fixed to each row in the Raspberry Pi rack
As can be seen in the above diagram, at full operation, a row (8 Raspberry Pis) was consuming around 15W of power (P = V x I). That boils down to less than 2 Watts of power per Pi! We also observed that even though the Raspberry Pi model B spec says it draws a minimum current of 700mA, when no peripherals are attached, in practice, a network connected Pi processing a load draws between 350-400 mA of current. 

In terms of power consumed per transaction, the Raspberry Pi is one of the most efficient out their.

--

Raspberry Pi Control Center

A screenshot from the completed Raspberry Pi Control Center is shown below. This simple yet powerful UI theme was developed by Chanaka, one of the UI technology experts at WSO2. We were able to track the load average, free memory & disk usage of each Pi in the cluster.



Tracking Raspberry Pis using GPIO functionality

Locating a Raspberry Pi on the Control Center also became a requirement. This comes in handy when you need to power off a Pi or replace a defective Pi. We built a small circuit which consisted of an LED, normally off switch, and 3 resistors. 

The diagram below show how a Raspberry Pi is highlighted in the Control Center when the switch is pressed.

A Raspberry Pi has been highlighted afte the switch on the GPIO board was pressed

GPIO Raspberry Pi selection board schematic

  • S1 - normally off switch
  • R1 - 10K
  • R2 - 100R 
  • R3 - 1K

GPIO Connector Board on R-Pi GPIO Pins

As shown in the above diagram, we built a small board and at the bottom of the board, we have a 16 pin IC base, which is used to connect the board to the GPIO pins.

A Python script (switch.py), which continuously runs, observes the status of the switch, and makes an HTTP call to the Control Center if the switch was pressed. The Control Center then sets the status of the Pi to selected, and highlights the Pi. There is another process (blink.py) which periodically does an HTTP call to the Control Center to check if it LED should blink, and blinks it a few times if so. Blinking the LED can be done by turning it on in the Control Center.

The following software is needed in order to run GPIO Python scripts.
  • Python
  • Python Dev (apt-get install python-dev)
  • RPi.GPIO library 

switch.py code

# Import the required module. 
import RPi.GPIO as GPIO 
import time
import urllib2, base64
import urllib2, base64
import fcntl, socket, struct

def getHwAddr(ifname):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    info = fcntl.ioctl(s.fileno(), 0x8927,  struct.pack('256s', ifname[:15]))
    return ''.join(['%02x:' % ord(char) for char in info[18:24]])[:-1]

username="xxx@xxx.xxx"
password="xxx"
mac=getHwAddr('eth0')
url="http://appserver.stratoslive.wso2.com/t/azeez.org/webapps/rpi/secure/selectpi?mac="+mac+"&selected=true"

def callSelect():
 request = urllib2.Request(url)
 base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
 request.add_header("Authorization", "Basic %s" % base64string)   
 response = urllib2.urlopen(request)

# blinking function
def blinkOnce(pin):
 GPIO.output(pin,GPIO.HIGH)
        time.sleep(0.25)
        GPIO.output(pin,GPIO.LOW)
        time.sleep(0.25)
        return

def blink():
 # blink GPIO17 50 times
 for i in range(0,10):
         blinkOnce(11)

try:
 GPIO.cleanup()
 # Set the mode of numbering the pins. 
 GPIO.setmode(GPIO.BOARD) 
 # GPIO pin 11 is the output. 
 GPIO.setup(11, GPIO.OUT) 
 #GPIO pin 10 is the input. 
 GPIO.setup(10, GPIO.IN) 
 # Initialise GPIO8 to high (true) so that the LED is off. 
 GPIO.output(11, False) 
 
 #initialise a previous input variable to 0 (assume button not pressed last)
 prev_input = 0
 while 1: 
      #take a reading
    input = GPIO.input(10)
    #if the last reading was low and this one high, print
    if ((not prev_input) and input):
       print("Button pressed")
   blink()
   callSelect()
    #update previous input
    prev_input = input
    #slight pause to debounce
    time.sleep(0.05)
  
except KeyboardInterrupt:
 GPIO.cleanup()

blink.py code

import RPi.GPIO as GPIO
import time
# blinking function
def blink(pin):
        GPIO.output(pin,GPIO.HIGH)
        time.sleep(0.25)
        GPIO.output(pin,GPIO.LOW)
        time.sleep(0.25)
        return
# to use Raspberry Pi board pin numbers
GPIO.setmode(GPIO.BOARD)
# set up GPIO output channel
GPIO.setup(11, GPIO.OUT)
# blink GPIO17 50 times
for i in range(0,50):
        blink(11)
GPIO.cleanup() 

Software Used in the Cluster

Here is the list of software we ran on each Raspberry Pi, in order to run the WSO2Con App.

  • OS: Raspbian “wheezy”
  • Kernel: Linux raspberrypi 3.2.27+ #250 PREEMPT Thu Oct 18 19:03:02 BST 2012 armv6l GNU/Linux
  • JAVA: Java(TM) SE Embedded Runtime Environment (build 1.7.0_10-b18, headless
  • WSO2:  AppServer with enhanced Carbon kernel

--

WSO2Con App

The WSO2Con App, developed by a group of User Experience specialists including Jonathan, Nuwan, Ruchira & others. This app was used for viewing & RSVPing sessions, setting up meetings with speakers, viewing #wso2con Twitter feed, chatting and so on. It was a big hit at this years' conference. The app was also available on Google Play & Apple Store. This app was running on the Raspberry Pi cluster.

WSO2Con App

Raspberry Pi Cluster in Action at WSO2Con London


Here is a video of this cluster in action at WSO2Con 2013, London.




Raspberry Pi - WSO2 Style

We rocked the floor in the last session of the first day of WSO2Con 2013. The presentation titled "Raspberry Pi - WSO2 Style" is embedded below.




View the recording of the presentation: http://wso2.org/library/wso2con2013/raspberry-pi-wso2-style/

... and the saga continues

This is not the end of this project, we have plans to run more intense performance tests. We managed to run lightweight Linux Containers (LXC) on the Raspberry Pi. The next steps would be to run an OpenStack based cloud on the Raspberry Pi cluster.

This was one of the most rewarding projects  have worked on. I hope that you enjoyed reading about it as much as we enjoyed building this cluster.
Post a Comment