This is a description of how I set up my pelicanHPC cluster and configured it to execute parallel programming in R. This describes every step, starting with just the hardware. Target audience is an R user with limited knowledge of pelicanHPC and some experience parallel programming in R. The instructions are far from perfect, but I hope they help.
Hardware Required:
-Worker nodes (Workstations to be used as worker nodes)
-Frontnode (workstation to be used as a frontnode should have two network cards)
-Unmanaged switch
-Ethernet cables
-Two ~4GB USB sticks
-Network the workstations together on the unmanaged switch (wired Ethernet).
-Enter the BIOS on each worker node and configure to boot from network card.
-Enter the BIOS on the frontnode and configure to boot from USB first in the boot sequence.
Software (on the forntnode):
-Download and install debian-testing-i386-CD-1.iso (any modern debian dist would probably work) from:
http://cdimage.debian.org/cdimage/weekly-builds/i386/iso-cd/-Add reference to debian packages in “/etc/apt/sources.list” add “deb ftp://ftp.debian.org/debian unstable main contrib non-free” at the end of sources.list
-$ “apt-get update” (with the frontnode connected to the internet)
-“sudo apt-get install live-build rsync wget debootstrap”
-Download the attached script “make_pelican_R-v1”
-Log into debian box as root
-Create a working directory “/Desktop/PelicanCustom1” for example
-In the working directory create a folder called “pelicanhome”
-Download the attached file flops.f and put it in pelicanhome (including flops.f is optional, but it provides a nice quick check to see that your cluster is working)
-move make_pelican_R-v1 into working directory (not pelicanhome)
-Open a terminal (if building from within gui environment)
-In the terminal navigate to the working directory /Desktop/PelicanCustom1 where you see make_pelican_R-v1 and the folder pelicanhome (which contains flops.f)
-Run $ “sh make_pelican_R-v1”
-The script will download packages. You will need at least 4GB of disk space on the frontnode. It will take some time.
-When the script finishes, navigate to /Desktop/PelicanCustom1/i386/frontnode there you will see a file called binary-hybrid.iso
-This file can be burned to CD or written to a USB stick. I prefer the USB method.
-Put one of the blank USB sticks into the frontnode
-It should auto-mount
-It will probably be called sdb by the operating system
-To confirm the name run $ “ls –l /dev/disk/by-id”. You should see the usb stick there.
-In the terminal navigate to /Desktop/PelicanCustom1/i386/frontnode where you can see binary-hybrid.iso
-Run $ “dd if=binary-hybrid.iso of=/dev/sdb” (not sdb1)
-The image will be written to the usb stick, and when it finishes you will get a confirmation of what was copied.
-WARNING: It will erase everything on the usb stick.
-Restart the frontnode.
-If the BIOS has been set to boot from USB and the image was built successfully, then you will see the Debian-Live menu appear.
-pelicaHPC is a customized distro of Debian-Live and I often use the terms interchangeably in these notes.
-At the menu, select Live, and Debian-Live will begin to load, and eventually you will see the first screen of pelicanHPC.
-The first time pelicanHPC loads, just use the default values to test that you can set up a basic cluster.
-Test pelican_setup and follow Michael’s excellent tutorial.
-If you choose to include flops.f at the end of pelican_setup you will get a quick test to see that your cluster is working. If you didn’t include flops.f you will get an error about a missing executable.
-Once you’re confident that the basic setup worked, then follow the next steps to set up our second usb stick as a permanent storage location for our R libraries.
-Ssh into each node “ssh pel30” then run $ “sudo shutdown –h now”
-Restart pelicanHPC (debian-live) frontnode.
Loading a permanent usb storage location:
-When the first pelicanHPC screen appears it will have “-t tmpfs –o size=200m tmpfs” in the input box.
-Erase this, and write “/dev/sdc1”.
-At this point insert the second usb stick.
-It will probably auto-mount and mess the screen up a bit.
-The operating system will probably call the second usb stick sdc.
-Hit “OK” after it has mounted.
-Once you have the second usb stick as permanent storage say No when asked to copy user configuration and examples to /home/user.
-Then continue loading pelicanHPC and follow the instructions on screen.
Configuring R:
-Log in as user with your password.
-By entering “df –h” you can see that /home is mounted on the usb stick sdc.
-The script make_pelican_R-v1 installs R 2.12.1. It only installs r-base-dev r-base-core and r-recommended.
-We will create a directory on our second usb stick as a permanent library for additional R packages.
-Run $ “mkdir /home/user/rlibs” this will be our permanent R library. You only have to do this the first time you use the second usb stick.
-Next navigate to /usr/lib/R/etc. Here you will find the file “Renviron”. We need to modify this file to point to our new R library (on our second usb stick).
-Change the R_LIBS_USER=… line to R_LIBS_USER=${R_LIBS_USER-‘/home/user/rlibs’}
-Change the R_LIBS_SITE=… line to R_LIBS_SITE=${R_LIBS_SITE-‘/home/user/rlibs:/home/user/rlibs:/usr/lib/R/library’}
-Copy the updated Renviron file to your usb key with “cp /usr/lib/R/etc/Renviron /home/user”
-In the future (on successive logins) you can just copy this file back from your usb stick to /usr/lib/R/etc to save having to edit the lines in Renviron each time.
-Before we update R and download new packages we need to make sure that the frontnode is connected to the internet (from within pelicanHPC). You can do this via the second network card, wireless or simply unplug the frontnode from the switch and connect to your household Ethernet.
-In debian-live most Ethernet cards should be supported by default.
-Next we will test that R knows of the new library location.
-At the command prompt run $ “R” then when R loads enter > “library()” to see the library path. It will be empty the first time but it should point to /home/user/rlibs and /usr/lib/R/library.
-Then from within R install all the packages you need for your work. If you can’t connect to a cran mirror, then it means your internet is not properly set-up.
-From within R install packages in the normal way with “install.packages(“foreach”). These packages will be saved to /home/user/rlibs
-To do parallel processing in R on your new cluster you will need at a minimum the packages doSNOW, snow, foreach, and iterators with their dependencies.
Use dsh to configure Renviron on the worker nodes:
-Before R can run in parallel, the worker nodes need the updated copy of Renviron.
-To copy Renviron to other nodes we will use a utility called dsh.
-After you have run pelican_setup and your cluster is working.
-Next “vi /etc/hosts” on the frontnode to see what hostnames were assigned.
-Hostnames will be of the form pel1, pel30,pel31 etc.
-Add the hostnames (pel30, pel31, etc) to /etc/dsh/machines.list
-Each hostname gets its own line.
-In machines.list remove “localhost” and “pel1”
-A quick test of dsh; “dsh –a –M echo hello”. Each node should reply with “hello”. If the node names are different the test was a success, if the node names are all pel1, then there is a problem. The first time you run it, it will look a little messy, so run the test twice.
-Copy our modified version of Renviron to all nodes $ dsh –a –M ‘sudo cp /home/user/Renviron /usr/lib/R/etc’. The single quotes here are essential.
-You can ssh into one of the nodes to confirm that the copy was successful and that the nodes have the proper configuation in their /usr/lib/R/etc/Renviron file. If you do check remember to ssh back to pel1.
-Once dsh is setup in this way it is also possible to shutdown all nodes other than the frontnode (pel1) with $ dsh –a ‘sudo shutdown –h now’.
Testing parallel R:
-To test you new cluster download the attached file rtest.R. Run the file with “mpirun –n 1 –hostfile /home/user/tmp/bhosts R –no-save –f rtest.R”
-Run the test from the directory that contains rtest.R for simplicity. You will need to save rtest.R to your second usb stick.
-rtest.R has each node return its nodename and then tests an example using doSNOW and foreach.
-The default values in rtest.R run the test on a cluster size of only 3 nodes. This can easily be modified in the line “cl<-makeMPIcluster(3)”.
-In the example you will notice a significant performance difference between dopar, do and the for loop.
-The do example will probably be slightly slower than the for loop example.
-While testing your cluster there will be times when mpi jobs are not properly closed. To clean up resources after such a failure use. “sudo orte-clean –v”.
Note:
-The make_pelican_R-v1 script is a customized version of make_pelican-v2.3 and may not work for you. It is for 32 bit intel architecture. It also contains some debs specific to my needs like my wireless NIC. It has some hard-coded links to some specific files that are sure to change over time, which will result in its failure. It is not supported by either Michael or myself, so use it at your own risk .
Good luck, and have fun.
Justin Moriarty
Attachments:
rtest.R
make_pelican_R-v1
flops.f
rtest.Rmake_pelican_R-v1flops.f
Locked
