Optimizing your windows environment for computing at the Demography Lab

Abstract:

How to setup you windows machine so that you can do science both on your PC and on Demography Lab servers using tools that are almost identical. In addition, this set up will make it easy to move projects back and forth between your PC and the Lab servers so you can collaborate with others or take advantage of more computing power.

STOP you might not need to do any of the things listed below. You can instead use freeNX to connect to the server from your personal computer in a way that is far simpler to setup and, for most people, much easier to use. See http://lab.demog.berkeley.edu/LabWiki for instructions on using FreeNX. The instructions below may be of slightly more than historical interest to those who like to install things and who want to maintain an independent ability to do science...from their caves..after the ApoCALypse.

What you need

The Microsoft business model is based on windows machines not inter operating with stuff that Microsoft does not profit from. Since almost all of the stuff you need to be a great scientist is open-source, it will take some effort to make your windows machine do what it should do.

You will need the following parts:

R: The best statistical computing environment for Demographers and other thoughtful people.
Emacs: The editor that you will soon learn to love.
ESS: An add-in for Emacs that understands R and your innermost thoughts.
rsync: A program for synchronizing directory trees across machines.
putty ( ssh): A secure shell program which allows you to run a terminal window on remote Unix machine or mac machine.
hummingbird exceed (X11 server): A windowing system that allows you to control remotely running applications.
Texlive (LaTeX): A document preparation system that works well for scientists. (We won't use this until November so you can safely wait to install it until later)

Once these bits are installed, and properly configured, you will be able to do the following useful tasks:

(with Emacs and R) create and execute R programs of moderate size on your portable computer.
(with rsync) easily create or maintain copies of directories that hold your project's data and R code so that you can run that code either on Demography Lab servers or on your portable.
(with ssh and X11) run programs such as R (or STATA or whatever) on Demography Lab servers while controlling them from your portable such that even fancy graphic output will display on your portable screen as if everything were running locally.

Emacs, ESS and etc.

The software fairies (in this case Vincent Goulet) distributes a single installation package that installs Emacs and several useful add-ins including ESS. So just download this and run it:

http://vgoulet.act.ulaval.ca/pub/emacs/emacs-23.3-modified-3.exe

R

R is downloadable from the Comprehensive R Archive Network.

http://cran.r-project.org/bin/windows/base/R-2.13.1-win.exe

As you work in R, you will discover that you often need a ``library'' of code in order to do a particular thing. These are easy to install, either with the install.packages() function, or via the menus in R's own windows interface.

rsync

The rsync version we want to use is called cwrsync. You can find it here:

http://sourceforge.net/projects/sereds/files/cwRsync/

The current version is 4.1.0. Get the one called cwRsync_4.1.0_Installer.zip do not bother with anything with the word ``server'' in it's filename.

Once you have succeeded in installing cwrsync, you will want to change your path ``environment variable'' so that windows knows where to find the executable and some stuff on which it depends. path is the list of directories that the computer searches through in order to find commands that you type. In order for rsync to work properly, the directory in which it lives must therefore be in your path

Generic instructions on how to modify environment variables can be found here:

http://geekswithblogs.net/renso/archive/2009/10/21/how-to-set-the-windows-path-in-windows-7.aspx

What you need to do is set modify the ``value'' of the ``path'' variable to include something like:

C:\Program Files (x86)\CWRSYNC\BIN;

Note that the individual directories in path are semi-colon delimited.

You might need to look in your Program Files directories to find out where cwRsync wound up. There may be two such directories under Windows 7. Note the ``(x86)'' in the above.

puTTY (ssh)

Putty is the secure shell program. The installer is here:

http://the.earth.li/~sgtatham/putty/latest/x86/putty-0.61-installer.exe

Once installed you will need to set your session parameters to enable X11 forwarding. In the left hand column look for and select Connection $\rightarrow$ ssh $\rightarrow$ X11. That should produce an opportunity to check the box for ``Enable X11 forwarding''.

After accomplishing the above, Select Session; then highlight Default Settings in the ``Saved Sessions'' area; and finally click Save.

Enabling X11 forwarding is essential for X11 to be worth anything. What you just did, should take care of X11 forwarding for ever and ever, but if you have trouble getting server applications to create windows on your screen (after successfully ssh'ing to the server). The first thing to check is X11 forwarding.

hummingbird Exceed (X11)

NOTE: unless you have a reason to prefer X11, we recommend instead installing the FreeNX client. See http://lab.demog.berkeley.edu/LabWiki

The Hummingbird Exceed X11 server is the only bit that is not open source. Luckily the University licenses it, so from your point of view, it is at least free. With your calnet ID, you can download it from here:

http://software-central.berkeley.edu/software/42-Exceed

select ``Exceed v14'' (or perhaps there is a higher number since this document was written).

This is huge suite of programs but the only one we care about is ``Exceed''. It's in the menu, but it is hidden among dozens of things that we will not use.

Texlive (LaTeX) Optional

LaTeX will be essential when you start writing your dissertation, but if you're tired of all this installing, you can put this off for at least a few months.

LaTeX is quite large. But the installer program takes care of downloading. Get the installer program install-tl from http://mirror.ctan.org/systems/texlive/tlnet/install-tl.zip

Put it on your desktop and click on it to unzip it; then run it-maybe overnight if you are doing this from home.

Test everything

Once everything is installed, let's make sure that it all works as I promised it would.

synchronizing directory trees (1).

Let's synchronize your 213 directory so that it is the same in your Demography Lab account as it is on your personal computer. Launch a cmd window and in it run this command OBVIOUSLY changing ``userid'' into your own userid on the Demography Lab system.

@: $\textgreater$ rsync -uva userid@coale.demog.berkeley.edu:213 .

NOTE the significant dot at the end of the above command.

NOTE also that just as you can run R from within Emacs, you can also run a ``shell''. In Emacs:

M-x shell

will do this. You might find this to be a very convenient way to run rsync on your windows machine.

Run a moderate (actually tiny) R session on your personal computer

Once you have downloaded the 213 directory you should be able to do the following:

Launch Emacs; find and open the 213/Week1/213ex1.r file.
Launch R in the usual way (from Emacs) and run the commands.
Make a trivial change to the 213ex1.r buffer; save it; quit R and exit Emacs.

synchronizing directory trees (2)

The procedure for synchronizing your Demography Lab copy of your 213 directory tree with what's on your personal computer is very similar BUT NOT identical to the command you ran to bring a copy of the 213 directory from the server to your personal computer. Here is the command to move the changes that you made on your PC to the 213 directory on the server (in other words to synchronize the two copies). Do this either in a cmd window or in shell running in Emacs:

@: $\textgreater$ rsync -uva -delete ./213 userid@coale.demog.berkeley.edu:

IMPORTANT: rsync copies the contents or the changes is the source directory onto the target directory. Which is the source and which the target is determined by order in which they appear in the command. In the above command, ./213 is the source and userid@coale.demog.berkeley.edu: (which translates to userid's home directory on coale.demog.berkeley.edu) is the target.

rsync will not change the source directory so as long as you get the source and target business right, you cannot do irreparable harm.

NOTE also the -delete argument (two dashes precede delete). In the present case this will not matter, but it tells rsync to delete files that exist in the target directory but not in the source. It is wise to use this flag because otherwise your life will fill up with useless files that you thought you had deleted already.

Running R on the server

Now let's connect to the server in such a way that you can run R inside Emacs on the server but have it behave as though you were running it locally on your PC.

Launch your X11 server Exceed - it's just called Exceed, but to launch it you have to find it among the many other things in the ``hummingbird connectivity'' universe. It is not far down in the menu system. When you find it and click on it, you should see a brief logo display and then a small floating tool bar, which you can generally ignore. What's important is invisible: the X11 server is now running on your personal computer.

Launch puTTY and open a connection to coale.demog.berkeley.edu provide your usual login credentials.

In the resulting window type

@: $\textgreater$ echo $DISPLAY

If that results in localhost:10.0 (the number is not important). Then continue, otherwise review Section 1.4 regarding enabling X11 forwarding.

You should ultimately have terminal window with a prompt that indicates that you are running on coale.

Launch emacs as you generally do in 213:

@: $\textgreater$ cd 213/Week1

@: $\textgreater$ emacs 213ex1.r &

Verify that the trivial change that you made to this file is indeed present.

If you want to, you can run all the commands in the file, but that's more boring than it's worth. To test the important feature of the X11 server, create graph in R