¶ Introduction to Linux systems and the Moriah cluster
Since the Moriah cluster nodes (the actual hardware making up our “cloud”) are running a Linux based operating system, the easiest way to connect to it is from another Linux-like system, which will be demonstrated below. Guides for other operating systems are also available outside of this tutorial.
¶ Terminals
Any Linux machine (Laptop, PC or High Performance Computing server) will have some sort of “Terminal” program.
The terminal is the program we launch in order to get to the “command-line-interface”, or CLI of the operating system, as opposed to a graphical user interface you may be more accustomed to from your Windows or Mac machines.
- Mac-OS is also historically based on a Linux like system, and so macbooks and their kin also come with a built in terminal, which is using the zsh shell environment. Don't be alarmed if this doesn't mean anything yet! We'll get to shell environments eventually.
- While Windows does offer a basic terminal (the windows command line, or “cmd”) as well as a more feature rich “power-shell”, these do not support direct access to linux machines as easily as Mac or Linux machines. Windows users will often need to download extra software (we recommend using MobaXTerm) to enable opening a terminal on a remote Linux machine.
- Starting with Windows 10, Microsoft has started working on a built-in “Windows Subsystem for Linux”, or WSL. The WSL offers a native Linux experience baked into your Windows laptop or PC. While having WSL activated on your machine allows opening a Linux terminal and connecting to the cluster nodes more easily, it will currently not be covered by this guide.
Technically, the terminal will allow you to run any program on the machine you are connected to.
While we will focus mostly on shell commands in the coming sections, opening graphical applications is also possible via some added configuration.
¶ Working In the Shell Environment
As we've seen above - the first program we encounter when working in the HURCS environment is the terminal. The terminal program gives us access to a command line interface called a shell. The shell interprets the commands written in the terminal and passes them on to the operating system.
¶ Why bother?
Why do you need to learn the command line anyway? Well, let me tell you a story. Many years ago we had a problem where I worked. There was a shared drive on one of our file servers that kept getting full. I won't mention that this legacy operating system did not support user quotas; that's another story. But the server kept getting full and it stopped people from working. One of our software engineers spent a couple of hours writing a C++ program that would look through all the user's directories and add up the space they were using and make a listing of the results. Since I was forced to use the legacy OS while I was on the job, I installed a Linux-like command line environment for it. When I heard about the problem, I realized I could perform this task with this single line:
du -s * | sort -nr > $HOME/user_space_report.txtGraphical user interfaces (GUIs) are helpful for many tasks, but they are not good for all tasks. I have long felt that most computers today are not powered by electricity. They instead seem to be powered by the "pumping" motion of the mouse. Computers were supposed to free us from manual labor, but how many times have you performed some task you felt sure the computer should be able to do but you ended up doing the work yourself by tediously working the mouse? Pointing and clicking, pointing and clicking.
The excerpt above is taken from the great Linux learning site- linuxcommand, where you can find some simple tutorials and explanations about Linux in general and shell commands in particular. This wiki also contains some evolving sections about useful commands and common features available for HURCS users.
¶ First encounters
When you first log-in to the HURCS cluster, your terminal program of choice will open and present you with some text and a blinking cursor. In this example, I will be connecting from my laptop (named “penguin” for some reason) to the Moriah Gateway Node.
The screenshot above is crowded with text, but most of it is not important.
- The first line is an
sshcommand. SSH is used in order to establish a connection between the local computer and the remote machine. - The second line is a prompt for the user's password. Since no username was provided, the local username (zivben) is used. luckily that is also my username at HURCS!
- The next 8 lines (not including blank lines) are the standard greeting message. It states:
- Which node you have connected to (moriah-gw-01)
- What OS version it is running (debian10-5782)
- Contact information for reaching support (system@cs.huji.ac.il)
- A note stating that this specific (host) machine is part of something called a SLURM cluster. We'll talk about that in a later unit.
- Login information - when was the last login performed and from which IP address.
The commands we type on the command line are mostly not single words, they are often “sentences” made of multiple parts.
The first word of each such sentence is the command itself, and the following words and symbols are arguments that affect the process of the command - same as you would interact with a graphical application, selecting settings from menus and typing values in fields. Below is a breakdown of each part of the first line, with the first four columns being the “command prompt” (the part that waits for input) and the following three being the actual typed command:
|
zivben |
@ |
penguin |
:~$ |
ssh |
-X |
moriah-gw.cs.huji.ac.il |
|
|---|---|---|---|---|---|---|---|
| part name |
user-name |
‘at’ |
machine-name |
command |
command |
option |
command argument |
| details | Currently connected user | Active machine | This field may change according to the shell environment but it always means “your command here!” | The first word typed in the prompt is the command being executed | Commands often have special arguments called ‘options’, behaving like check-boxes in a Graphical Interface | Other arguments can be free-text. In this case, the address of the server we want to connect to. |
While this is a bit complicated for a first example, it sets up the basic premise for the next section!
¶ Commands and basic navigation
The linux-command website also has a great tutorial about navigation, but we'll give some relevant examples and explanations below. The three most basic commands for navigating a Linux file-system are:
pwd- “Print Working Directory” which will simply tell you where you are.ls- “LiSt directory content” is used for printing (to the screen) the content of the current working directory.cd- Will “Change Directory” to the current user's home folder.
Don't worry if you don't exactly understand all of the above, trying is the best way to learn so we'll do some exercises, but for that we need to discuss arguments.
¶ command line arguments
When reviewing the three commands above, you may have been wondering - all I see is the current working directory and the home directory! How do I actually move around?
The answer to that question lies with arguments. While it's true that you can use the cd command on its own (to get to your home folder), it will usually be used along with an argument. Specifically, the argument will be the path of the Directory to which we want to Change to.
So another way to describe the cd command can be:
cd [directory]
Where “directory” is the name of the argument, and the square brackets mean that it is optional. This sort of notation is common in program manuals, and cd is, in fact, a program!
You can see the complete program documentation for cd in this link, where you may notice that we have omitted the [-L|-P] part (go back and check if you didn't notice!). These are command options that can be passed to the program to affect the way it carries its work, specifically the way they handle things called “dot” and “dot dot” operands. While the function of these options (-L and -P) is not really important to us for the time being, the operands they handle are, since they are an inherent part of the directory tree.
¶ Files and folders, users and permissions
You may already be familiar with the concept of a directory tree, even if you didn't know it!
At one point or another, both macOS and Windows had an option to navigate the content of your drive in the form of nested icons like so:
The regular windows file explorer is probably one of the most familiar windows applications. We use(d) it to browse our documents, music and movies (before everything went streaming…).
You may be used to the process of navigating the file system using your mouse - clicking on the “Pictures” icon and then “Thailand 2012” to look at pictures of a particular vacation, or “Documents” then “Thesis” if you're feeling particularly adventurous…
In the example above, some log files related to the computer manufacturer's (Dell) driver update package. The file explorer lists the files in the main window as well as some information like their modification date and size. Note that the folder called “Log” is highlighted in the left panel. This means that this is the current folder being inspected.
Our friendly terminal command prompt can help us get the same information using a few simple commands. We'll use the ls, cd, pwd and tree commands to brows the file system:
- After a terminal screen is opened, we'll use
pwdto Tell us where we are. - Using the
lscommand, we'll list the contents of that folder. - The
lscommand only gives us the names of the files and folders, but if we want more details, we'll have to use the-longformat option by typingls -l. - Many more flags are available for the
lscommand. Typingls --helpwill print out some basic information and instructions about it.
Try and figure out how to sort the output list according to date and size, both ascending and descending order. - Finally, we can use the
treecommand to display a nice directory tree like in the above left panel. You can usetree -L l, wherelis any positive integer, to limit the level to which the tree is drawn. Try using increasing values ofl, liketree -L1thentree -L 2etc. to see the effect yourself.
Example output will like this:
Note that while there are no column names like in the windows file explorer, most values produced by ls -l are pretty self-explanatory.
From right to left we have:
- The entry name. note that the colors of the entry names change according to their type, with yellow for folders, teal for links to files or folders, along with an arrow pointing to the target of the link, white for documents, peach for scripts and dark blue for archives. These color conventions are true for the CS and HURCS environments when using the default c-shell. while other shell environments and other Linux builds may behave differently.
- The next column displays either the year the file was last modified at, if it was in the previous year or before, or the exact hour and minute, if within the current year.
- Day of last modification.
- Month of last modification.
- File size in bytes. (try using
ls -lhand see what happens!) - Name of the “group” to which the entry belongs
- Username for the entry owner.
- Entry permissions
The last 5 columns are quite clear then. But the concept of users, groups and permissions may be new to most people.
The tree output can look something like so:
-L argument with the value of 2 limits the output of the tree command to two levels deep