Linux Useful Commands

From CsWiki
Revision as of 13:32, 11 April 2018 by Naamaplay (Talk | contribs) (Created page with "== Storage Policy == === Backups === First of all, it is important to understand that none of our main file systems are backed up on other devices. This means that in case of ...")

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Storage Policy

Backups

First of all, it is important to understand that none of our main file systems are backed up on other devices. This means that in case of a hardware crash data will be lost.

Snapshots

Some file systems, have snapshots, which allow a rollback of the file system to a certain point in time. Snapshots reside on the same file system as the data itself, and without backup will be lost as well if a crash occurs.

The main storage systems are:

Home directories: These reside on a highly capable file server with provided snapshots and 
have the lowest chance of crashing but are relatively more limited in space. 
Storage for labs:  These are directories that reside under /cs/labs. They provide snapshots
and are the volume of choice for large amounts of data. 

Storage limits

Each faculty member is allocated up to 2 terabytes of space, which should cover most users. Anyone requiring more space will be asked to pay a yearly amount that will cover the price of additional storage resources. It is important to note that for each terabyte of data 500 gigabyte is reserved for snapshots.

Other file systems can have varying snapshot policies. Contact us if you want to better understand their status.

Additional recommendations

To safely store code, we suggest using our Github. The github virtual machine and data are backed up on two different file-systems, providing high-level resilience.

Another easy backup option is using the unlimited space Google provides along with your mail HUJI account. Some students and faculty members are using this option and are pleased with the results.

Snapshots do not behave well with heavy writes. If your lab needs to write a lot of data or do heavy reads please contact us for further assistance.

Disk quotas

Users are allocated disk quota as follows:

File System Type
Account typeGroupPrimarySafeSecondaryTertiary
undergraduatestud1G200M00
graduategrad2G2G00
doctoralphd4G04G0
post-doctoralpdoc5G05G0
visting lecturervisitor5G01G0
facultystaff20G0--
guestguest1G01G0
external graduatecgrad2G2G00
external phdcphd4G04G0

To see how much quota you have, type nquota. Note that the data for nquota is updated once every few minutes.

Home directories

User's home directories are located as follows:

File System TypePathShorthand
Primary/cs/usr/login~
Safe/cs/+/usr/login~/safe
Secondary/cs/+/usr/login~/.+
Tertiary/cs/++/usr/login~/.++

Note: ~ is a UNIX alias to one's home directory. ~/safe, ~/.+ and ~/.++ are symbolic links under ~. See ln(1).

N.B. Groups stud, studap, grad and cgrad have ~/safe directories.

Snapshot and Backups

Snapshots

Snapshots are short-term online backups. There are three types of snapshots: hourly, daily and weekly. Snapshots cycle out -- i.e. if N hourly snapshots are defined for a filesystem, when the next snapshot is created, the oldest ([N+1) snapshot disappears permanently. Note that online means that the snapshots are part of the filesystem. If a filesystem becomes unavailable due to a fileserver failure, the snapshots are unavailable as well.

Snapshots are available in the special .snapshot directory and can be accessed using the snapshot utility or directly (e.g. "cd .snapshot"). Note that the .snapshot doesn't show up when searching a directory (i.e. "ls -a" won't show it), but it exists and can be "cd"-ed into.

The current snapshot schedule is as follows:

File System TypePathHourlyDailyWeekly
Primary~24148
Safe~/safe24148
Secondary~/.+24148
Tertiary~/.++000
Graduate Labs/cs/labs/supervisor/login010

Note that tertiary directories (~/.++) have no backups whatsoever.

Various labs have specialised filesystems. The snapshot policy of each and every such filesystem is determined according to the lab's needs/requests.

Backups

Currently, undergraduate and graduate directories have snapshots only.

In the future, offsite mirroring of primary, safe and secondary directories is planned.

Also in the future, offsite backups for longer-term retention is planned in coordination with the Computation Authority.

Disk space shortage

Logging in without quota

If there is insufficient disk space, some window managers might not start at all (e.g. xfce4, or KDE). When this happens you'll receive some message of insufficient quota, or an error message saying the Xsession has terminated too quickly. In this case, to be able to login and clean up your quota, you'll have to login using xterm.

There are 2 ways to login in xterm:

  1. When the chooser appears, choose XTerm instead of the normal window manager you use.
  2. Before logging in (i.e. entering the username and password), click on the Session button, and choose Failsafe Terminal

After you've logged in, you'll get a simple xterm window. There you can start cleaning up your quota as described in the following sections. When finished, simply type exit, and re-login again.

Note: Without a window manager, to write commands to the xterm window, you'll have to place the mouse courser on that window.

Basic cleanup

Some basic utilities to manage disk space:

  1. Are you using safe or secondary storage area (~/safe and ~/.+)?
    If not there's more space there you can use. Note that with ~/.+ the backup is less often, so it is not advisable to use it on active projects.
  2. Use the cleanup script to remove files that can be safely deleted:
    cleanup
  3. Use du to find how space is distributed within your home directory:
    du ~ | sort -n
    Note that this will also show you the space taken by files beginning with '.' which do not normally show in directory listings.
    A more elaborate usage of the du utility is:
    \ls -A1 | sed "s#'#\\\'#" | sed 's/ /\\ /g' | xargs du -sckx -- | sort -n
    Which will calculate the sum of each directory (in the current directory). New students have an alias for this as dua.

Mail

This section only applies for those using a local program for reading mail (e.g. thunderbird, kmail, etc.), if you are using a web browser to read the mail (e.g. the mail.huji mail account's web interface) this does not apply.

  • By default, your mail is saved under your home directory. Old mail and mail with big attachments (multimedia, PDFs, junk) can cause space shortage. To avoid it, remember to periodically clean up your inbox.
  • You should be able to see if the mail is taking your disk space by using the above du command, either ~/Mail, or ~/.mozilla-thunderbird will take a lot of space (depending on your mail client).
  • On some mail clients (e.g. thunderbird/icedove, kmail) simply deleting the mail isn't enough. You need to compact the inbox folder (or other folders, if you use any). You can do this (in kmail and thunderbird/icedove), by right clicking on the folder, and choosing "Compact this folder".
  • On thunderbird/icedove, you can tell the program to do this automatically by setting:
    Edit -> Preferences -> Advanced -> Offline & Disk Space -> Compact folders when ...
    For new users, it is already set (so old users can use reinstall thunderbird, to set this configuration).
    • Sometimes, when very low on memory compacting doesn't work. In that case, you can either:
      1. Better: reinstall thunderbird - reinstall also compacts the folders (specifically, the Inbox, Sent, and Trash).
        or
      2. Risky: Run the script /cs/share/scripts/compact-thunderbird.sh.
Warning: Please note that you need to close thunderbird before running these scripts.
These methods should be executed only once, since after changing the preferences and compacting the folders, thunderbird should be able to manage the compcat by itself (and it will do so each time you open it).