In the afternoon of Monday, November 6, 2023, the Isilon cluster experienced a serious crash as a result of both hardware problems and a software glitch. When the cluster came back online on Sunday, November 12, 2023, a significant amount of files had been found (by an internal Isilon consistency programme) to have been damaged.
To illustrate how to recognise corrupted files in a directory, we will use the example of the directory /sci/labs/ben.madan/julie.bergman/images/proj5.
In this directory, there were five files before the crash:
% cd /sci/labs/ben.madan/julie.bergman/images/proj5
% ls
red.jpg
orange.jpg
yellow.jpg
green.jpg
blue.jpgFile orange.jpg was corrupted.
The Linux user may see the corruption is four ways:
1. The file has disappeared:
% ls
red.jpg
yellow.jpg
green.jpg
blue.jpg2. ‘ls’ gives an error “No such file or directory”:
% ls
ls: orange.jpg: No such file or directory
red.jpg
yellow.jpg
green.jpg
blue.jpg3. It is possible, ‘ls’ will not show the file, but ‘ls -l’ will:
% ls -l
ls: orange.jpg: No such file or directory
-rw-rw-r-- 1 julie.bergman ben.madan_lab 4405 Dec 9 2021 red.jpg
-rw-rw-r-- 1 julie.bergman ben.madan_lab 3217 Dec 7 2021 yellow.jpg
-rw-r--r-- 1 julie.bergman ben.madan_lab 5986 Dec 10 2021 green.jpg
-rw-rw---- 1 julie.bergman ben.madan_lab 3409 Dec 3 2021 blue.jpg4. The error may be “Input/Output error” instead of “No such file or directory”
% ls
ls: orange.jpg: Input/Output error
red.jpg
yellow.jpg
green.jpg
blue.jpgDifferent clients may show, for the same directory, different errors from the above four ways. The public host “pita” is best to use to show corruption in a consistent manner.
If your software can work with without using the file, then open another file with a different name and keep working.
However, if your programme requires that particular filename, the only solution is to move the entire directory to another name and copy the contents back to the original name – you cannot modify or erase a corrupted file. This will allow you to create a new file with the same name as the corrupted file.
For example:
% pwd
/sci/labs/ben.madan/julie.bergman/images/proj5
% cd ..
% pwd
/sci/labs/ben.madan/julie.bergman/images
% mv proj5 proj5.corrupted
% ls
proj1
proj2
proj3
proj4
proj5.corrupted
proj6
% cp -rp proj5.corrupted proj5
% ls
proj1
proj2
proj3
proj4
proj5
proj5.corrupted
proj6
% cd proj5
% ls
red.jpg
yellow.jpg
green.jpg
blue.jpg
% echo Hello World > orange.jpg
% ls
red.jpg
orange.jpg
yellow.jpg
green.jpg
blue.jpg
International efforts are being made to try to recover the data of the corrupted files. However, it will only be known towards the end of 2023, the extent of data that can be recovered.