Boston Linux & Unix (BLU) Home | Calendar | Mail Lists | List Archives | Desktop SIG | Hardware Hacking SIG
Wiki | Flickr | PicasaWeb | Video | Maps & Directions | Installfests | Keysignings
Linux Cafe | Meeting Notes | Blog | Linux Links | Bling | About BLU

BLU Discuss list archive


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Entire file system goes read-only



On Fri, 2010-05-28 at 19:41 -0400, edwardp-mh2Nk+tgbQM at public.gmane.org wrote:
> On one system while it's running, something occurs which causes the 
> /entire/ Ubuntu (10.04 LTS) file system to become read-only.  It was 
> first discovered two days ago when I tried to clear out the Trash 
> folder, when a window appeared indicating files were read-only.  I
> then 
> opened the file manager and /every/ icon had a lock on it.
> 
> At this point, the only options were to shut down or reboot.  Upon
> doing 
> so, when the system comes back up, it goes straight into a file
> system 
> check, then when that has completed, the system reboots itself
> without 
> user intervention.
> 
> Is there any possible explanation as to what (and why this) is 
> happening, or maybe something I can look at?

I would recommend that you look in your syslog for any indication that
disk I/O operations are failing.  If you don't see anything in your
syslog (I'd be surprised if you didn't) I'd still check your disk and
I/O controller hardware/cables.

If you don't find anything, you might have some file system corruption
that fsck can't fix.  Or, you have have a hardware problem somewhere
else can can interfere with disk I/O (like a bad ram chip or processor).

The reason I say this is that the Linux kernel will transition file
systems to read only mode to try and prevent damage in the event it
detects a hardware failure.  I'm, unfortunately deeply familiar with
this process and how it occurs because of some recent experiences I've
been having. 

We have some Linux servers running under vmWare Server 2.0 on host Linux
systems with SATA drives.  When the I/O controllers become saturated the
guest Linux kernel I/O SCSI ops on the virtual machine disks soft fail
with a timeout because the host kernel is blocking waiting for disk I/O
to complete.  After a few re-tries the Linux kernel of the guest virtual
machine assumes that the I/O SCSI commands are failing because the I/O
controller or because the disk has suffered a hardware failure.
Surprisingly, this happens with as little as 12.5% I/O wait reported by
the host kernel.

Since the guest VM's Linux kernel assumes it is running on damaged
hardware, it immediately puts all the file systems on the
disk/controller it thinks failed into read only mode to protect against
any possible further data loss or corruption.  

The end result is a guest VM with all read only file systems that needs
to be rebooted and have all of its file systems fsck'd in order to get
them remounted in r/w mode.  Of course, most of the applications running
on the VM crash or report failure to any incoming requests because of
the read only file systems.


      - VAB








BLU is a member of BostonUserGroups
BLU is a member of BostonUserGroups
We also thank MIT for the use of their facilities.

Valid HTML 4.01! Valid CSS!



Boston Linux & Unix / webmaster@blu.org