Entire file system goes read-only

[gaf-mNDKBlG2WHs at public.gmane.org (Jerry Feldman)]

On 05/28/2010 10:17 PM, V. Alex Brennen wrote:
> On Fri, 2010-05-28 at 19:41 -0400, edwardp-mh2Nk+tgbQM at public.gmane.org wrote:
>> On one system while it's running, something occurs which causes the
>> /entire/ Ubuntu (10.04 LTS) file system to become read-only.  It was
>> first discovered two days ago when I tried to clear out the Trash
>> folder, when a window appeared indicating files were read-only.  I
>> then
>> opened the file manager and /every/ icon had a lock on it.
>>
>> At this point, the only options were to shut down or reboot.  Upon
>> doing
>> so, when the system comes back up, it goes straight into a file
>> system
>> check, then when that has completed, the system reboots itself
>> without
>> user intervention.
>>
>> Is there any possible explanation as to what (and why this) is
>> happening, or maybe something I can look at?
>
> I would recommend that you look in your syslog for any indication that
> disk I/O operations are failing.  If you don't see anything in your
> syslog (I'd be surprised if you didn't) I'd still check your disk and
> I/O controller hardware/cables.
>
> If you don't find anything, you might have some file system corruption
> that fsck can't fix.  Or, you have have a hardware problem somewhere
> else can can interfere with disk I/O (like a bad ram chip or processor).
>
> The reason I say this is that the Linux kernel will transition file
> systems to read only mode to try and prevent damage in the event it
> detects a hardware failure.  I'm, unfortunately deeply familiar with
> this process and how it occurs because of some recent experiences I've
> been having.
>
> We have some Linux servers running under vmWare Server 2.0 on host Linux
> systems with SATA drives.  When the I/O controllers become saturated the
> guest Linux kernel I/O SCSI ops on the virtual machine disks soft fail
> with a timeout because the host kernel is blocking waiting for disk I/O
> to complete.  After a few re-tries the Linux kernel of the guest virtual
> machine assumes that the I/O SCSI commands are failing because the I/O
> controller or because the disk has suffered a hardware failure.
> Surprisingly, this happens with as little as 12.5% I/O wait reported by
> the host kernel.
>
> Since the guest VM's Linux kernel assumes it is running on damaged
> hardware, it immediately puts all the file systems on the
> disk/controller it thinks failed into read only mode to protect against
> any possible further data loss or corruption.
>
> The end result is a guest VM with all read only file systems that needs
> to be rebooted and have all of its file systems fsck'd in order to get
> them remounted in r/w mode.  Of course, most of the applications running
> on the VM crash or report failure to any incoming requests because of
> the read only file systems.

It is very possible that the system remounted the filesystem as readonly 
before a syslog entry could be written.


-- 
Jerry Feldman <gaf-mNDKBlG2WHs at public.gmane.org>
Boston Linux and Unix
PGP key id: 537C5846
PGP Key fingerprint: 3D1B 8377 A3C0 A5F2 ECBB  CA3B 4607 4319 537C 5846