Entire file system goes read-only

[edwardp-mh2Nk+tgbQM at public.gmane.org (edwardp-mh2Nk+tgbQM at public.gmane.org)]

V. Alex Brennen wrote:

> I would recommend that you look in your syslog for any indication that
> disk I/O operations are failing.  If you don't see anything in your
> syslog (I'd be surprised if you didn't) I'd still check your disk and
> I/O controller hardware/cables.
>
> If you don't find anything, you might have some file system corruption
> that fsck can't fix.  Or, you have have a hardware problem somewhere
> else can can interfere with disk I/O (like a bad ram chip or processor).
>
> The reason I say this is that the Linux kernel will transition file
> systems to read only mode to try and prevent damage in the event it
> detects a hardware failure.  I'm, unfortunately deeply familiar with
> this process and how it occurs because of some recent experiences I've
> been having.
>
> We have some Linux servers running under vmWare Server 2.0 on host Linux
> systems with SATA drives.  When the I/O controllers become saturated the
> guest Linux kernel I/O SCSI ops on the virtual machine disks soft fail
> with a timeout because the host kernel is blocking waiting for disk I/O
> to complete.  After a few re-tries the Linux kernel of the guest virtual
> machine assumes that the I/O SCSI commands are failing because the I/O
> controller or because the disk has suffered a hardware failure.
> Surprisingly, this happens with as little as 12.5% I/O wait reported by
> the host kernel.
>
> Since the guest VM's Linux kernel assumes it is running on damaged
> hardware, it immediately puts all the file systems on the
> disk/controller it thinks failed into read only mode to protect against
> any possible further data loss or corruption.
>
> The end result is a guest VM with all read only file systems that needs
> to be rebooted and have all of its file systems fsck'd in order to get
> them remounted in r/w mode.  Of course, most of the applications running
> on the VM crash or report failure to any incoming requests because of
> the read only file systems.
>    

I looked at the syslog last night and did not find anything that 
appeared to be I/O errors specifically.

However, if something else appeared to be an error of some type, it was 
researched and the same indicators were found to have been filed as bugs 
with Ubuntu against past versions of the distro, so obviously those were 
not actual errors specific to this problem.

I then looked in the BIOS settings and discovered a "32-bit I/O" setting 
underneath the indicator for the hard drive, it was set  to "disabled" 
all this time.  I changed this to "enabled" and saved it.

Since enabling 32-bit I/O, up to now (fingers crossed), the file system 
has not gone back to read-only again and two LiveCD's (one Ubuntu-based, 
the other Mandriva-based) that previously would not load on this 
particular system (due to a change in the Linux IDE/SCSI drivers during 
2008 - in fact, no full distro released since the fall of 2008 would 
install on this system because of that change), all of a sudden, loaded 
fine and ran without problems or displaying errors.  A welcome surprise.

In attempting to find out why these LiveCD's and distros would not run 
or install previously on this, I had replaced the hard drive as well as 
the ribbon cables going to it and the optical drive, which did not 
result in any change.  Back then, the LiveCD's had reported numerous I/O 
errors while loading in the information and the installers themselves 
would crash during an installation.  The only way to get Linux installed 
on this, was to use an Ubuntu 7.10 LiveCD to install, then incremental 
upgrades to 8.04 LTS and then to 10.04 LTS.

Perhaps enabling the 32-bit I/O /may/ have fixed this, but I will 
continue to monitor.