Journaling file systems revisited

[mark at buttery.org (Mark J. Dulcey)]

Glenn Burkhardt wrote:
> 
> P.S.  Some of our older systems have only ext2, and have gotten into a funky 
> state that fsck doesn't fix.  No files can be created (file system full), but 
> existing files can be read.  'df' reports plenty of free space, and I can't 
> find unusually large files in /tmp, /var/tmp, or /var/spool.  I've tried to 
> force an 'fsck' with an abrupt powerdown, and the user reports the file 
> system being checked (I can't look myself, the system is in Italy).  So no 
> temporary files can be created, and some things don't work.  What can I do, 
> short of sending a new hard drive, or re-building the file system 
> (re-installing Linux)?

One thing to try:

Boot the system from a rescue disk. (If the computers are new enough to 
be able to boot from CD-ROM, you can use the installation CD from most 
Linux distributions for this purpose. For older systems, you'll have to 
make floppies.) Then, without mounting the damaged file system, run 
fsck. The "standalone" version that you run from the command line is 
more thorough than the version that is run at boot time on most Linux 
systems, it might fix problems that the other one missed.

Of course, this may be difficult to arrange, since the systems are 
remote. You might be able to get the same effect, however, by dropping 
the system down to the single-user run level, mounting the damaged file 
system read-only, and running fsck from there. Alas, that may be 
difficult, too, since going to single user mode disables the network 
support on most Linux distributions.

It looks as though, whatever you do, you'll be partly at the mercy of 
getting the user at the other end to do stuff for you. Good luck...