System cracked, a story

[sweetser at TheWorld.com (Doug Sweetser)]

Hello:

I have a six year-old 200 MHz box connected to the Internet by a 56k
modem.  This is like driving a car with over a hundred thousand miles
with no money spent to repair dings.

Last Sunday, someone with a root kit was able to replace my
/etc/passwd file.  I had left my computer connected and unattended
(forget why), but when I returned, tcsh was not there, only bash.
/bin/tcsh generated a segmentation fault, so it was clear something
was very wrong, particularly when su did not work.  I was able to
close down the machine because a suid bit had been set on halt. 

I had a bootable Linux on a floppy.  After mounting the hacked hard
disk, I saw the "new" password file.  There was a new user added,
kodok.  I replaced it, and rebooted.  Still I was unable to get tcsh
back.  The computer was now booting into a new kernel.  In the root
home directory, I found a compiled program called psybnc.  Apparently
it is a program to hide IP addesses.

There were many programs that looked newly installed: 

depmod  e2fsck  fdisk   fsck    fsck.ext2  fsck.nfs     fstab
getty   group   halt    init    inittab    insmod       killall5
ld.so.cache     ld.so.conf      ldap       ldconfig     mke2fs
mkfs    mkfs.ext2       mkswap  nsswitch.conf           pam.conf
runlevel        shutdown        slamet      sulogin     swapon
tempfile        update

There may be others, I did not run a script to find all files created
on that date.  The root kit had a mission.

My computer was still bootable, I could still log into the internet
and get email.  Yet I was getting just a bit paranoid
(understandable), usually moving things I no longer trusted to a file
I called "virus".  It looked like my problem with tcsh was with with c
library.  In haste, I tossed away libc-2.3.1.so.  That deletion made
the box unbootable.  Oops.

All my data in my home directory has been burned to a CDROM.  As a
part of that process, I record all the installed packages.  For Debian,
that is a one line program: 

    dpkg --get-selections > ~/OS/Linux/selections.`date +%Y.%m.%d

A friend helped me reinstall all of those packages, and I added back
all the data from my hard disks.  One partition was wiped out by the
intruder, 3 MB of space on hda that had all of 7 mp3's, nothing else.

I had the book "Securing and optimizing Linux" by Gerhad Mourani which
I considered good, but had not followed through all the advice.  The
one thing I think would have halted this attack was not mentioned in
the book.  For the ext2 file system, /usr/bin/chattr +i /etc/passwd
would change the attributes of passwd to be "immutable".  I have a
program that makes a series of files immutable (passwd, shadow, group,
fstab, should I add others?) and can reverse the process.  This
sometimes complicated installations if a new user needs to be added,
but then I make things mutable for a moment, reinstall, and go back to
immutable.

In /etc/passwd, all the shells are set to /etc/false except root and
the two users of the machine.  Shadow passwords are used.  hosts.deny
reads ALL: PARANOID, host.allow ALL: 192.168.0.0/24, so only machines
local to the ethernet can get any services.

It took me about five days to recover (that's why I missed the last
meeting).  I don't rebuild Linux boxes often, but this time I took
notes, so getting a machine working the way I like should take much
less time in the future.

I don't know what security flaw was exploited.  I had snort installed,
but it is a program I never got to understand.  There are a bunch of
alerts that read:

# less alert
[**] [1:499:3] ICMP Large ICMP Packet [**]
[Classification: Potentially Bad Traffic] [Priority: 2]
05/17-19:00:30.168506 199.172.62.5 -> 192.168.0.2
ICMP TTL:253 TOS:0x0 ID:26246 IpLen:20 DgmLen:1500 DF
Type:0  Code:0  ID:39612  Seq:57072  ECHO REPLY
[Xref => http://www.whitehats.com/info/IDS246]

Whitehats are suppose to be the good guys, and the exact page
referenced is about Large ICMP Packets (I don't know what that
means).  There are snort logs, but I don't know how to read them.
Anyway, it is hard to trust the information from any logfile.  The
root kit tools are probably designed to cover up tracks.

The intruder wasted my time, but no data was lost.  If people have
other ideas about stopping root kits, I'd like to know.

doug
quaternions.com