Best methods to clean up a hacked site with no clean version available?

Question

I have been asked to fix a hacked site that was built using osCommerce on a production server.

The site has always existed on the remote host. There is no o

Answer 1

Obtain a fresh copy of the osCommerce version the site was built with, and do a diff between the new fresh osCommerce and the hacked site. Also check for files which exist on the server but not in the osCommerce package.

By manually comparing the differences, you can track down all possible places the hack may have created or modified scripts.

Answer 2

I know this is a little late in the day to be offering this solution but the official fix from osCommerce developement is here: http://library.oscommerce.com/confluence/display/OSCOM23/(A)+(SEC)+Administration+Tool+Log-In+Update

Once those code changes are applied then most of the actual work is in cleaning up the website. The admin login bypass exploit will be the cause that has allowed attackers to upload files via the file manager (usually) into directories that are writable, often the images directory.

There are other files that are often writable too which can have malicious code appended in them. cookie_usage.php and includes/languages/english/cookie_usage.php are the usual files that are affected, however on some server configurations, all site files can be susceptible.

Even though the official osCommerce fix is linked to above, I would also suggest to make this change as well: In the page above, scroll down till you see the link that says "Update PHP_SELF Value". Make those changes as well.

This will correct the way $PHP_SELF reports and prevent attackers from using malformed URLs in attempts to bypass the admin login.

I also suggest that you add htaccess basic authentication login to the admin directory.

Also check out an addon I authored called osC_Sec which is an all in one security fix, which while works on most php backed websystems, it is specifically designed to deal to the issues that exist in the older versions of osCommerce. http://addons.oscommerce.com/info/8283

Answer 3

Because you cannot trust anything on the web host (it might have had a rootkit installed), the safest approach is to rebuild a new web server from scratch; don't forget to update all the external-facing software before bringing it online. Do all the updating on the happy side of a draconian firewall.

When you rebuild the system, be sure to pay special attention to proper configuration. If the web content is owned by a different Unix user than the web server's userid and the permissions on the files are set to forbid writing, then the web server cannot modify the program files.

Configure your web server's Unix user account so it has write access to only its log files and database sockets, if they are in the filesystem. A hacked web server could still serve hacked pages to clients, but a restart would 'undo' the 'live hack'. Of course, your database contents could be sent to the Yakuza or corrupted by people who think your data should include pictures of unicorns. The Principle of Least Privilege will be a good guideline -- what, exactly, does your web server need to access in order to do its job? Grant only that.

Also consider deploying a mandatory access control system such as AppArmor, SELinux, TOMOYO, or SMACK. Any of these systems, properly configured, can control the scope of what can be damaged or leaked when a system is hacked. (I've worked on AppArmor for ten years, and I'm confident most system administrators can learn how to deploy a workable security policy on their systems in a day or two of study. No tool is applicable to all situations, so be sure to read about all of your choices.)

The second time around, be sure to keep your configuration managed through tools such as as puppet, chef, or at the very least in a revision control system.

Update

Something else, a little unrelated to coming back online, but potentially educational all the same: save the hard drive from the compromised system, so you can mount it and inspect its contents from another system. Maybe there's something that can be learned by doing forensics on the compromised data: you might find that the compromise happened months earlier and had been stealing passwords or ssh keys. You might find a rootkit or further exploit tools. You might find information to show the source of the attack -- perhaps the admin of that site doesn't yet realize they've been hacked.

Be careful when inspecting hacked data -- that .jpg you don't recognize might very well be the exploit that cracked the system in the first place, and viewing it on a 'known good' system might crack it, too. Do the work with a hard drive you can format when you're done. (Virtualized or with a mandatory access control system might be sufficient to confine "passive" data-based hacks, but there's nothing quite like throwaway systems for peace of mind.)