Site Restored

Beginning in the late afternoon of Monday, July 21, I received emails from a number of HASA site visitors that something was terribly wrong with the site. Because of regulations at my workplace, I had to wait until I got home to view the site. My heart sank when I saw the errors affecting almost every page of the site. It was obviously malicious work, so I shut off the site to prevent further access.

Looking at source code, I saw information that should not have been there - JavaScript snippets, strange URLs, and other text garbage. A quick Google search revealed the problem. There had been a worldwide SQL injection attack on ColdFusion sites that ran on IIS web servers. Many sites had been brought down and a lot of data had been corrupted. Luckily, the attacker had only tried to insert code to convert site links to go to his own web site, and had not tried to delete data or erase databases. The ColdFusion community had responded quickly. By the time I was looking for information, the exact attack had been identified, various clean up scripts had been posted, and I had the tools necessary to fix the database.

What is a SQL injection attack? It is the nastiest, most destructive web site attack out there. In general terms, a coder takes a site form or URL, looks for a variable that appears to be an ID used by a database to locate a record, such as a story. Most database driven web sites work like this. The coder then resends the URL or form with a string of database commands (Structured Query Language statements) and tries to take over your database. If their SQL string is written well enough and if the database under attack is not protected, these commands can do anything from insert text to delete the entire database to (under very rare conditions) take over the server itself.

ColdFusion has a number of features to foil SQL injections right on the server and there are coding practices to help prevent the attacks. Just over a year ago, I had spent a month retrofitting much of the site to protect the main data tables and search functions from attacks. However, real life demanded my attention, and I did not get all the tables protected. Some of the lookup tables, like Story Era, were still vulnerable, and that's where HASA was attacked. In addition, the SQL statement used for the attacks was very sophisticated and bypassed most of the server-level protections.

The first thing to do was assess the damage. I ran some scripts and discovered there were 29 fields in 11 tables where trash data had been inserted. None of them affected the main story, forum, comment or review tables, which was a great relief. Also, non-HASA databases had not been affected, which meant damage was contained. Next, I ran a scrubber on those tables which located the bogus text and stripped it out. The biggest advantage here was that I did not have to run a restore from a backup, which would have lost content added to the site since the time the backup was run.

Data restored, I now had to track down those vulnerability points in the site. That is what has taken so long as every individual page in the site that an external (not logged in) user can reach had to be examined for all database connections and then those connections had to be evaluated for whether they were injection points. After the pages were reviewed, corrections made, and new code posted to a hidden site for the testers to check for bugs, all the forms had to be gone over. Forms are more dangerous than URLs, though more difficult to compromise, because they are the most common way to insert data into a database. The forms were completed yesterday. After that, the revised code was merged with the production code, uploaded again to another protected test area and put through its paces. That went on until pretty late last night.

There are a few glitches left. One form, the Knowledge Base search in the Research Library, could not be fixed and has been taken offline. The index search and all of the individual section lookups the Research Library are working. Also, the pop-up forum window attached to story pages will throw an error message if you try to use it. The Who's Online feature for logged in members does not appear to be working in production. There are a few conditions on a few forms we were not able to reproduce, so it's not 100% certain that they are working.

More testing will continue over the next month to be certain all access points were caught and to fix any bug accidentally introduced during the fixes. If you encounter an error message or you see something that just seems wrong on the site, please use the Contact form and send me an email at Tech Support.

This was the worst technical problem the site has experienced in its seven year history because of the data corruption. We got away with correctable damage, but it could have been much worse had the attacker been trying to destroy data. My deep thanks and appreciation for the site members who alerted me to the attack and provided their assistance to get the site fixed.

Anglachel