After four years of development, I’m still a big CodeIgniter fan, although some other frameworks are starting to look mighty interesting.  Despite the bugs in Version 2.0.2, however, CI is heading in the right direction.

One CI-related issue that has caused problems for my workplace in the past has been the ability to setup distributed development environments.  Everybody has a different local setup, and some of the configuration directives inside of CI’s application/config folder must change from machine to machine.

Of course, we can’t just tell Mercurial to ignore that folder during commits, because there are some configuration directives that should remain the same across all environments, such as language, charset, enable_hooks, etc.

So, how to make some configuration settings consistent across different computers, and how to make some local settings?

Enter: config.local.php.

The solution is to create a local configuration file that overrides default config.php settings.  This way, local environments can set certain settings locally, and use the defaults in the application/config folder by default.  Here’s how to do it:

Step One: Create a local configuration file, and add certain configuration settings that you want to override.  You can put it in your root folder, and name it ./config.local.php For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

<?php if ( ! defined('BASEPATH')) exit('No direct script access allowed');
 
/* Local Configuration Options */
$config["base_url"] = "http://localhost/projects/my_project/";
$config["index_page"] = "index.php";
$config["uri_protocol"] = "AUTO";
$config["email_protocol"] = "sendmail";
$config["mailpath"] = "/usr/bin/sendmail";
$config["smtp_host"] = "";
$config["smtp_user"] = "";
$config["smtp_pass"] = "";
$config["smtp_port"] = "25";
$config["smtp_timeout"] = "5";
$config['log_threshold'] = 4;
//..and whatever else you want to override..
 
/* EOF */

Step Two: Set your version control system to ignore this file.  If you’re using Mercurial, simply add a line to the .hgignore file:

1

^config\.local\.php$

Step Three: Subclass the core/Config.php library to read the local configuration file after reading the default one every time.  To do this, create the file application/core/MY_Config.php in your application folder, and add the following code inside that file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?php  if ( ! defined('BASEPATH')) exit('No direct script access allowed');
 
class MY_Config extends CI_Config
{
	/**
	 * Load a config file - Overrides built-in CodeIgniter config file loader
	 * 
	 * @param string $file
	 * @param boolean $use_sections
	 * @param boolean $fail_gracefully 
	 */
	function load($file = '', $use_sections = FALSE, $fail_gracefully = FALSE)
	{
		parent::load($file, $use_sections, $fail_gracefully);
 
		//Local settings override permanent settings always.
		if (is_readable(FCPATH . 'system/config.local.php'))
			parent::load('system/config.local.php', $use_sections, $fail_gracefully);
	}
}
 
/* EOF: MY_Config */

Using local settings for database Configuration

Another file that is likely to be completely different from one environment to the next is the application/config/database.php file.  Since CodeIgniter seems to read the database settings a bit differently than normal settings, we have to go through one extra step to override those.  It’s a bit messier than I’d like, but gets the job done:

Step One: Add a line to your config.local.php for each database setting you want to override, but put the database settings in their own sub-array:

1
2
3
4
5
6
7

//...inside the local config file...
$config["db"]["hostname"] = "localhost";
$config["db"]["username"] = "localuser";
$config["db"]["password"] = "";
$config["db"]["database"] = "oeg_dev";
$config["db"]["db_debug"] = "TRUE";
$config["db"]["dbdriver"] = "mysql";

Step Two: Add the following code to the end of the application/config/database.php file to read those settings after reading the default settings:

1
2
3
4
5
6
7

//If there is a local config file, overwrite the settings with that..
if (is_readable(FCPATH . 'config.local.php'))
{
	include_once(FCPATH . 'config.local.php');
	foreach($db['default'] as $key => $val)
		$db['default'][$key] = (isset($config['db'][$key])) ? $config['db'][$key] : $val;
}

There you have it!  From now on, you can your local configuration file to override any configuration setting in CodeIgniter.  This makes sharing code a breeze.  You developers can set their own base_path, database, index_file and other settings that are likely to change from one environment to the next without modifying the main configuration files.

PS. Happy Friday!

PPS. Have a better method? I’d love to hear about it.

So where have I been these days? In between my six jobs, I’ve decided that instead of updating this blog, I’d drink some beer. However, we’ve run out of beer, so here I am. Which is good for you, because I have a great big awesome announcement:

Command Line Fridays start today!

GUIs are great and all, and they certainly have their place, but for those of us nerds who love our keyboards (I’m the guy on the right), it doesn’t get any better than the old CLI.

So, here are the rules of Command Line Fridays:

• You must use the CLI to perform all of your core duties all day long
• You may not open a GUI at any point during the day. If you open a GUI, YOU LOSE!
• You can use any CLI you like, but seriously, Windows CLI? Come on…

The cool thing is, even though it is 2011, you can still not only be ridiculously productive on the CLI, you can even slack off. You can Tweet and Facebook from the CLI, write documents, listen to music, code, run applications, view web sites, and generally get stuff done.

Here are a list of programs to get you started:

• WordPress CLI – Where do you think this blog post was written from? Post media and content to WordPress, and view WordPress blogs.
• VIM – Code in any programming language, or just write stuff. As powerful as any GUI IDE, and all keyboard based!
• Twidge – Post to, and read Twitter
• PianoBar – Listen to Pandora from the Command-Line (!)
• Lynx – An up-to-date, text-based web browser for the CLI
• FBCMD – A command-line Facebook client. Read and Post
• CenterIM – A full-featured IM client for Google, Yahoo, etc.
• Mutt – An email client that works with IMAP, POP3, and can be connected to Gmail

Look at all that cool stuff you can do! With all of that (and trust me–the list goes on..), why would anybody want to dink around with an inefficient mouse–the most inferior of all input devices (well.. maybe the trackball is worse).

So, go forth, CLI-lovers. I’ll see you in BASH. Happy Friday, and if I forgot to mention it three months ago, happy 2011.

About a year ago, I wrote an article on this website about Version Control.  In that article, I touted the glory of Subversion, and mentioned:

“I’ve since tried other VCS systems like Mercurial, Git, and Bazaar, but have not found any compelling reason to switch.”

Well, I’ve since found my compelling reasons to switch.  SVN lost some of our files, and it was a pain to do all sorts of stuff (get setup, keep clean, branch, merge, etc).  I won’t get into; there are enough rants about Subversion on the ‘net already.  We use Mercurial now, and boy has our workflow improved.

Mercurial is my new favorite child, because it’s easy to setup, easy to use, easy to branch and merge, and easy to teach to my students.  Oh, and it just works.  Sometimes I think it runs off magic.

In order to help my colleagues make the transition from SVN to Hg, I came up with a few diagrams to explain the workflow.  I thought I’d share them here.

Mercurial Recipe for Success

You will Need:

1. Mercurial.  Grab and install it from the official site.
2. Your files.  You know, all of those css, html, and php files that you’ve been working so hard on.
3. A repository server.  If you don’t have one, go get a free account at Bitbucket.
4. A web server.  Install Mercurial here too.

The Workflow:

Setting up a Mercurial Repository

Day-to-Day Developer Workflow

Do you love HeidiSQL as much as I do?  It’s really the best MySQL front-end I’ve found so far for Windows.  Sure, MySQL Workbench has all kinds of super-fancy features and GUI tools and what-not.  But, in Workbench, I find that it takes sixteen mouse clicks to perform the same task that you can do in five with HeidiSQL.

I use HeidiSQL for nearly everything, and only have had one major beef with it.  I could never get SSH Tunnels to work with it until today I figured it out! I’m super-stoked.  If you’re having problems with Heidi and SSH too, I’ll provide a step-by-step below (keep reading).

If you want to skip the background junk, and go right to the procedure, be my guest!

First: Why SSH Tunnels?

Okay, if you’re reading this article, I assume you are at-least familiar with MySQL and HeidiSQL.  Chances are, you connect to your database server via the default port 3306.  You open up HeidiSQL, enter your DB Username, password, and host, and away you go.

The problem with this approach is that the MySQL protocol that you are using to transfer your data around is inherently insecure.  Folks sniffing network traffic can intercept your data as it travels over the network.  So, if you care about your data or your database not being hacked, it’s a good idea to encrypt the traffic between your client (HeidiSQL) and your server (MySQL Server).

Insecure MySQL Traffic Flows Across the 'Net as Plaintext

SSH, on the other hand, is a secure protocol.  All traffic between the client and the server is encrypted so that nefarious network sniffer folks can’t decrypt the traffic (easily).

What you want to do is to convert the insecure MySQL traffic to secure SSH traffic on your computer before it hits the Internet.  On the server-side, you want to decode the SSH traffic and pass it along the MySQL server.   Something like this:

SSH Tunnel. All data travels across the 'net as SSH

Setting it Up

First, make sure you know what your SSH username and password are on the server where your MySQL database lives.  Then, follow along:

1. Download Plink.exe.  Plink is a nifty little SSH tool for Windows that allows you easily setup a SSH tunnel.
2. Place the downloaded file anywhere on your hard-drive you wish.  It’s probably a good idea to put it somewhere inside your home directory.
3. Now (this is the step that kept tripping  me up), before HeidiSQL can use Plink to connect to your server, you must download the server’s public key to your computer.
4. So, fire up your command-line, and browse to wherever you put the plink.exe file.
5. Type: plink.exe -L 3307:localhost:3306 [USERNAME]@[YOURSERVER.COM]
6. If it worked, you’ll get a big long message that ends with “Store key in Cache? (y/n)”
7. Say “yes”, of course.  Where does it put this key?  That was a mystery to me too!  It turns out, when you say yes, Plink will put the key into your Windows Registry (at HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys).
8. Type your password, and then type “exit” to finish up this step.
9. Okay, now that you’ve downloaded the key, you’re ready to use SSH Tunneling in HeidiSQL!
10. Fire up HeidiSQL
11. Create a new connection, and choose “SSH Tunnel” from the “Network Type” dropdown.
12. Inside the “Settings” tab, use “127.0.0.1″ for the hostname.  Then, enter your normal database username and password.  Yep, 127.0.0.1 is correct.  You enter the address as if you were logged-in to the server via SSH.
13. Next, flip to the “SSH Tunnel” tab.  Tell HeidiSQL where the plink.exe file is on your hard-drive, then put the actual IP address or DNS name for your database server.  Use 22 (SSH) for the port.
14. Enter your SSH username and password, and then choose “3307″ (or some other unused port) on your computer for the “Local Port”.
15. Let ‘er rip!

If all is well, and the powers-that-be are smiling down on you, you’ll be able to login to click “Open” and everything will work.  Congrats!  Now all your database traffic is travelling through the tubes securely.

If it doesn’t work, it’s time to check your steps or go bother somebody in the HeidiSQL forums.

Happy Friday!

I finally installed a caching plugin for this site, because I know that I’m so awesome that any day now, millions of beautiful women will be clamoring to visit my website at the same time.  Some things you are just sure of.

While I wait for that day, it makes sense to have a caching plugin anyway.  Let’s count the reasons:

1. It’s super easy, so time investment costs almost nothing,
2. I’m on shared hosting, so every saved bit counts,
3. Caching makes everything faster even when traffic is low, and
4. I never know when I may actually write something useful, and thousands of denziens from the across the Internet will come and bring my poor little Dreamhost server to its knees.

So what is caching anyway?

The Problem

Most websites are data-driven.  That’s a good thing, because without data, you don’t have content, and without content, you don’t have a very good website.  On a web server, your content can be stored basically in one of three places:

1. A Database
2. The Hard-drive
3. The RAM

The thing is, getting data from a database is much slower and requires a ton more resources than getting it from the hard drive.  Likewise, getting data from the hard drive is super slow compared to getting it from the RAM.  Here’s a handy diagram for you visually-oriented folks:

You can’t save your content in the RAM, because it’s not permanent storage.  You could use flat-files, but then you would lose all of the security, multi-access, and other benefits that database systems provide.  So, most sites store their data in slow, but reliable and robust databases.

This means, every time a user wants to view a blog post or other content on your site, the web server has to connect to the database, find the content, retrieve it, and process a bunch of code to turn it into HTML and make it look pretty.  Slow.  If there are 800 raving fans accessing that blog post at the same time, the server has to perform this process 800 times — simultaneously!  And… then the site goes down.

What Caching Does

So: Database = Slow, Robust, Reliable.  RAM = Fast, Volatile, Not permanent.

Can we have the good things without the bad?  Yes we can!  Caching flies in like a superhero on crack to deliver a solution that will work.

Basically, a cache library takes the content in your database and copies it into a flat-file or the RAM.  Then, it instructs your website to load the data from the RAM or file rather than the database when users visit your site.  This way, your server uses fewer resources for each request, but you still get to manage the data using the database, with all of its super-cool features.

The caveat is, if you update the data in your database, the server needs to know to also copy the data in the cache.  Fortunately, most caching systems handle these kinds details automatically.  Sometimes however, you have to manually delete and recopy the cached versions, depending on your cache system and how it is configured.

In most cases, the cached versions are set to expire after a certain amount of time has passed, and then regenerate with the latest fresh data in the database.  You can configure this amount of time to be 30 seconds, five minutes, eight days, or whatever.  If your users can live without up-to-the-second data, then this method of doing things won’t be a problem.

RAM Caching vs Hard Drive Caching

Memory caching is faster, but the cache will have to be regenerated if the server is restarted (not a big deal, IMHO).  File caching is a slower, but will remain in-tact if the server is restarted.

Memory caching requires that you or your systems administrator installs extra software on the server.  Ususally memcache (free).  File caching requires that your webserver software can write files to the hard drives.  Usually involves setting file permissions in Linux.

How to do it in WordPress, Drupal, Etc.

The cool thing about being a web developer in 2010 is that, if you can think of something cool, like caching your WordPress or Mediawiki installation, somebody has probably already created a tool for it and published it as open source.

Prerequisites

Before you start using these cool tools, however, you need to make sure your server can handle caching.  If you’re like 58.07% of websites on the Internet, you use Apache to serve up web pages.  So, the following tools would be good to have already setup.

If you’re on shared hosting, you’ll need to make sure that your server supports these (most good ones do).  If you manage your own server, you can install them yourself.  I’ve included the Ubuntu server commands below for those of you in the latter category:

• Memcache (sudo apt-get install memcached php5-memcache) — This is a piece of software that runs in the background on your server and allows you to store data in the RAM.
• PHP APC (sudo apt-get install php-apc) — This is a cache tool built specifically for PHP.  It does a lot of cool speedy-uppy things for PHP that other, more general cache tools cannot.
• mod_expires (sudo a2enmod expires) — A module for Apache that allows your website to instruct browsers to cache pages and other site resources for a certain length of time before downloading them again from the server.
• mod_headers (sudo a2enmod headers) — A module for Aapche that allows your website to modify headers before they are sent to the client.
• mod_setenvif (sudo a2enmod setenvif) — A module for Apache that allows you to change the way the webserver behaves based on different criteria, such as what browser the visitor is using.
• mod_deflate (sudo a2enmod deflate) — A module for Apache that allows the webserver content to be compressed before sending it over the network; not exactly like caching, but also a performance booster.  Many caching libraries use this.
• mod_rewrite (sudo a2enmod rewrite) – A module for Apache that allows the server to rewrite requests as they come in from the client.  Useful for redirecting clients to cached versions of files without having to invoke PHP, which saves even more time.

Not sure if you have these?  Don’t worry too much — most good caching libraries will let you know.

Popular Caching Libraries

There are a ton of good ones; here’s a sampling:

• WordPress SuperCache — So far as I can tell, this cache system uses files, and it helps guide you through the process of setting up your cache.  There’s even an easy mode with a simple “On/Off” switch, so it couldn’t be simpler:  Install the plugin and turn it on.
• WordPress Total Cache — A more powerful cache tool that lets you cache to memory or files (your choice) depending on what server software you have setup.  There are a ton of settings to manipulate.
• MediaWiki — File caching is built-in and well-documented.
• Drupal – Also has caching built-in, and Lullabot does a better job than I would explaining it.  Also, there are modules for Memcached, and APC.
• Frameworks — There are zillions of PHP frameworks, and most of the good ones have caching baked right in.  If you’re using a framework, you’ll want to refer to your documentation.

The End

Well, there’s your two-cent tour.  If you’re interested in learning more about speeding up websites, I recommend this book as a good starting place.

So, yesterday, I spent a few hours trying to create a robust command-line interface for one of my CodeIgniter applications.  I learned a lot about PHP-CLI.  I also came to remember why it’s a bad idea to try and fit a round peg in a square hole.

The thing is, CodeIgniter is a great little web framework.  See the key word in that sentence?  Yes, you can hack some CLI functionality in, but if you want to start doing fancy stuff like adding multiple levels of interactive menus and such, then its time to rethink what you’re doing.

On the other hand, there are many reasons to create a simple CLI-interface to your web application.  The main use would be for running of maintenance tasks and automated CRON scripts.  For this kind of task, it’s very easy to create a CLI entry-point for your app.

How to Do It.

I started off by grabbing some advice from NetTuts and downloading Phil Sturgeon’s excellent CLI library.

In the same folder as the main index.php file, I created another file named entry.php to serve as my CLI entry to the application.  That file looks like:

Notice that I am accepting command-line arguments.  The above code will halt execution and exit if any unknown command-line arguments are given.  It assumes the first non-switch command line argument is the action the user wants to run.  It automatically prepends the “utils” controller to the route, thereby sandboxing CLI access to a single controller. The controller can do whatever you want. Here’s a small example wherein I present a main menu if no route was specified when the user runs the CLI interface:

Notice the main menu!  The design goal for the Controller was for the main menu index method to be interactive, but every other method to be able to run unattended if parameters are sent, otherwise to display prompts.

I also made some heavy modifications and additions to the CLI.php library, adding the ability to run any form_validation validation function against command-line inputs (w00t), adding the ability to display view files easily, and adding a few other helper method here and there.

The code is too long to display in this post, but here’s a link to the file source on Bitbucket.

Wrap-Up

It’s definitely not a good idea to attempt a full, robust, multi-level CLI interface for your CodeIgniter application, but this framework will help you setup a simple, even slightly-interactive CLI interface to make running maintenance tasks and providing basic tools easier for your CLI users.

The repository with all of the code is available on my Bitbucket page.

A subject that has intrigued me lately is this topic of asynchronous PHP processing.  The idea is you have a web application that includes a task which takes a long time, like, say processing video files or sending mass emails.  Your users should be able to click a button, get a notification that their job is waiting to be processed, and then go about their business.

I’ve run across all sorts of really cool queuing technologies, all of which have PHP libraries. So creating and managing job queues was no problem for me. My confusion was how to setup a worker script to process those queues.

There is absolutely no way to get around the fact that you will need at-least two distinct, simultaneous processes running on your server in order to make this happen.  The process that initiates the queue jobs is presumably Apache (or whatever webserver you like).  The second process runs in the background, and it just sits and waits for jobs and processes them as they come in.

Of course, the only way to have a process continually listening for jobs on a Linux server is for it to be a daemon.  And, of course, unless you are a system administrator with root privileges, setting up a reliable daemon is not really an option.  So, if you are using a shared hosting provider, you are kind of out-of-luck.

Or, are you?

I’ve been toying with one way to make it happen using CRON, MySQL, and PHP-CLI, all of which good shared hosting providers allow you to use.  The idea is this: Every minute, a cron-task spawns a PHP-CLI script.  The script is smart enough to see if prior instance of itself is still running, and if not, it processes the job queue.  The queue can be a simple MySQL database table, or something fancier like a Beanstalkd queue or some implementation of Zend_queue.

Here’s how the script would run:

And, below is really rough example of how I might begin to implement this in PHP, with logging thrown in. I’m sure there are ways to tweak this, or there may already be libraries out there that do the same thing better; anybody care to chime in?

#!/usr/bin/php -q
<?php
 
/*
 * CRON Command-Line Processor for job queues...
 *
 * Requirements:
 *  1. *nix-based system
 *  2. Cron
 *  3. Shell access
 *  4. MySQL
 */
// Implement by setting permissions on this script to exeuctable, and adding
// by adding the following to your crontab: */ 
// */1 * * * * /path/to/this/script.php >/dev/null 2 >& 1
 
 
define('CRON_PROCESS_LOGFILE', '/path/to/logfile.log');
define('CRON_PROCESS_LOCK_FILE', '/path/to/lockfile.lock');
 
//------------------------------------------------------
 
class Cron_process
{
	public static function run()
	{
		//Log that we're getting started.
		Cron_process::log('Attempting to start cron_process of job queue...');
 
		//Check to see if a prior instance of the process is already running..
		if ($my_special_database_connector_class->query("SELECT `status` FROM `cron_process_status`;")->row() == 'in-process'
				OR file_exists(CRON_PROCESS_LOCK_FILE))
		{
			Cron_process::log('Aborted processing.. Prior job already running');
			return FALSE;
		}
 
 
		//Create a lockfile and indicate in the database that the process is started..
		$my_special_database_connector_class->query("UPDATE `cron_process_status` SET `status` = 'in-progress';");
 
		if ( ! file_exists(CRON_PROCESS_LOCK_FILE))
			touch(CRON_PROCESS_LOCK_FILE);
		$fp = fopen(CRON_PROCESS_LOCK_FILE);
		flock($fp, LOCK_EX);
		fputs($fp, 'cron_process_began_' . time());
		fflush($fp);
 
		//Process the jobs using whatever queue technology you want.
		while($my_special_queue_adapter_class->get_number_of_pending_jobs() > 0)
		{
			// *****************
			// HERE is where the magic happens!!
			//  All the code for processing the job goes
			//  right in here..
			// *****************
		}
 
		//Clear the lockfile and mark the job complete in MySQL
		$my_special_database_connector_class->query("UPDATE `cron_process_status` SET `status` = 'idle';");
 
		flock($fp, LOCK_UN);
		fclose($fp);
		unlink(CRON_PROCESS_LOCK_FILE);
 
		//Log that we're done
		Cron_process::log('Succesfully ran the cron_process. Exiting...');
 
		//All done!
		return TRUE;
	}
 
	//------------------------------------------------------
 
	public static function init_logfile()
	{
		if ( ! is_writable(dirname(CRON_PROCESS_LOGFILE)))
			return "Cannot write cron process logfile.  Directory " . dirname(CRON_PROCESS_LOGFILE) . " is not writable.  Check permissions";
 
		if ( ! touch(CRON_PROCESS_LOGFILE))
			return "Cannot write to cron process logfile.  Check permissions.";
 
		return TRUE;
	}
 
	//------------------------------------------------------
 
	public static function log($msg, $level = 'info')
	{
		$str = date('Y-M-d H:m:s') . " - $level - $msg\n";
 
		return file_put_contents(CRON_PROCESS_LOGFILE, $str, FILE_APPEND, LOCK_EX);
	}
}
 
//------------------------------------------------------
// Processing starts here..
 
//Initial Checks
if (php_sapi_name() != 'cli' OR ! empty($_SERVER['REMOTE_ADDR']))
	exit('This script can be run only from the command line!');
 
//Check to ensure we can write to a logfile.
$init_result = Cron_process::init_logfile();
if ($init_result !== TRUE)
	exit($init_result);
 
//Include any boostrap files
require('/path/to/any/bootstapfiles/my_special_db_class_and_others.php');
 
Cron_process::run();
exit(0);
 
 
 
/* EOF: cron_async.php */

I love, love, love getting rid of crap.  Maybe that’s why I like to read Unclutterer, or think that the 100 Things Challenge is such a great idea.  In real-life, I’m a minimalist, but when it comes to programming web apps, I’ve become a digital pack-rat.

I don’t care if my database grows into a giant data-monster, ready to ravage my server’s hard disk and slow queries down to a crawl; I’ll deal with it.  The key here is to not allow any important history of data to get lost.

Why Deleting is Bad

Here’s an example, to get us started:

Let’s say you have a simple online store, and you’re selling important consumer items to important people.  You have a database that keeps track of your customers, your products, and your transactions.  It may look something like this:

So, you’ve been in business for a while, and you have a bunch of customers in the system, quite a few products, and a lengthy transaction history.  Recently, you’ve decided to stop selling your Jell-O Brain Mold and you’re ready to remove it from your online store.  Common sense says that you would go to your database, find the record referring to your product, and delete it.

But wait!  If you do this, then the entire history of transactions linked to those products become orphaned (theproducts_id field will point to a non-existent record).  So, if you want to go back in at a future date to see the history of how many of those items that you sold, you will be in trouble.  You haven’t just deleted the item from your website per se, you’ve actually deleted the record of the item ever existing in the first place.

And, you don’t want to do that.

Flag It!

The useful alternative would be to add a flag to the data, so that instead of deleting the record, you could mark it as inactive.  This way, it wouldn’t appear on your website, but it would still exist in the database for future reference.

Your tables will now look something like:

Now you can tell your application to only show the products with `active` = “TRUE” on your website, but refer to products with `active` = “FALSE” when viewing older transaction histories.

This works pretty well, and if you program your User Interface correctly, your website users should never actually be deleting records; instead they should be marking them inactive so they “go away” but never truly die.

Audit History

Okay, now it gets a bit more complex.  In some cases, you don’t just want to mark a record “active” or “inactive” (that is, give it the ability to have two states), you also want to keep a complete revision history of that record, including what change was made, who made it, and when.  This way you can see what a record looked like at a particular moment in time.

This is not a new concept.  Think about your bank account.  It’s pretty much a bunch of numbers stored in a database somewhere.  Do you think that your checking account balance is simply a single field in a single record stored in a database?  Heck no!  Every single debit and credit is stored as a separate transaction, showing a complete history of changes to the account.

There are many ways to accomplish this, really, and for me to put up a tutorial, would be to reinvent the wheel.  The key is to think about which data you want to have complete histories on, and which data is not expendable.  Design your application around these constraints.

Sometimes you Just Have to Kill It

I wrote an application that contains patient records.  When an administrator clicks “Delete”, the patient doesn’t really go away; all of the information remains in the database.  Sometimes, however, we must delete the record permanently. Usually this is because a patient opts out of the program and requests to be removed from the study.  I use a special word for this kind of delete: Purge.

But, before the record is purged, the user has to click a checkbox, and go through two “Are You Sure?” screens informing him or her that not only will the patient record be deleted, but also the complete history of that record, and all data associated with it.  I try to have the application show the data in question on the screen the user has to click yes to.

…well, that’s enough for now.  Hope you’re thinking about data consistency, so you too can become a digital pack rat.

It’s the perfect time to think about it.  With disaster on everybody’s mind, and this being the season of resolutions, it’s time to think about your disaster recovery strategy.

Recently, a major department right here at FSU suffered catastrophic loss to their servers, with scant means of quick recovery.  Their site was down literally for weeks as staff from all over the University scrambled to assist them in recreating the critical pieces.  They lost of a lot of time and money.

I’m absolutely sure they’ll be super-prepared next time, but you don’t have to wait until something tragic happens to be prepared!

Disaster Recovery Enterprise Planning = Big Words (Scary!)

When I was an IT Manager n00b, I thought that planning for disasters was intimidating and complicated.  I thought it took a team of folks working around the clock, military-style, to prepare for incidents.  I was encouraged to believe this by colleagues constantly talking in business-speak, conference sessions talking about things like “DR evaluation metrics”, and by big ugly books that take days to even read through.

In the meantime, while I was being intimidated by all this “enterprise” stuff, we went without reliable backups.  We would have been Scre-ewed if something happened.

1. Embracing Common Sense

Okay, just ignore all that business-speak.  Let’s start with the basics:  If you’re not backing up your data, put down the big ugly DR books right now and go setup a backup.

No, seriously, do it right now.

This does not have to be complicated.  Go find an external hard drive, then find out where your data is on your computers or servers.  Plug the hard-drive into the back of the computers and copy the folders with your data.  Do this every day, and you have a basic disaster recovery strategy!

Pro Tip: If you’re using Windows, I’m a big fan of using Teracopy when I have to copy lots of data at a time.

You get bonus if you copy the data to two hard-drives and take one off-site every day.  Now your data is in two geographic locations on three devices.

See? Easy.  Now that you’re off and running, you can make your life even easier by automating your backups.  If you’re not excited about doing a bunch of geeky tech stuff to get your automated backups working, go convince your boss to spend a few bucks a month on cheap, easy-to-setup online backup services.  If he doesn’t want to spend the money, you can scare him with many horrifying stories of data loss on the Internet.

2. Being Prepared

Next Step: Just sit down in a quiet place for a few minutes and think about the things that could go wrong:

• “My server could suffer a catastrophic hardware failure”
• “My backup data could be corrupt”
• “The building could burn down”
• “We could get robbed”
• “An employee could go apeshit and format the C: drive of the server”
• etc..

Now consider which three or four of those scenarios are most likely to happen in your department. You’ve just considered your risks.  Now, for each one of those scenarios, think about (a) what you can easily do to guard against them and (b) what you would have to do to recover from them:

• Buy a second, backup server (or set an old computer aside to act as server temporarily in case of disaster).
• Keep two backup copies instead of one and periodically test opening files on the backup.
• Take backups offsite so that the data doesn’t burn down with the rest of the building.
• Lock the servers in a room without windows.
• Keep backups (already done), and make sure employees don’t have too much access to the server.
• etc..

If you write all of this stuff down somewhere and tell your boss about it, you have a basic DR plan!  Go have a beer.

3. Building Your Kit

Now that you are creating backups and thinking about our risks/responses, it’s time to build your Disaster Recovery Kit! Put this kit somewhere out of harm’s way, and have it handy in case you need it.  Here’s mine:

1. A camcorder bag — I like this, because it’s easy to carry and has a lot of compartments for all the stuff below:
2. Operating System Boot discs – In my case, Ubuntu Server 8.04
3. GParted Boot disc — For formatting and partitioning systems on-the-fly.
4. Cables – Standard power cable and USB cable.
5. Backup hard drives — Two.  They are my offsite copies of all our data.  I rotate them and keep them encrypted.
6. Flash light – Small and compact, for poking around inside computers or navigating server rooms in the dark
7. Thumb drive — Includes a backup of my server setup procedures/changelogs, systems passwords (in KeePass), and a bunch of portable apps.  Updated weekly.
8. Printed copy of contact information for key stakeholders — My IT colleagues, clients, and my boss, so I can keep them updated.
9. Geek tool kit –  Something like this.  Includes screwdrivers, etc.
10. External SATA Hard Disk Reader – Something small and compact.  In case I need to rip a hard drive from a computer and get the data off quickly.
11. A big sheet of paper that says “Don’t Panic“
12. Music CD with high-energy power music on it.
13. Energy supplement — Or a flask of whiskey, depending on how you roll.
14. “Getting started” guide – A piece of paper, right on top, that explains briefly how to use the kit.
15. Checklist — With all of these items on it, and with weekly DR Kit maintenance procedures.

There, now I can sleep at night much better.  What’s in your DR Toolkit?

As a Christmas Holiday present to those of you that are using my CodeIgniter tools, CI-UserTracking and GeSHi for CI, I did something very philanthropic: I updated them and tested them on the latest version of CodeIgniter.

Oh, and I also fixed the links…

..and I put them in their very own Google Code pages, so you can peruse the source code, download from Google servers, and leave comments and tickets for yours truly.

AND.. as your Hanukkah/New Years/Participation gift, I also have added a brand new CodeIgniter project that I’m working on called CI Headstart with all kinds of helpers, libraries, and other goodies to help speed up development even more.  Roll on over to the Code page to check it all out.

Now that Christmas break is upon us, and I since I work for a University that gives employees like a bazillion days off each December, I hope to clean these up even more and provide some quality documentation.

Happy Holidays from me to you,

PS >> If anybody is interested in contributing code to these projects, let me know.