pelago

I’ve been developing for the web long enough — since my foray into PERL and HTML in 1994 — to know that there will always be a certain “rules” developers will adhere to when banging out lines of code. They are usually good rules because it is important for developers to follow them. However, developers need to know when to break from convention, when a higher purpose requires them to sacrifice the brilliance and elegance of their code. I personally have a few favorite examples from my own years of experience and from working with other developers…

The Almighty Framework

Frameworks are great for web development. The Model-View-Controller (MVC) framework, in its many interpretations for languages such as PHP, Ruby and Python, has made building web sites so much faster. And MVC frameworks make for nice clean code and a logical separation of data, business logic, actions and content. The view layer in and of itself is a godsend and the MVC framework has finally provided recognition for all the hard work put in by templating engine developers (i.e. Smarty).

What happens when the framework becomes a performance barrier? While developing our online project management software, Intervals, we’ve come across two main areas where we’ve had to lift the hood and require some of the framework internals.

First, web-based applications can be complex enough that the SQL queries can become cumbersome if they are not fine tuned. Using the default SELECT and JOIN conventions provided by the framework is not always ideal. When you start getting into the granular levels of optimizing SQL queries you have to get your hands dirty at the Model level. This means writing new queries, and tweaking and tuning them until they run as fast as possible. In some circumstances, this sometimes means associating a Model, especially list models, with a database table other than what it was intended. In addition, the framework is not going to optimize your database structure for you. Once the database is built you will need to tune the indices and learn about vacuuming and clustering.

Second, frameworks consume memory as they sift data up from the database, through layers of business logic, actions, and finally, into the view layer. For basic web-based applications that serve up limited information on a page, this is not a big concern and can usually be overcome using memcache if it does become one. However, if your web-based application is churning through a lot of data and presenting it to the user in real time, you will hit memory limits. This can happen, for example, with reports that contain a lot of data over a large date range (probably why Basecamp limits report data to a given number of months). In this case you will get the best performance with the View layer accessing the database directly using cursors. Yeah, I know developers won’t like this, but there comes a time when providing speed to your customers is more important than the framework upon which it is built. Anyways, cutting out the middle man you remove most of the strain on memory and increase the speed of the reports.

The Normalized Database

When we design databases our primary goal is to reduce the redundancy of data through the use of multiple tables, foreign keys, and queries that rely on JOINs. Developers will nitpick over a normalized database until every last bit of redundancy is ironed out. This approach works great for most web-based applications but when traffic increases all of those carefully crafted tables and keys, along with the JOIN-heavy queries begin costing you milliseconds, then seconds, before the app becomes unusable.

The solution is to begin denormalizing data. We called this a “necessary redundancy” at Pelago. You begin by identifying the slowest queries and removing their JOINs by placed the JOINed data in multiple tables. Than it’s up to stored procedures at the database level or developers at the code base level to make sure the redundant data is always kept redundant. The entire database doesn’t have to be denormalized all at once. Just the tables requiring JOINs that are causing you performance issues.

One Database to Rule Them All

Another harsh reality of web development is that the database may become too large and unwieldy for handling the number of people using the web-based application. Sys admins will start throwing around the four-letter word “sharding” as developers begin to cringe. If your app starts growing large enough, sharding may become a necessity for the app to scale. Breaking up your database onto multiple servers and keeping each copy of the database in sync with the others is a laborious task and should be a last resort. However, to dismiss sharding altogether in favor of throwing hardware at the problem is shortsighted. If your web-based application is growing you should be thinking about how you would shard the database if it becomes necessary in the long term. It’s better to have a plan in place before it’s needed than to be scrambling at the last moment to relieve an overloaded web application.

In fact, all of the “rules” I mentioned above should be addressed by web developers at some point if they have plans on scaling their web-based applications. Meanwhile, let’s hear from other web developers out there. What are some of the “rules” you’ve had to break?

Built using only Ubuntu, Crunchbang Linux has boiled down its distribution offering only the bare minimum needed to have a good balance between speed and functionality. Though not intended for older hardware, it is reported to work well in outdated environments where there is a premium placed on eking out as much speed as possible. Read more about it at http://crunchbanglinux.org/wiki/about and share with us any stories you might have in using it.

If you google around on the web you will find there are several tutorials on how to make your Ubuntu Linux installation run faster — especially on older hardware. These tips are very useful and range from minor tweaks to major overhauls. Being an intermediate Linux user myself, I found some of the more difficult optimizations to be overwhelming and not something I would want to try at home — changes to how Linux writes to the hard drive, for example. While researching and implementing the optimizations I felt comfortable with on my older Dell laptop and my new Asus eeePC 901, I jotted down all of my notes for future reference. Below are some tips for the intermediate user on how to optimize your laptop (or desktop) running Ubuntu Linux.

Reduce Swappiness

Most laptops have enough RAM installed that the swap space on the hard disk shouldn’t really be used. Yet the default setting for swappiness in Ubuntu is 60. By lowering it to 10 we can reduce the number of read/writes the the hard disk. This is especially handy for netbooks with solid state drives in them. To reduce the swappiness, follow these steps:

1. sudo sysctl -w vm.swappiness=10
2. Add the following line to /etc/sysctl.conf
   vm.swappiness=10

Preload

Preload is an adaptive read-ahead daemon that monitors running applications and analyzes them for commonalities. It then uses this data to predict what applications you might run and preloads them into memory. The result is faster startup times for commonly used programs. Installing preload is as easy. Just run the following line in a terminal window:

sudo apt-get install preload

» For further explanation, read this article on loading applications quicker in Ubuntu using preload.

Boot-Up Manager

The Boot-Up Manager (BUM) is a useful utility for starting and stopping boot-up scripts. Once installed it will appear in the System -> Administration menu. You can use BUM disable unnecessary boot-up scripts that may be slowing down your boot time. For example, I disabled saned, an API library for scanners, because I know I’ll never be attaching a flatbed scanner to my laptop. To install BUM:

sudo apt-get install bum

» Read the documentation for Boot-Up Manager

Startup Applications

If you are using Gnome for your desktop manager there will be an option to select which applications are started when you login. Go to System -> Preferences -> Startup Applications and uncheck the programs you don’t need. For example, I disabled the Evolution Alarm Notifier because I don’t use Evolution for setting alarms. I also disabled the Remote Desktop server, UME Desktop Launcher, and UNR Launcher. Just be careful to uncheck them and not remove them. In the case that you’ve removed something critical, you’ll want to be able to get it back.

Replace Metacity with Openbox

Openbox is a minimalistic and lightweight window manager that is known to run much faster than metacity, its bulkier counterpart. And you can easily get it working while still using Gnome as your window manager / desktop environment. By running Openbox inside the Gnome environment your desktop will become cleaner and faster. To install it, follow these steps:

1. Install openbox using:
   sudo apt-get install openbox obconf openbox-themes
2. Setup openbox as the default window manager by adding an entry in Startup Applications.
   To do this, go to System -> Preferences -> Startup Applications and enter the following:
   • Name: Openbox
   • Command: openbox ––replace

   Note: According to the Openbox documentation you can make it the default by choosing the GNOME/Openbox session when logging in and saving this as your default session. However, this functionality is broken in 9.04. The above steps are a workaround until this is fixed.

3. Choosing the Openbox theme and other configuration settings
   Go to System -> Preferences -> Openbox Configuration Manager to choose a theme you like and to update other settings such as Appearance and Windows.
   Note: changing the Desktops setting doesn’t effect the Gnome applet controlling the number of desktops. To change the number of desktops, you will need to revert back to Metacity and change them, then re-enable Openbox

» More information, documentation and screenshots available at the Openbox web site

That’s all for now

I found that making these few simple changes decreased load times on my older laptop significantly and made my newer netbook more minimal. Hopefully, these tips will help you as well.

Face detection in pure PHP

Technical description.

When running PHP as a cron, the $_SERVER['HOSTNAME'] variable is not set, nor are any other variables that will identify which server your cron is running on. This can be problematic if you are running the same cron on multiple servers, such as in a load balanced environment, and you need the cron to report back or log information about the server on which it ran.

Here is some code for getting the hostname value from your linux network configuration, assuming you have setup /etc/sysconfig/network properly.


//get hostname info from /etc/sysconfig/network
preg_match('/HOSTNAME=(.*)/', file_get_contents('/etc/sysconfig/network'), $network);
$hostname = split("\=", $network[0]);
echo $hostname[1]; //this equals the value of your HOSTNAME


For the intervals API, we’re wrestling with issues surrounding data input validation. This recently became interesting when the matter of date validation came up. Ordinarily, Intervals allows many, many different date formats, dependent on the locale that the customer is using (for example, Intervals may expect the date format ‘mm/dd/yyyy’ for US customers, ‘dd.mm.yy’ for a customer in Austria).

For our API developers, we wanted to use a common, universal format, one that would be easily compatible with our application and database layers. For that we selected ISO 8601, which is great in terms of widespread use, but not so great in terms of how complicated its specifications are.

Generally, ISO 8601 looks something like ‘2009-05-20′ for dates and ‘2009-05-20 12:30:30′ for date/time combinations. These two examples encompass 98% of the user input we’re likely to encounter. But we wanted to make sure that if we told developers they could use ISO 8601 dates, our system would support it. Read the rest of this entry »

Background

Pelago is a web design and development agency in Santa Barbara, California. Since our humble beginnings in August of 2000, we’ve seen the Internet landscape evolve exponentially in the last nine years. Our most current challenge as a creative and engineering agency is in embracing diversity in platforms and the inevitable shift towards the remote office. We use three different operating systems — Windows, OS X and Linux — on a daily basis. In addition to developing web-based project management software that is compatible with all three platforms, our designers and developers rely on them inside and outside of the office. Tunneling through the firewall from outside the office was our next requirement for embracing a diverse and distributed remote office.

Our SonicWall router makes it easy enough to establish a VPN connection using Windows. There is a client that can be downloaded from the SonicWall web site. What about other operating systems? Our developers often use Ubuntu Linux from home and required a way to VPN into the office. Once established, a VPN connection allows access to development servers, remote desktops, and other network resources inside the firewall; so you can see why it would be essential for the remote office.

The problem, as most Linux users out there already know, is that setting up a VPN connection in Ubuntu Linux is not very easy. After much trial and error, here is how we got the VPN working on Ubuntu Linux using Openswan.

Configuring the SonicWall Router

Login to your SonicWall router admin and make the following adjustments to the VPN settings.

1. Click on the VPN link and note the Unique Firewall Identifier for your SonicWall router. You will need it later for the value sonicwall.unique.identifier.
   
2. Under VPN Policies, create or edit the ‘GroupVPN’ policy.
3. Click on the General tab and set the following:
   
   • IPSec Keying Mode: IKE using Preshared Secret
   • Shared Secret: shared.secret.key (enter your secret key here)
4. Click on the Proposals tab and set the following:
   
   • IKE (Phase 1) Proposal
     DH Group: Group 5
     Encryption: 3DES
     Authentication: SHA1
   • Ipsec (Phase 2) Proposal
     Protocol: ESP
     Encryption: 3DES
     Authentication: SHA1
   • Enable Perfect Forward Secrecy (checked)
     DH Group: Group 5

Installing Openswan

If you are using Ubuntu, open a terminal window and type in:
sudo apt-get install openswan
The install will ask you a few questions about how you want to set it up. Select the suggested default for each step. This will install Openswan and create the ipsec.conf and ipsec.secrets configuration files.

Configuring Openswan

Add the following connection parameters to your /etc/ipsec.conf file:
conn sonicwall
     type=tunnel
     left=192.168.2.31 #your IP
     leftid=@home
     leftxauthclient=yes
     right=xxx.xxx.xxx.xxx #IP address of your sonicwall router
     rightsubnet=192.168.1.0/24 #gateway IP for your LAN. This will work for most
     rightxauthserver=yes
     rightid=@sonicwall.unique.identifier
     keyingtries=0
     pfs=yes
     aggrmode=yes
     auto=add
     auth=esp
     esp=3DES-SHA1
     ike=3DES-SHA1
     authby=secret
     #xauth=yes


Update: After upgrading to Ubuntu 9.10 a few things changed in the conf file. First, I had to comment out ‘xauth=yes’ as it was throwing an error. Second, I had to use my own static IP instead of %defaultroute. If you are using DHCP, %defaultroute may still work, but I have a static IP and had to change this.

Add the following line to your /etc/ipsec.secrets file
@home @sonicwall.unique.identifer : PSK "shared.secret.key"

Starting and stopping the VPN connection

Starting ipsec and opening the VPN connection
sudo ipsec setup ––start
sudo ipsec auto ––add sonicwall (note: if you change the configuration files, you'll need to run 'sudo ipsec auto ––replace sonicwall' to reload the file)
sudo ipsec whack ––name sonicwall ––initiate

Closing the VPN connection and stopping ipsec
sudo ipsec whack ––name sonicwall ––terminate
sudo ipsec setup ––stop


References

The following links were useful for getting the VPN connection up and running on Ubuntu Linux:
Openswan wiki page for SonicWall routers:
http://wiki.openswan.org/index.php/Openswan/SonicWall
SonicWall PDF instructions for using Agressive Mode and IKE with Pre-shared Keys
http://www.sonicwall.com/downloads/SonicOS_Enhanced_to_Openswan_Using_Aggressive_Mode_IKE_with_PreShared_key.pdf

The last two hours have seen the Jesusita fire take a turn for the worst due to hi temps, hi winds, and low humidity. Our offices are two blocks from the evactuation warning zone and most of downtown Santa Barbara is now in the evacuation warning area. Here is an updated Google map with the evacuation zone.

Here is the view from our office.

http://airbnb.com/ – “Rent nightly from real people in 809 cities in 71 countries.”

http://www.popcuts.com/ – “Every time a song you bought sells, you get a cut of the proceeds.”

http://bountii.com/ – “Find a lower price on any of the products below and we’ll pay you the Bountii.”

People are discussing whether it’s a complete copy of maxmind’s free database, but some are saying it’s not, either way nothing beats free.

IP address geolocation SQL database