The Almighty Framework

Frameworks are great for web development. The Model-View-Controller (MVC) framework, in its many interpretations for languages such as PHP, Ruby and Python, has made building web sites so much faster. And MVC frameworks make for nice clean code and a logical separation of data, business logic, actions and content. The view layer in and of itself is a godsend and the MVC framework has finally provided recognition for all the hard work put in by templating engine developers (i.e. Smarty).

What happens when the framework becomes a performance barrier? While developing our online project management software, Intervals, we’ve come across two main areas where we’ve had to lift the hood and require some of the framework internals.

First, web-based applications can be complex enough that the SQL queries can become cumbersome if they are not fine tuned. Using the default SELECT and JOIN conventions provided by the framework is not always ideal. When you start getting into the granular levels of optimizing SQL queries you have to get your hands dirty at the Model level. This means writing new queries, and tweaking and tuning them until they run as fast as possible. In some circumstances, this sometimes means associating a Model, especially list models, with a database table other than what it was intended. In addition, the framework is not going to optimize your database structure for you. Once the database is built you will need to tune the indices and learn about vacuuming and clustering.

Second, frameworks consume memory as they sift data up from the database, through layers of business logic, actions, and finally, into the view layer. For basic web-based applications that serve up limited information on a page, this is not a big concern and can usually be overcome using memcache if it does become one. However, if your web-based application is churning through a lot of data and presenting it to the user in real time, you will hit memory limits. This can happen, for example, with reports that contain a lot of data over a large date range (probably why Basecamp limits report data to a given number of months). In this case you will get the best performance with the View layer accessing the database directly using cursors. Yeah, I know developers won’t like this, but there comes a time when providing speed to your customers is more important than the framework upon which it is built. Anyways, cutting out the middle man you remove most of the strain on memory and increase the speed of the reports.

The Normalized Database

When we design databases our primary goal is to reduce the redundancy of data through the use of multiple tables, foreign keys, and queries that rely on JOINs. Developers will nitpick over a normalized database until every last bit of redundancy is ironed out. This approach works great for most web-based applications but when traffic increases all of those carefully crafted tables and keys, along with the JOIN-heavy queries begin costing you milliseconds, then seconds, before the app becomes unusable.

The solution is to begin denormalizing data. We called this a “necessary redundancy” at Pelago. You begin by identifying the slowest queries and removing their JOINs by placed the JOINed data in multiple tables. Than it’s up to stored procedures at the database level or developers at the code base level to make sure the redundant data is always kept redundant. The entire database doesn’t have to be denormalized all at once. Just the tables requiring JOINs that are causing you performance issues.

One Database to Rule Them All

Another harsh reality of web development is that the database may become too large and unwieldy for handling the number of people using the web-based application. Sys admins will start throwing around the four-letter word “sharding” as developers begin to cringe. If your app starts growing large enough, sharding may become a necessity for the app to scale. Breaking up your database onto multiple servers and keeping each copy of the database in sync with the others is a laborious task and should be a last resort. However, to dismiss sharding altogether in favor of throwing hardware at the problem is shortsighted. If your web-based application is growing you should be thinking about how you would shard the database if it becomes necessary in the long term. It’s better to have a plan in place before it’s needed than to be scrambling at the last moment to relieve an overloaded web application.

In fact, all of the “rules” I mentioned above should be addressed by web developers at some point if they have plans on scaling their web-based applications. Meanwhile, let’s hear from other web developers out there. What are some of the “rules” you’ve had to break?

Reduce Swappiness

Most laptops have enough RAM installed that the swap space on the hard disk shouldn’t really be used. Yet the default setting for swappiness in Ubuntu is 60. By lowering it to 10 we can reduce the number of read/writes the the hard disk. This is especially handy for netbooks with solid state drives in them. To reduce the swappiness, follow these steps:

1. sudo sysctl -w vm.swappiness=10
2. Add the following line to /etc/sysctl.conf
   vm.swappiness=10

Preload

Preload is an adaptive read-ahead daemon that monitors running applications and analyzes them for commonalities. It then uses this data to predict what applications you might run and preloads them into memory. The result is faster startup times for commonly used programs. Installing preload is as easy. Just run the following line in a terminal window:

sudo apt-get install preload

» For further explanation, read this article on loading applications quicker in Ubuntu using preload.

Boot-Up Manager

The Boot-Up Manager (BUM) is a useful utility for starting and stopping boot-up scripts. Once installed it will appear in the System -> Administration menu. You can use BUM disable unnecessary boot-up scripts that may be slowing down your boot time. For example, I disabled saned, an API library for scanners, because I know I’ll never be attaching a flatbed scanner to my laptop. To install BUM:

sudo apt-get install bum

» Read the documentation for Boot-Up Manager

Startup Applications

If you are using Gnome for your desktop manager there will be an option to select which applications are started when you login. Go to System -> Preferences -> Startup Applications and uncheck the programs you don’t need. For example, I disabled the Evolution Alarm Notifier because I don’t use Evolution for setting alarms. I also disabled the Remote Desktop server, UME Desktop Launcher, and UNR Launcher. Just be careful to uncheck them and not remove them. In the case that you’ve removed something critical, you’ll want to be able to get it back.

Replace Metacity with Openbox

Openbox is a minimalistic and lightweight window manager that is known to run much faster than metacity, its bulkier counterpart. And you can easily get it working while still using Gnome as your window manager / desktop environment. By running Openbox inside the Gnome environment your desktop will become cleaner and faster. To install it, follow these steps:

1. Install openbox using:
   sudo apt-get install openbox obconf openbox-themes
2. Setup openbox as the default window manager by adding an entry in Startup Applications.
   To do this, go to System -> Preferences -> Startup Applications and enter the following:
   • Name: Openbox
   • Command: openbox ––replace

   Note: According to the Openbox documentation you can make it the default by choosing the GNOME/Openbox session when logging in and saving this as your default session. However, this functionality is broken in 9.04. The above steps are a workaround until this is fixed.

3. Choosing the Openbox theme and other configuration settings
   Go to System -> Preferences -> Openbox Configuration Manager to choose a theme you like and to update other settings such as Appearance and Windows.
   Note: changing the Desktops setting doesn’t effect the Gnome applet controlling the number of desktops. To change the number of desktops, you will need to revert back to Metacity and change them, then re-enable Openbox

» More information, documentation and screenshots available at the Openbox web site

That’s all for now

I found that making these few simple changes decreased load times on my older laptop significantly and made my newer netbook more minimal. Hopefully, these tips will help you as well.

Technical description.

Here is some code for getting the hostname value from your linux network configuration, assuming you have setup /etc/sysconfig/network properly.


//get hostname info from /etc/sysconfig/network
preg_match('/HOSTNAME=(.*)/', file_get_contents('/etc/sysconfig/network'), $network);
$hostname = split("\=", $network[0]);
echo $hostname[1]; //this equals the value of your HOSTNAME


For the Intervals API, we’re wrestling with issues surrounding data input validation. This recently became interesting when the matter of date validation came up. Ordinarily, Intervals allows many, many different date formats, dependent on the locale that the customer is using (for example, Intervals may expect the date format ‘mm/dd/yyyy’ for US customers, ‘dd.mm.yy’ for a customer in Austria).

For our API developers, we wanted to use a common, universal format, one that would be easily compatible with our application and database layers. For that we selected ISO 8601, which is great in terms of widespread use, but not so great in terms of how complicated its specifications are.

Generally, ISO 8601 looks something like ‘2009-05-20′ for dates and ‘2009-05-20 12:30:30′ for date/time combinations. These two examples encompass 98% of the user input we’re likely to encounter. But we wanted to make sure that if we told developers they could use ISO 8601 dates, our system would support it. Unfortunately, there’s not a lot of code out there for the validation of ISO 8601 dates (especially regular expressions), and most of the stuff that is out there doesn’t encompass the entirety of the ISO 8601 spec.

Starting off, here are some dates that the validator should match (all these are valid ISO 8601 dates to the best of my knowledge):

2009-12T12:34
2009
2009-05-19
2009-05-19
20090519
2009123
2009-05
2009-123
2009-222
2009-001
2009-W01-1
2009-W51-1
2009-W511
2009-W33
2009W511
2009-05-19
2009-05-19 00:00
2009-05-19 14
2009-05-19 14:31
2009-05-19 14:39:22
2009-05-19T14:39Z
2009-W21-2
2009-W21-2T01:22
2009-139
2009-05-19 14:39:22-06:00
2009-05-19 14:39:22+0600
2009-05-19 14:39:22-01
20090621T0545Z
2007-04-06T00:00
2007-04-05T24:00

Added Feb 19 2010:
2010-02-18T16:23:48.5
2010-02-18T16:23:48,444
2010-02-18T16:23:48,3-06:00
2010-02-18T16:23.4
2010-02-18T16:23,25
2010-02-18T16:23.33+0600
2010-02-18T16.23334444
2010-02-18T16,2283
2009-05-19 143922.500
2009-05-19 1439,55

And here are some of the strings that the validator should not match (ie. reject):

200905
2009367
2009-
2007-04-05T24:50
2009-000
2009-M511
2009M511
2009-05-19T14a39r
2009-05-19T14:3924
2009-0519
2009-05-1914:39
2009-05-19 14:
2009-05-19r14:39
2009-05-19 14a39a22
200912-01
2009-05-19 14:39:22+06a00

Added Feb 19 2010:
2009-05-19 146922.500
2010-02-18T16.5:23.35:48
2010-02-18T16:23.35:48
2010-02-18T16:23.35:48.45
2009-05-19 14.5.44
2010-02-18T16:23.33.600
2010-02-18T16,25:23:48,444

The code we came up with was the following:

Updated Feb 19 2010:
^([\+-]?\d{4}(?!\d{2}\b))((-?)((0[1-9]|1[0-2])(\3([12]\d|0[1-9]|3[01]))?|W([0-4]\d|5[0-2])(-?[1-7])?|(00[1-9]|0[1-9]\d|[12]\d{2}|3([0-5]\d|6[1-6])))([T\s]((([01]\d|2[0-3])((:?)[0-5]\d)?|24\:?00)([\.,]\d+(?!:))?)?(\17[0-5]\d([\.,]\d+)?)?([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?)?)?$

I guess I should add the caveat that this code doesn’t support the time interval or duration part of the ISO 8601 spec, so I didn’t include it. And it only supports dates or date/times, since right now we don’t have to deal with time input (for the Intervals API, all time is input in decimal format, rather than ISO 8601). But it should support everything else. Please let me know if this works for you or doesn’t, of if you can fine tune it.

• Background
• Configuring the Sonicwall Router
• Installing Openswan
• Configuring Openswan
• Starting and stopping the VPN connection
• References

Background

Pelago is a web design and development agency in Santa Barbara, California. Since our humble beginnings in August of 2000, we’ve seen the Internet landscape evolve exponentially in the last nine years. Our most current challenge as a creative and engineering agency is in embracing diversity in platforms and the inevitable shift towards the remote office. We use three different operating systems — Windows, OS X and Linux — on a daily basis. In addition to developing web-based project management software that is compatible with all three platforms, our designers and developers rely on them inside and outside of the office. Tunneling through the firewall from outside the office was our next requirement for embracing a diverse and distributed remote office.

Our SonicWall router makes it easy enough to establish a VPN connection using Windows. There is a client that can be downloaded from the SonicWall web site. What about other operating systems? Our developers often use Ubuntu Linux from home and required a way to VPN into the office. Once established, a VPN connection allows access to development servers, remote desktops, and other network resources inside the firewall; so you can see why it would be essential for the remote office.

The problem, as most Linux users out there already know, is that setting up a VPN connection in Ubuntu Linux is not very easy. After much trial and error, here is how we got the VPN working on Ubuntu Linux using Openswan.

Configuring the SonicWall Router

Login to your SonicWall router admin and make the following adjustments to the VPN settings.

1. Click on the VPN link and note the Unique Firewall Identifier for your SonicWall router. You will need it later for the value sonicwall.unique.identifier.
   
2. Under VPN Policies, create or edit the ‘GroupVPN’ policy.
3. Click on the General tab and set the following:
   
   • IPSec Keying Mode: IKE using Preshared Secret
   • Shared Secret: shared.secret.key (enter your secret key here)
4. Click on the Proposals tab and set the following:
   
   • IKE (Phase 1) Proposal
     DH Group: Group 5
     Encryption: 3DES
     Authentication: SHA1
   • Ipsec (Phase 2) Proposal
     Protocol: ESP
     Encryption: 3DES
     Authentication: SHA1
   • Enable Perfect Forward Secrecy (checked)
     DH Group: Group 5

Installing Openswan

If you are using Ubuntu, open a terminal window and type in:
sudo apt-get install openswan
The install will ask you a few questions about how you want to set it up. Select the suggested default for each step. This will install Openswan and create the ipsec.conf and ipsec.secrets configuration files.

Configuring Openswan

Add the following connection parameters to your /etc/ipsec.conf file:
conn sonicwall
     type=tunnel
     left=192.168.2.31 #your IP
     leftid=@home
     leftxauthclient=yes
     right=xxx.xxx.xxx.xxx #IP address of your sonicwall router
     rightsubnet=192.168.1.0/24 #gateway IP for your LAN. This will work for most
     rightxauthserver=yes
     rightid=@sonicwall.unique.identifier
     keyingtries=0
     pfs=yes
     aggrmode=yes
     auto=add
     auth=esp
     esp=3DES-SHA1
     ike=3DES-SHA1
     authby=secret
     #xauth=yes


Update: After upgrading to Ubuntu 9.10 a few things changed in the conf file. First, I had to comment out ‘xauth=yes’ as it was throwing an error. Second, I had to use my own static IP instead of %defaultroute. If you are using DHCP, %defaultroute may still work, but I have a static IP and had to change this.

Add the following line to your /etc/ipsec.secrets file
@home @sonicwall.unique.identifer : PSK "shared.secret.key"

Starting and stopping the VPN connection

Starting ipsec and opening the VPN connection
sudo ipsec setup ––start
sudo ipsec auto ––add sonicwall (note: if you change the configuration files, you'll need to run 'sudo ipsec auto ––replace sonicwall' to reload the file)
sudo ipsec whack ––name sonicwall ––initiate

Closing the VPN connection and stopping ipsec
sudo ipsec whack ––name sonicwall ––terminate
sudo ipsec setup ––stop


References

The following links were useful for getting the VPN connection up and running on Ubuntu Linux:
Openswan wiki page for SonicWall routers:
http://wiki.openswan.org/index.php/Openswan/SonicWall
SonicWall PDF instructions for using Agressive Mode and IKE with Pre-shared Keys
http://www.sonicwall.com/downloads/SonicOS_Enhanced_to_Openswan_Using_Aggressive_Mode_IKE_with_PreShared_key.pdf

Here is the view from our office.

http://www.popcuts.com/ – “Every time a song you bought sells, you get a cut of the proceeds.”

http://bountii.com/ – “Find a lower price on any of the products below and we’ll pay you the Bountii.”

IP address geolocation SQL database