Breaking the “rules” of web development

I’ve been developing for the web long enough — since my foray into PERL and HTML in 1994 — to know that there will always be a certain “rules” developers will adhere to when banging out lines of code. They are usually good rules because it is important for developers to follow them. However, developers need to know when to break from convention, when a higher purpose requires them to sacrifice the brilliance and elegance of their code. I personally have a few favorite examples from my own years of experience and from working with other developers…

The Almighty Framework

Frameworks are great for web development. The Model-View-Controller (MVC) framework, in its many interpretations for languages such as PHP, Ruby and Python, has made building web sites so much faster. And MVC frameworks make for nice clean code and a logical separation of data, business logic, actions and content. The view layer in and of itself is a godsend and the MVC framework has finally provided recognition for all the hard work put in by templating engine developers (i.e. Smarty).

What happens when the framework becomes a performance barrier? While developing our online project management software, Intervals, we’ve come across two main areas where we’ve had to lift the hood and require some of the framework internals.

First, web-based applications can be complex enough that the SQL queries can become cumbersome if they are not fine tuned. Using the default SELECT and JOIN conventions provided by the framework is not always ideal. When you start getting into the granular levels of optimizing SQL queries you have to get your hands dirty at the Model level. This means writing new queries, and tweaking and tuning them until they run as fast as possible. In some circumstances, this sometimes means associating a Model, especially list models, with a database table other than what it was intended. In addition, the framework is not going to optimize your database structure for you. Once the database is built you will need to tune the indices and learn about vacuuming and clustering.

Second, frameworks consume memory as they sift data up from the database, through layers of business logic, actions, and finally, into the view layer. For basic web-based applications that serve up limited information on a page, this is not a big concern and can usually be overcome using memcache if it does become one. However, if your web-based application is churning through a lot of data and presenting it to the user in real time, you will hit memory limits. This can happen, for example, with reports that contain a lot of data over a large date range (probably why Basecamp limits report data to a given number of months). In this case you will get the best performance with the View layer accessing the database directly using cursors. Yeah, I know developers won’t like this, but there comes a time when providing speed to your customers is more important than the framework upon which it is built. Anyways, cutting out the middle man you remove most of the strain on memory and increase the speed of the reports.

The Normalized Database

When we design databases our primary goal is to reduce the redundancy of data through the use of multiple tables, foreign keys, and queries that rely on JOINs. Developers will nitpick over a normalized database until every last bit of redundancy is ironed out. This approach works great for most web-based applications but when traffic increases all of those carefully crafted tables and keys, along with the JOIN-heavy queries begin costing you milliseconds, then seconds, before the app becomes unusable.

The solution is to begin denormalizing data. We called this a “necessary redundancy” at Pelago. You begin by identifying the slowest queries and removing their JOINs by placed the JOINed data in multiple tables. Than it’s up to stored procedures at the database level or developers at the code base level to make sure the redundant data is always kept redundant. The entire database doesn’t have to be denormalized all at once. Just the tables requiring JOINs that are causing you performance issues.

One Database to Rule Them All

Another harsh reality of web development is that the database may become too large and unwieldy for handling the number of people using the web-based application. Sys admins will start throwing around the four-letter word “sharding” as developers begin to cringe. If your app starts growing large enough, sharding may become a necessity for the app to scale. Breaking up your database onto multiple servers and keeping each copy of the database in sync with the others is a laborious task and should be a last resort. However, to dismiss sharding altogether in favor of throwing hardware at the problem is shortsighted. If your web-based application is growing you should be thinking about how you would shard the database if it becomes necessary in the long term. It’s better to have a plan in place before it’s needed than to be scrambling at the last moment to relieve an overloaded web application.

In fact, all of the “rules” I mentioned above should be addressed by web developers at some point if they have plans on scaling their web-based applications. Meanwhile, let’s hear from other web developers out there. What are some of the “rules” you’ve had to break?

Tags: , , ,
Bookmark: Post to Del.icio.us Post to Digg Post to Google Post to Ma.gnolia Post to MyWeb Post to Newsvine Post to Reddit Post to Simpy Post to Slashdot Post to Technorati

11 Responses to “Breaking the “rules” of web development”

  1. kreativenews.com Says:

    Breaking the “rules” of web development…

    Developers need to know when to break from convention, when a higher purpose requires them to sacrifice the brilliance and elegance of their code….

  2. haaze.com Says:

    Breaking the “rules” of web development…

    Developers need to know when to break from convention, when a higher purpose requires them to sacrifice the brilliance and elegance of their code….

  3. Obiora Okwudili Says:

    Reviewing your blog, most of the tech speak is pretty heavyweight;

    PHP crons, linux, and the hostname
    Face detection in pure PHP (towards: web trends presence, communities and analytics?)
    Configuring Sonic Wall Routers
    consumer application of VPN connections?

    - I guess towards more discussion of your API for administration / customisation of intervals?

    - I’d be interested to see how this all integrates with your hosting requirements / how flow of TRAFFIC impacts on server loads – end performance concerns to the client etc.

  4. John Says:

    Obioria,

    Thanks for commenting. Yes, the Pelago blog is very tech-oriented and heavy on details. Perhaps you were looking for our Intervals blog? It is geared more towards our online time, task and project management software and its various attributes, such as the API.

    Check it out at:
    http://www.myintervals.com/blog/

    Meanwhile, to address your question about performance. Intervals runs pretty lean considering the comprehensiveness of its feature set. We are continually monitoring, tweaking and tuning the server and software to optimize performance. In fact, each Intervals release includes code to make the application run a little faster.

  5. programming Says:

    Kramer auto Pingback[...] hours ago by raymcbride2 commentssharecancelloading…27•••Breaking the "rules" of web development (pelagodesign.com)submitted 1 minute ago by jjriv1 [...]

  6. programming Says:

    Kramer auto Pingback[...] horas por raymcbride2 comentariossharecancelarloading…27•••Breaking the "rules" of web development (pelagodesign.com)enviado hace 2 minutos por jjriv1 [...]

  7. Breaking the “rules” of web development Says:

    Kramer auto Pingback[...] johnjreeve via pelagodesign.com Submitted: Jan 19 2010 / 19:09 Breaking the “rules” of web development Developers need to know when to break from convention, when a higher purpose requires them to [...]

  8. Php Mvc Framework - Blogs, Pictures, and more on Blogged Says:

    Kramer auto Pingback[...] Breaking the "rules" of web development Pelago  … . The Model-View-Controller (MVC) framework, in its many interpretations for languages such as PHP, Ruby and Python, has made building web sites so much faster. And MVC frameworks make for nice clean code and … list models, with a database table other than what it was intended. In addition, the framework is not going to optimize your database structure for you. Once the database is built you … related tags: design, engineering, mantra, pelago, quality [...]

  9. kreativenews.com - Sites Linking in - from Alexa Says:

    Kramer auto Pingback[...] http://www.pelagodesign.com/blog/2010/01/19/breaking-the-rules-of-web-development/ [...]

  10. What Qualifies as Project Management Software? | Project management and time tracking blog for web designers and small business :: the Intervals Blog by Pelago Says:

    Kramer auto Pingback[...] Web development rules and when to break them :: Pelago :: web design & development blog Says: October 2nd, 2010 at 7:38 am [...]

  11. Web-based Applications ~ Design and Development Tips | Project management and time tracking blog for web designers and small business :: the Intervals Blog by Pelago Says:

    Kramer auto Pingback[...] Another benefit of frameworks is that they enforce well established design patterns, which flows well with web designers and developers who appreciate clean and elegant code. Your customers, however, don’t really care what your code looks like. They want a web-based application that is responsive and reliable. There are times when we have to break the rules to make this happen. As a project manager, it is not fun having to twist a developers arm to get them to write faster, uglier code. But a project manager is also an advocate for the customer’s user experience. Web designers and developers need to understand the relationship between their work, customer satisfaction, and ultimately, company revenue. Additional resources for breaking the rules: Breaking the “rules” of web development. [...]

Leave a Reply