Graffiti

You’ve probably heard that web pages can do a lot more than they used to … but I haven’t really shown you that yet. Simple Javascript applications can be fun and useful, but it’s about time to move on to bigger, better things.

Graffiti is a program I wrote that is designed to be an online chat/forum that allows people to make posts which will be propagated in real-time to everyone who has the page open. It has a Javascript interface that you’re used to by now, a PHP middle layer, and a SQLite database. It communicates with PHP using JSON, which I’ll explain in a later post.

The Idea

Why would you want a realtime chat/forum? Well, what if you’re at work, and the AIM port is blocked? Or you want to have a chat with several people, but not all of them are on the same IM network, or one of them has an old version of the client software that doesn’t support group chat? What if you’re reading a forum, and would never know if there have been new posts made since the last time you loaded the page? In any of these cases, it would be nice to have a program that will get around those problems. Graffiti is one such program.

Adding a new post appends it to the bottom of your page and sends it off to the server. In a standard client-server situation, it would be the server’s responsibility to send that post to every connected user. However, HTTP is limited, in that the server cannot “push” data down to clients. The clients must “pull” it from the server. How can we simulate a “push” when we can only “pull”?

By polling for new content. Each client will sit in a loop and check repeatedly for changes to the state of the program. In the interest of not killing the server, we have the clients checking once per second for new content. If there is no new content, the server will say so. If there is new content, the server will send it. This way, everyone who has the page open will be looking at exactly the same page, regardless of who edits it and when.

The Data

The first thing we’ll do is design the database. There aren’t a lot of demands for our data, so it won’t be too much of a problem. I’m using SQLite as my database, but this is SQL 92 compliant code, so you should be able to use it (or very easily port it) to whatever database you prefer.

Our database will have 3 tables and 3 triggers. The first table is the “entries” table, which holds all the posts that are displayed on the page. It needs a field for the unique id, the author, the content of the post, and the time the post was made. The other two tables are the “instracker” and “deltracker” tables, which are to keep track of the posts that have been made and deleted over the course of the program running. These tables help keep everything sync’d up, by giving the client a place to look when they need to know if anything has changed since the last update.

create table deltracker (id integer primary key, entry integer);
create table entries (id INTEGER PRIMARY KEY, author VARCHAR(255), data TEXT, time DATE);
create table instracker (id integer primary key, entry integer);

To use this code in SQLite, just open up your SQLite command line interface, and type these lines in verbatim. Other databases will obviously have different ways of doing it, so you’re on your own there.

% /usr/local/bin/sqlite graffiti.db

Now let’s put our triggers in. We have one trigger that will put the date into the entries table, so that each new post has an exact date that goes along with it. The other two triggers update the instracker and deltracker tables whenever a post has been added or deleted.

CREATE TRIGGER delete_entries_tracker AFTER DELETE ON entries
BEGIN
        DELETE from instracker where (entry=old.rowid);
        INSERT into deltracker (entry) values (old.rowid);
END;
CREATE TRIGGER insert_entries_time AFTER INSERT ON entries
BEGIN
        UPDATE entries SET time = DATETIME('NOW') WHERE rowid = new.rowid;
END;
CREATE TRIGGER insert_entries_tracker AFTER INSERT ON entries
BEGIN
        INSERT into instracker (entry) values (new.rowid);
END;

You can add these triggers to your SQLite database from the command line.

% /usr/local/bin/sqlite graffiti.db < triggers.sql

Our Javascript code needs to support this database as well. We will define a class, called Entry, that corresponds to the data in the entries table.

function Entry() {
    this.type = "";
    this.id = "";
    this.author = "";
    this.data = "";
    this.time = "";
}

Note that there is a member in our class for each field in the table.

The Interface

Now that our data is all set up and ready, let’s define the interface. It’s not very complicated, but we will have to modify the page at runtime quite a bit.

The page itself doesn’t need much in the way of code. We simply have two div’s, one of which will hold the list of all the posts, and the other will be where the user can add new posts.

<div id="outer"></div>

<div id="add_div">
    <input type="submit" value="Add new" onclick="openAdd();" />
</div>

The CSS to style this program is equally simple. We just define classes for an entry, author, entrypanel, and the bottom. The entrypanel is where the delete button for each post will go, and the bottom is where the content of the post will be displayed.

.entry {
    position:relative;
    border:1px solid black;
}

.author {
    position:relative;
}

.entrypanel {
    position:absolute;
    top: 0px;
    right: 0px;
}

.bottom {
    position:relative;
}

The Functionality

Now that we have the basic interface scoped out a little, what do we want it to do? The button in the add_div will open the Add Post region of the page at the bottom, below all the posts. The user types his name and a message, and clicks the submit button. The post shows up immediately on his page, and is sent off to the server. When someone else posts a response, it will automatically show up at the bottom of the list. If he decides he doesn’t like what he wrote, he can hit the delete button on his post, and it will disappear from everyone’s page.

Communication between the application and the database is enabled by the XMLHttpRequest object. Originally developed by Microsoft and implemented by the other browsers, this allows the browser to request and receive data from a server while the page is loaded, and the browser can do something with said data without refreshing the page. XMLHttpRequest can work synchronously or asynchronously, which means that you can define whether a request should stop the application from running while it waits for data, or if it should continue on its business even if it is waiting for a response from the server. We’re going to use both in this program.

You’ve probably heard the hype already, and I won’t go over it any more. Now … have you actually used the XMLHttpRequest object? Naturally, Microsoft does it a bit differently from everyone else. We’ll use a function that detects whether the browser supports Microsoft’s version or the Mozilla version (Safari and Opera use the Mozilla version), and returns the proper object.

function xmlhttp() {
    var treq;
    if (window.XMLHttpRequest) {
        try {
            treq = new XMLHttpRequest();
        } catch(e) {
            treq = false;
        }
    } else if (window.ActiveXObject) {
        try {
            treq = new ActiveXObject("Microsoft.XMLHTTP");
        } catch(e) {
            treq = false;
        }
    }
    return treq;
}

You can do several things with the XMLHTTP object, but you only actually need a few of them. After you get an object from xmlhttp(), you’ll open it, set its header (needed on some browsers for some types of requests), and send a parameter string. When you open it, you define the action type (GET or POST), the URL to which to send the request, and whether the request should be processed asynchronously or not. If you’re using POST, you have to set the request header, otherwise, you don’t. If you’re using asynchronous, you have to define a callback function. Otherwise, you don’t.

Before I get too far into this and forget to set you up on PHP/SQLite, you need to create a PHP file that can talk to your database. For an application using SQLite and JSON, as we are, the top of your PHP file will look like this:

<?php
require_once('JSON.php');

$json = new JSON();
$file = "graffiti.db";

$db = new SQLiteDatabase($file) or die("Could not open database");

...

unset($db);
?>

This tells PHP to use the object-oriented version of its SQLite adaptor, which we will find very convenient when we need to transfer data. The way JSON works is that it packages up an object, and since our Entry class exactly mirrors our entries table, PHP will see the exact same class structure coming out of the database and the application.

The first thing we need to do when we load the page is load all the entries that already exist in the database. We want this to be a synchronous request, because the program should not run until all the posts have been successfully loaded and put onto the page.

function loadEntries() {
    var req = xmlhttp();
    var param = "load=1";

    //this is synchronous, because we want it to block until all the entries
    //are loaded.
    req.open("POST", "graffitisub.php", false);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);

    var res = req.responseText;
    entries = JSON.parse(res);
    for (var i=0; i < entries.length; i++) {
        buildEntry(entries[i]);
    }
}

In case you’re wondering what that “load=1” business is, you’re about to find out. That’s the parameter string for the request. If you were loading this in a browser window, that would follow the “?” in the URL. It is also known as a query string. In this case, we’re sending a message to our PHP page that tells it to return an array of Entry objects.

if (!empty($_POST['load'])) {
    $query = "select * from entries";
    $result = $db->query($query) or die("Error in loading");
    $arr = array();
    if ($result->numRows() > 0) {
        while ($obj = $result->fetchObject()) {
            $arr[] = $obj;
        }
        echo $json->encode($arr);
    }
    unset($arr);
}

This sends a query to the database, requesting everything from the entries table. It puts each row in the database into its own object, and puts all those objects into an array, which it returns to the application.

You may also have noticed that for each post that is in the database, it calls a function called “buildEntry”. What that function does is simple: it builds an entry and puts it onto the page. There are two ways to put something onto the page; one way is to use an element’s innerHTML property to simply place HTML into an object and have the browser render it. The other way is to build elements using the DOM’s createElement and appendChild functions. For the buildEntry function, we’ll be using the latter.

We create the div for the entry, then its author span, its entrypanel span, and its bottom div, populate all of them with the proper content, and put it into the page by appending it to the outer div (defined in the HTML).

function buildEntry(entry) {
    var outer = document.getElementById("outer");
    
    var div = document.createElement("DIV");
    div.id = "entrydiv_" + entry.id;
    div.setAttribute("class", "entry");
    
    var top = document.createElement("DIV");
    
    var a_span = document.createElement("SPAN");
    a_span.setAttribute("class", "author");
    var at = document.createTextNode(entry.author);
    var tt = document.createTextNode(entry.time);
    a_span.appendChild(at);
    a_span.appendChild(document.createElement("BR"));
    a_span.appendChild(tt);
    top.appendChild(a_span);
    
    var t_span = document.createElement("SPAN");
    t_span.setAttribute("class", "entrypanel");
    var del = document.createElement("input");
    del.type = "submit";
    del.value = "Delete";
    del.onclick = function() {
        delEntry(entry.id);
    }
    t_span.appendChild(del);
    top.appendChild(t_span);
    
    var bottom = document.createElement("DIV");
    bottom.setAttribute("class", "bottom");
    var d_span = document.createElement("SPAN");
    d_span.innerHTML = entry.data;
    bottom.appendChild(d_span);
    
    mostRecent = entry.id;
    
    div.appendChild(top);
    div.appendChild(bottom);
    outer.appendChild(div);
}

Now that we can load all the pre-existing entries, we need to be able to add our own. The single button isn’t really going to allow that, so we need a function that will expand the add_div div so that new posts can be entered. openAdd() uses the innerHTML property to build its HTML.

function openAdd() {
    var add_div = document.getElementById("add_div");
    var ih = "";
    ih += "<input type='submit' value='Cancel' onclick='closeAdd();' />";
    ih += "<input type='submit' value='Save' onclick='addEntry();' /><br />";
    ih += "Author: <input type='text' id='add_author' /><br />";
    ih += "<textarea id='add_data' cols='100' rows='15'></textarea>";
    add_div.innerHTML = ih;
}

If you can open it, you should be able to close it, so we have a corresponding closeAdd() function.

function closeAdd() {
    var add_div = document.getElementById("add_div");
    add_div.innerHTML = '<input type="submit" value="Add new" onclick="openAdd();" />';
}

As you see above, the openAdd function calls addEntry() when its button is clicked. We now need to write that function. We get the values of the author and data fields in the add_div, and send it off to the PHP page using XMLHttpRequest. For this we want to use asynchronous processing, so that the page doesn’t get slowed down when we click the Save button. This function also updates the global mostRecent variable, which is always the id number of the post most recently added to the page.

function addEntry() {
    var req = xmlhttp();
    var author_e = document.getElementById("add_author");
    var data_e = document.getElementById("add_data");
    
    var e = new Entry();
    e.author = author_e.value;
    e.data = data_e.value;
    var param = "insert=" + JSON.stringify(e);
    req.open("POST", "graffitisub.php", true);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);
    req.onreadystatechange = function() {
        if (req.readyState == 4) {
            if (req.status == 200) {
                var res = JSON.parse(req.responseText);
                
                buildEntry(res);
                data_e.value = "";
            }
        }
    }
}

Our PHP now has to be updated to support this. We build our query from the passed Entry object, ship it off to the database, and (this part is kind of tricky) send another request to the database to get the entire object that was just inserted. We return this object to the application. The reason we do this is because we want to get the time and id number of the post, which are only known after it has been inserted. Notice above that the addEntry function is designed for this.

if (!empty($_POST['insert'])) {
    $input = $_POST['insert'];
    $value = $json->decode($input);
    $insQuery = "insert into entries (author,data) values (\"".$value->author."\", \"".$value->data."\")";
    $insResult = $db->query($insQuery) or die("Error in query");
    $query = "select * from entries where (id=".$db->lastInsertRowId().")";
    $res = $db->query($query) or die("Error in query 2");
    if ($res->numRows() > 0) {
        $obj = $res->fetchObject();
        $value->id = $obj->id;
        $value->time = $obj->time;
        echo $json->encode($value);
    }
}

Now that we can add posts, we need to be able to delete them. We wouldn’t want our page to get cluttered up with useless crap (especially when we’re testing it out, right?). The delete button on each post calls the delEntry function. This sends a request to delete an entry to the PHP. We use asynchronous processing here because we don’t want to block the rest of the program, but it doesn’t need a callback because nothing is returned from a delete request.

function delEntry(id) {
    var req = xmlhttp();
    var param = "delete=" + id;
    
    req.open("POST", "graffitisub.php", true);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);
    //don't need to set a callback, because nothing needs to be done after
    //the post has been removed.
    
    removeEntry(id);
}

The PHP code to handle a deletion is probably the simplest thing you’ll have to do. It just takes the id number you sent it from the application and deletes it from the database.

if (!empty($_POST['delete'])) {
    $delQuery = "delete from entries where (id=".$_POST['delete'].")";
    $delResult = $db->query($delQuery) or die("Error in deletion");
}

Once the post has been deleted from the database, we have to remove it from the page. We do this by calling the removeEntry function. This updates the global deletions array (which is supposed to match the deltracker table) and removes the post from the page using the DOM removeChild function.

function removeEntry(id) {
    deletions[deletions.length] = id;
    var outer = document.getElementById("outer");
    var div = document.getElementById("entrydiv_"+id);
    if (div) {
        outer.removeChild(div);
    }
}

Keeping the different clients sync’d with the database and each other is the hardest part of this application. We’ll do it with a check to the database once per second for each client (the timer uses a global timerID variable). We send the most recently added entry id to the database, and it responds with a carefully crafted array. There are always two items in this array. The first is an array of new entries added after the one most recently added by this client (this will often be empty). The second is another array of entry id numbers that have been deleted. This mirrors the deltracker function.

function checkForEntries() {
    var req = xmlhttp();
    
    if (timerID) {
        clearTimeout(timerID);
    }
    
    var param = "last=" + mostRecent;
    req.open("POST", "graffitisub.php", false);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);
    if (req.responseText) {
        var res = JSON.parse(req.responseText);
        for (var i=0; i < res[0].length; i++) {
            buildEntry(res[0][i]);
        }
        mergeDeletions(res[1]);
    }
    
    timerID = setTimeout("checkForEntries()", 1000);
}

The PHP for this is fairly complicated. It has to create the three arrays I mentioned above, check the instracker table for new entries, get all their id numbers, get those posts from the entries table, and put them into their positions in the arrays. It then needs to return the deltracker array.

if (!empty($_POST['last'])) {
    $last = $_POST['last'];
    $totalArr = array();
    $insArr = array();
    $delArr = array();
    
    $trackQuery = "select * from instracker where (entry=".$last.")";
    $trackResult = $db->query($trackQuery) or die ("Error in tracker");
    if ($trackResult->numRows() > 0) {
        $track = $trackResult->fetchObject();
        $tq2 = "select * from instracker where id>".$track->id;
        $tr2 = $db->query($tq2) or die ("Error in tracker");
        if ($tr2->numRows() > 0) {
            while ($obj = $tr2->fetchObject()) {
                $eq = "select * from entries where id=".$obj->entry;
                $er = $db->query($eq) or die ("Error in tracker");
                if ($er->numRows() > 0) {
                    $o2 = $er->fetchObject();
                    $insArr[] = $o2;
                }
            }
        }
    }
    
    $dq = "select * from deltracker";
    $dr = $db->query($dq) or die("Error in deltracker");
    if ($dr->numRows() > 0) {
        while ($obj = $dr->fetchObject()) {
            $delArr[] = $obj->entry;
        }
    }
    
    $totalArr[] = $insArr;
    $totalArr[] = $delArr;
    echo $json->encode($totalArr);

    unset($totalArr);
    unset($insArr);
    unset($delArr);
}

When it gets the data back from the database, the program now has to make sure everything is sync’d. It loops through the new entries and builds each one. It also needs to take the deltracker array (the second item in the returned array) and merge it with the deletions array (global). The mergeDeletions function simply loops through the given array, starting at the index that would be past the end of the deletions array, and removes the entries specified.

function mergeDeletions(del) {
    for (var j = deletions.length; j < del.length; j++) {
        removeEntry(del[j]);
    }
}

We’re almost done now. The only thing left is to write the init function, which will run when we load the page. It needs to initialize the global deletions array, load the existing posts, and then start checking for new ones.

function init() {
    deletions = new Array();
    loadEntries();
    checkForEntries();
}

I mentioned the three global variables we needed as we went along. They are defined here:

//global variable for the timer
var timerID;

//global variable to track the most recent received id
var mostRecent;

//global deletions array, to keep track of all the deletions made
var deletions;

Recap

If you’ve followed along and built this program, it should now be working for you. Clearly, using XMLHttpRequest and JSON, you can do things with web pages that weren’t possible at all just a few short years ago. This Graffiti program is one such page. Previously, if you wanted to do something like this, you would have had to write a Java applet or a Flash program, both of which require clunky plugins (which I personally hate). Now you can do it with Javascript, which ends up being much nicer for the user.

Beyond the application itself, what this tutorial should have shown you is a kind of development process. The process goes from the IDEA phase, where you sit and dream up what you want your program to do, to the DATA phase, where you define your data structures (including the database), to the INTERFACE phase, where you write your HTML and CSS and decide how the program is going to look, to the FUNCTIONALITY phase, where you write your Javascript (and PHP) and give your program the desired behavior.

To Do

• Make it look nicer. It’s either a forum or a chat, each of which have their own display requirements.
• New posts come in at the bottom of the page. This isn’t always what you want. So update the program so it prepends the new posts (puts them at the top). And make it an option!
• Think about users … what will have to be done to this program to create a login system so that only the author of a post can delete it?
• Dynamically change the timeout interval based on usage to maximize efficiency.