The fact is, it’s rarely PHP that crashes a website. It’s usually overloading MySQL and/or bandwidth limitations. Obviously, we can’t do much about bandwidth. If you’re on a really cool host, they may be psyched for your Digging or Slashdotting, but most are not particularly keen on you using too much processor, too much bandwidth, or too many MySQL connections. Although MySQL is capable of amazing scaling, rarely do web hosts leverage it properly, and even if they did, rarely do most programmers know how to read into slow queries, find inefficient code like loops within loops, and debug a site performance before it’s too late.

The site linked from Digg shares a very lame trick, let’s examine:

//put this code at the very beginning of your page, before anything else
$randomdigg = rand(1,4);
if($randomdigg != 1) {
exit("Thanks for visiting, but the site is under a great deal of 
stress due to being on the front page of Digg. 
Please try back in a few moments. Thanks!");
}

Ok, so essentially, approximately 25% of the time, the page will work, the other 75% of the time, the user will get the message and be asked to refresh later. That’s really lame. First off, unless you want your website to be functional only 25% of the time, you have to be there babysitting it when it hits front page. Also, you need to remove this code. But what if you wanted everything to happen automagically? What if your site could withstand a digging, a slashdotting, a redditing, an OSNews’ing, a Farking, whatever…. all at once, without being forewarned?

Although there are lots of [[http://www.danga.com/memcached/|tools that can do magic for you]], you can leverage plain old PHP to do this dirty work for you, and it’s easy to integrate into your site.

Here’s the premise, we examine the referrer of the page, and if the page is from one or many domains that you specify, you build a cache ON REQUEST. Then, you serve the cached version. When the cache stales, say after minute or two, you refresh it, as needed, again, on request. And the best part is, it’s easy to do.

So, let’s review the code. First, at the top of the page, let’s do this:

$rfr = $_SERVER['HTTP_REFERER']; //get the referring site
$op = '';
if(strstr($rfr,'digg.com')) { $op="do-cache"; } 
$cache= "/path/to/mysite/cache/".$postid.".html";

Let’s take it line by line. First, we get the referring site. If it is digg.com, we’re going to turn the cache on by setting $op to “do-cache.” Note that we preset the $op variable to null to prevent abuse. Next, we define the file we’re going to use as the cache. Since we might want to do this for any entry/page, we’ll give it a unique address. Remember that you do not have to use .html as an extension, since it’s static code anyway. I’ve seen “id.cache”, “id.txt”, and even just “id”. It doesn’t matter, it will be included in a PHP page and spit out as X/HTML anyway.

if(file_exists($cache)) {
	$fmt = filemtime($cache); 
	if($fmt > time()-180) { 
		header("Location: http://mysite.com/cache/".$id.".html"); 
		exit(); 
	} elseif($fmt > time()-1800) { 
         //cache will reload if it's been reloaded 
         //in the last half hour
	   $op=="do-cache"; 
	} else {
	   $op='';
	}
}

This might seem overwhelming, but it’s really not. Check it out. If the cache exists, regardless of referrer, we’re going to check it. The filemtime() function will tell us when the cache was last refreshed. If it reports a time in the last 180 seconds (you can change that number), it will redirect you to the cache. PHP will barely have to do any work, Apache will barely have to do any work, and MySQL will not be touched at all. Almost any host can hang in there to serve static traffic like this.

But what about if the cache is old? Once your site is no longer getting hits every three minutes, for the next 27 minutes, it will blindly reload the cache. It will keep the cache going as long as the page has been hit in the last 30 minutes. After no one hits the cache for 30 minutes, it will go back to fully dynamic. When someone hits the page from digg again, it will start the caching process again. In this case, we’re going to let the cache rebuild. Enter the next part of the code.

if($op=="do-cache") {
	ob_clean(); 
	ob_start(); 
}

If the $op variable is set to “do-cache”, our mission is… uh… to build or rebuild the cache. So, we begin with two simple PHP standard functions, ob_clean() and ob_start(). The “ob” in these functions stands for “output buffering.” When you buffer your output, nothing is echoed by the script, but rather written to a buffer. In short, it’s all stored up in memory for eventual dumping later. Now we stop the caching scripts and just write our normal HTML via PHP and MySQL.

At the end of the page – the very end, after your html element is closed, you’ll need this last bit.

if($op=="do-cache") { 
	$buffer = ob_get_contents(); 
	$fp = @fopen($cache,'w'); 
	if($fp) { 
		fwrite($fp,$buffer); 
	}
	@fclose($fp); 
	ob_get_flush();  	
}

What’s up with this? Again, if we’re building the cache, we’re getting the contents of the buffer, which contains all of our page. Then we’re opening the cache file we specified above, and lastly, dumping the contents of the buffer into the cache file. Our final action is to clear the buffer, which is unnecessary to us now anyway.

That’s it. Your cache is reloaded, and will be served for the next three minutes, after which it will reload the cache every 3 minutes until no one hits the site for 30 minutes. Then it will go back to dynamic. If another digg visitor comes, it will recache and serve the cache. Having written several types of caching mechanisms for large sites myself, I can tell you this “cache on demand” method works. On OSNews v4, we use this for things like user-specific RSS files, which aren’t plausibly built via cron jobs and, frankly, we don’t want to allow access to MySQL without limit. They reload every 60 minutes, regardless of referer.

If you maintain a site and have any concern about being Dugg, I encourage you to research some sort of caching mechanism. Your web host will thank you, and you will be thankful for the maintained uptime and high hit count.

In colloquial use, a spoiler is revealing something either as yet unrevealed or any major plot twist. Today is Saturday and people are complaining that the headline contains a spoiler, that Charlie died. Forget for a moment that anyone who has seen Lost in the last few months knew this moment was coming – my question is, “is this actually a spoiler?”

By strictist definition, Darth Vader being revealed as Luke’s father is a spoiler, despite its presence as a pop culture reference. By loosest definition, spoilers are revealed every day.

First off, the submitter on Digg was quoting an article on E Online, which, for the record, shared the same title. Secondly, well over 1,000 people dugg it up – do they not share any blame. And thirdly, is this even a spoiler? This information was in several online articles the day after the Wednesday finale.

The fact is, nearly every major news outlet “spoiled” American Idol within a day of the finale, but I didn’t hear people complaining about that. And although I’ve heard people say “Digg isn’t just for the US, and other places are broadcast behind you,” ABC does share the full length episodes on their website and… well… this is the INTERNET! It’s the age of instantaneous information.

So what is an acceptable amount of time to wait before something is no longer a spoiler? I believe spoilers are real time only for TV, and pretty close to it for movies. It would’ve been a spoiler on Wednesday day or before, but once the episode airs, it’s no longer a spoiler. And if you don’t want to know, stay off the internet, certainly sites that will features reviews of a show that is very popular.

Movies are close to real time; it’s bad class to give away the twist to a movie, but how long until people generally know the twist? Is “The Sixth Sense” still fooling anyone?

Spoilers are only spoilers until the general public gets access. Then, I’m afraid, it’s every man for himself. You are responsible for navigating yourself away from the data you’re trying to avoid, because the world doesn’t owe it to you to not discuss something popular because you didn’t get a chance to watch it.

Read this comments on this inane, embarassing mess. Folks, this is proof that user generated content is not necessarily anything more than just volume. So is this, by the way.

Sites like Slashdot and OSNews are mostly valuable because their users contribute so much quality in the comments. Slashdot has been better since I moved my threshold to +3. OSNews is pretty good at 0. But most sites that make it too easy to participate are complete crap.

I think it is. It’s clear, concise, and well worded. It’s not moronic, it doesn’t rely on AOL-speak or l33t. It doesn’t play off the same techie cliches. It’s just good, old fashioned quality content. It makes a point and it makes a point well.

The quality of comments on digg.com are generally low, but every now and again you find a diamond in the rough.

We know that often times, once they make the front page, stories continue to receive diggs even if the site is unresponsive. Why? How can this be? This exposes a major flaw in the system – if the title and description are good enough, people will digg it, no matter what is on the other side. And that suggests we need two things.

First and foremost, we need bookmark functionality. Make my digg history almost worthless by allowing me to arbitrarily bookmark posts – seriously. This way, I can digg posts that I think should be in the front page without worrying that my history will be diluted. Maybe a bookmark (but not a digg) counts as a quarter-digg in the algorithm (or something like that). This way, I can bookmark a post to revisit it later without digging it now. I can choose to digg it later or not.

The idea of “digg” I think has always been to play the word “dig,” such as “Yeah, man, I really like this site. Ya dig?” How can you digg something you haven’t seen?

If bookmarks were implemented, posts wouldn’t necessarily continue to accumulate diggs even if the page goes down. Loyal readers suckered by a good headline, as I was yesterday when I kept going back to a posts entitled “Uber-Nerd Calls 911” which pointed to this clip from Reno 911, wouldn’t be forced to digg something just to remember to come back to it. The site went down mere seconds after being front paged, and then to rescue to the poor site, the video was temporarily removed and links to other videos were put in its place. Now, sure, there’s DuggMirror, but DuggMirror can be blocked by robots.txt, so it’s not a sure thing, and it requires me to remember or bookmark, rather than simply click a link on digg.com itself and have a pending list of links to revisit – my digg bookmarks.

This is very different from the recently introduced “My #1″ feature, which is mostly useless to me. This allows you to chose a story you’d like to feature, for some unknown reason, but only from the stories you’ve dugg. Furthermore, since I added something to “My #1″ the day it was rolled out mistakenly (due to a *Digg* coding error at that!), it’s stuck there, the first story in my myone section. I can’t remove it, even though I haven’t even dugg it.

Now, this could go the other way. While some may be more reserved in their digging once they can bookmark, it can be argued that adding bookmarks could potentially mean more diggs. Once you can segregate your important stories, a digg is easy to hand out with no limits. So the next step is that instead of the links going right to the site, the links should all be measured. I wouldn’t mind seeing a digg link point to http://digg.com/link/ae59f907b23 and then having my own personal “clicked link” history too.

On top of that, the number of actual clicks could factor into the algorithm. I’m betting that several people digg each front page story without ever having looked at it. Like all trends here, this is good and bad. It’s possible that I may feel as though this piece should be shared, particularly if it’s a headline like “Rumsfeld Quits!” that I know to be true whether I read the article or not. But for most stories, the number of clicks should matter. If the number of diggs paces the number of clicks, for most stories, this is suspicious. And I bet it happens a lot. This goes double for comedy videos or submitters who conjure great headlines.

Yes, I always check out the window.status onmouseover, but this could easily be replaced even if the link doesn’t point to the URL is redirects to. Perhaps this feature could even be toggled for the ultra paranoid who are freaked out about being monitored. I just assume that since HTTP is sniffable by anyone who uses TCP/IP and has access to routers, my click history – at least at digg.com – isn’t that sensitive.

The last big problem I see with Digg is the ever-growing stream of duplicates. Yesterday, someone submitted this genius piece. It was probably the 20th article about the Macbook Core 2 Duo release. Since it was a dupe, the author got around it by pointing to the site: http://www.apple.com/powerbook. Of course, since there is no Powerbook anymore, it redirects you to a meaningful page, but one that has already been submitted. This is silly – the author almost assuredly knew this was a duplicate story and purposely entered this URL to fool the digg engine. 2300+ diggs later, it paid off. Digg needs a better way of monitoring dupes. Wikipedia has an interesting practice – when a potentionally controversial entry is linked from an external source that drives large amounts of traffic in, the administrators lock down that entry for some period. Perhaps digg needs some site admins whose job is solely to monitor the site, a few smart operators to scan the Matrix. When there’s an announcement such as new Macbooks or Vista’s RTM, they look for dupes and simply blow em away. Sure, is is community run. But I’m sure the community agrees that softing through the same stupid announcment 20+ times makes digg.com/view/all/upcoming a real chore.

Digg is a great site, its only shortcoming is that various methods of using the tools provided to us make for results that are inconsistent. Furthermore, bad links can be, and often are, dugg up if the site is down but the title is good enough. A few changes to the core of the site would make the entire thing much more usable and would make the concept of a digg more valuable.

So, in summation:

1. Digg should add a bookmarking system independent from the digg system
2. Outgoing clicks should be measured
3. Hire community moderators to weed out duplicate posts

So I slammed the pedal to the metal and did it. I now have a working API. It currently supports only 1 method reliably – blogger2.newPost() – however, I have tested blogger.getRecentPosts and both corresponding methods under the metaWeblog API. That said, I will probably hack something up that explains how to add an API to your homegrown PHP weblog. There are functions out there that do the hardwork, and there are functions I’ve written that do the rest of the work, so you just need to write the correct queries for your database.

Anyway, I can now post from digg.com, which is cool. Ultimately, if we ever support user blogs on OSNews, we’ll roll out an API so people can comment on OSNews and Digg stories via the API. Could be interesting.

Gmail|Slashdot|OSNews|My Yahoo!|Work|Bloglines|Digg

I’ve added to the tabs in the last year or two: I added the work tab (email when at home, our helpdesk at work), then added Bloglines when I got into RSS aggreggation. Then Digg is new as of the middle of last year or so.

But I realized recently that I don’t really read Slashdot much anymore. For one, their news is way too slow, especially compared to Digg. The same COULD be said for OSNews, although we’re typically MUCH faster (faster even than Digg can promote to the front page for huge news), we feature many more originals, and the comments are readable, whereas the Slashdot arrogance and the volume of comments has gotten to be too much for me. What has been my #2 tab since about 2002 (I started reading Slashdot sometime in 2000) is now used the least of all. So, with the introduction of tab’s on Google’s customizable homepage, I pimped out my Google and made it tab #2. I’ve got my main tab, my Tech tab, my Sports tab, and my Entertainment tab rockin’. It’s pretty cool. So, from now on, I’ll get my Slashdot fill from google.com/ig.

I wonder if Slashdot is losing people, or if I’m just a single person moving on.