If you’re interested in some revealing behind-the-scenes info, feel free to ask questions now.

I don’t believe anyone out there is still using Flip, but if there is, this is how to defeat the script: simple add this line at line 102 of account.php:

 if(strstr($_POST['em'],"][")) { die('Fail'); }

and this at line 162:

 if(strstr($_POST['nem'],"][")) { die('Fail'); }

Alternatively, you can download the modified file here or download Flip 3.0.1 here.

It may sound odd, but I would highly recommend that you do *not* use this code. It’s now 7 years old and the web is a much different place. The code here is really not suited for running a website today. That said, it was odd to unzip and install it and see that it actually works. The rendering of most of the “themes” is weird (Fudge works great), but otherwise, everything worked.

If you are still a Flip user, I recommend you update your account.php page immediately, and if you have the time and inclination, upgrade to 3.0.1. The following files have some minor changes:

• account.php
• index.php
• inc/config.php
• README.html

Once again, this code is aged not particularly well suited for today’s web.  If you want a simple weblog, I recommend WordPress.

I got the first few steps knocked out, I selected WordPress,  gave it my MySQL username and password, and let it go.  It installed PHP for Windows, the MySQL connector, and WordPress.  Then I launched my browser and pointed to http://localhost:81 and… no.  Error 402.  I monkeyed with the site in IIS and was able to generate an error that simply says:  Parameter not found.

PHP is installed.  IIS assicates .php files with PHP.exe.  But Wordpress no worky.

Fail.

Directions are this easy: set the path of $tw_File with a static, writable file.  Set $tw_userid to your Twitter user id.  Done. 

Download firsttube.com “get latest tweet” php snippet.

First, snag your Twitter user id.  Then, open up your theme file.  I put mine in sidebar.php found in /wp-content/themes/<THEMENAME>/.    Use the below code.  If you want the output wrapped in a list, you would need to put <ul> and <li> tags around this code.

Carefully set your variables.  The cache file should be writable.  Note that you can use a decimal value for $tw_BlankAfter and $tw_Minutes if necessary.   That’s it.

Due to what must be a bug in Wordpress, please ignore the closing “</text></created_at>” at the end of this post.  It’s trying be smart and “fix” broken tags, but the code is right.

NOTE (2/20/09): I have updated the below code.  The new version can be found at “Posting Your Latest Tweet in Wordpress“.

/* ~~~~ Custom Twitter Bit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ */
/* ~~~~ Adam S, firsttube.com, twitter @sethadam1 ~~~~~~~~~~~~~~~~~~~~ */
 
$tw_File = '/path/to/a/static/writable/file/twitter.html';
$tw_Userid='XXXXXXX'; //set to your Twitter user id
$tw_BlankAfter = 30; //blank out status if it's older than this many days
$tw_Minutes = 10; //minutes between reloads
 
$tw_Offset = FALSE; //leave as is
// uncomment below time if you want to allow a manual reset via ?twitter-reset
// if($_SERVER['QUERY_STRING']=='twitter-reset') { $tw_Offset=0; } 
 
/* Do not edit below this line */
if(filemtime($tw_File)>time()-floatval($tw_Offset)) {
	include $tw_File;
} else {
	if(is_writable($tw_File)) { $tw_iswritable=1; }
	$tw_time = (86400*floatval($tw_BlankAfter));
	if($tw_Offset) { $tw_time=$tw_Offset; }
	$tw_hyperlinks = true;
	$tw_c = curl_init();
	curl_setopt($tw_c, CURLOPT_URL,
		"http://twitter.com/statuses/user_timeline/"
		.intval($tw_Userid).".xml");
	curl_setopt($tw_c, CURLOPT_RETURNTRANSFER, 1);
	$tw_src = curl_exec($tw_c);
	curl_close($tw_c);
	preg_match('/(.*)< \/created_at>/', $tw_src, $tw_d);
	if(strtotime($tw_d[1]) > time()-$tw_time) {
		preg_match('/(.*)< \/text>/', $tw_src, $tw_m);
		$tw_status = htmlentities(str_replace("&","&",$tw_m[1]));
		if( $tw_hyperlinks ) {
			$tw_status = ereg_replace(
			"[[:alpha:]]+://[^<>[:space:]]+[[:alnum:]/]",
			"<a href="\">\\0</a>",
			$tw_status);
		}
		$tw_output = $tw_status;
	} else {
		if($tw_iswritable==1) {file_put_contents($tw_File,''); }
	} 
 
	if($tw_iswritable==1) { file_put_contents($tw_File,$tw_output); }
	echo $tw_output;
}
/* ~~~ /Custom Twitter Bit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ */

Please note that portions of this code come from the twtter_status() function that was not written by me, but is available from various sources online.

Update: Removed function and put code inline.

//an array of some songs I like
$songs =  array(
		'1' => array('artist'=>'The Smashing Pumpkins', 'songname'=>'Soma'),
		'2' => array('artist'=>'The Decemberists', 'songname'=>'The Island'),
		'3' => array('artist'=>'Fleetwood Mac', 'songname' =>'Second-hand News')
	);

The problem is thus: I’d like to echo out the songs I like in the format “Songname (Artist),” and I’d like to do it alphabetically by artist. PHP provides many functions for sorting arrays, but none will work here. ksort() will allow me to sort by key, but the keys in the $songs array are irrelevant. asort() allows me to sort and preserves keys, but it will sort $songs by the value of each element, which is also useless, since the value of each is “array()”. usort() is another possible candidate and can do multi-dimensional sorting, but it involves building a callback function and is often pretty long-winded. Even the examples in the PHP docs references specific keys.

So I developed a quick function to sort by the value of a key in a sub-array. Please note this version does a case-insensitive sort. See subval_sort() below.

function subval_sort($a,$subkey) {
	foreach($a as $k=>$v) {
		$b[$k] = strtolower($v[$subkey]);
	}
	asort($b);
	foreach($b as $key=>$val) {
		$c[] = $a[$key];
	}
	return $c;
}

To use it on the above, I would simply type:

$songs = subval_sort($songs,'artist'); 
print_r($songs);

This is what you should expect see:

Array
(
    [0] => Array
        (
            [artist] => Fleetwood Mac
            [song] => Second-hand News
        )
 
    [1] => Array
        (
            [artist] => The Decemberists
            [song] => The Island
        )
 
    [2] => Array
        (
            [artist] => The Smashing Pumpkins
            [song] => Cherub Rock
        )
 
)

The songs, sorted by artist.

Onto my gripes. I can’t get inline replying/threading to work.  There is very little documentation on it so far. The functions are called comment_reply_link() and get_comment_reply_link(), and there’s nothing anywhere in the codex that helps, there’s little on the internet, the only place to get any real detail is the code itself, which explains:

from wp-includes/comment-template.php starting at line 949 on WP 2.7.0

 * Retrieve HTML content for reply to comment link.
 *
 * The default arguments that can be override are 'add_below', 'respond_id',
 * 'reply_text', 'login_text', and 'depth'. The 'login_text' argument will be
 * used, if the user must log in or register first before posting a comment. The
 * 'reply_text' will be used, if they can post a reply. The 'add_below' and
 * 'respond_id' arguments are for the JavaScript moveAddCommentForm() function
 * parameters.
 *
 * @since 2.7.0
 *
 * @param array $args Optional. Override default options.
 * @param int $comment Optional. Comment being replied to.
 * @param int $post Optional. Post that the comment is going to be displayed on.
 * @return string|bool|null Link to show comment form, if successful. False, if comments are closed.

It doesn’t matter much, because it doesn’t work, period, even though I’ve followed the instruction here to a t. So I’ll have to fix that in time.

My last gripe is with the new wp_list_comments() routine. I understand this is all new, but the idea that templating comments requires a callback function as a wrapper to all comments, pings, and trackbacks is clumsy at best. The codex on wp_list_comments() have nothing to explain it to people, so while I’ve dug in and gotten things working, it’s not for the feint of heart just yet, since you need to build a PHP function in your theme in your functions.php file (or create one if it doesn’t exist, which cannot be done via the Dashboard). I’m a little sad, since the theme system is so flexible and the new plugin system is just incredible, to see the new comment loop be so manual compared to the single file approach used so successfully in the past.

I know that Scoble says Wordpress 2.7 rocks, and it does. Scoble doesn’t realize the shortcomings because he hasn’t tried to play with the new features, and fortunately, it very gracefully degrades. But it’s got some work to do to be perfect, for me at least.

Themes were entirely unbroken. Upgrading firsttube.com may be a bit more of a challenge since I’ve manually changed a few fore WordPress files, which may prevent in place automatic upgrades.  However, all in all, I think the 2.7 release is looking really great.  

When 2.7 final is released, I expect to be updating my live site pretty quickly.

I wrote an application some time ago for my company that looks up the longitude and latitude of an address for use in our geocoding initiative. It relied on yahoo_geo(), a function written by PHP creator Rasmus Lerdorf and the Yahoo Maps API. It was largely dependent on this function:

function yahoo_geo($location) {
	$q = 'http://api.local.yahoo.com/MapsService/V1/geocode?appid=rlerdorf&location='
		.rawurlencode($location);
	$tmp = '/tmp/yws_geo_'.md5($q);
	request_cache($q, $tmp, 43200);
	libxml_use_internal_errors(true);
	$xml = simplexml_load_file($tmp);
	$ret['precision'] = (string)$xml->Result['precision'];
	foreach($xml->Result->children() as $key=>$val) {
		if(strlen($val)) $ret[(string)$key] = (string)$val;
	}
	return $ret;
}

This function worked for over two years for us with no problems at all. Then suddenly, in the last month, it started getting spotty. I fixed things by commenting out the caching parts of the function and forcing each execution to run again. Then I got errors about the libxml_use_internal_errors() function, so I commented that out. But today, the function just flat out failed, every single time returning the same error:

Warning: file_get_contents(http://XXXXXXXXXX/XXX) [function.file-get-contents]: failed to open stream: HTTP request failed! in /home/intranet/html/fetch.php on line X

What the heck? This code is all over the web. I’ve tried a million permutations of this function, including using fopen() and ob_get_contents(), and none have worked. And most frustratingly, I could load the URL successfully in Lynx and eLinks, so the machine could quickly and easily fetch the URL.

So I ventured into a sandbox I’ve never really played before: cURL. cURL is an interesting animal. But the interesting thing is, once I got it working, it worked faster than ever! So, without further ado, here is the new and improved yahoo_geo() function:

function yahoo_geo($location) {
	$q = 'http://api.local.yahoo.com/MapsService/V1/geocode?appid=rlerdorf&location='.urlencode(trim($location));
	$ch = curl_init($q);
	curl_setopt($ch, CURLOPT_HEADER, 0);
	ob_start();
	curl_exec($ch);
	$stream = ob_get_contents();
	ob_end_clean();
	if($stream) {
		$xml = simplexml_load_string($stream);
		$ret['precision'] = (string)$xml->Result['precision'];
		if($xml) {
			foreach($xml->Result->children() as $key=>$val) {
				if(strlen($val)) $ret[(string)$key] =  (string)$val;
			}
		}
		curl_close($ch);
		return $ret;
	} else {
		return FALSE;
	}
}

Note: If you’re reproducing these functions elsewhere, be careful – WordPress may have converted the quotes into smart quotes that will need to be fixed before this script will run properly.

The interesting thing here is that I’ve browsed the source of Slash, Scoop, phpNuke, and now WordPress, and all of them are definitively more complex and much heavier than the entire OSNews codebase. Now, before you jump all over me – firstly, Slash and Scoop are Perl, and I don’t really read Perl, so I can’t speak as an expert there.  Secondly, WordPress and Nuke both are very portable and dynamic, whereas OSNews has a narrow focus and, location-wise, is very static.  But that aside, OSNews has withstood simultaneous link bombs from Slashdot and Digg.  As amazing as WordPress is, it’s mostly amazing that it functions at all and loads in less than 2 minutes per page with as much going on as I can see behind the scenes.   That’s not a cut on WordPress, by the way.

In fact, if anything , what is really impressed upon me is how smooth and simple OSNews code is, if I may be so bold.  OSNews runs superfast due, in part, to lots of creative caching, some on-demand, some via cron.  But it also does so because of highly efficient queries that are measured for speed on their JOINs, meaning in some cases, it’s faster to do 20 simple queries than one complex one, or build a long and scary chain of “OR x=a OR x=b OR x=c OR x=d…”  Watching WordPress code in action is really fun for me, but watching OSNews work knowing what I now know about how much work PHP can cram into its threads is even more fun.

Why? Let’s argue for a second that everything people say about PHP is true, as many of the complaints are sound.

It’s true the primary namespace has way too many functions – over three thousand, I’m told. It’s true that the function names are inconsistent, some have underscores, some don’t. It’s true that the function names are often verbose. It’s true that OOP was weak until recently, it’s true that register_globals was a security nightmare. All those things are potential issues, and all languages have them. As the “real programmers” who write Perl would never admit, reading other people’s terse Perl is often a f’ing disaster, even for seasoned Perl-ites. And when using compiled ASP.net – for best performance, natch – you must update your entire site (well, all the concerned ASPX pages and DLLs) to make elementary changes.

That said, PHP is easy. Really easy. And it’s a trivial task to get a website up and running fairly quickly. And you can serve enormous amounts of traffic as proven not only by OSNews (who have been dugg and Slashdotted concurrently), but by Yahoo!, Wikipedia, Flickr, Facebook, and many, many others. And why are there so many open source PHP frameworks, apps, CMSes, etc? Because PHP is installable virtually everywhere, it’s very portable, and it’s really simple to hack up. Try installing something dependent on mod_perl (e.g. Slash or Scoop) and get back to me on the ease of the install.

The fact is, even if everyone’s fears about writing insecure code is true, the ability to make mistakes does not mean everyone does, and those who would forsake “the right tool for the job at hand” shouldn’t be trusted even to water your plants, because they are obviously nitwits. If you can’t concede that PHP can be the right tool some of the time for some situations, you shouldn’t be trusted to code or make adult decisions. No, I argue that the reason they dislike PHP is because many start with PHP and thus, admitting to liking it would make them appear to be a “noob.” It’s because they must appear to be seasoned pros. It’s the bragging rights on the 21st century.

Nobody has ever claimed PHP is the solution to everything, but it is a remarkably easy tool for scripting dynamically generated HTML. And, in my opinion and experience, it does so better than Perl, better than Ruby, and a hell of a lot better than both ASP.net and JSP.

I’ll spare you the details: I got hacked. Someone either brute forced their way into the admin site (which is now pretty locked down, until I figure this all out) or brute forced into SSH and uploaded several malicious PHP scripts. They are scary, I actually have them intact in a backup from a few days ago. How much has been revealed? My MySQL passwords? It’s impossible to tell. Virtually everything will need scrubbing.

In the meantime, excuse any wonkiness until all is repaired. The good news is this finally forces me to finish work on the new administrative area I’ve been playing with.

The fact is, it’s rarely PHP that crashes a website. It’s usually overloading MySQL and/or bandwidth limitations. Obviously, we can’t do much about bandwidth. If you’re on a really cool host, they may be psyched for your Digging or Slashdotting, but most are not particularly keen on you using too much processor, too much bandwidth, or too many MySQL connections. Although MySQL is capable of amazing scaling, rarely do web hosts leverage it properly, and even if they did, rarely do most programmers know how to read into slow queries, find inefficient code like loops within loops, and debug a site performance before it’s too late.

The site linked from Digg shares a very lame trick, let’s examine:

//put this code at the very beginning of your page, before anything else
$randomdigg = rand(1,4);
if($randomdigg != 1) {
exit("Thanks for visiting, but the site is under a great deal of 
stress due to being on the front page of Digg. 
Please try back in a few moments. Thanks!");
}

Ok, so essentially, approximately 25% of the time, the page will work, the other 75% of the time, the user will get the message and be asked to refresh later. That’s really lame. First off, unless you want your website to be functional only 25% of the time, you have to be there babysitting it when it hits front page. Also, you need to remove this code. But what if you wanted everything to happen automagically? What if your site could withstand a digging, a slashdotting, a redditing, an OSNews’ing, a Farking, whatever…. all at once, without being forewarned?

Although there are lots of [[http://www.danga.com/memcached/|tools that can do magic for you]], you can leverage plain old PHP to do this dirty work for you, and it’s easy to integrate into your site.

Here’s the premise, we examine the referrer of the page, and if the page is from one or many domains that you specify, you build a cache ON REQUEST. Then, you serve the cached version. When the cache stales, say after minute or two, you refresh it, as needed, again, on request. And the best part is, it’s easy to do.

So, let’s review the code. First, at the top of the page, let’s do this:

$rfr = $_SERVER['HTTP_REFERER']; //get the referring site
$op = '';
if(strstr($rfr,'digg.com')) { $op="do-cache"; } 
$cache= "/path/to/mysite/cache/".$postid.".html";

Let’s take it line by line. First, we get the referring site. If it is digg.com, we’re going to turn the cache on by setting $op to “do-cache.” Note that we preset the $op variable to null to prevent abuse. Next, we define the file we’re going to use as the cache. Since we might want to do this for any entry/page, we’ll give it a unique address. Remember that you do not have to use .html as an extension, since it’s static code anyway. I’ve seen “id.cache”, “id.txt”, and even just “id”. It doesn’t matter, it will be included in a PHP page and spit out as X/HTML anyway.

if(file_exists($cache)) {
	$fmt = filemtime($cache); 
	if($fmt > time()-180) { 
		header("Location: http://mysite.com/cache/".$id.".html"); 
		exit(); 
	} elseif($fmt > time()-1800) { 
         //cache will reload if it's been reloaded 
         //in the last half hour
	   $op=="do-cache"; 
	} else {
	   $op='';
	}
}

This might seem overwhelming, but it’s really not. Check it out. If the cache exists, regardless of referrer, we’re going to check it. The filemtime() function will tell us when the cache was last refreshed. If it reports a time in the last 180 seconds (you can change that number), it will redirect you to the cache. PHP will barely have to do any work, Apache will barely have to do any work, and MySQL will not be touched at all. Almost any host can hang in there to serve static traffic like this.

But what about if the cache is old? Once your site is no longer getting hits every three minutes, for the next 27 minutes, it will blindly reload the cache. It will keep the cache going as long as the page has been hit in the last 30 minutes. After no one hits the cache for 30 minutes, it will go back to fully dynamic. When someone hits the page from digg again, it will start the caching process again. In this case, we’re going to let the cache rebuild. Enter the next part of the code.

if($op=="do-cache") {
	ob_clean(); 
	ob_start(); 
}

If the $op variable is set to “do-cache”, our mission is… uh… to build or rebuild the cache. So, we begin with two simple PHP standard functions, ob_clean() and ob_start(). The “ob” in these functions stands for “output buffering.” When you buffer your output, nothing is echoed by the script, but rather written to a buffer. In short, it’s all stored up in memory for eventual dumping later. Now we stop the caching scripts and just write our normal HTML via PHP and MySQL.

At the end of the page – the very end, after your html element is closed, you’ll need this last bit.

if($op=="do-cache") { 
	$buffer = ob_get_contents(); 
	$fp = @fopen($cache,'w'); 
	if($fp) { 
		fwrite($fp,$buffer); 
	}
	@fclose($fp); 
	ob_get_flush();  	
}

What’s up with this? Again, if we’re building the cache, we’re getting the contents of the buffer, which contains all of our page. Then we’re opening the cache file we specified above, and lastly, dumping the contents of the buffer into the cache file. Our final action is to clear the buffer, which is unnecessary to us now anyway.

That’s it. Your cache is reloaded, and will be served for the next three minutes, after which it will reload the cache every 3 minutes until no one hits the site for 30 minutes. Then it will go back to dynamic. If another digg visitor comes, it will recache and serve the cache. Having written several types of caching mechanisms for large sites myself, I can tell you this “cache on demand” method works. On OSNews v4, we use this for things like user-specific RSS files, which aren’t plausibly built via cron jobs and, frankly, we don’t want to allow access to MySQL without limit. They reload every 60 minutes, regardless of referer.

If you maintain a site and have any concern about being Dugg, I encourage you to research some sort of caching mechanism. Your web host will thank you, and you will be thankful for the maintained uptime and high hit count.

So, I got a phone call from the director of our customer contact department this week. He was concerned about the reports. He made a decision last week that when a call came in that was a lead – in other words, a non-customer, that his people would fill the phone number from the caller ID into the account number field. But when he ran his export reports, he found that hisn techs had entered this phone number for ALL of the calls: 429-496-7295. That’s weird, he said. So he called me and asked why that was. I checked all the calls and most were from one woman, so my first instinct was “Check if her browser has autocomplete turned on”. But he swore that he tried it too and gotten the same result.

I checked the database and sure enough, it was right there: 429-496-7295, in all of the fields. So I went back to the code. In short, I took the input from the form, and declared it like this:

$accountnum = (int) $_POST['accountnum'];

Pretty straightforward: explicitly declare the type. So, I started my debugging by attempting to manually enter the data into the database. Sure enough: the account key field showed this: 4294967295.

So, I went back to the PHP and started by dumping out the raw SQL query:

INSERT INTO calls ('','x','x','x','4294967295','x','x');

What? So the database automatically converts it to this weird phone number and PHP does too? Suddenly it occured to me. One of the benefits of 64-bit computing is the ability to address more memory. There are limits to what can be done in 32-bit computing, and one is that integers have a limit! In this case, a database field called “integer” is limited to numbers between -2,147,483,648 and +2,147,483,647. It just so happens that the number is the same length as a US phone number – 10 digits. Changing the db field to “BIGINT” allowed me to manually run the SQL query and it worked. But the app still didn’t.

PHP’s int() and (int) $var syntaxes both conform to the integer limit. So I devised a work around:

$ac = $_POST['accountnum'];
if(!is_numeric($ac) { $ac = (int) $ac; }

It’s not gorgeous, but it will more than suffice for an internal app. We web programmers don’t usually have to deal with big integers, so it’s totally possible that web developers would never have had occasion to handle a situation like this. Here’s looking forward to native 64-bit for our next server, though.

Imagine my surprise when on Monday, my wife said, “Hey, your site isn’t working!” The index page worked, but none of the other pages.

In short, my webhost, Hostgator decided to implement PHPsuexec. Here’s the gist of this awesome program: typically, your web server runs as the “nobody” user on a server, but you login as yourself, say your username is “jdough.” You need to use certain tricks, like using .htaccess files and chmodding to get around certain limitations. PHPSuexec makes php run *as you,* removing the need for world writable directories and creating a need for custom php.ini files to replace certain php directives in your .htacess files.

Since my site doesn’t use file extensions on most files, I used a directive called DefaultType to make everything PHP. This stopped functioning when Hostgator made the changes on Monday. Instead, every one of the pages that relied upon that value for parsing stopped working and started displaying HTTP error 500.

When I returned into town on Sunday, I opened a high priorityt ticket with Hostgator. An hour later, I called the support line and was told an admin would reply presently. An hour later, I replied to my confirmation email to their email support line. Another hour later, I called again. After 35 minutes on the phone, they finally helped me get the pages running. But images across the site were broken. They were generating parsing errors! They were being interpretted by PHP. Yikes! Another 25 minutes on the phone today resulted in new .htaccess files everywhere. I should tell you that today’s phone calls were with two “gators” who were both very friendly and helped me very enthusiastically.

Hostgator did not email me about these changes, even though they have my email address. They did not call me, even though they have my phone number. They did not post anything in my control panel, even though they can. Instead, they posted it in their own support forums and expected me to check it. A major change to the very core of the server behavior and they simply didn’t tell me. And as a result, my sites were down for a week plus. So if you tried visiting firsttube.com in that time, I’m sorry for the interruption: the view page, the print page, the comments page, and nearly every other meaningful page failed to parse.

If I were a business and monetized my site in any way, I would immediately cancel. But to be fair, Hostgator has unparalleled uptime, unmatched availability, awesome tools (cpanel based), a competitive rate, and a friendly support staff. So I decided to give them one more chance. They have burned all the trust they gained with me, and I will not be recommending them to anyone right now, but I am not taking my business elsewhere just yet.

PHPsuexec is a great tool that provides a nice security boost, but do some serious testing before you implement it. It can dramatically alter the way your websites work.