A few notes on bbPress

I have been spending a lot of time off and on the past week creating a bbPress (message boards, or “forum software,” or “bulletin board”) theme for eMusic. Theming is fun because you get to touch almost every feature of the product, bbPress and/or WordPress. In the midst of this, I’ve been discovering how bbPress does things and making some modifications along the way. Those mods may only exist in eMusic and never make it into bbPress, but I have shared ideas with JJJ so I figured I would share them here as well for anyone to check out and supply feedback.

Queries within Queries with Queries

bbPress abstracts almost everything. For most WordPress-y things, there is a bbPress-y thing. One thing that will immediately confuse and potentially wreak havoc is the way “topics” are queried. Because the default WordPress query is the page or custom post type you are on, the topics for a forum are queried using a second mechanism: bbp_has_topics( ), which upon success, returns a WP_Query object.

That’s all well and good, but at any given time, there is only 1 bbPress query and no hardened reference to the original query. If you are using the default theme or just modifying its presentation, you probably don’t care. If you are implementing a design that has multiple “topics” queries, you are kinda up shit’s creek unless you roll your own code. The template tags / functions provided by bbPress will work to an extent, but they will also shift bbPress’s entire context every time you call bbp_has_topics( ).

The solution espoused to me was to roll my own WP_Query objects to replicate what bbPress is doing in the background, but I don’t want to do that. I want to use bbPress functions and fix whatever context is set along the way when necessary. WordPress maintains state and stores the main query by using $wp_the_query and $wp_query, for the main query and then the current query, respectively. bbPress clobbers $bbp->topic_query every time bbp_has_topics( ) is run. bbPress is treating bbp_has_topics( ) like query_posts( ) in WordPress, not like new WP_Query( ). Ask Nacin how he feels about query_posts( ).

bbPress makes an attempt to provide query context by providing the function bbp_set_query_name( $name )

Guess what it does internally… basically nothing. So I wanted to fix this, here’s some code I am using:

global $emusic_the_bbp_query;
$emusic_the_bbp_query = array();
function emusic_set_bbp_query( $name = 'main', $params = array() ) {
    global $emusic_the_bbp_query;
    $bbp = bbpress();

    bbp_set_query_name( $name );

    if ( !empty( $bbp->topic_query ) && empty( $emusic_the_bbp_query ) )
        $emusic_the_bbp_query['main'] = $bbp->topic_query;

    if ( !empty( $name ) && isset( $emusic_the_bbp_query[$name] ) ) {
        $bbp->topic_query = $emusic_the_bbp_query[$name];
        return $bbp->topic_query;
    }

    if ( !empty( $params ) ) {
        bbp_has_topics( $params );
        $emusic_the_bbp_query[$name] = $bbp->topic_query;
        return $bbp->topic_query;
    }
}

function emusic_reset_bbp_query() {
    global $emusic_the_bbp_query;
    $bbp = bbpress();
    $bbp->topic_query = $emusic_the_bbp_query['main'];
    bbp_set_query_name();
}

My code does a few things:

  • Sets $emusic_the_bbp_query whenever you start to change context the first time
  • Always allows you to retrieve the default context when you are done with a sub-query for topics
  • Non-persistently caches topic queries by name

So where’s my use case? I want to show a “snapshot” of your subscribed-to topics, favorite topics, popular topics, and super-sticky topics in the sidebar on every page, and I want to reuse theme code (template parts) to do so. Imagine calling query_posts( ) 5 times in a template on the WordPress side…

Could I have accomplished this with WP_Query? Yeah, but… I want to use the bbPress stuff. So here’s what I do:

<?php if ( is_user_logged_in() ):   ?>
<div class="meta-block">
    <h3>Your Topics</h3>
    <span class="double-line-narrow"></span>
    <?php
    if ( bbp_is_subscriptions_active() ) : ?>
        <h4 class="sub-head">
            Subscribed
            <a class="aux" href="<?php bbp_user_profile_url( get_current_user_id() ) ?>">view all</a>
        </h4>
        <span class="double-line-narrow"></span>
        <?php
        $subscriptions = bbp_get_user_subscribed_topic_ids( get_current_user_id() );
        if ( !empty( $subscriptions ) ):
            emusic_set_bbp_query( 'bbp_user_profile_subscriptions', array( 'post__in' => $subscriptions, 'posts_per_page' => 3 ) );

            while ( bbp_topics() ) : bbp_the_topic();

                bbp_get_template_part( 'loop' , 'single-sidebar-topic' );

            endwhile;
        else:
            printf( '<p>%s</p>', __( 'You haven't subscribed to any posts.' ) );
        endif;

        emusic_reset_bbp_query();

    endif; ?>

    <h4 class="sub-head">
        Favorites
        <a class="aux" href="<?php bbp_user_profile_url( get_current_user_id() ) ?>">view all</a>
    </h4>
    <span class="double-line-narrow"></span>
    <?php
    $favorites = bbp_get_user_favorites_topic_ids( get_current_user_id() );
    if ( !empty( $favorites ) ):
        emusic_set_bbp_query( 'bbp_user_profile_favorites', array( 'post__in' => $favorites, 'posts_per_page' => 3 ) );

        while ( bbp_topics() ) : bbp_the_topic();

            bbp_get_template_part( 'loop' , 'single-sidebar-topic' );

        endwhile;
    else:
        printf( '<p>%s</p>', __( 'You haven't favorited any posts.' ) );
    endif;

    emusic_reset_bbp_query();
?>
</div>
<?php endif ?>

<?php
$super_stickies = get_option( '_bbp_super_sticky_topics', array() );
if ( !empty( $super_stickies ) ): ?>
    <div class="meta-block">
        <h3>Featured Discussions</h3>
        <span class="double-line-narrow"></span>
        <?php
        emusic_set_bbp_query( 'bbp_super_stickies', array( 'post__in' => $super_stickies, 'posts_per_page' => 4 ) );

        while ( bbp_topics() ) : bbp_the_topic();

            bbp_get_template_part( 'loop' , 'single-sidebar-topic' );

        endwhile;
        ?>
    </div>
<?php endif ?>

Meta Queries like whoa

First of all, JJJ has done an awesome job making bbPress a plugin, and it is super sweet how seamlessly it integrates with everything else in WordPress. The two areas I would like to help make improvements in are 1) cache and non-persistent cache 2) Meta Query performance in WordPress as a whole. bbPress makes Meta Queries like (holy shit) whoa. For 99% of installs, who cares. For us, my current dataset is already 100s of 1000s of posts and 2-3 million rows of postmeta. A meta query basically says:

  • I need stuff from the posts table
  • I need stuff joined from the postmeta table
  • I have indexed columns in both
  • Fuck that noise, let’s join on an unindexable LONGTEXT column and pray for mercy

I wish meta_query never existed in WordPress. But it does, and bbPress sings its song throughout. I have a few ideas for reducing number of queries made and also splitting the query into 2, so that PHP can compare integers instead of trying to sort them as text in MySQL. I have done a lot of performance testing on meta_queries, and I actually ditched them in a few places for WP_Query because they do not scale out of the box.

But I’m not just gonna whine about it, I’m gonna try to contribute and I’ll report back when I do.

WordCamp NYC 2012: “Cloud, Cache, and Configs”

Here are the slides from my talk today:

I spoke for 40 minutes to a room full of people that had no idea what I was saying. Seriously.

Have any of you made a plugin before? Silence / crickets. Cool, well let me dive into scaling HTTP parallelization for 15 minutes…

Installing libmemcached on CentOS

I recently updated the Memcached WP Object Cache plugin to use the Memcached PHP extension – it currently uses the Memcache extension. My plugin is called Memcached Redux, and it’s delicious. The Memcached PHP extension implements methods from libmemcached, the C / C++ Memcached library.

Installing the extension on MacPorts is beyond easy:

port install php5-memached

Installing on Amazon EC2 instance running CentOS 5 is WAY more esoteric. I spent about 2 hours mucking around until I figured it out, so here it is (this assumes you are already using memcached and have libevent, libzlib, etc installed):

cd /etc/yum.repos.d/
wget http://rpms.famillecollet.com/remi-enterprise.repo
wget http://syslogserver.googlecode.com/files/epel-release-5-3.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-5.rpm
rpm -Uvh remi-release-5*.rpm epel-release-5*.rpm
yum --enablerepo=remi install libmemcached*
pecl install memcached

vi /etc/php.ini

// (under Dynamic Extensions) Add "extension=memcached.so":

extension=apc.so
extension=http.so
extension=memcache.so
extension=memcached.so

Servers have to be restarted – this has be done before code is deployed or the class won’t exist, which will cause a fatal error.

This process looks harmless, but I had to read 3.7 million blog posts before I found the right path forward.

Memcached Redux

I’ve been reading about Couchbase, and I knew it was compatible with Memcached out of the box. One of the features I wanted to start using was Memcached::getMulti and Memcached::setMulti. I knew the Memcached WP Object Cache plugin had a get_multi method, but I didn’t know what it did or how it did it. Turns out, it doesn’t implement Memcached::getMulti.

When I looked under the hood, I realized that the Memcache extension is loads different than the Memcached extension in PHP. Memcached has the getMulti method, Memcache does not. So I set out to change this: I have altered the famed Memcached plugin to actually use the Memcached class.

Because I did this, you can now use methods like this:

wp_cache_get_multi( array(
	array( 'key', 'group' ),
	array( 'key', '' ),
	array( 'key', 'group' ),
	'key'
) );

wp_cache_set_multi( array(
	array( 'key', 'data', 'group' ),
	array( 'key', 'data' )
) );

Rather than making many calls to grab data, especially related data, you can grab it in one hop.

I’m gonna keep working on the plugin, and drop a line if you install it and have comments / concerns.

The Plugin: Memcached Redux

Audio Redux + Updated Plugins

I had the pleasure recently of realizing that some of my plugins had disappeared from the WordPress dot org plugins repo because I haven’t updated them in 1.5ish years. Last year at WordCamp San Francisco, I used part of my talk to explain why plugins aren’t always awesome, and me not updating my own plugins is a great example of why. You might think to yourself, “Scott’s pretty good, I’m sure his plugins are awesome, I could probably get pregnant just by activating one of them!” To which I would reply: “If you are using my Movies plugin, I have absolutely no idea if it still works, and I’m sure every javascript library in it is 50-75 versions behind.”

So I am trying to right this wrong and update my code. Back in 2010, I was still under the impression that procedural programming was da bomb. Needless to say, the plugins should work a lot better now and will be maintained / are more maintainable.

Here are some of the plugins that are updated, along with their companion blog posts:

Minify
Plugin: http://wordpress.org/extend/plugins/minify/
Blog Post: http://scotty-t.com/2012/05/24/minify-redux/

Shuffle
Plugin: http://wordpress.org/extend/plugins/shuffle/
Blog Post: http://scotty-t.com/2010/11/15/my-first-plugin-shuffle/

Audio
Plugin: http://wordpress.org/extend/plugins/audio/
Blog Post: http://scotty-t.com/2010/11/22/new-plugin-audio/

Audio is awesome, and here’s why:

  • WordPress doesn’t come packaged with an mp3 player
  • Your player can be styled 100% with CSS – (drop in a replacement: STYLESHEETPATH . ‘/audio.css’)
  • If you use Shuffle, you can attach .ogg files to your .mp3s, and they will be used in native HTML5 browsers that don’t natively support .mp3
  • If you use your own styles, you can style a playlist automatically, and with Shuffle, attach image(s) to each MP3

BOOM. I will try to be a better man in the future and keep my plugins bleeding edge.

xoxo

Minify Redux

As you may or may not know, I wrote a plugin a while back called Minify. Its purpose was to automatically concatenate your JS and CSS files for you in your WordPress theme. Concatenation and minification are good front-end performance optimization techniques.  The thinking is simple: why request 10 files when we can request 1 optimized file? So that plugin existed for a little while, I said nothing about it, 20 people downloaded it, and then it completely disappeared from the WordPress plugin repo – random.

As of today, it’s back, and it’s way different. When I originally wrote it, I wasn’t working in a load-balanced environment with WordPress. I was working on one or 2 servers at a time on sites that were inconsequential as far as traffic goes. So initially, I was generating flat files.

When you get into the cluster business, you need to give up the notion of working with dynamically-generated flat files. Unless a user is “pinned” to a particular server in the environment, there is no way to determine if you are on the same server from page to page without some logic you shouldn’t feel like messing with – ah, the statelessness of HTTP.

When you are on many servers, Memcached is a great conduit for maintaining state and sharing resources. If that is true, why not use it for JS and CSS? As with all static assets, in the end, you are producing a URL that points at data. Where that data comes from shouldn’t matter. It should be produced fast, and if it can be distributed, great. Especially if you are behind a CDN like Akamai, once the data is requested, it will be cached there, where your local flat files aren’t being utilized.

How Does It Work

WordPress provides tools for enqueue’ing scripts / styles and resolving dependencies among them. When all is said and done, WP will output these scripts / styles when wp_head() and wp_footer() are called. Minify works by using add_action() and output buffering. Let’s take a look:

<?php
/**
 * Essentially, what happens
 *
 */
function minify_start_buffer() {
    ob_start();
}
add_action( 'wp_head', 'minify_start_buffer', 0 );
add_action( 'wp_footer', 'minify_start_buffer', 0 );

function minify_combine_scripts() {
...... a bunch of stuff happens / buffer is released .......
}
add_action( 'wp_head', 'minify_combine_scripts', 10000 );
add_action( 'wp_footer', 'minify_combine_scripts', 2000 );

From there, JS and CSS files are slurped out. Using JS as an example:

  • All srcs are retrieved
  • They are placed side by side in a string, which is then md5‘d
  • At this point – we either already Minify’d them, or they need to be Minify’d
  • Provide locking to serve old scripts temporarily to prevent cache miss stampedes on the new script generation

Cache Busting

Minify will essentially serve your files from the cache into infinity unless:

  • the cache expires
  • you edit the file to indicate expiration
  • you change the name of any file that is enqueued in the set

A Minify tab is specified in the admin to let you update your Minify increment. This is crucial for cache-busting a CDN like Akamai that won’t always request a new version of the file when the query string changes. The increment is included in the list of files that is stored and the generated source, so updating it will create a new list and new source.

Loading WordPress Without Loading ALL of WordPress

Using Andrew Nacin’s favorite constant, SHORTINIT, you can require wp-load.php and not load a billion plugins along with it. This is useful when you are pointing at a PHP file with a RewriteRule and need some WordPress functionality, but you have no desire to generate HTML source, or use plugins and themes. Since plugins are loaded by calling require( $plugin_file ) anyways, after you load wp-load.php, you can selectively load any plugin you need just by requiring it. That’s what I do in Minify:

<?php
define( 'SHORTINIT', 1 );

/**
 * load bare-bones WP so we get Cache / Options / Transients
 *
 */
$load_path = $_SERVER['DOCUMENT_ROOT'] . '/wordpress/wp-load.php';
if ( !is_file( $load_path ) )
	$load_path = $_SERVER['DOCUMENT_ROOT'] . '/wp-load.php'

if ( !is_file( $load_path ) )
	die( 'WHERE IS WORDPRESS? Please edit: ' . __FILE__ );

require_once( $load_path );
/**
 * SHORTINIT does NOT load plugins, so load our base plugin file
 *
 */
require_once( 'minify.php' );

Anyways, we use Minify at eMusic. It has dramatically sped up our site / Javascript execution, perhaps more than any other single optimization. Take a look and drop a note if anything offends you.

Fix for “The plugin does not have a valid header”

If you’ve ever seen this error when trying to install a plugin in your WordPress Admin Panel, you have probably considered going on a murderous rampage:

“The plugin does not have a valid header.”

This error is probably due to your list of Plugins being cached in whatever WP Object Cache your blog / site is using. None of the sites you will find when searching Google will tell you this. They will tell you that you have an invalid Plugin Header. You might. But probably, your list of files is cached.

There are 2 problems here:

1) The list of plugins ( $cache_plugins ) is not pluggable

2) the generic error for all missing plugins is “The plugin does not have a valid header”

function validate_plugin($plugin) {
    if ( validate_file($plugin) )
        return new WP_Error('plugin_invalid',
            __('Invalid plugin path.'));
    if ( ! file_exists(WP_PLUGIN_DIR . '/' . $plugin) )
        return new WP_Error('plugin_not_found',
            __('Plugin file does not exist.'));

   $installed_plugins = get_plugins();
   if ( ! isset($installed_plugins[$plugin]) )
       return new WP_Error('no_plugin_header', 
           __('The plugin does not have a valid header.'));
   return 0;
}

TO FIX THIS GARBAGE = in /wp-admin/includes/plugin.php at line 219-ish, comment out the lines below then visit your Plugins list:

// if ( ! $cache_plugins = wp_cache_get('plugins', 'plugins') )
$cache_plugins = array();

// if ( isset($cache_plugins[ $plugin_folder ]) )
// return $cache_plugins[ $plugin_folder ];

When you’re done and everything works, un-comment those lines. If this didn’t fix the problem, then yes, you probably fucked up when creating the Plugin header comment.