Measuring content effectiveness

Recently I began work on a small analytics project. I wanted to establish an a KPI measurement of our site content effectiveness. The plan was to develop a scorecard that could be leveraged on a regular basis by our communications staff (the content writers) to help them determine what pages and content needed to be fixed. I wanted to be able to use this scorecard to set priorities without having to do a ton of additional analysis (something like a number that yelled ‘fix me’ at a glance) and wanted to be able to monitor pages over time to see if the changes being made were in fact resolving problems.

Robbin at LunaMetrics agreed to help (and was a fantastic sounding board throughout this project).

We began by looking at a number of options for measurement. A number of metrics and KPIs came to mind: Ratio page views to visits; Exit ratio; Bounce rate and Percent one page visits. Here’s an explanation of each of the KPIs I’ve mentioned in more detail:

Ratio page views to visits
Formula: page views of a particular page / visits to the same page
A high ratio, especially on deep pages, seemed to indicate that the visitor was stuck and using the back button.

Exit ratio
Formula: exit visits / visits to the same page
While we might have a high number of visits for a page (meaning that we were getting the traffic there), this KPI would highlight which pages were causing the user to leave when they couldn’t find the information they were looking for.

Bounce rate
Formula: single access visits / entry visits to the same page
A measure of how many people leave without viewing any other pages. A low number tells you that the page is effective in moving the user deeper into the site, a higher bounce rate, the less effective a page is at keeping the user engaged. (Note: HBX, unlike Google Analytics, doesn’t display a bounce rate metric like Google Analytics does, so you have to calculate this in Excel).

Percent one page visits
Formula: single access visits / visits to the same page
Jim Novo noted this one in a Google Conversion University article. It’s the percentage of visitors bouncing off the site (like ‘plexiglas’ says Jim). It’s and usually is tied to global navigation issues and should trend down over time as you make changes to the site or copy. Robbin took the Bounce Rate and Percent One Page Visits KPIs and charted them and they correlated nicely, so using either KPI would work for us.

Looking at each of this KPIs on a per page basis told us we had some problems. Each by themselves was partially effective, but missing bits of information. So we set out to create a KPI mashup. This became known as the Steif Mashup Content KPI. Here’s what we came up with:

Visits * Bounce Rate / Time Spent on Page in Seconds

We liked this. Visits were a good indicator of the traffic volume and page importance to users, Time Spent on Page was a good indicator of the engagement of the user with the page content when they got there and Bounce rate was a good indicator of whether the page itself was working or not.

• A high visit page with a high bounce rate needed to be fixed ASAP. People were going here and we were losing them
• A low visit page with a high bounce rate, could be attended to when priority allowed
• A High visit page with a low bounce rate we could leave for now and come back to tweak later
• A low visit page with a low bounce rate we’d likely not even look at.

Using HBX Report Builder I created an Excel worksheet that pulled in the page path and the metrics related to that page path. For those who have never done this using Report Builder, here’s what you need to do.

• For the page path select ‘Most Requested Pages’ and bring in the page name and page path. If you set a filter you can bring in only pages that meet certain criteria (e.g. I used this to bring in only pages for the ‘Banking’ section of our site – which you can set either within the Report Builder interface or reference to a cell in your Excel spreadsheet)
• Adding dependent requests off the page name, I grabbed each of the other values (visits, single access, entries, exits and time spent on page) and added them to their own columns.
• Since HBX reports time spent on page as D:HH:MM:SS you need to break this apart to get a value for the time overall in seconds. That’s easy using the MID function in Excel (cell G3 in the example is the time spent on page cell):
  (MID(G3,6,2)*60)+(MID(G3,9,2)

Then doing the calculations for each KPI was just a matter of referencing the data pulled down into Excel.

Problem was that this still wasn’t giving me an “at-a-glance” view of what pages were working or not. For example for a high traffic sick page, the resulting number was higher than that of a low traffic sick page, even if the bounce rate was the same for both pages. So setting priority at a glance was difficult. So we tried changing the mashup KPI taking out the ‘visits’ and ended up with this:

Bounce Rate / Time Spent on Page in Seconds

This was starting to look better. Now if a page had a higher “score” the more this page needed attention.

• A high bounce rate page with high read time could mean that the content was missing some information the visitor was looking for
• A low bounce rate page with a high read time meant that the content was likely working.
• A high bounce rate page with a low read time meant we were losing the visitor quickly. Content wasn’t working or providing the necessary information.
• A low bounce rate page with a low read time wasn’t getting used much, but was meeting the needs when it was accessed.

Now with this new KPI we’re starting to prioritize content changes that need to be made. As an overlay on this we’ve also segmented these KPIs looking at how the values change based on the traffic source, new vs. returning visitor, member vs. non-member, etc. Where we have pages with lite traffic volumes, we can extend it out to cover a longer period (e.g. last quarter instead of the last month).

So what do you think? Have we tackled this the right way? How do you measure your site’s content effectiveness?