SilverStripe

User testing the SilverStripe website

Posted by Kerstin Schuman on 27 September 2012

 

You might have heard it through the grapevine; we decided to face the challenge to redevelop our community hub at silverstripe.org.

The current site was created in late 2008 after separating the SilverStripe web services content from the actual open source product. As you may or may not know, SilverStripe makes their money building websites and applications based on the SilverStripe CMS and Framework. Communication wise, these services needed to be separated from the actual open source product and the community. Silverstripe.org as you see it today was born, well not quite as you see it today but almost.

The site got a refresh two years later in 2010 and now another two years in it is time for a major review. And major this will be. The site has grown over the years, we’ve added more sub-sites like doc.silverstripe.org and we now use third party tools such as github, trac and google groups. SilverStripe information and code is scattered all over the place and new users have trouble finding what they are looking for. We are aware of issues people have with the current site and are determined to take on the challenge to make silverstripe.org a better place. So with 2012 being the year in which we released 3.0 we figured let’s also upgrade silverstripe.org.

But where to start? Sitting together with a team of core developers, a designer, a project manager and myself we all had more ideas about what to do with it than there was space on the whiteboard. We quickly realised that there just wasn’t enough time to do all the things we wanted to do. When prioritising we quickly realised the obvious; YOU have to tell us what you want, because the site is built for you and not just for ourselves. We decided to take our own medicine. We keep preaching to our clients to start with a usability test, to better understand what their users need. And that’s what we did.

We invited nine users representing a cross-section of the site’s audience to come into the office for one-on-one interviews with us; comprising of developers, front-end developers/designer and content editors/marketing people.

But we also have a big international community. Most of you can’t just come along and visit our office and we were keen to hear from you what you would like to see improved. So we set up an online test with Loop11, which supports remote user testing. Setting up the test was easy; you can ask questions or give people tasks they have to fulfill via the website. Users can click a button when they are done or can skip a task if they find it impossible to complete.

We set up three different tests, one for each of our different target groups and let them run for two weeks. The reporting is clear and well structured and on a quick glance you can see which tasks were the most challenging.

The tool also offers a clickstream and heatmap analysis to understand where people were looking for information.

 

Good on ya mate! 1,243 SilverStripers started the test! That is awesome and was way more than we had expected. Thank you, guys! Well only 23% of you actually completed the test, but apparently that is within a normal range. ;)

The allocation among the user groups looked like this - developer 50%, front-end developer/designer 35% and content editors/marketing people 15%. I assume this reflects the SilverStripe community quite well and also demonstrates the potential we have in terms of attracting more front-end devs/designers to SilverStripe.

Here are some of our key findings:

The location, complexity and tone of content can make the content hard to understand for some of our audience types. For example:

• The amount of information in the introduction and features pages for both the CMS and Framework are far too wordy. Users found that it was too much information to read in order to get a quick overview of the SilverStripe product.

• When reading the features page, users found that there wasn’t a clear enough hierarchy between each heading and the body text.

• Different user groups used and navigated silverstripe.org in different ways, even when looking for the same content as they had different needs and different levels of understanding of information we provide. Often labelling has different meanings for the different audience types and often results in unwanted results.

• The “Contribute” area provides technical details on what is required to contribute but not where to contribute themes, modules, and widgets.

• Users were often unsure if pages or content areas were applicable to them and their audience group. Something we have observed is that by the way we organise, label and write content we could better acknowledge the wider range of SilverStripe community members.

 

Some of the site’s navigation and area labels were not intuitive, confused users, and did not accurately reflect their content. For example:

• Users found that the “Modules” section is confusing to navigate (for what should be one of the simplest areas) and it is also lacking the promotion of better and more recent modules.

• The hardest task to complete was to find information about meeting others in the community e.g. links to SilverStripe Meetups

• Multiple sites within silverstripe.org (discussion groups, documentation, etc) confused users and made it hard for them to navigate back to silverstripe.org

• Developers found the label of ‘Help’ unclear - they did not expect to find documentation under this section. They would have expected something like ‘support’ or ‘documentation’

• “Developer Network” sounds too much like a place for developers to network, rather than a directory of developers

• The homepage’s links to the blog and the forum are labelled differently from their content (“More News & Events” goes to the blog but doesn’t contain the same news items, and “Community” goes to forums).

 

Some structural and functional aspects of the site are not meeting users’ needs. For example:

• Search functionality results are often vague, misleading, or incorrect. Users prefer to search using Google search.

• Sites within “Showcase” are hard to browse because of the site filtering and because they are grouped by month and not type of website.

• When new users navigated to the Forums section they found it difficult to find how to comment on or create a new post. This is due to the fact that the user is given no clear direction that they need to sign in or register in order to do so.

 

Design

• In general the user experience was not affected majorly by the visual design of the site, although some users mentioned it was time to give ss.org a tidy up to better reflect the direction the CMS and Framework is going.
•  
  Users found that layout and hierarchy of elements sometimes made it difficult to navigate and find the information they needed and thought the use of better font sizing and colour could enhance their experience. 

Interestingly enough, many of the experienced SilverStripe users don’t even use silverstripe.org anymore. They go straight to Github, the dev list on Google groups or the IRC channel. And as far as the newbies go, they have trouble finding their way around altogether. Well, I guess that is a bit of a simplification, but pictures the situation nicely.

We want silverstripe.org to be a fit for everyone, no matter what their level of experience with SilverStripe. With this in mind we then completed a Card Sort Exercise. In this exercise, you create a separate card for each area of content within your site and then give the cards to different groups to sort.

We did this exercise with five groups (each comprising of 3 to 4 people) and each group came up with slightly different results, but we did manage to identify overlaps and patterns and came up with a new site structure that took the best of all the results and looks really good. I am personally really happy with the result and think it makes a lot more sense.

It is too early to share this result with you, but it will be the basis for our redevelopment and I can’t wait to share this new baby with you. This project is a lot about restructuring what we have and making it suitable for all of our audiences, rather than developing all new features.

It will still be new and shiny, but most of all it will better fulfill your needs and we will continue to expand over time.

Needless to say, it will run on SilverStripe 3. Thanks for all your support, you guys rock!

P.S. In addition to the local and remote user testing, we checked the Google Analytics data we have for silverstripe.org. And guess what? The key findings were pretty much all the same across all three methods. Of course you get more detailed information the more personal it gets. From the data analysis, to the local, to the remote testing. Now we are certain that the results are reliable. If you face your next project and you don’t have any budget for user testing, start on checking your data. That already gives away a lot of information.