Overview

• I hope to leave you with a better understanding of
  • How simple design decisions for database pages and MetaLib affect usability
  • How to conduct and Usability Scenarios and interpret the results
  • How usability scenario results can feed into design decisions
• I will talk about:
  • UCF's design choices for database and MetaLib pages
  • The Usability Scenario study and results
  • Design changes made or planned based on results

Usability Scenario's in a Nutshell

• Ask users to try to try to complete tasks using your site or system
• Record the efforts and whether the task could be completed.
• UCF's Usability Scenario
  • Used Camtasia to recorded the screens as students tried to use the web and our database pages to find articles on a topic
  • Ask students about their level of satisfaction with the process and results for each task
  • Analyzed the recordings to see the choices students made, what worked, what problems were encountered
  • Made design changes

UCF's Database Page Choices

• Easy Access v. Authentication
  • Access will be as open as possible - no login needed to see our title lists, easy to link to, findable in web search engines
• Site Integration v. Distinctive Look and Navigation
  • Pages will integrate into our overall website with consistent graphical elements and navigation
• Flexibility v. Controlled Content and Organization
  • Pages will include any resources in whatever order and organization is appropriate
• Embed Functionality v. Linking
  • Highly used functionality should be immediately available on the pages, i.e. catalog search, online journal title search, and database search
• Quick view of UCF's database pages prior to the study

UCF's MetaLib Choices

• Access and Privacy v. Identification and Personalization
  • Login should not be required unless user decides to personalize the account
• Power and Functions v. Consistent Interface
  • Federated searching prevents access to specialized options and retrieval sets may not include the most relevant hits
• Design Flexibility, Control v. Single Database Discovery Site
  • Maintain Database discovery pages on our website and deemphasize MetaLib Find Database
• Consistent Result Options v. Inclusiveness of Databases
  • Focus on titles that are fully functional in MetaLib
• Speedy and Focused v. Comprehensive Searching
  • Keep search sets small, using a few sources for each subject.

Realities of UCF's Database Pages and MetaLib

• Do not have much staff time to dedicate to database pages (a fraction of my time)
• Consortial MetaLib installation hosted by FCLA
• MetaLib was purchased at the same time as Aleph, so not specifically chosen by UCF for functionality
• Do not have much staff time to dedicate to MetaLib ( a fraction of Ying's time)
• Very little interface customization in v.3

Daabase Page Stats

• Database pages are heavily used: 281,607 visitors in 2006
  

QuickSearch and MetaLib Stats

• Quick Search is used much more than any other MetaLib service
  

Purpose of the Usability Project

• Find out what students try first when given a research task
• Get an overall picture of how students use the database pages
• See if and how students were interacting with MetaLib search forms
• Determine which options on the database pages lead to the most success and user satisfaction

Methodology: Usability Scenarios

• Collaborated with Meg Scharf, Associate Director for Public Services, and Karla Kitalong, UCF professor usability researcher
• Wrote scenarios - brief tasks that users will try to complete
• Beta tested the scenarios with the help of a graduate student
• Created web pages for a demographic survey and scenario instructions
• Employed an undergraduate student, Jodi, to serve as recruiter and interviewer
• Took a laptop into student areas to recruit participants and record each session
  • How to use Camtasia - 6K in Quicktime

Usability Tasks

• Starting from four different points on the web: Find articles or other resources on the career benefits of internships for college students.
• Tasks start points are:
  • Blank page
  • Database page main area
  • Database page side menu
  • Detailed Subject Resource Guides
• Users were not specifically steer users to MetaLib

Sample Task Description and Satisfaction Survey

Sample Recordings: in Quicktime and How

• Demographic and Task Satisfaction Surveys - 3K
• Task A - use the information source of your choice. - 1.6 K
• Task B - use the main portion of the database page - 2.3K
• Task C - use the side menu on the database page - 1.6K
• Task D- use the Research Guides page

Evaluation of the Usability Scenario Process

• Focus on one or two Tasks - 4 was too many
• Beta test to work out unclear wording, awkward transitions, starting and stopping recording
• The interviewer may have to coax verbal comments out of participants
• Each interaction takes 20-40 minutes
• Provides very rich information source

Data Gathering Process

• Demographic survey and satisfaction ratings were captured into CSV files
• Screens and voices were recorded for entire scenarios
• Jodi wrote "reflections" of her observations for each interaction.
• We reviewed each recording, entered data for search options chosen and the search terms used

Behavior Observations

• Students usually entered the topic exactly as we gave it - none used synonyms
• Students often don't relate specific terms or topics with broader concepts
• Students report high satisfaction if words on the web site exactly match their concept
• Students tend to use a search box before browsing subject or alphabetical database lists
• Students click on what comes up fastest rather than waiting for full results to load
• Off-campus login process often hindered use of the embedded form
• Prior instruction influenced student search behavior

Digging into the Data: Demographics

How often do you use the library?

How often do you use the library website?

Have you had library instruction?

How many library assignments?

Digging into the Results: Resource Chosen and Full Text Found

Chart of Resources Chosen Task B and C

Resource Selected and Satisfaction

Other Data Analysis Possibilities

• Path taken per task
• Time taken per task and per resource selected
• Relationship between full text found and satisfaction
• Relationship between prior library experience and satisfaction
• And much, much, more!

Design Concerns Revealed And Potential Fixes

• MetaLib form attracts users who don't know what it does
  • Reduce the prominence of the MetaLib form
  • Add explanation of what MetaLib searches do
• Duplication of MetaLib form (in the main area and in the side menu of the database page confused users
  • Remove form from side menu
• SFX was ignored - students did not know what it does
  • Add explanations of SFX on the database page
  • Add explanations of SFX on MetaLib results pages

Design Concerns Revealed Continued

• Students don't login to EZproxy before using the forms
  • Redesign the consortial QuickSearch screen
  • Make login form more noticeable
  • Figure out how to make embedded forms use the proxy
• Students try to use the back button to return to the database pages after using MetaLib
  • Open MetaLib search results in a new window
  • Add warning that they are leaving Library web site
  • Add obvious navigation back to the Database page
  • Hope MetaLib 4 works with the back button
• Students do not know what to do after reaching a MetaLib result record
  • Add help on the results screen explaining how to check for full text, search the catalog, etc.
  • Add a research steps guide on the database pages that reveals what to do after getting a citation

Database Page Changes

MetaLib Changes Planned or Desired: Retrieval Screen

MetaLib Changes Planned or Desired: Results Screen

MetaLib Changes Planned or Desired: Record View

More MetaLib Changes Coming in v4

• Recreate embedded forms
• Use the improved CSS to better implement UCF's style in MetaLib
• Customize help screens
• Add facets and match the look of our Endeca catalog as much as possible

Thank you!