Metrics and User Studies Group Discussion Notes

The Reference, Access, and Outreach Section's annual meeting included breakout sessions for members to discuss issues of interest or concern to them. This is the second post sharing notes from those discussions. The notes from the RAO mission discussion are available with the others to follow.We hope those in attendance as well as those who could not attend will share further thoughts, suggestions, and reflections on topics of interest in the comments or by contacting a member of the RAO Steering Committee directly. This feedback will then be used by the Steering Committee in formulating the section's agenda and priorities as we move forward. If you have any thoughts on the small group discussion format or the annual meeting, feel free to share your thoughts here or with a member of RAO's Steering Committee.

Notes from Metrics and User Studies Discussion, Notetaker: Amy Schindler

The group began by introducing ourselves, sharing our interests in the area as well as what our repositories are currently doing in the area of metrics and/or user studies. Responses included:

• collecting stats, not doing anything with them
• collecting basic reference stats, particularly interested in web stats
• collect lots of statistics, but really to justify their existence, haven't used to answer questions about doing better
• do collect stats, but not happy with how and what they're collecting. Would like better metrics; have begun doing usability testing on home page directed at most likely users and hope to expand
• just implemented a circulation system and can now assemble stats, don't have good website info; using statistics for security purposes
• have been developing projects for researchers, consortium study report on subject-based access; wondering how to do valid statistical research as an information professional

Participants discussed how time and money often drive the process and that there is a need to justify and produce for activities.

Discussion of what kind of statistics repositories collect - has there been a recent survey?

Mention was made that Paul Conway's article included a form that has provided some guidance.

The Archival Metrics website was brought up - no one in the group is using the user survey toolkits available from AM at this time. The profession is in need of an update from the research team.

The fear by resource allocators that knowing numbers may cause allocation questions was discussed.

Question: How are we tracking email?

• Central email account, by hand, access db tracking
• any staff may receive & handle email inquiry, track via paper printout
  
• stats based on hashmarks
• in spreadsheets with other reference statistics
  

Question: What tools are there to track reference interactions and other statistics?

• Questionpoint
• We need to tools to track "traditional" and online, including 2.0/social media, interactions
• Aeon from Atlas (some attendees were not familiar with this product and no one had seen it demonstrated)

Some of the needs and further questions for RAO to work on from the group:

• Need for consistency in statistics: people day & people hours, etc.
  
• Knowing what is out there as far as products, databases, tools, etc. for collecting data and
  
• What numbers are we keeping?
• What are we doing with those numbers (and other information)? Sending to boss only? Do they make it further up the chain? And then using them in more interesting ways.
• Impacts of collection-level surveys on determining what will be processed further
• A clearinghouse of user studies should be better shared including not just the studies, but also further follow-up
• Training for post-grad archivists on user studies, research statistics; How to do qualitative & quantitative research? Research methodology courses available/appropriate for archivists?