Library Technology Guides

Document Repository

Developing statistics and performance measures to describe networked information services and resource for ARL Libraries: discussion prospectus

[December 15, 1999]



The move to a networked environment has significantly increased the range of services and resources that the library provides its users. The library has become a 24 hour a day access point to information services where users obtain services and resources on their terms and when they want such services -- oftentimes not coming to the library physically nor interacting directly with library staff. The costs to provide these networked services and resources, however, can be significant. Librarians' inability to develop reliable and accurate methods to describe these services costs injures their ability to make good resource allocation decisions and develop strategic plans for the development and operation of networked services and resources.

On an experiential basis, most ARL libraries will describe the use of their networked information services with terms such as "exponential growth" or "we can't keep up with demand." At the same time, a number of ARL libraries have also seen stagnant or declining statistics of traditional indicators of library service such as turn-stil counts, in-house reference transactions, circulation, etc. While there is a need to develop new statistics such as "virtual visits," "full-text downloads," "electronic reference transactions as a percentage of all transactions," etc., there is little agreement on how to compute such statistics and measures.

There are other factors affecting the development of these types of statistics and performance measures.

  • Librarians do not control access and use to a range of data that describe vendor-supplied information services and resources. Some vendors are unwilling or unable to provide the types of statistics and use data that librarians request. Statistics and measures for database use and services, nonetheless, are essential.
  • The rapidly changing nature of the networked environment also affects the types of services and resources that can be provided by libraries. As the networked services change, new types of statistics and measures may be needed.
  • Networked services costs and use may be difficult to "unbundle" if the library obtains these services through a consortium or as part of another purchase plan. Costs can either be hidden or be extremely difficult to allocate to individual libraries.

Librarians may be entering a period of time where statistics and measures for networked services may be useful for two-four years and then will have to be re-developed or discarded. Such an environment is quite different than the statistics collecting environment in which academic libraries previously existed. Despite these factors, ARL libraries need to move forward and learn how best to produce and use such statistics and measures in this new environment. To some extent, the effort will need to be experimental and developmental as the complexities of developing such statistics are only now being examined.

The Key Term

The working definition of network-based information services is: Those electronic information resources and/or services that users access on-site in the library, from their office, dorm, or home, or from regional/statewide networks. Examples of electronic networked resources include: local, regional, or state-wide library hosted or authored web sites or library-licensed databases (e.g., Infotrac, SearchBank, EbscoHost). Especially important are statistics that describe the use of unique and often-times interactive remote scientific and technical databases. Examples of electronic networked services include: provision of access to networked services such as email, listservs, online reference/assistance, and training in the use of these resources and services. In addition, libraries increasingly provide interactive services such as requesting services via online forms (e.g. interlibrary loans, etc.) It is likely that the term "networked information resources and services" will continue to evolve as the network evolves.

Key Research Issues

The factors identified above as well as the significant ARL commitment of resources and services to the networked environment leaves library administrators with the need to:

  • Describe the purpose for specific types of statistics and measures describing networked services and who the intended audience might be for these statistics and measures.
  • Agree on (1) which types of networked services and resources should be described and counted and (2) how they should be defined and operationalized.
  • Describe and "count" the amount and type of services use and resources access via the networked environment (1) currently occurring, and (2) on a longitudinal basis.
  • Develop performance or quality indicators that describe the impact and success of such networked services.
  • Document trends in the provision of library services and resources in the networked environment.

This list of key issues is intended to be illustrative only -- clearly there are other topics and issues that the move to the networked environment pose for ARL library staff.

One Approach

A number of approaches can be taken to address these and related issues. One approach, however, is for a group of interested ARL libraries to work together as a consortium in conjunction with a research team at the Information Management Use and Policy Institute at FSU and with ARL staff. The members of this group would agree to make a commitment of a maximum of $10,000 to fund a project with the objective of obtaining at least 10 to 15 member libraries that would support such a project. The study would be one year in length, address the research issues (and perhaps others) as outlined above, and the libraries would participate in the project by providing data and serving as site visits for a range of project activities. In this approach, an overview of key tasks would include:

  • Detail project tasking and schedule in light of the project funding level and project objectives.
  • Identify the measurement needs of members for selected networked information services and resources.
  • Identify current counting and measurement approaches in use by ARL libraries.
  • Propose definitions and operationalized methods for counting and describing selected networked services and resources.
  • Field test and revise and refine the methodology.
  • Write a short descriptive guide for the collection of data that can describe selected networked services and resources as well as suggest performance indicators that assess the quality of such service provision.
  • Develop an on-going process that can revise and update the guide to keep the process current in light of new developments and services provided via the networked environment.

In such an approach, an advisory committee comprised of a small group of selected individuals from the participating libraries would offer advice and suggestions and would review selected documents and project papers. A liaison at ARL would serve as a single point of contact for the study team identified to complete the project.

It may be useful to design the project in two Phases. Phase I would concentrate on developing, field testing, and refining basic descriptive statistics to describe services use and resources access in the academic networked environment. Phase II would build on Phase I and develop, test, and refine measures that would assess and describe impacts from these services. In the one year project timeline proposed here, the extent to which Phase II activities could be completed is unclear.

Importance of the Project

To some extent, ARL libraries are under-representing and under-counting the range of services and resource provision in which they are engaged because there are no agreed upon means to count and describe service provision in the networked environment. In addition, resource allocation decisions are further complicated by a limited understanding of the use and costs of services provision in the networked environment. Given this situation library administrators are unable to demonstrate the importance and impact of such networked-based services. Such is true for individual libraries as well as for ARL libraries as a group.

Clearly, the problems and issues identified in this prospectus regarding counts and measures of services and resources in the networked environment will not disappear in the near future. Indeed, these issues are only likely to increase in importance as the networked environment evolves with a range of new services and resources. It is essential that ARL libraries have the ability to demonstrate their involvement and use of the networked environment and be able to communicate this involvement, use, and impact to faculty, students, and university officials/governing boards.

Support for this project provides ARL and a group of ARL libraries to take a leadership role in the process of developing statistics and measures for services and use in the networked environment. Many libraries need such statistics and measures now. The sooner work is initiated on a project such as that outlined in this prospectus, the sooner such statistics and measures can be used to support resource allocation decisions, services provision and assessment, and strategic planning.

View Citation
Publication Year:1999
Type of Material:Article
Language English
Issue:December 15, 1999
Publisher:Association of Research Libraries
Series: ARL New Measures
Place of Publication:Washington, DC
Notes:Charles R. McClure is the Francis Eppes Professor, and Director Information Management Use and Policy Institute, School of Information Studies, Florida State University. Association of Research Libraries, Washington, DC
Subject: Electronic Resources -- measuring use
Online access:
Record Number:9668
Last Update:2022-08-25 04:43:40
Date Created:0000-00-00 00:00:00