Perceptions 2011: An International Survey of Library Automation

by Marshall Breeding. January 28, 2012

Introduction

The ability for libraries to carry out their operations efficiently and effectively depends—at least to a certain extent—on having an automation system well suited to their needs. In any given year, we see a certain level of churn of libraries migrating from automation systems deemed obsolete or ill-fitting to their current expectations, moving on to new alternatives. The pool of choices for new automation products spans a set of diverse options, though within a limited pool, narrowed by the continual process of industry consolidation. Libraries in immediate need of replacing their current system, or in the process of making longer term technology strategies, need data from a variety of sources as they evaluate options. Technical documentation, marketing materials, product demonstrations, product vision statements and functionality checklists represent some sources of information to help libraries evaluate automation products.

Another important avenue of investigation involves data from libraries involved with any given product that describes their first-hand experience. This survey aims to measure the perceptions libraries have regarding their current automation products, the companies that support them and to capture their intentions about future migration options. It also explores interest in open source library automation systems, a key issue for the industry. This survey aims to gather sufficient numbers of responses, reflecting the subjective experience of any given library to create meaningful results, reasonably informative about the collective experience of libraries with this set of products and companies.

This year’s edition of the survey provides the ability to view the response data not just in a broad aggregation, but to also according to factors such as the library type (public, academic, school, etc) and by the collection size. By looking at these subsets we are able to derive more nuanced conclusions. In the broad results some of the products that serve very small libraries dominate as top performers. A different dynamic emerges when considering specific demographic categories. We provide some examples of the category combinations possible in the tables placed into this report. We have also provided the tools to apply these factors across all of the categories in the interactive version of the survey results.

Some libraries may refer to the results of this survey as they formulate technology strategies or even consider specific products. I urge libraries not to base any decision solely on this report. While it reflects the responses of a large number of libraries using these products, I hope this survey serves more as an instrument to guide what questions that a library might bring up in their considerations and not to make premature conclusions based on subjecive responses. Especially for libraries with more complex needs, it's unrealistic to expect satisfaction scores at the very top of the rankings. Large and complex libraries exercise all aspects of an automation system and at any given time may have outstanding issues that would naturally result in survey responses short of the highest marks.

The survey results also aim to provide useful information to the companies involved in the library automation industry. While each company likely performs its own measures of client satisfaction, this survey may show perceptions in different ways. I hope that the rankings in each category and the published comments provide useful information to help each of the companies hone in on problem areas and make any needed adjustments to their support procedures or product directions.

This year marks the fifth annual edition of the survey, with 2432 responses, more than any previous year (2010 = 2,173, 2009=2,099, 2008=1,453, 2007=1,779). In very broad terms, the survey results are similar this year, but with some interesting new trends. Libraries of all sizes responded, including 251 will collections over 1 million items, 1,189 with collections over 100,000, 1,083 with collections under 100,000, 750 with collections under 750, and 376 libraries with under 20,000 items. There were 131 responses with no collection size provided.

General Information about the Survey

The survey has been conducted annually since 2007. Previous editions continue to be available: 2007, 2008, 2009, and 2010.

This year, the survey attracted 2432 responses from libraries in 77 different countries. The countries most strongly represented include the United States (1,756 responses), followed by Canada (160), United Kingdom (104), Australia (104) and New Zealand (30). As with the general demographics of the libraries.org database, the respondents of the library primarily come from libraries in English-speaking countries. Survey results were gathered between November 14, 201 and 1January 27, 2012 (Full demographic summary).

The survey attracted 50 or more responses from libraries using Millennium (458), Symphony (333), Voyager (154), Horizon (150), ALEPH 500 (140), Library.Solution(132), Polaris (106), OPALS (81), VERSO (67). Koha -- ByWater Solutions (62), Apollo (54), Evergreen -- Equinox Software (52), Many other products were represented in the survey with few number of responses. Systems with less than 15 did not appear in the main statistical tables. These responses can be seen through the individual ILS Product Reports available.

This article is an original publication of Library Technology Guides and is not slated to appear in any print publication. Please direct any comments or enquiries to the author.

This survey and its analysis reflect my ongoing interest in following trends in the library automation industry. It is designed to complement, and not replace, the annual Automation Systems Marketplace feature that I have written the last seven years for Library Journal. The survey underlying the Library Journal article relies on information provided by the companies that offer library automation products and services. The survey that serves as the basis for this article collects data from the libraries themselves.

Survey Results

Statistics related to the question: How satisfied is the library with your current Integrated Library System (ILS)?

An interactive version of the statistical reports, is available which includes the ability to view the responses for each of the ILS products, along with the redacted comments.

---

General Observations

In one of the more interesting observations of the survey, respondents tend to blur their perceptions across many of the questions. Those with generally positive satisfaction, tend to answer the questions about ILS satisfaction, company satisfaction, support satisfaction, and ILS functionality without major distinction. Few responses will indicate high satisfaction in one question and moderate or low satisfaction in others. Despite this tendency, the variation between questions does result in some differences in rankings in the companies across categories. The blurring across categories is especially conspicuous in the question asking whether service has improved in the last year. Aggregate responses in this category tend to be quite high for products with strong satisfaction. Taken literally, high numbers in this category imply that great improvement had taken place since the previous year. Numbers in the middle range would indicate that support had maintained an existing level of high performance. Since this question was introduced last year, it does not seem that it has been especially successful in measuring changes in support performance.

Small libraries give measurably more positive responses on the survey than larger libraries, bearing out a trend that the size and complexity have a downward impact on overall satisfaction with automation scenarios. Larger libraries press the limits of functionality and encounter more support issues for which there may not be an easy answer or immediate response possible. Across all libraries with collections less than 25,000 items, the average satisfaction score for their ILS was 7.13, compared to 6.47 for both the groups of medium-sized and very large libraries. The top performers in the small library group received superlative ratings: Apollo with 8.57 and OPALS at 8.31; in the large library arena top scoring Polaris averaged ILS satisfaction of 8.25 and second-place Millennium at 6.87. In the middle tier of libraries Polaris received 8.03 followed by Library.Solution at 7.95.

This survey has not attracted a great deal of interest from the K-12 school library arena, with only 126 out of the 2,432 total responses. Of these, 57 were from libraries using OPALS, 14 from Destiny, 11 using some version of Koha, 8 from Library.Solution, 5 using Millennium, 4 using SirsiDynix Symphony, with a few other ILS products represented in smaller numbers. Destiny holds dominant market share among school libraries. OPALS represents a smaller portion of the K-12 school market, though is much more strongly represented in this survey. Libraries using OPALS express superlative perceptions of the software and for support arrangements.

Perceptions Open Source ILS Products and Support Companies

Consistent with results from previous years, the scores in the "Interest Level in Open Source" naturally run high for those libraries already involved with an open source ILS, ranging from a 8.41 given by those running Koha implemented independently, 8.06 when supported by ByWater Solutions, through 7.53 for Evergreen. Libraries using LibLime’s version of Koha reported a bit less enthusiasm for open source ILS with a mean response score of 6.34. For those libraries running a proprietary ILS, the interest in open source ILS seems mostly indirectly proportional to satisfaction with the ILS, company, and support. Libraries running proprietary products that rate high satisfaction with the ILS, company, and support categories selected lower levels of interest in open source alternatives, while those more dissatisfied show at least some higher interest. The scale of interest in open source from those running proprietary systems tops out at 4.50 (Talis Alto) compared to scores greater than 6.34 from existing open source practitioners.

Overall there was a slight softening in the enthusiasm for open source relative to last year, with mean scores of the question probing interest in open source ILS dropping since 2010. In the case of LibLime, the drop was significant, down to 6.34 in 2011 from 8.50 in 2010.

The survey does not support the notion that libraries running open source are generally more satisfied than those using proprietary systems. Among small libraries, Apollo, a proprietary product offered through software as a service beats any of the open source products, though the open source OPALs ranks only a tiny notch below. Open source and proprietary products are both dispersed throughout the result tables. Neither proprietary nor open source products dominate the top rankings. In the new question category added this year asking about the completeness of functionality, we saw some trends rating the open source products slightly downward and proprietary ones slightly upward. For example, Evergreen ranked fourteenth in overall ILS satisfaction and fifteenth in functionality, SirsiDynix Symphony ranked sixteenth in ILS satisfaction and fourteenth in functionality; Koha supported by ByWater Solutions was perceived as ninth in functionality though eighth in overall satisfaction; Millennium placed twelfth in overall satisfaction and tenth in perceived functionality; Polaris ranked fifth in overall ILS satisfaction and third in functionality.

Libraries that have implemented Koha independently reflect higher satisfaction than those that rely on commercial support arrangements. One might suppose that this reflects their enthusiasm toward open source and that they are essentially evaluating themselves rather than an external organization. ByWater Solutions continues to reap high praise from their support customers, though their scores dropped a fraction since last year. Libraries using the versions of Koha supplied and supported by LibLime reflected some of the lowest satisfaction scores on this year's survey across all categories. Satisfaction with Evergreen as supported by Equinox Software fell into the middle tier of results in most categories.

It’s important to note that despite any softening in the scores rating interest in open source ILS products, many libraries indicating that they are interested in a migration mention open source products as a replacement candidate: Evergreen (87) Koha (74)and Kuali OLE (16).

Comments on the Products and Companies specializing in the Automation of Smaller Libraries

We noted above that small libraries tend to give higher satisfaction ratings than do larger libraries. 463 of the total of responses came from librareis with 25,000 or fewer items in their collections. Products that appeared only in the small library category included: Apollo, OPALS, VERSO, Koha implemented independently or with support from ByWater Solutions, and Destiny. Some small libraries use products generally oriented toward larger libraries such as Millennium or SirsiDynix Symphony, but as members of a consortium.

Apollo, a hosted ILS provided by Biblionix used by small public libraries, received top rankings for ILS satisfaction (8.60), company satisfaction (8.65); ILS functionality (8.37), customer support (8.67); and company loyalty (8.69). The number of responses for Apollo (53) was down this year relative to previous years. No libraries using Apollo indicated consideration of moving to a new ILS, none plan for a discovery interface, and 51 out of 54 report on-schedule installation. Comments offered from libraries using Apollo were overwhelmingly positive.

VERSO from Auto-Graphics received positive ratings in each of the categories. The average ILS satisfaction ratings for this product improved over last year, moving up to 7.72 from 7.40 last year. Perceptions for ILS satisfaction ranked sixth, support ranked Fourth (8.34), sixth in ILS functionality (7.49) sixth for loyalty to company (7.70). Libraries using VERSO ranked very low in interest to an open source ILS (2.59). Overall survey results confirm very high perceptions for VERSO as an ILS and Auto-Graphics as a company and comments offered mostly reflect high praise, though some reflect some transition issues related to the company's new Iluminar interface.

Comments on the Products and Companies specializing in the Automation of Larger Libraries

Polaris tops the rankings within its sector of larger public libraries. This product has attracted very positive rankings from its users across all five years of this survey. In the first years of the survey, Polaris earned top placement. A set of products has emerged in the survey in recent years that serve smaller libraries with even higher ratings. But the performance of Polaris has remained remarkably constant (ILS Satisfaction: 2011=7.77, 2011=7.77, 2009=7.79, 2008=7.73, 2007=7.78). While Polaris’ ranking in the general pool of responses for ILS satisfaction has slipped to fifth place this year, its performance has in no way diminished. When limiting the results to public libraries, Polaris ranks second. In the category of libraries with collections over one million items, Polaris ranks first, as it does among libraries with collections over 200,000 items. Only 1.89 percent of libraries using Polaris indicate interest in migrating to a new ILS.

Millennium from Innovative Interfaces, Inc. received quite respectable rankings, generally placing near the top of the middle tier of most categories. Libraries running Millennium responded to the survey in higher numbers than any other ILS (454). ILS satisfaction for Millennium slipped a small fraction since last year (ILS satisfaction: 2011=6.88, 2010=7.11, 2009=7.13, 2008=7.08, 2007=7.17). The number of libraries running Millennium indicating that they are considering migrating to a new ILS is up sharply this year (31.22 percent) compared to last year (18.73 percent), but sixty-five of these libraries indicate consideration of Innovative's own Sierra platform. Twenty-five show interest in Evergreen, with another thirteen including Koha as a migration candidate. The comments offered were diverse: common themes included complaints about system costs, others spoke highly the new Sierra platform.

Libraries using Library.Solution from The Library Corporation responded with rankings that generally fall around the top of the middle tier. In the ILS Satisfaction category, the 7.45 mean rating was ninth best; company satisfaction (7.56) and support satisfaction (7.52) scores earned similar placement. Support satisfaction for Library.Solution has improved each of the five years of the survey. Libraries gave average company loyalty ratings of 7.06 and 14.39 percent indicate interest in migrating to a new ILS.

SirsiDynix offers two ILS products represented in the survey, Symphony, (316 responses) and Horizon (149 responses). The two ILS products received remarkably similar scores. Though SirsiDynix promotes Symphony as its flagship ILS, it offers continued support and development for Horizon. Satisfaction with Symphony has remained relatively constant across all five years of the survey (ILS satisfaction: 2011=6.18, 2010=6.15, 2009=6.06, 2008=5.68, 2007=6.41). Horizon has seen a slight diminishment in satisfaction. (ILS Satisfaction: 2011=5.91, 2010=5.99, 2009=6.07, 2008=5.68, 2007=6.13). Libraries running Horizon showed an improvement in company loyalty since last year, up to 5.72 from 4.94. Interestingly, this year Symphony libraries reflect a slightly lower company loyalty score of 5.47. 54 percent of Horizon libraries and 22.52 percent of Symphony libraries indicate that they are considering migrating to a new ILS. 39 out of 90 libraries running Horizon that were considering migrating mentioned SirsiDynix Symphony a replacement option.

Two products from Ex Libris were represented in the survey results, Aleph, (139 responses) and Voyager, (154 responses); the company continues support, marketing, and developmlent for both of these ILS products. Ex Libris has announced Alma as its next generation system. Aleph and Voyager serve large and diverse organizations and offer very complex functionality, placing them into a tier of products that tend not receive superlative marks. Aleph received a mean score of 6.28 on ILS satisfaction, just a bit higher than that of Voyager (6.07). Perceptions of the company between these two products show slightly higher ratings from Voyager libraries (6.21) than those running Aleph (6.04); A simlar pattern emerged for customer support with libraries running Voyager slightly happier (6.16) than Aleph sites (6.07). 25.71 percent of Aleph libraries indicated interest in migrating to a new ILS with 14 mentioning Alma as a replacement candidate. 38.31 percent of Voyager libraries indicated interest in migrating to a new ILS with 21 mentioning Alma as a replacemetn candidate.

---

Details about The Survey

The survey instrument included five numerical ratings, three yes/no responses, and two short response fields, and a text field for general comments. The numeric rating fields allow responses from 0 through 9. Each scale was labeled to indicate the meaning of the numeric selection.

Four of the numeric questions probe at the level of satisfaction with and loyalty to the company or organization that provides its current automation system:

• How satisfied is the library with your current Integrated Library System?
• How satisfied is the library overall with the company from which you purchased your current ILS?
• How satisfied is this library with this company’s customer support services?
• How likely is it that this library will purchase its next ILS from this company?

A yes/no question asks whether the library is considering migrating to a new ILS and a fill-in text field provides the opportunity to provide specific systems under consideration. Another yes/no question asks whether the automation system currently in use was installed on schedule.

Given the recent interest in new search interfaces, a yes/no question asks “Is the library currently considering a search interface for its collection that is separate from the ILS?” and a fill-in form to indicate products under consideration.

The survey includes two questions that aim to gauge interest in open source ILS, a numerical rating that asks “How likely is it that this library would consider implementing and open source ILS?” and a fill-in text field for indicating products under consideration.

The survey concludes with a text box inviting comments.

View the survey. (This version of the survey does not accept or record response data.)

In order to correlate the responses with particular automation systems and companies, the survey links to entries in the lib-web-cats directory of libraries. Each entry in libraries.org indicates the automation system currently in use as well as data on the type of library, location, collection size, and other factors that might be of potential interest. In order to fill out the survey, the responder had first to find their library in lib-web-cats and then press a button that launched the response form. Some potential respondents indicated that found this process complex.

The link between the lib-web-cats entry and the survey automatically populated fields for the library name and current automation system and provided access to other data elements about the library as needed. The report on survey response demographics, for example, relies on data from libraries.org.

A number of methods were used to solicit responses to the survey. E-mail messages were sent to library-oriented mailing lists such as WEB4LIB, PUBLIB, and NGC4LIB. Invitational messages were also sent to many lists for specific automation systems and companies. Where contact information was available in lib-web-cats, and automated script produced e-mail messages with a direct link to the survey response form for that library.

The survey attempted to limit responses to one per library. This restriction was imposed to attempt to sway the respondents to reflect the broad perceptions of their institution rather than their personal opinions.

The survey instrument was created using the same infrastructure as the Library Technology Guides web site—a custom interface written in perl using MySQL to store the data, with ODBC as the connection layer. Access to the raw responses is controlled through a user name and password available only to the author. Scripts were written to provide public access to the survey in a way that does not expose individual responses.

In order to provide access to the comments without violating the stated agreement not to attribute individual responses to any given institution or individual, an addition field was created for “edited comments.” This field was manually populated with text selected from the “comments” text provided by the respondent. Any information that might identify the individual or library was edited out, with an ellipse indicating the removed text. Comments that only explained a response or described the circumstances of the library were not transferred to the Edited Comments field.

Statistics

To analyze the results, a few scripts were written to summarize, analyze, and present the responses.

In order to avoid making generalizations based on inadequate sample sizes, the processing scripts included a threshold variable that would only present results when the number of responses exceeded the specified value. The threshold was set to a value of 20.

For each of the survey questions that involve a numeric rating, a set of subroutines was created to calculate and display simple statistics.

• Responses indicates the number of survey responses that made a selection for this question.
• A Response Distribution array lists the number of responses for each possible value from 0-9.
• The Mode indicates the numeric response that receive the most selections.
• The Mean is the average response, calculated by adding together all the responses and dividing by the Responses value, rounding to two significant decimal places.
• The Median is the middle response, calculated by placing each of the responses in a sorted array and selecting the middle value.
• The Standard Deviation was calculated by subtracting each response value from the mean, squaring the difference, summing the squares and dividing by the number of responses to determine the variance. The standard deviation is the square root of the variance.

The "report-by-category.pl" script processes each of the numerical ratings, displaying each of the statistical components listed above for each product that received responses above the threshold value. This report provides a convenient way to compare the performance of each ILS product for the selected question. The report sorts the statistics for each product in descending order of the mean. The report categories available correspond to the survey questions with numerical scale responses.

The “product-report.pl” script provides the results for each of the ILS products mentioned in the responses. This report also provides the statistical components for each of the numeric question. It also provides the percentage of yes responses to the two yes/no questions:

• Is this library currently considering migrating to a new ILS?
• Is this library currently considering acquiring a search interface for its collection that is separate from the ILS? The report then lists the “EditedComments” text.

[The text of this section mostly replicates what appeared in the 2007 version of this article. For for both editions of the survey I followed the same methodology for collection and and statistical analysis.]

---

Caveat

As I noted with previous editions of the survey, one should not read too much into the survey results. Responders to the survey provide their subjective impressions to fairly general questions. Although the survey instructions encourage responders to consider the broader institutional perceptions, it’s usually the case that multiple opinions prevail within any given library. While I believe that this survey does provide useful information about the experiences of libraries with their current integrated library systems and the companies that provide support, it should not be used as a definitive assessment tool.

Maintained by Marshall Breeding Copyright 1999-2023