CENDI PRINCIPALS AND ALTERNATES MEETING
Department of Commerce
National Technical Information Service
Washington, DC
May 15, 2007
Minutes
Metrics
Metrics and Evaluation of Website Usage: The State of the Internet
Assessment, Evaluation and Value: Current Trends
NTIS Showcase
Metrics and Evaluation of Website Usage: The State of the Internet
(Lara Fisher, Director, Marketing Solutions, comScore Networks)
comScore tracks the online activity of one million Internet users in the U.S. and one million outside the U.S. They track only the activity of people who agree to participate. The users load a client on their machines and comScore is able to “watch” everything. The clickstreams are captured and the private information is stripped. Unlike web logs, this approach targets a natural population, and they see the aggregate usage across transactions that are included in a number of web logs. This also eliminates the impact of spiders, crawlers and bots; the comScore approach is more reflective of actual usage by people. There are some possible gaps, including public access in libraries and Internet cafes.
comScore’s results are provided in a worldwide context. The online population outside the U.S. has skyrocketed at five times the growth inside the U.S. There was a 12.9 percent increase from 2005-2006. Seventy-six percent of the people online are located outside the U.S., but a lower percentage outside the U.S. use health or government resources.
In terms of the usage of government sites, there was a 33 percent growth for NIH sites over the last year. There was also substantial growth at the Department of Education and the Internal Revenue Service (IRS). Not one particular group (heavy, medium or light users) is dominating the use of government or health sites.
As of December 2006, comScore is basing their U.S. data on a projected 174 million adults over 18 years of age. The characteristics of their one million panelists are matched against Commerce and Census data to ensure a representative cross section. They are able to project out to the 174 million users by weighting their panelists based on demographic characteristics. The comScore client can be loaded on home, work or school machines, which creates a gap in following a person’s individual usage across these environments.
In December 2006, the average U.S. user was online 22 days during the month for a total of 29 hours. They viewed over 2500 pages. (A visit is defined as at least one minute in duration and it ends if there is inactivity for more than 30 minutes.) Thirty-six percent of the online time was spent at communication sites such as e-mail and instant messaging. Eighty-four percent of these users did at least one search; the average user conducted 41 searches. Home machines were up 19 percent in December. E-commerce spending was up 20 percent from 2005-2006. Non-travel spending was $102 billion, an increase of 24 percent, with the average buyer spending $224 in December 2006.
The Internet is increasingly the preferred source for most types of information. Sports and grocery product information are still dominated by other media. The usage is very event driven. For example, the increased visits to nasa.gov and noaa.gov can be correlated to specific events.
There is an increased interest in social network (Web 2.0-type) content such as wikis and blogs. Streaming is represented across all demographic segments. Streaming presents good opportunities for messaging because viewers can be targeted by location or content. This approach to marketing works!
Forty-six percent of the population engaged in eCommerce in December 2006. This is up 36 percent from December 2005. The largest growth was in non-travel sales. Government sites such as irs.gov are clearly driving traffic to other commercial sites to which they link.
In a comparison of sponsored versus algorithmic clicks, sponsored were up 37 percent from 2005 to 2006. Algorithmic clicks (those based on the search results) are up 31 percent. Twenty percent of the online search population makes up 62 percent of the searches. However, these heavy searchers aren’t more likely than medium or light searchers to delve into a page of results more deeply.
In an analysis of heavy, medium, and light searchers per month, heavy users are only slightly more likely to visit health or government sites. Some conditions, such as flu and cold, are seasonal. For some health conditions, such as “weight gain” and “quit smoking,” people conduct many searches using the same keywords, but the results aren’t clicked. People are seeing the results but they are not inclined to click on individual results, probably because they aren’t getting what they expect.
Assessment, Evaluation and Value: Current Trends
Rosalind Dudden, Library Services Director, Gerald Tucker Memorial Medical Library, National Jewish Medical and Research Center
Ms. Dudden began her book, “Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement and Library Standards,” (Neal-Schulman, 2007) in response to library cuts. Her work has been funded by the National Library of Medicine’s Grant for Scholarly Works in Biomedicine and Health since October 2004. While there has been a great deal of work done by the Association of Research Libraries (ARL) and others on assessment and evaluation techniques and metrics for large libraries, there is a need to help librarians in small settings with a step-by-step approach to the use of metrics. No previous book seemed to do what was needed. This how-to book will be published in July 2007. It contains three essay chapters on the theory of evaluation, along with workbook chapters on each of the topics and tools for doing the evaluations.
The practice of evaluation in libraries owes much to F.W. Lancaster’s “The Measurement and Evaluation of Library Services” published in 1977. This type of evaluation was geared toward management and used by head librarians. However, the results were unlikely to be noticed by those outside the part of the library being evaluated. During this period, libraries were generally supported, so there was little impetus to evaluate more thoroughly if the library had a basic value to the society or the corporation. To assure or perhaps prove quality, libraries became involved in the quality improvement trends in the 1980’s and 1990’s. Library associations, such as the Association of Research Libraries (ARL), started collecting comparative data on library operations. The Medical Library Association (MLA) began a similar benchmarking activity for non-academic health sciences libraries with surveys in 2002 and 2004 with another planned for late 2007.
In the 1980’s, libraries began to feel the paradigm shift from the printed word to other media. By the 1990’s, the impact of the Internet on library operations was almost unprecedented. Technology has had a major impact, particularly with regard to the integration of all types of information. This integration is industry wide and is reflected in electronic health records, industry-wide portals and federated search engines that aggregate databases. This integration complicates the assessment and evaluation process.
There has also been a paradigm shift away from the engineering model of the 1900’s, which was very linear. In this century, the paradigm is one of biology, which is one of diffusion, webs, volatility, and adaptability. These paradigms have an impact on how evaluation might be done. The approaches are still somewhat linear but with more circular feedback loops.
User behavior has become more multi-dimensional and information resources are used in a multiplicity of ways. Traditional library resources are only one avenue to gain information. Other avenues increasingly involve connecting with experts via e-mail, browsing the Internet, locating blogs and wikis of interest, and sending inquiries to ask-a-services and friend-of-a-friend networks. This complex web of relations provides a richer context for the impact libraries are making. The user doesn’t really respond to the linear evaluation model anymore, and the use of the usual linear model of input, activities, output, and outcome, is limiting our understanding of a model that is much more cyclical, haphazard, and uncontrolled. This model becomes even more complex when you introduce the idea of motion for both the user and the changing information resources. Such an idea creates a model that looks like a spiral that swirls up and down. It becomes a challenge to tell what is affecting what.
In this environment, the key question is “What do we measure?” Factors that can be measured include needs, inputs, quality processes, outputs, quality of service, outcomes, and impacts. The measurement of these factors is complicated by the fact that evaluation is a “semantic jungle” in which there is little agreement on definitions of key terms. Ms. Dudden gave definitions for many of these factors, realizing that they are often dependent on the context.
Measuring inputs, such as the size of the staff and the number of e-journals, tells only part of the story, but people expect and they generally understand these numbers. The traditional input counts are needed for good management, budgeting and resource allocation, and comparative benchmarking. However, numbers reflect the input and what really matters is what people do with the information. She recommends a blended approach.
This includes measuring the quality of processes using Total Quality Management (TQM) and quality improvement techniques. Measuring outputs answers the question “What do you do?” These measures can be used in communications to stakeholders. They also help identify trends and areas where further evaluation is needed. Measuring service quality is supported by the LibQual suite of services from ARL. The program’s core is a Web-based survey. However, for small settings, this approach is expensive and it is hard to get users to answer all the questions. Small libraries could use a simple satisfaction survey on SurveyMonkey, for example.
Measuring outcomes started in 1993 with the Government Performance and Results Act (GPRA), which required most federal agencies to develop objective, quantifiable, and measurable goals and to report on their achievements. The influence of this law spread to local governments and other organizations. Outcome measurement is promoted by the Institute for Museum and Library Services (IMLS), which supports library development through grants. The IMLS has funded two web-based toolkits to help libraries measure outcomes. This activity has become more important over the last 15 years for public libraries where there is increased political pressure to prove their worth in competition for tax dollars.
However, outcomes and impacts are farther removed in time (and perhaps space) for academic and health libraries. For health libraries, several resources were mentioned, including MLA’s Vital Pathways Task Force slide show that could be modified to help libraries talk to their administrators about the true value of libraries and library services. An approach that seems to be successful, called “Myths and Truths about Library Services,” poses a myth, provides a truth statement that is supported by research, quotes the source of the research, and then uses locally gathered information based on a small survey to report on the outcomes at the local level. Telling a specific story or anecdote is very powerful, because that is what many administrators remember more than the data.
The “Culture of Assessment” brings it all together. This is a movement within ARL to create systemized structures and a customer focus. Assessment occurs at all stages from strategic planning to daily operations. In support of this type of culture, library management Information systems need to be developed to collect and report on library information in support of assessment. The University of Arizona Is working on a prototype called PEMS.
Practicing librarians in small settings need to be generalists. They need to learn to do evaluations without being experts. It is important to tie the library planning to the organization, to focus on customers, to establish a culture of assessment, to use a variety of measures, both old and new, and to use outcomes to tell the story. Ms. Dudden’s full bibliography and a copy of the PowerPoint is available at http://info.nationaljewish.org/libraryinfo/
Discussion
There was a wide ranging discussion about other resources and approaches, particularly for economic model development. Rate of Return (ROI) is one approach. Another is to look at alternative costs of not having libraries. Jose Marie Griffith used a regional economic model in her recent Florida studies which will soon be reported in the literature. There are several very stable and well-tried economic models within the Department of Agriculture and the Corps of Engineers that might also apply.
NTIS Showcase
Ellen Herbst
NTIS is tasked with collecting, organizing, and disseminating scientific, technical, and business information from across the federal government to strengthen the U.S. economy and promote economic growth. One enabling piece of legislation is the ATPA. Several years ago, they were charged with helping agencies to disseminate information. This resulted in a new set of services. NTIS is a fee-based organization. It has been self-sustaining for many years. There are currently about 147 employees and a few contractors, with facilities in Springfield, Virginia, and a warehouse in Brandywine, Maryland.
NTIS has three methods for collecting the information. 1) A human acquisition effort is used. Increasingly, these individuals are talking to agency webmasters in the agencies. 2) They perform manual harvesting and electronic harvesting using NTIS-developed tools. 3) They have had long-term relationships with the major science agencies, and working closely with DTIC, NASA, DOE and EPA. All information collected is unclassified.
NTIS approximately 40,000 products annually, including reports, international technical reports, conference proceedings, journal articles, non-print, and multimedia. Approximately 95 percent of the materials are received electronically. The average electronic document is 138 pages. The database is made up of COSATI (Committee on Scientific and Technical Information) metadata records.
Last year, users requested approximately 12,000 copies of reports in paper. In addition, libraries continue to want fiche, despite various educational campaigns to emphasize the benefits of moving from fiche to CD. NTIS also uses fiche as a backup medium, providing a set of fiche to NARA as the deep archive. The continued maintenance of the Computer Output Microfilm (COM) devices is increasingly difficult. This is a concern which is shared by many of the other CENDI agencies.
There are numerous services provided to agencies, including customized ones. The services range from doing some or all of the distribution processes, including the online web site to physical mailing. They are a registered e-Training Service provider for the Office of Personnel Management (OPM). A prominent e-government initiative was the Wage Determination Online system, which is now being maintained by NTIS.
They have authority to work with the private sector in joint projects and to share in the results. For example, the World News Connection is a major, long-standing partnership with Dialog.
There are several key NTIS initiatives. The Strategic Planning Process, begun in 2005 and continuing, is designed to update understanding of the information environment and determine what NTIS needs to change to. The process has led to the development of Strategic Roadmap Projects There are more than 12 active projects and about five to six projects that are awaiting resources. They range from assessment of the input processing system to a review of human capital and the retirement rate.
Of key interest is the way that NTIS can increase the value it provides. They must optimize the value in existing products. Approval is being sought to run a survey in order to learn what customers want. They will survey what users need that they aren’t getting now.
Enhancing “findability” (rather than “searchability”) involves projects on short-term web site redesign, long-term redesign, new search functionality and looking for additional distribution channels. In all cases, they are conducting experiments and pilots. One pilot that challenges previous assumptions about their business model involves making 240,000 titles available to the Federal Depository Library Program (FDLP) for free. So far, 300 libraries have signed up. The feedback, to date, is very positive. The early signs are that increased visibility will increase the interest and use of NTIS products. NTIS is being more proactive in working with search engines and will be starting a pilot to make a small portion of the database searchable by search engines.
In terms of collecting material, NTIS has been doing harvesting for almost 10 years. In the short term, they are investigating alternatives for better acquisition. In the long term, they want to tie this more closely to the abstracting and indexing processes.
In 2006, NTIS spent time on the digitization of the collection. Given the size of the task, NTIS can not fund the effort by itself, but is looking into ways to work with partners to achieve digitization as a major initiative. NTIS does some scanning to meet customer requests but this effort will not result in a fully digitized collection in the foreseeable future. The digitized part of the collection goes back to 1995.
NTIS continues to hold its value and it is growing on the collection side. The subscription-type product, which provides output by category every two weeks, remains very popular. However, NTIS is part of the “long tail” with more than 50 percent of the requests being for materials that are more than seven years old, and 50 percent of the collection is ordered only once. They have tried several approaches to identify a “best seller” in advance with limited success. NTIS is now focused on identifying new value propositions for its customer base.
