Web Magazine for Information Professionals

The JISC User Behaviour Monitoring and Evaluation Framework

Jenny Rowley introduces the JISC User Behaviour Monitoring and Evaluation Framework.

The JISC User Behaviour Monitoring and Evaluation Framework (hereafter the Framework) is an ambitious project to design and use a research framework that monitors and maps the development of user behaviour with electronic information resources in higher education. Initially approved for three years, the Framework is currently in its third and final annual cycle. The methodology and findings relating to the first and second cycles are located in the First Cycle and Second Cycle Annual Reports, available through the JISC web site. Aspects of the work conducted under the Framework have also been disseminated variously through conference papers, and professional and academic journal articles.

The concept of the Framework was sponsored by JISC’s Committee on Awareness, Liaison and Training in collaboration with the JISC Committee on Electronic Information. The Framework has three strands which are managed through the two projects: JUSTEIS and JUBILEE, as described below. In the second cycle pilot studies were also conducted under each of the strands in further education and these will be expanded to full studies with both ‘survey’ and action research components during the Third Cycle.

Framework Projects

JUSTEIS (JISC Usage Surveys: Trends in Electronic Information Service)

Strands A and C of the Framework were contracted to the Department of Information and Library Studies at the University of Wales, Aberystwyth.

Under the JUSTEIS project, Strand A is an annual survey which seeks to measure and evaluate the overall awareness, uptake, usage and usefulness of information technologies and information services in higher education in the United Kingdom. The survey was conducted by telephone interview, e-mail and paper-based questionnaire. Several hundred interviews have been conducted with students and staff drawn from institutions and subjects across the entire higher education spectrum. The interviews use a critical incident technique to investigate the types of EIS that are accessed, approaches to searching, and success in searching, and the reasons for use of the EIS. A taxonomy of EIS is used to define the types of EIS covered.

Strand C is a general survey of EIS provision, with the aim of developing profiles of current and planned service provision. The basis for this survey is a Web survey of resources access provided by individual HEI’s. This has been supplemented by telephone interviews with senior LIS managers to explore purchasing determinants and intentions.

JUBILEE (JISC User Behaviour in Information Seeking: Longitudinal Evaluation of EIS)

Strand D of the Framework has been contracted to The Information Management Research Institute, School of Information Studies at the University of Northumbria at Newcastle. The JUBILEE Project team have completed the Second Cycle of a linked programme of ongoing, longitudinal monitoring of the information behaviour, needs and opportunities for specific academic and students communities and for academics and students in general. This ongoing programme of qualitative monitoring centres around a selection of actual (HEI’s, Departments) and virtual (discipline/subject communities/networks) ‘sites’, with an annual reporting cycle. Six institutions and three disciplines are studied per cycle. The focus of monitoring changes over time. For the first cycle the disciplines were: Health Sciences, Business Studies and English. Second Cycle disciplines were History, Computing and Sociology. Research methods have included questionnaires, interviews, focus groups, electronic communication (Chat room and observatory) and feedback on case study reports.

The Development of the Framework

Work on the Framework has two components:

As indicated in Table 1 the key focus in Cycle 1 of the Framework was to design and test the methodology. By Cycle 2 the data sets start to be sufficiently well populated to increase confidence both in the methodology and in findings. Cycle 3 will continue with population of data sets, but also focuses more strongly on integration as a basis for a holistic set of deliverables.


Key Activities

Level of Integration


Design and testing of project data collection and access methodologies, leading to the generation of frameworks for understanding the data at project level

Loosely linked set of project with an overarching set of objectives.


Further refinement of data collection and access methodologies. Further population of data sets.

Partial integration based on enhanced understanding of the relationships between projects.


Further population of data sets

Refinement of project data collection, and analysis to generate a holistic picture of user behaviour

Full integration, evidenced through a holistic set of deliverables.

Table 1 : Framework Cycles

Profiling User Behaviour

The findings from the Second Cycle replicate those from the First Cycle to a significant extent. These are:

Author Details


Jenny Rowley