User studies on end-user service composition: A literature review and a design framework

Context: End-user service composition (EUSC) is a service-oriented paradigm that aims to empower end users and allow them to compose their own web applications from reusable service components. User studies have been used to evaluate EUSC tools and processes. Such an approach should benefit software development, because incorporating end users' feedback into software development should make software more useful and usable. Problem: There is a gap in our understanding of what constitutes a user study and how a good user study should be designed, conducted, and reported. Goal: This article aims to address this gap. Method: The article presents a systematic review of 47 selected user studies for EUSC. Guided by a review framework, the article systematically and consistently assesses the focus, methodology and cohesion of each of these studies. Results: The article concludes that the focus of these studies is clear, but their methodology is incomplete and inadequate, their overall cohesion is poor. The findings lead to the development of a design framework and a set of questions for the design, reporting, and review of good user studies for EUSC. The detailed analysis and the insights obtained from the analysis should be applicable to the design of user studies for service-oriented systems as well and indeed for any user studies related to software artifacts.