Page - 37 - in Applied Interdisciplinary Theory in Health Informatics - Knowledge Base for Practitioners

Evaluation can take place at each of these steps in the value chain, but it is not correct to assume that a good result at one step necessarily translates into a good result at the next. Nor should we even desire that improvements at one step in the chain flow downstream. For example, a new telecommunication system between a primary care physician and patients at home may allow a doctor to talk to their patients without the need for a physical visit. We might be able to demonstrate high system utilization and user satisfaction with this telehealth system, but also be surprised to find that there is no significant change to the survival or quality of life for patients. Why might this be so, and should we consider such a system a ‘failure’? There are many reasons why benefits at one step in the value chain do not manifest in later steps. Sometimes a technology intervention behaves as a substitute for an existing service process, but does not to improve it. So, when the quality of normal care is already of a high standard, any telehealth substitute for face-to-face interaction should aim to be non-inferior i.e. be no worse. All that we are doing is replacing face-to-face interactions with online ones. If the goal of this telehealth system were only to reduce the need for a patient to travel to the office, then demonstrating a cost-effective reduction in such visits (once we add in the costs of the telehealth system) would be considered a success. There should be no expectation that benefits at the initial interaction stage of the chain translate to clinical outcomes. We should just be mindful to not see any deterioration in outcomes. 1.2. Different evaluation measures may be used at different steps in the value chain Which stages of the value chain are formally evaluated will depend on the type of system in question, and the purpose of the evaluation. Unsurprisingly, the processes that are studied, and their related measures, can vary with both the step in the chain, and the type of system being developed (Table 1). For example, we might evaluate the quality of interaction with an information retrieval search engine using metrics for the ease with which a query can be formulated to retrieve relevant information, whilst measuring the quality of a telehealth interaction would perhaps focus on the quality of the video call, the rate of technical disruptions to the call, or a user’s perceived satisfaction with the call. 1.3. The value of information can be quantified Value chain analysis makes clear that creating and accessing information alone does not always lead to a change in process or clinical outcome. We know from Shannon’s Information Theory that not every additional piece of data is as informative as another [3]. The amount of Shannon information is a measure of how “surprising” new data are compared to our expectation. If data do not tell us anything new, they bring little or no new information. Another way of thinking about this is to ask how many times information must be read before there is a measureable impact on clinical outcomes. Metrics such as the number needed to read [4] and the number needed to benefit from information [5] are related attempts to correlate access to information such as clinical guidelines with their impact on process or outcome. E.Coiera /AssessingTechnologySuccessandFailureUsing InformationValueChainTheory 37