L643: Evaluation of Information Systems

L643: Evaluation of Information Systems

S519: Evaluation of Information Systems Analyzing data: value and importance Ch6+7 Values in evaluation (DCh6) Adding value to descriptive data to make our evaluation explicit Our goal Using quantitative value to evaluate the quality or value of the evaluand in a particular context. Build up our conclusions based on a level of

certainty What are values: good, valuable, worthwhile Values in evaluation Adding values to descriptive data collected about Process, outcomes, costs, comparisons, exportabilities; or

Situated dimensions or components Weighting all the strengthens or weaknesses of these values to draw overall conclusion about the evaluand. How Importance weighting Merit determination synthesis methodology Subjective Before we go for methodology, we have to answer the question:

Whether our data are subjective Three types of subjectives (Scriven, 1991) Subjective 1: Inappropriate application of personal or cultural preferences/biases: Subjective 2: assessment or interpretation by a person, rather than guidelines

arbitrary, idiosyncratic, unreliable, highly personal (i.e., based purely on personal preference, cultural biases, gender biases) Using well-founded expert judgments Robust evidence Subjective 3: about a persons inner life or experiences (e.g., headaches, fears, beliefs, emotions, stress) Usually not independently verifiable Avoid Subjective We provide our conclusion based on certainty in the relevant decision-making context

Keep the whole evaluation well documented and justified All evaluations, especially high-stakes ones, should be meta-evaluated (i.e., evaluation itself should be evaluated) Determining importance (Dch7) Importance determination is the process of assigning labels to dimensions or components to indicate their importance. Importance weighting

Prioritize improvements Identify whether identified strengths or weakness are serious or minor Work out whether an evaluand with mixed results is doing fairly well, quite poorly, or somewhere in between. Determining importance (Dch7) Different evaluations Dimensional evaluation Looking at multiple dimensions of merit that pertain to the evaluand

as a whole rather than separately to its parts. Component evaluation Looking at each of the evaluands components (or parts) separately and then synthesizing these findings to draw conclusion about the evaluand as a whole. Each component can be evaluated on several dimensions that pertain to this component only rather than to the evaluand as a whole. Holistic evaluation Looking evaluation as a whole without division into dimensions or components When to use what Component analysis

Evaluating policies, programs, or interventions that have several quite distinct parts An international program consisting of projects implemented in different locations (e.g. WIC in IU) A government policy includes multiple policy measurements (e.g. Juvenile delinquency) An organizational transformation includes several distinct interventions (e.g. Career support) When to use what Dimensional

evaluation Entities whose quality or value is experienced by consumers on multiple dimensions that pertain to the evaluand as a whole Product evaluation (i.e. Car evaluation) When to use what Holistic evaluation Unusual in the evaluation of programs, policies

and other large complex evaluands. More common in personnel, product and service evaluation (expertise-oriented evaluation) Judging the overall quality of a sample of writing Grading essays Classroom teaching Athletic performance cosmetics Determining the importance of dimensions Determining the importance of

componentss Determining importance Weak performance on minor criteria (e.g. dimensions, components) may be no big deal, But weak performance on important criteria can be very serious issues. Determining importance: 6 strategies 1. having stakeholders or consumers vote on importance

Commonly used in both participatory and nonparticipatory evaluations Collecting opinions from everybody Assumptions Each person is well informed Stakeholders belief what (s)he chooses is important Stakeholders important should be treated equally Pros and cons?

Determining importance: 6 strategies 2. Drawing on the knowledge of selected stakeholders Using selected stakeholder input to guide the assignment of importance weightings Collecting opinions from selected experts Setting up the Bars A bar is a defined minimum level of criterion performance below which the evaluand is considered

completely unacceptable, regardless of performance on other criteria. Determining importance: 6 strategies 2. Drawing on the knowledge of selected stakeholders Assumptions: The stakeholders should be sufficiently well informed to provide valuable relevant information The combination of stakeholder input will provide

sufficient certainty about importance for the given decision-making context Pros and cons? Determining importance: 6 strategies 3. Using evidence from the literature

Literature review Evaluations of similar evaluations in similar contexts Research documenting the key drivers (or strongest predicators) of success or failure with this type of evaluand. Assumptions The volume and quality of the available research is sufficient to judge the importance The context of other research is sufficiently similar to yours and therefore that the findings can be reasonably applied to your setting Pros and cons? Determining importance: 6

strategies 4. Using specialist judgment When you have tight timeline, no time for gathering stakeholders and looking for literature Identify one or two (or two or more) well-known specialists in the domain Better be supplemented with other evidence Pros and cons?

Determining importance: 6 strategies 5. Using evidence from the needs and values assessments Determining the importance of criteria (dimensions) Any frequently mentioned characteristics? Looking for poor-performing evaluators that cause serious problem Looking for top-notch evaluators that have dramatic

impacts on success Determining importance: 6 strategies 5. Using evidence from the needs and values assessments Determining the importance of components Severity of dysfunction addressed (primary consideration) Scarcity of alternatives: no other options for addressing the need.

Intent to use alternatives: if the evaluand component in question did not exist. Rubrics to measure (Table7.4, 7.5 and 7.6, 7.7, 7.8 (combined)) Prons and cons? Determining importance: 6 strategies 6. Using program theory and evidence of causal linkages When criteria or components are linked to needs through a complex logic chain.

Such as soft skills or attributes (e.g., inspirational leadership, selfesteem, stress management, a kind of instrumental needs) More upstream variables (see Exhibit 7.5) How to estimate the strengths of the links Interview Analyze your previous data ... Strengths and weaknesses of the six strategies

Always think whether they are applicable Choose mulitple of them Exercise Table 7.10 (class dissusion) Form a group Discuss which strategies you will choose to determine the importance for the student services in the school health program (see Table 7.8) Discuss which strategies you will choose to

determine the importance for your group project

Recently Viewed Presentations

  • Telemedicine in Maldives - 123seminarsonly.com

    Telemedicine in Maldives - 123seminarsonly.com

    Telemedicine in Maldives. Vision: Provide a telemedicine solution to Maldives which is compatible, scalable, reliable and inter-operable. The objective is to upgrade the quality of health care and minimize the cost of medical care through effective acquisition of relevant clinical...
  • ASE 2020 GCSE science networking hubs - filestore.aqa.org.uk

    ASE 2020 GCSE science networking hubs - filestore.aqa.org.uk

    ASE Annual Conference: The GCSE Science Hub network. Jane Bryant. Spring 2020. 11/29/2019
  • The Bride Comes to Yellow Sky - fju.edu.tw

    The Bride Comes to Yellow Sky - fju.edu.tw

    " The Bride Comes to Yellow Sky Outline 1. Main & Minor Characters 2. Setting 3. Plot -turning points 4. The Climax: the encounter Main & Minor Characters How are the bride and the groom presented here? Are they a...
  • Engineering multilingual eCommerce on the WWW

    Engineering multilingual eCommerce on the WWW

    3.10.95. Semantic Web Tools Paula Silvonen VTT Information Technology www.vtt.fi/tte
  • CABLE SYSTEMS - Archi-fied!

    CABLE SYSTEMS - Archi-fied!

    The thrust is inversely proportional to the sag; halving the sag doubles the thrust. This raises an interesting question of economy through. OPTIMAL SAG : A large sag increases the cable length, but reduces the tensile force & allows a...
  • India's Achievements

    India's Achievements

    Artists from both of India's major religions, Hinduism and Buddhism, drew on their beliefs to create their works. As a result, many of the fi nest paintings of ancient India are found in temples. Hindu painters drew hundreds of gods...
  • Sunshine Health Plan - HomeTown Health University

    Sunshine Health Plan - HomeTown Health University

    Facility. Address. City. Zip. Phone. County. Region. Bay Medical Center . 615 N Bonita Ave. Panama City. 32401. 850-769-1511. Bay. 2. Sacred Heart Hospital on the ...
  • Social Discount Rate - University of Arizona

    Social Discount Rate - University of Arizona

    Social Discount Rate But in reality not a single interest rate for all savers and investors Many factors affect interest rates. What factors? Transactions costs Differences in risks Different time horizons Expectations of future - inflation Policies - controls on...