Whats EPB? Whats so good about RCTs? (And what are we reading?) Philosophy of Social Science Phil 152 Winter 2011 Week 8 A theory of evidence for use Foundation for a guide by C&H for the use of evidence in evaluating policy effectiveness. The (C&H) guide should be Simultaneously well-grounded and practicable. More comprehensive than those currently available For policy-makers not expert in natural and social
science with limited amounts of time and resources. 2 The question of effectiveness Will the proposed policy produce the targeted outcomes were it to be implemented in the targeted setting in the way it would in fact be implemented there? So, whats evidence good evidence for answers? 3
The rise of evidence-based policy In the UK, USA and increasingly in Europe we see a huge drive to use evidence to inform policy and practice mandated by executive and legislative branches at international, national and local levels pushed by national and international organisations like the Campbell and Cochrane Collaborations with accompanying institutions and regulations to ensure evidence is appropriately considered. 4 Some guides for use of evidence to
judge policy effectiveness IARC: International Agency for Research on Cancer SIGN: Scottish Intercollegiate Guidelines Network What Works Clearinghouse USEPA: US Environmental Protection Agency
CEPA: Canadian Environmental Protection Act Cochrane Collaboration Oxford Centre for Evidence-Based Medicine Daubert decision!! (US Supreme Court) 5 For an evidence-based policy We want Evidence of high quality That speaks for the policy. 6
High Quality Evidence We do not want to build an argument for a policy on shaky premises. 7 High Relevance Evidence No matter how sturdy this foundation: It wont support these houses: 8
High Relevance v High Quality ____________________________________________ Evidence The Weight of Argument When do symmetry arguments provide the best evidence? When are detailed calculations found more convincing? When does modeling appear conclusive? For how much does societal acceptance account? (From Martin Harwit, lecture, 12 Nov 2007, IAS Durham. Ital added.) _____________________________________________ not QUALITY: The likely truth of evidence. but RELEVANCE: Evidence that supports the conclusion.
9 High Quality Evidence We want to admit as evidence only claims that are true, or highly probable: P(e) is high. Ranking schemes rank evidence according to the method by which it is produced. Top-ranked evidence is produced by methods that make it likely that the result is true: P(e) is high. 10 Evidence-ranking Schemes: SIGNS (Scottish
Intercollegiate Guidelines Network) 11 Levels of evidence 1++ High quality meta analyses 1, systematic reviews2 of RCTs3, or RCTs with a very low risk of bias 1+ Well conducted meta analyses, systematic reviews of RCTs, or RCTs with a low risk of bias 1 - Meta analyses, systematic reviews of RCTs, or RCTs with a high risk of bias 2++ High quality systematic reviews of case-control or cohort studies High quality case-control or cohort studies with a very low risk of confounding, bias, or chance and a high probability that the relationship is causal
2+ Well conducted case control or cohort studies with a low risk of confounding, bias, or chance and a moderate probability that the relationship is causal 2 - Case control or cohort studies with a high risk of confounding, bias, or chance and a significant risk that the relationship is not causal 3 Non-analytic studies, e.g. case reports, case series 4 Expert opinion 12
The RCT RCTs establish causal conclusions: In Tennessee reducing class size caused better reading scores. In two Zambian hospitals introducing tri-co increased survival rates in HIV-positive children. If properly done, they guarantee That there was the difference indicated. That it was caused in the way indicated (and not, e.g., by accident or by some other factor introduced at the same time). The method itself makes it very probable that the resulting conclusion is true.
13 Whats an RCT for T causes O? A Mills method-of-difference study: 2 groups where all causal factors for O are distributed the same except for T and its downstream effects. T is universally present in the treatment group, universally absent in the control. If ProbT(O) > ProbC(O), T must be responsible for the difference. SO: T was a contributing cause towards O in at least some members of study population.
14 Whats special about RCTs? They clinch causal conclusions: If the assumptions for the study are met and ProbT(O) > ProbC(O), it follows deductively that T causes O in some study members. They are self-validating: Blinding (quadruple maybe) Random assignment Placebo control aim to make the distribution of other factors the same in treatment and control groups.
15 Other methods Can be clinchers but none are self validating: they require substance-specific assumptions. The assumptions of the RCTs are supposedly all based on method. Econometric modelling Process tracing Derivation from theory Some methods merely suggest a conclusion or vouch for it. Causal relations from analogue systems Observational studies.
Standard advice guides tell you to ignore other clinchers and vouchers. 16 Daft advice HARD WON KNOWLEDGE 17 The RCT The RCT can produce high quality claims claims we have good reason to judge true that we may adduce as
evidence. But what are they evidence for? Thats our second requirement. We want high quality claims that speak for the policy. Showing that a claim is very likely true goes no way to showing that it is relevant to the truth of our policy hypotheses. 18 Turn now to Relevance How do we decide which evidence supports which conclusions?
19 US Dept of Education Website Strong evidence for your policy = two or more high quality RCTs in settings similar to that of your schools/classrooms. Later elaboration adds 4 lines trials on white suburban populations do not constitute strong evidence for large inner city schools serving primarily minority students. 20 SIGNS Grades of recommendation
A B C D At least one meta analysis, systematic review, or RCT rated as 1++, and directly applicable to the target population; or A systematic review of RCTs or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population, and demonstrating overall consistency of results
A body of evidence including studies rated as 2++, directly applicable to the target population, and demonstrating overall consistency of results; or Extrapolated evidence from studies rated as 1++ or 1+ A body of evidence including studies rated as 2+, directly applicable to the target population and demonstrating overall consistency of results; or Extrapolated evidence from studies rated as 2++ Evidence level 3 or 4; or Extrapolated evidence from studies rated as 2+ 21 Practical advice about relevance??? Solook for high quality evidence (1++) that
is directly applicable to the target population!!!! For better advice read Cartwright & Hardie 22 23 RCTs cannot hand evidentiary support directly to effectiveness claims
24 The Theory and the Cakes it Calls For Philosophy 152 Philosophy of Social Science Week 9 2011 Causes are INUS conditions INUS condition: an Insufficient but Necessary part of an
Unnecessary but Sufficient condition. Unnecessary Theres more than one way to skin a cat. Many different causes can produce contributions to the same effect. No single one of them is necessary to get a contribution. Some will be negative. California class-size reduction programme Important side lesson: Beware what you do in implementation. Sufficient
Sufficient = enough. A bundle of factors is sufficient for a contribution means that you dont need to add any more to get a contribution. Notice I say bundle. This brings us to PARTS of unnecessary but sufficient conditions. And to causal cakes. Cakes and their ingredients Its like making pancakes. Given the 3 base ingredients flour, milk and eggs baking powder can turn the whole mix into pancakes. +
= But you cant make pancakes without flour, without eggs or without milk no matter how much baking powder you pour in the bowl. + Necessary but Insufficient parts Each causal factor in a bundle that makes up a sufficient condition is necessary if
one casual factor is missing you get no contribution at all. But each is insufficient by itself. You need all of them together to get any contribution at all. Smoking causes lung cancer, but not all smokers develop lung cancer. Genetic and environmental factors contribute. Sufficient Cause A is a constellation of factors, including smoking, that together cause lung cancer; smoking is C3. But people develop lung cancer without smoking. Sufficient Cause B is the constellation of factors, not including smoking (no C3), that together cause lung cancer. Working in a coal mine is C8. CN
CM C1 C5 C2 C4 C3 Sufficient Cause A.
C1 C8 C2 C7 Sufficient Cause B. C6 Homework & test scores
Homework is one INUS condition contributing to higher test scores. Other conditions are necessary to ensure that homework is maximally effective. These include student motivation and student ability, having access to a proper study space, a supportive family, getting a consistent message from teachers and parents, and
receiving teacher feedback on assignments Other Work feedback Homework Student motivation Consistent lesson Supportive
family Study space Student ability Other practices might achieve the same outcome as assigning homework These would be different pies altogether. For example, directed inclass tutorials, while resource intensive, could achieve the aim of higher
test scores We could depict more pies for the other factors that we expect to affect test scores (like smaller class sizes). Other Tutorial space Tutor's ability x hours of
tutorial Student motivation Student ability Why fuss about INUS conditions? They remind us that - There are usually a number of distinct causal bundles (cakes) that contribute separately to the effect. Other factors are necessary along with the policy variable if the policy is to have any effect at all.
Our theory of evidence for EBP Focuses on relevance: What kinds of facts matter to the truth of an effectiveness prediction? Has two simple ideas. Three presuppositions 1. Law-governedness: Outcomes are produced under the governance of causal principles. 2. Analyticity: Causal principles allow that different distinguishable sources can contribute to the same outcome, some
positively, some negatively. So a causal principle for O in S dictates what factors can contribute positively or negatively, to O in S what the form of the contribution is what it takes for that contribution to occur. 3. Causes are INUS conditions. So causal principles can be represented like this: O c= C11C12C1n + + Cm1Cm2Cmr P11P12P1s Pt1Pt2Ptu. where the meaning of + can vary. When the focus is on a specific treatment/ programme T, this is often shortened to O c= T + W.T + W.
Caution: T + W. is not a constant. [Causes are INUS conditions.] T + W. represents the support team without which T contributes nothing. The support team You need all those factors without which the policy variable cannot act. Theory claim 1 T as implemented will contribute positively to O
in S iff 1.There is a causal principle that holds in S from implementation till time of outcome in which T figures as a cause of O. 2.All the factors obtain that are required in that principle for T to contribute to O obtain in S at the required times. Theory claim 2 The facts relevant for predicting T will contribute positively to the production of O in S are those that must obtain if this claim is to be true. So 1. Is there a causal principle that holds in S from implementation till time of outcome in which T figures
as a cause of O? 2. Do all the factors obtain in S that are required in that principle for T to contribute to O and at the right times? These are the two kinds of facts that are directly relevant to predicting effectiveness. Other facts are relevant only if they support these. Two cautions T will make a positive contribution does not tell us what will actually happen. What happens depends on the totality of causal factors present. 1.
Both positive and negative. The magnet may contribute but the final effect may not be what one hopes for. Though better than what would happen otherwise. Watch out for what happens when you implement. Your efforts might make things worse. 2.
California class-size reduction : Implementing a positive cause introduced a strong negative cause as well. So what can RCTs do for you? Recall Theory claim 2: The facts relevant for predicting T will contribute positively to the production of O in S are those that must obtain if this claim is to be true. So 1. Is there a causal principle that holds in S from implementation till time of outcome in which T figures as a cause of O? 2. Do all the factors obtain in S that are required in that
principle for T to contribute to O and at the right times? These are the two kinds of facts that are directly relevant to predicting effectiveness. Other facts are relevant only if they support these. So what can RCTs do for you? They are at best conditionally indirectly relevant. 1. They support [very well] the claim that there is a causal principle L for the study situation X in which T figures as a cause of O. They are relevant to T figures as a cause of O in S only
conditional on separate support that L is shared with S. 2. They provide no support for the occurrence of the requisite support team in S. They are relevant to S only conditional on separate support that the requisite support team occurs in S (and does so postimplementation). Three kinds of causal claim 1. It-works-somewhere claims: T contributes positively to O somewhere under some
conditions (e.g. in study population X, administered by method M). 2. General claims: T contributes positively to O widely. 3. Effectiveness claims: T would contribute positively to O in S administered as it would be administered in S given policy P. Hierarchy of evidential support Effectiveness claim about S Presence of support team in S ??
? General claim It-workssomewhere claim ?? ? ? ? R
C T ?? R C T ? ?? ? ?
Finding ingredients for the cake Tell a causal roll-forward story from T to O. How is T to produce O step-by-step? This will help identify the factors needed at each step to get the next. Taken all together these constitute the requisite support team. Imagine the policy has failed and try to explain why. Simple decision tree: Try to think of the most salient features without which the policy wont work. If you dont have/cant obtain those, you dont need to think further.
Finding ingredients for the cake Tell a causal roll-forward story from T to O. How is T to produce O step-by-step? This will help identify the factors needed at each step to get the next step. Taken all together these constitute the requisite support team. Imagine the policy has failed and try to explain why. Simple decision tree: Try to think of the most salient features without which the policy wont work. If you dont have/cant obtain those, you dont need to think further.
First is synkleroma—"co-heirs." This legal term suggests not just an equality of shared privilege, as in two business but instead it is more like two "blood-brothers" who have utterly bound themselves together, totally dependent on each other for whatever inheritance,...
If a broadcaster indorses a candidate an opposing candidate has opportunity to reply. Popular Press. A section of the media specifically designed to reach a large audience. Prior restraint. A view that the press is guarantied freedom of censorship.
An investment company that issues its portfolio shares to investors. Money from shareholders are pooled and invested in a wide range of stocks, bonds, or money market securities. ... Issues a fixed number of shares at a given point in...
Unsought Products. These products may be unknown to the buyer or not normally considered for purchase. Unsought goods require special marketing effort. Consumer Product Classifications. This CTR corresponds to Table 8-1 on p. 240 and relates to the material on...
The dog alerted to the presence of drugs, and a large bag of methamphetamine was found. ... David Leon Riley belonged to the Lincoln Park gang of San Diego, California. On August 2, 2009, he and others opened fire on...
Short-term notes payable are a common form of financing. A short-term note payable represents a written promise by the business to pay a debt, usually involving interest, within one year or less. Assume that on May 1, Smart Touch Learning...
Ready to download the document? Go ahead and hit continue!