Stop 'bullying' the social sciences
Jul. 18th, 2012 09:32 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
eve11And a quick discussion afterward that I might like to turn into a real and citable argument, but must first get up to speed on current thoughts about this (here, likely, Gelman's blog will prove useful).
http://www.latimes.com/news/opinion/commentary/la-oe-wilson-social-sciences-20120712,0,4672619.story
My father is an experimental physicist who says "if it has to put 'science' in the name, it isn't." I think that the notion that controlled experiments are the only recourse for true objective scientific pursuit is flawed (ETA: simplistic? Depends on the methods). Observational studies can provide information, though not always causation which is one of the main reasons that experiments are done. In observational studies, generalizability to the population of interest or prediction of future events can still be quantified along with its uncertainty (possibly even objectively, or near enough). I also think that it is important to understand the analytic tools involved. When observation replaces experimentation, factors can be isolated by controlling for (eg estimating) sources of variation that arise from confounding factors. Random effects models are a good example. Prediction then is based on the generalizability of the model to the population of interest. Interpretations can be tested, sufficient statistics devised, sampling methods quantified, and levels of confidence at general statements provided. Not everything is going to be generalizable to the degree that a law of physics is. The important thing is correctly outlining the utility and scope of the result. This is definitely harder to do in disciplines involving human behavior.
My experiences with psychology research were that when the discipline tried to emulate physics, it ignored the sources of variation that arose with human factors and indirectly observable phenomena, and applied simplistic measurement models--that were adequate in some situations in physics for the most efficient use of the data in learning about their hypotheses and assumptions, but that were not adequate in their own studies. On the other hand, my dad called me the other day about what I'm pretty sure should be some kind of two-way random effects ANOVA analysis of some data, muddled through the lens of someone used to being able to always take independent measurements and conceptualize variation solely as 'error bars'. Many times, what someone considers a "data point" is actually an aggregation: a function of the data, eg a statistic by definition, that may or may not have retained all useful information about the question that the person wishes to answer. The guiding principle of this delineation in all data analysis is likelihood; but it's buried so far beneath the surface of analytic tools, data processing software, canned packages etc, that very few people truly explore it or understand that it is there.
So it seems I have been thinking about the relationship between models, the scientific method, definitions of probability, theory testing, experimental design, the relationship of all of this to "Big Data" and the science of cybersecurity. In particular the range of utility of modeling and of the inference involved: generative vs. empirical models, and the scale of interpretability that puts emphasis of inference in the model space (defined by interpretable parameters) vs. the data space (prediction of future outcomes). Big data wants to use the "black box", wants to say that models are no longer necessary for inference of what has occurred. When all possible confounding factors can be observed and taken into account, an experiment is no longer needed, or so the story goes. And yet, under what basis and assumptions are these results used to predict future outcomes? There, I do not think we can escape the need for assumptions in order to make predictions.
In physics those assumptions are not generally too bad: 'the sun will still rise tomorrow'. Even in medicine: "human biology will not change appreciably in the next ten years". In cybersecurity, they may be more troubling: attacks will continue to follow certain patterns. People will innovate in certain ways. I begin to understand better Geer's discussion of unknown unknowns, and the importance of understanding the "bell cow" culture of hackers, while still at the same time allowing some possibility that these principles may in the future be flung out the window. Perhaps financial analysts were on to something when they couched everything in terms of volatility; at the same time, no one truly predicted what would happen in 2008. I don't count the iconoclasts and doomsayers; those people are always right when disaster occurs.
Is there a notion of precision of measurement and utility of theory that goes along with "science"? Predictions are useful if they let us do things, expect things, imagine. Predictions come with associated imprecision. Sometimes that is assumed to be solely measurement error and other nuisances, but sometimes it is wrapped up in the random variation within the population. The utility of a prediction: what it is used for, is not particularly the purview of good science or intellectual curiosity, but it is the purview of good, directed research. So, when you start having to predict distributions instead of constants, distributions that will not happily dissolve away to the infinitesimal with the increase of sample size: what does that mean?
Of course I think controlled experiments are a worthwhile undertaking; I think though, that we will end up searching more in the sociology, economic, psychology and psychometric literature for how best to design them: to understand the latent concepts and constructs that we wish to measure, to learn the methods and models needed for teasing out the relationships we wish to study, to quantify and verify the extent that the results obtained are generalizable, and to develop predictions that nonetheless will always be subject to random variation. Of course in terms of security I don't think this is necessarily a death blow to the notion of objectivity, knowledge, or science; all of security should, in my mind, be defined in terms of relative risk. If we try to follow the example of physics, we must do so with caution; their tools and methods have been developed for their own interests.
Why listen to me? Well, of course I am the product of my experiences and particular educational background, but I also feel that it has given me a good wide-angle perspective. Because the goal of all of this is extracting knowledge from empirical data; how to analyze it no matter what the hypothesis, variation, sampling frame. Statistics started out with the experimentalists, with a measure of uncertainty pertaining to a relative frequency definition of probability, but even that has been expanded to good effect for addressing certain kinds of problems, using Bayesian methods. Gelman posits that scientific discovery is about resolving and understanding outliers in models. Science has always been an iterative process and it always will be. Measurement and metrics will always be difficult but if done right can be standardized, interpretable, and used to direct the future seeking of knowledge. Science also always has included the element of imagination; even Feynman would agree. Imagination is driven by information but not necessarily bound to the logic of inference. Imagination drives iterations.
In all pursuits, we must be able to link the questions that we want answered to the data that is needed to answer it, and to determine the utility and scope of the predictions that result. That is fundamentally the realm of statistical analysis and statistical thinking.
http://www.latimes.com/news/opinion/commentary/la-oe-wilson-social-sciences-20120712,0,4672619.story
My father is an experimental physicist who says "if it has to put 'science' in the name, it isn't." I think that the notion that controlled experiments are the only recourse for true objective scientific pursuit is flawed (ETA: simplistic? Depends on the methods). Observational studies can provide information, though not always causation which is one of the main reasons that experiments are done. In observational studies, generalizability to the population of interest or prediction of future events can still be quantified along with its uncertainty (possibly even objectively, or near enough). I also think that it is important to understand the analytic tools involved. When observation replaces experimentation, factors can be isolated by controlling for (eg estimating) sources of variation that arise from confounding factors. Random effects models are a good example. Prediction then is based on the generalizability of the model to the population of interest. Interpretations can be tested, sufficient statistics devised, sampling methods quantified, and levels of confidence at general statements provided. Not everything is going to be generalizable to the degree that a law of physics is. The important thing is correctly outlining the utility and scope of the result. This is definitely harder to do in disciplines involving human behavior.
My experiences with psychology research were that when the discipline tried to emulate physics, it ignored the sources of variation that arose with human factors and indirectly observable phenomena, and applied simplistic measurement models--that were adequate in some situations in physics for the most efficient use of the data in learning about their hypotheses and assumptions, but that were not adequate in their own studies. On the other hand, my dad called me the other day about what I'm pretty sure should be some kind of two-way random effects ANOVA analysis of some data, muddled through the lens of someone used to being able to always take independent measurements and conceptualize variation solely as 'error bars'. Many times, what someone considers a "data point" is actually an aggregation: a function of the data, eg a statistic by definition, that may or may not have retained all useful information about the question that the person wishes to answer. The guiding principle of this delineation in all data analysis is likelihood; but it's buried so far beneath the surface of analytic tools, data processing software, canned packages etc, that very few people truly explore it or understand that it is there.
So it seems I have been thinking about the relationship between models, the scientific method, definitions of probability, theory testing, experimental design, the relationship of all of this to "Big Data" and the science of cybersecurity. In particular the range of utility of modeling and of the inference involved: generative vs. empirical models, and the scale of interpretability that puts emphasis of inference in the model space (defined by interpretable parameters) vs. the data space (prediction of future outcomes). Big data wants to use the "black box", wants to say that models are no longer necessary for inference of what has occurred. When all possible confounding factors can be observed and taken into account, an experiment is no longer needed, or so the story goes. And yet, under what basis and assumptions are these results used to predict future outcomes? There, I do not think we can escape the need for assumptions in order to make predictions.
In physics those assumptions are not generally too bad: 'the sun will still rise tomorrow'. Even in medicine: "human biology will not change appreciably in the next ten years". In cybersecurity, they may be more troubling: attacks will continue to follow certain patterns. People will innovate in certain ways. I begin to understand better Geer's discussion of unknown unknowns, and the importance of understanding the "bell cow" culture of hackers, while still at the same time allowing some possibility that these principles may in the future be flung out the window. Perhaps financial analysts were on to something when they couched everything in terms of volatility; at the same time, no one truly predicted what would happen in 2008. I don't count the iconoclasts and doomsayers; those people are always right when disaster occurs.
Is there a notion of precision of measurement and utility of theory that goes along with "science"? Predictions are useful if they let us do things, expect things, imagine. Predictions come with associated imprecision. Sometimes that is assumed to be solely measurement error and other nuisances, but sometimes it is wrapped up in the random variation within the population. The utility of a prediction: what it is used for, is not particularly the purview of good science or intellectual curiosity, but it is the purview of good, directed research. So, when you start having to predict distributions instead of constants, distributions that will not happily dissolve away to the infinitesimal with the increase of sample size: what does that mean?
Of course I think controlled experiments are a worthwhile undertaking; I think though, that we will end up searching more in the sociology, economic, psychology and psychometric literature for how best to design them: to understand the latent concepts and constructs that we wish to measure, to learn the methods and models needed for teasing out the relationships we wish to study, to quantify and verify the extent that the results obtained are generalizable, and to develop predictions that nonetheless will always be subject to random variation. Of course in terms of security I don't think this is necessarily a death blow to the notion of objectivity, knowledge, or science; all of security should, in my mind, be defined in terms of relative risk. If we try to follow the example of physics, we must do so with caution; their tools and methods have been developed for their own interests.
Why listen to me? Well, of course I am the product of my experiences and particular educational background, but I also feel that it has given me a good wide-angle perspective. Because the goal of all of this is extracting knowledge from empirical data; how to analyze it no matter what the hypothesis, variation, sampling frame. Statistics started out with the experimentalists, with a measure of uncertainty pertaining to a relative frequency definition of probability, but even that has been expanded to good effect for addressing certain kinds of problems, using Bayesian methods. Gelman posits that scientific discovery is about resolving and understanding outliers in models. Science has always been an iterative process and it always will be. Measurement and metrics will always be difficult but if done right can be standardized, interpretable, and used to direct the future seeking of knowledge. Science also always has included the element of imagination; even Feynman would agree. Imagination is driven by information but not necessarily bound to the logic of inference. Imagination drives iterations.
In all pursuits, we must be able to link the questions that we want answered to the data that is needed to answer it, and to determine the utility and scope of the predictions that result. That is fundamentally the realm of statistical analysis and statistical thinking.
