Explanatory reasoning ...

... or why natural experiments aren't experiments ...

... or why knowledge of really complex systems is different than regular scientific knowledge ...

... or one of the reasons why social knowledge is different from scientific law based knowledge ...

Experimental vs explanatory reasoning: in experiments, you choose the input and the categories with which you think about the input, so not only can you test the relationship between an input and an output, but you can test the utility of your categories as well. You can also limit the complexity of the input and output. This pre-structuring of epistemological categories is not possible in observation, and very difficult in complex experiments (say a terrarium).

Interestingly, one of the things that is interesting is trying to build complex systems out of simple laws, is that we test the integration of smaller scale concepts into larger scales. This integration often fails.

To show that explanatory reasoning is inherently different, I would like to model an inferential process that is structured around an input, an output, and a new inference connecting the two after one "tick" of time (t -> t+1). Then we count the number of possible rearrangements of input and output given plausible categories used to classify them, finding some huge number of possible input and output pairs depending on categorizations. All of these pairs would have to be explored in order to really understand the causality of the process, and a search over them would be intractable. This is for discreet and countable input and output, and only one time "tick"; continuity and more time would make it harder.

Obviously, there are heuristics for this search, and relationships between sets of input/output pairs, but we should shoot down the idea that there can be easy proofs about causality in complex systems.

In a laboratory experiment you: (1) manage tractability by having a small number of inputs and outputs; (2) decide on categorizations before even setting up the system and running the experiment, and thus the experiment tests both the in/out relationship AND your categorization system (leading to the demise of Aristotelian physics as a side effect of trying to predict how fast the cannon ball falls compared to the orange). In a "natural experiment" (say a one-time toxic leak in a complex eco-system that had been monitored before and after), causality is more squirrely because there are so many ways to classify the before and the after states, especially accounting for interaction; at least here the basic processes are fairly well characterized, though the interaction is not. But in a social "natural experiment", the social processes are anything but well characterized, and the signaling between processes is almost a free-for-all (not to mention the problem that people are changing all the time, which is another blog entry)!

Anyway, we are left with heuristics and rhetoric and a moving target - just fine with me, but kind of disappointing to the social physics types.