Fooled by Deep Non-Random

If you had to pick one word to describe the standard result of the standard social, political or economic policy for countries in the 20th century, it would be ‘Wrong‘.

Anyone paying attention at all sees that things around us go wrong at all levels from small decisions in our individual lives up to international relations.  Anyone reading politics and political-sociology and -psychology over the last few decades has read many different diagnoses and recommended treatments for an amazing variety of problems.  We citizens believed savants on every issue.  Thus the drug war, war on poverty, innumerable shooting wars, 3 strikes and you’re out, …  These have all turned out to be wars against ourselves and we citizens have lost each one.

Every one of those diagnoses and treatment recommendations were very wrong,  judging from the outcomes of the treatments applied.  Clearly we, as a civilization, are doing something quite wrong.  In fact, the wrong thing entirely, this is not the pattern of a local failure from doing a some particular thing wrong.

Obviously, from the many examples, it is real easy to get things wrong about almost any big question.  Obviously, it is astonishingly easy to go along with the program, no amount of preaching diverts a system pursuing the wrong path, all of us can point to prophets who made hard predictions that proved accurate.  The list of subsystems of government and industry and society that are going dramatically wrong here in the USofA is long.  Many of those were known to be headed in the wrong direction at the beginning, and many recommendations have been made for remedy, but no effective changes are made.

Why so hard?  Why is it so difficult to get things working well?  Or perhaps why is it difficult to diagnose problems?  Is this a problem that can only be dealt with at the level of groups, or can we make corrections as individuals, in our own thinking and social behavior?.

This post is 3 different things at once.  First, it is a warning about how sort-of-regularities are surprisingly common and hard to recognize, so lead to incorrect diagnosis of problems. Second,  it mentions new tools that are becoming available for everyone to easily use to solve problems we couldn’t have even conceived of a few years in the past.  All such tools, of course, produce or reveal their own kinds of noise that looks like signal, the real expertise is knowing those details.  Third, it is the beginning of an discussion about how our tools shape all of our understandings, and how the new tools provided by computer+internet provide a design palette for the social-political system designer thinker that will ultimately reorganize our entire socio-politico-economic system.  Finally, it is a small beginning of rethinking goals and individual actions.

The brain that writes these random-walks through interesting stuff on the net and from my bookshelves, making connections.  This post is certainly an indication of how many connections can be made between items that come to my attention and how easily.  It is thus a warning about the probability that you can be fooled by happenstance patterns interpreted as meaningful.  The noise that is most difficult to separate from the signal is the noise that is most like the signal.  Is this post signal or noise?  How can you tell?   History of science and the recent projects to replicate important scientific studies suggests that “It makes sense to me” is an error-prone metric, however scaled.

Nassim Taleb’s “Fooled By Randomness” is an entertaining story (mixed with deep philosophy and math-of-reality) of how an external environment may, for a time, match expectations by a human.  And how that matching may stop, un-expectedly.  We knew that money managers were basically survivors of a random-walk, suspected that many high-level managers in corporations and bureaucracies had also random-walked into their positions.   Once you understand it, you suspect random results to have produced a lot of what you see.  Certainly my career has been without design, I always just took the next opportunity offered.

These Numberphile videos show that there are many more traps, some so clever it would be difficult NOT to believe your decisions had been right.  These first 2 are examples of very simple functions which, when iterated, produce numbers whose prime factors have astonishing regularity.  The implication is that such regularities happen with unknown, but higher than expected, frequency in many parts of math.  Things that happen in math often have analogs in reality.

The Mandelbrot-set and mathematical chaos are other.  If your random data was being modified by such a generator (logistics functions generate chaos and are everywhere) you would have a hard time dis-believing it.

Stephen Wolfram is a serious original thinker, a person with very big ideas and the ability to make them happen.   His Mathematica and Alpha have already made him a big contributor to science.  In math, there can’t be many generalizations bigger than the one Wolfram is working on.  His models of physical reality can only be compared to current models in the insights one can attain using them.  That will take a generation of studies to evaluate.

Wolfram gets a lot of criticism from people who see the world through their specialty’s pin-hole.  Ignore all that, attend only to the usefulness of his work in generating new knowledge.  That is the only metric that counts with models of reality, whatever you think reality might be today.

Several times I have been part of projects that measured very detailed behaviors of complex systems — physical systems, not just iterated math.  In each of the very different systems and types of measurements, the problem was always deciding what was real and relevant : data collection was no problem and the data could be summarized in many different ways easily, but selecting meaningful data (“real patterns”) and fitting the results into some frame that helped understand anything about the system was difficult-to-impossible : that is the research problem in every case.

Natural systems don’t come with subsystems or meaningful patterns outlined in chalk, you make up the divisions as you study the system, waves of different understandings sweep through the field, combine, are obsoleted, interests spark and die and are resurrected.  It is a very dynamic process, making progress cannot be assumed.

It is easy to see new phenomena in recording data from real systems : large-scale patterns of packet traffic in routers, throughput in data processing systems, latencies in responding to messages in clusters of systems, every area of research produces them reliably.  In networking, it is easy to modify processing algorithms and protocols and relate the patterns to them.  But it is nevertheless very difficult to understand why from the what, as difficult as understanding the role of a plant or animal in the overall operation of a rain forest.

All of those systems I worked with were mostly isolated from surrounding systems.  Attempts to analyze working systems within larger systems have a much larger problem, as there are more sources of noise patterns masquerading as meaning.  One of Wolfram’s points is that all interesting systems are past a level of complexity where they can be easily understood.  My experience and those of friends confirm that, all the easy stuff has been understood. Also, all understood stuff is easy. Expect it to be hard.  Expect to be fooled.

The only way to protect yourselves from these false patterns, noise seeming signal, is to focus on finding disconfirming evidence.  That is not easy, for example, there is no natural rate of coincidence that you can measure against.  So no matter how many anomalies and coincidences we find in investigations such as the various deaths, assassinations and bombings in recent US history, those alone do not and can not confirm their author’s hypotheses about the actual culprits.  OTOH, many of those facts seem to disconfirm the hypothesis “the government’s investigation was well-conducted and produced a comprehensive understanding of the event”.

The answer to “Why is it so hard” to make things work is that the world is increasing in scale and complexity at unprecedented rates.  The modern world is very new.  Its institutions are failing to deal effectively with the changes, failing to protect their members from predation of elites.  That is the next challenge for civilization, to manage a very large social change with minimal problems.  I think we haven’t been doing that well, better start using Engineering Thinking and as many modern tools for understanding as we can manage.

As for individuals, the only possible recommendation is to prepare for an uncertain future, exactly what any wise person would have said at any point in all of human history.  We are individuals, have no choice about either being individuals or groups, we are almost always both.  We have to act that way, the goal is to increase our own personal and family social safety net : resources we can use for our goals.  How is no mystery, it is what people do, it is why we have groups.  So go gossip and group, there is safety in numbers, if you choose the correct people.  “People just like you” has been the usual choice, but has often been wrong.  Big changes such as converting to Old Order Amish (one of the few genuinely Conservative cultures in the world, and it nevertheless pretty modern, only traditions are from 1632, late in the Protestant Reformation) may be better, depending on the nature of the future.


Found this later, so many sequences.

Later still, a pattern of non-random in prime numbers.

1 thought on “Fooled by Deep Non-Random

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s