Bayesian Sampling Theory*

Engineers and scientists like facts because they hint at underlying processes.  If we collect enough facts of the right sort, we can understand some new phenomenon and be able to control it in an economical way.

“What could this mean?” is always a springboard for interesting, possibly meaningful, connections.

So when I see news stories of major leaks, e.g. Siebel Edmonds, Chelsea Manning, Edward Snowden or of major hacks that sucked out entire data stores, e.g. Sony, OMB, Hacking Team, or market riggings such as [ long list here ], what each incident might mean is a major part of my thinking.  What does it tell me about the workings of the systems they are part of?

Past the significance of the event in itself, the first question always is “is this a general phenomena?”.  If so, your interpretation of the world’s structure and functioning might be quite different.  A case of Ebola on your Continent, for example.

Bayesian probability theory has to do with using probability of things and events related to the particular thing or event you are trying to evaluate or interpret.  The probability Ebola could ever be transmitted to medical personnel in a modern, first-world hospital system, given CDC’s claims that could never possibly happen would be another example.***

Bayesian statistics is popular just now, but the reasoning is not new.  Your mind automatically uses known information to guide its thinking, search for new knowledge.  In this case, we use it to think about samples of data.

Using Snowden as the example : There is a 100% probability of Snowden having happened.  That is a strong prior for something very like itself.  It is an existence proof. By logic, the null hypothesis must be that the first revealed event of a new class is NOT unique.

In fact, are first-revealed events found to be the first-ever events ? No. When you look back through the history of ideas and inventions, always there were several people doing or thinking the same things at the same time in different parts of the world. What could have changed to make that not true?  Communicating facts once known is indeed easy and fast, but making them known is of variable difficulty.  We can have no way of estimating that.  We know our government hides things from the various leaks and that things hidden are of major importance, but those leaks cannot indicate the range nor size of other things it hides.

So there is both logical and empirical support for assuming the first example of [anything] to be revealed was not the first to exist nor the first found.

Thus, any ‘unique event’ may be the leading edge of a trend or indication of a hidden process.  If that could possibly be a significant trend or process, it must be investigated.  You should not minimize its importance merely because it is unique.  Any fact is a 100% verification of itself.  It is a data point, not a theory, and ‘unique event’ is a hypothesis whose probability is low, given the above reasoning and that there is in fact an example.

Further Baysian-directed thinking may suggest reasons for previous examples to be suppressed and why the event appears unique.  If entities with the power to do so have motives for suppressing an event, then our estimate of the probability that information is hidden is much larger than otherwise.

If NSA, CIA, … ever found information that someone had taken information as Snowdon told the world he did, would they have told anyone?  Would not most of their executives have argued for a slow tightening of the security procedures?  After all, wouldn’t want to signal to anyone that we had problems, wouldn’t want to validate those files. That information would be buried in their darkest dungeon and deleted as soon as possible.  There is no upside to that information being known outside the agencies, so they will take care that it does not get out.  The other intelligence services also, of course.  China’s spys certainly won’t inform NSA that the OMB database is in their possession, if it is.

NSA’s people are partially responsible for security in other government agencies.  Doesn’t the same reasoning apply to their work there ?

In the light of all of that background, was Snowden the first such theft from NSA to ever happen?  To do that theft of data, Snowden must have been, would have had to be, and certainly appears to be,  highly intelligent, motivated, an excellent planner, able to carry off long-term plans well, an idealist.  The question is how many people there are at his level of access who are all of those, vs just enough of that list less ‘idealist’?.

Associated probabilities : Do you associate NSA with idealism?  Me neither, but rational analysis would require dispassionate assessment of the proportion of people having such extreme idealism.  By our own argument, if there was one, there very likely are more.  Security people know this.

From what Snowden taught us about NSA’s internal security procedures, ‘luck’ would not have been needed, so smart-enough, easily pissed off, opportunistic, and stays pissed long enough would produce anti-Snowdens who didn’t talk, who did go to Russia, China, … and live off their store of documents.

As a prior, a Lebowski Enlightenment*** would spotlight the idea that that was not a new idea, professionals must understand this and far more and have done so for some period of time.  Snowden was not inconvenienced by the security measures that resulted from those understandings.

As a prior, professionals knowing plus the fact that that is not a widely known fact, we should want to know : Why did they not tell us?

After another few steps on such sequences, priors revealing new aspects of the system, we may be able to answer the question of whether such intelligence services make any sense at all?  Great stockpiles of secrets attract attacks. In this stage of the technology of protecting secrets, it is better not to have secrets.  Secrets are expensive of treasure and liberties.  And they are often not secret after all.

Did nobody notice?  That was the prior that Snowden established.  NSA cannot keep America’s secrets secret.  If your secret-ferreting agency can’t hide your secrets, you better hang it up.

*No statistician will recognize that phrase and it may induce whiplash in your concept space.  Sampling has to do with selecting measures to be statisticized and limitations in interpreting the resulting statistics, which may produce probabilities.  Bayesian analysis is how to use probabilities of facts and events less-than-certain in reasoning.

***Completely random sequence, I thought of Ebola as a dramatic finale, and the CDC as the next logical following point.  As a prior, it biases me to thinking that sequences like that are easy to find.

***Lebowski Enlightenment.

That moment when The Dude realizes the only reason Lebowski would have hired him was if Lebowski didn’t want an answer.

The Dude: No, I’m saying, if he knows I’m a fuck-up, why does he leave me in charge of getting his wife back? Because he doesn’t fucking want her back! He no longer digs her, it’s all a show!

When you realize that no matter how pleasing your newest insight, every knowledgeable person understands the situation better than you do, and that that fact in itself is a prior to something important.

A complete aside here of no relevance to any of the above.  Just found the Rob Agar film analyses of The Big Lebowski.  As compared to a Bayesian prior-directed forward search, already philosophically questionable, that is anywhere-to-anywhere associations by this guy and his friends.  Any of N things paired with any of N-1 things is exponential.  If you can’t find enough associations in a movie to make a long movie review, you don’t know much and it was a dull movie. The next in the series is “Is it Film Noir?”.   Wow, the things that pass for intelligent commentary.  Written by random associations on a Google.  I knew there was a reason the ?name unimportant? meetings seemed strange, it must have been the lack of any referent outside of their minds.


One of my great joys is saying “I told you so“. Don’t be too impressed, because with so very many people with clearances working in installations on the NSA payroll, and their security being so incredibly lax, as shown by the vetting scandals, the several generations of which happened BEFORE the OMB database scandal. Get your mind around it, it is just another big federal project, and their vaunted security protocols and operations are largely facade. If they gave a gd, would Clinton’s email server not have been detected? Of course it was, and of course they have all of the emails. Cross-referenced in the searchable blackmail database, of course.

If they are competent, they are corrupt, as they chose to allow the Russians to see all rather than correct Clinton and tip off the Russians that they read everything on the network.

Your choice : competent or deeply corrupt, a rot at the core of our deeply corrupt government?


6 thoughts on “Bayesian Sampling Theory*

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s