Our Friends at the NSA

Have you ever thought about the capabilities of the NSA and their ability to spy on US citizens?  Many people do not have the How to spy on americansexperience to comprehend the potential ability of the world’s greatest spy agency.  I am not an employee or contractor for the NSA and as such I do not have any direct knowledge of their capabilities.  However, I have been in IT for well over 3 decades in varying capacities.  That experience gives me the ability to provide some history of IT capabilities and to provide some scope for the size of the NSA.

By Dave H, a reader of SHTFBlog & Survival Cache

Moore’s Law

For the IT savvy forgive me.  For the rest of you let’s start with some background information.  What is the difference between data and information?  An easy way to think of this is your phone book.  The collection of names, numbers and addresses is data.  In of itself it has no value.  However, when you look up a number you retrieve information.  Information is created when data is used to answer a question.

A query is a question or an interrogation of data that provides information.  Metadata is simply data about data.  When you NSA Spies on americanscall someone your conversation is the data.  The metadata is the data that this number called that number on this date for this duration.  Another concept is Moore’s Law (Gordon Moore pictured right). Moore’s law states that computing power will double every 2 years.  Moore’s law can be roughly applied to data storage as well.  Keep in mind the power of 2.  So if you start with 1 in 2 years it will double to 2 in 2 more years it grows to 4 and in 2 more years it grows to 8.  Enough definitions let’s talk about some history. Back in the early 1980’s I worked on an inverted list architecture data base called Model 204.  When I worked on this system we were told that it was developed by the CIA.  It used a bit map index structure to provide extremely fast answers to queries.  However, it had problems.  It was very slow to store data and if it was presented with a query that was not indexed, it would attempt to resolve the request by walking through the database one record at a time.

For example if you presented a query that said “Tell me all the oil wells drilled in Oklahoma?” M204 would instantly provide that response.  That would probably be too much data.  So let’s narrow the scope.  Tell me all the wells in Oklahoma in these 5 counties that were operated by Anadarko.  It would also give that answer instantly because all of these fields in the query were indexed when the data was stored.  If we added another field to the query that was not indexed the database would walk through every record to resolve the query.  So, tell me all the wells drilled in Oklahoma in these five counties operated by Anadarko and used a 3” diameter pipe would take the system hours to resolve.  So, 1970’s technology allowed for instant responses for queries that were predefined with the proper index structures.  Another point of reference for the 1980’s is in data storage.  We kept track of 3 million oil wells with 45GB of storage.  To put this in context you can put 128GB of storage in an iPhone 6.

Two Kids & Enterprise Search

Let’s fast forward to the middle of the 1990’s.  Google comes on the scene.  It can crawl through the Internet and provide an NSA Spyinginstant response to ad hoc queries.  Let’s think about what happens with a Google search.  It crawls through the Internet and provides millions of pages and information about those pages instantly.  Those algorithms are so complex they are listed as one of the top ten most guarded corporate secrets (Video).

So how did a 21 and a 22 year old student at Stanford come up with something so complex that to this day they remain a secret?  They didn’t.  These 2 guys would need to be so smart that they would make Einstein look like a drooling moron.  This was the work of teams of brilliant people that took decades to complete and probably cost millions if not billions of dollars to develop.  To complete this work they would have needed intimate knowledge of the Internet.  They would also need to develop new index structures never seen before.

They would need to develop find new ways to store and retrieve the data.  They would need to develop this in a way to scale up to millions if not billions of transactions without losing performance.  But you say it could be possible that these geniuses did develop Google.  You may even think that given enough monkeys with enough typewriters and enough time they too could write the World Book Encyclopedia.  Well these 2 monkeys would have needed to develop new groundbreaking work in 3 years.  There simply weren’t enough monkeys or enough time to pull it off.  They probably developed the front end.  One last point, if they were so smart where is their other ground breaking work? Chrome? Google Glass?  No, they are extensions of the previous technology.  So, the doubters will say that they had momentary flashes of genius. Whatever!

So, who did develop the algorithms for Google?  Who would have the need to provide instant responses to ad hoc queries?  Who would have the money to put the mathematicians and scientists together for an extended period of time to create this body of work?  It was the CIA, DARPA or the NSA.  The story on the street is it was the NSA.

Why would the NSA give these two guys their trade secrets?  The answer is pretty simple.  The NSA was still bound by the 4th amendment and they could not collect information on US citizens.  Private corporations are not bound by the 4th amendment.  They can collect any information they want and sell it to whomever they want.  Just like the credit agencies collect information on you and sell it.  Google can collect information on you and sell it to the NSA.  This is how they got around the 4th amendment.  The Patriot Act ended the restrictions and your 1st and 4th amendment rights (reference).


Back to 1990’s technology we can see that we can do instant ad hoc queries.  However, it cannot correlate the data.  If we do a NSA SpyingGoogle search about the number of oil wells in Oklahoma we will get a page that provides that information.  We can even do a query about the number of oil wells in Oklahoma operated by Anadarko.  When we start looking at specific counties and the response starts to fall apart.  The other problem with Google from 1995 is it cannot handle natural language queries.  So what about today?  We can see the natural language barrier has been broken.  IBM’s Watson was at the forefront and now that technology is available from Apple and Google on telephones.  There is other technology that is becoming available called data analytics and predictive analytics.  Here is a very good explanation of data analytics (click here).

In short data analytics is the process of developing conclusions from raw data.  This means that we can now ask natural language queries against large quantities of raw data and correlate conclusions.  Data analytics does not predict future behavior.  That is where predictive analytics comes into play.  Predictive analytics uses various statistical methods as well as data mining to determine the probability of someone taking a future action. For example: Let’s take something easy, say ammunition purchases. Someone could look at your past purchasing history to make a determination if you would purchase ammunition at a certain price and how much ammunition you would purchase.  So, if you saw Lake City 5.56 ammo at $1000/1000 you probably would not purchase any ammunition.  However, if Lake City .556 ammo was $200/1000 you would probably purchase several thousand rounds.  Predictive analytics would give the statistical probabilities of your personal purchases.

So, where does this go?  Let’s say I am the Feds and I want to know the anti-government groups in Idaho, I also want to know who is in each group and how likely they are to take action against the government.  Data and predictive analytics provide this answer. How do they do it?  The query could be done in a natural language like the question posed above.  I would start by finding people in Idaho who visit anti-government websites.  I would then look at the metadata for their phone calls and emails to determine links between the individuals.  This would give me the groups.  I would then look at their purchasing habits to determine their resolve as a rebel.  Do they buy survivalist gear?  Do they write for these anti-government sites?  How much long term food storage have they purchased?  How many firearm purchases did they make?  Did they buy large quantities of ammunition?  Did they purchase survivalist books?  Did they check them out from the library?  So, you take the metadata from website visits along with the metadata from phone and email records to determine groups then you use the information from purchasing history to determine the seriousness of the groups.

But what about the likelihood that any of these groups would take action against the government?  Once I have determined the active anti-government groups in Idaho, I would use predictive analytics to determine the likelihood they will attack.  This means I would use the actual contents of their emails and phone calls to determine their intent.  The amazing part is this can be done automatically for all 330,000,000 Americans.

So, how much data does the NSA have?  Let’s start with the Total Information Awareness program? (Reference)

Then there is the Patriot Act. (Reference)

Both of these programs opened the door to unprecedented surveillance on US citizens.  They are a direct assault on our personal freedoms and liberties but it actually started with the Echelon Project. (Reference)

So, how much does the NSA have?  Their new data center in Bluffdale, Utah is a 1,000,000sqft facility.  They say that 100,000sqft is currently data center and the remaining space is administrative space.  Let’s translate, that’s another 900,000sqft for expansion.  The data center currently consumes 65 megawatts of power.  Also, keep in mind that this is only the newest of the NSA’s data centers. They obviously had other facilities before they built this one. (Reference)  This is right off the NSA website.

Data Storage Capacity

In February 2012, Utah Governor Gary R. Herbert revealed that the Utah Data Center would be the “first facility in the world expected to gather and house a yottabyte”.  Since then, conflicting media reports have also estimated our storage capacity in terms of zettabytes and exabytes. While the actual capacity is classified for NATIONAL SECURITY REASONS, we can say this: The Utah Data Center was built with future expansion in mind and the ultimate capacity will definitely be “alottabytes”!

The steady rise in available computer power and the development of novel computer platforms will enable us to easily turn the huge volume of incoming data into an asset to be exploited, for the good of the nation.

So, What Is A Yottabyte?

1 YB = 1000(8)bytes = 10(24)bytes = 1000000000000000000000000bytes = 1000zettabytes = 1trillionterabytes

Now keep in mind Moore’s law. That capacity will double roughly every 2 years.  What kind of communication line would you need to fill a yottabyte?  Let’s look at it like it was a swimming pool.  Could you fill a 1,000,000sqft swimming pool with a garden hose?  How about a fire hose?  You would need something much larger to feed this capacity.  When you hear that the NSA has direct links into carrier gear you should probably believe it.  Going back to Moore’s law 1YB will turn into 2YB and then 4YB and 8YB and on and on and on.  I was in a class a while back and one of the people was from Yahoo. They slipped and said that we would be in shock at the amount of data they were required to send to the NSA. (Reference)


Should we be afraid? You bet. Look at the quote from the NSA page.

“The steady rise in available computer power and the development of novel computer platforms will enable us to easily turnour friends at the nsa the huge volume of incoming data into an asset to be exploited, for the good of the nation.”

Who benefits from this surveillance?  Obviously, both the Republicans and the Democrats are okay with the NSA spying or they would not have funded the facility.  What happens if our government becomes malevolent?  What happens if a Hitler comes to power in the US?  Why can’t we get our 1st and 4th amendments back?  What does it say about the capabilities of the NSA, if natural language, data analytics and predictive analytics have been released to the common public?  I hope this article gives you enough information to open your eyes and ask questions about our friends at the NSA.

Photos by:
Steve Rhodes

Support SHTFBlog.com by shopping @ Amazon (Click Here)

Visit Sponsors of SHTFBlog.com




11 comments… add one
  • kevin October 18, 2014, 12:55 pm

    WHOA that is some SCARY SHIT

  • Peek-a-Boo October 18, 2014, 1:30 pm

    Hi Dave,

    I am a datacenter manager specializing in building budget-minded data center teams with collaborative leadership that produces projects on time and under budget. My strengths include a strong combination of leadership, operations management and working with diverse groups at all levels.

    Do you think I have anything to worry about?

  • Dave H October 18, 2014, 2:54 pm

    I don’t believe your occupation places you at any more risk than anyone else. We all have a reason to fear the end of our constitutional rights. We should all work hard to ensure all of our rights are passed down to our children and their children. It is much easier to try and change the system while we have a government that will allow us to change it. If our government evolves into something that oppresses its people then we will have a real problem trying to reverse it back into a government of the people, by the people and for the people.

  • irishdutchuncle October 18, 2014, 3:20 pm

    pretty chilling, Dave.

    this is your best work on SHTFblog so far…
    mostly factual, not over the top politically. keep up the good work.

  • Studebaker Hawk October 19, 2014, 7:23 am

    Very nice article Dave. Well put in terms of understandability. Sadly I believe our government has already become malevolent. Just look to the news for story after story of outright lying to us. Our politicians for the most part are more concerned with finding a way to line their own pockets, and an apathetic public keeps electing them over and over. Just the fact that the data center in Bluffdale was even built should be proof enough. This country can not be saved. It’s done and we are watching it crash and burn. Barring a coronal mass ejection that knocks us back to the 18th century I think it’s time to start looking for another place on this planet that can be settled. An en-masse emmigration of such a magnitude that we completely crash the host country’s immigration system. Do exactly what has been done to us, learning of course from the mistakes that have been made here.

    • Dave H October 19, 2014, 12:19 pm

      Where do you go? There are no safe places on the planet. Have you ever heard of the Bank for International Settlements? Go to http://WWW.BIS.ORG
      Check out the board of directors. Check out the member central banks. This problem is global.

      Our best option is to try and fix our own issues. Running and hiding only allows the problem to spread.

  • late2theParty November 12, 2014, 5:00 pm

    On Google being great, for this IT person, it sucks. I generally look for pure data, not anything slanted towards my location or previous searches. Each search may have nothing to do with the previous. I.e. “Replacement crown O-ring Thayer and Chandler” then “Home grown NAS using OpenBSD” then “HHO bubbler brass” and maybe “Neurolinguistic programming studies 2014.” I used to use AltaVista before I was forced into ‘Googling’, I’ve hated the change since. There’s no real control over the data returned, they send what they want, even if you say “DON’T send me info on Mayor Thayer.”

    I can appreciate that in a lot of cases, the search by a person is very much “humano-centric” (personocentric?) “Where is Big Hero 6 playing” – a person generally wants the nearest theatre to their present geographic location and only movies in the future (you can’t go back and watch a movie last night), but the assumptions aren’t always valid. At least when I used AV I knew I could get ONLY the data I wanted (some of it wasn’t useful and so I say it wasn’t information till I culled it.) Say “+Thayer +Chandler +ring +airbrush +crown -mayor -puddsville -Friends -royal” and I might get no results, but no results meant I needed to restate my query; I was asking the wrong question. Most of the time I could get very valid and detailed information even though it was work.

    Thanks for the info, I know a lot of non-IT people definitely can understand better what the National Sneaky Agency is up to. It just burns my buns that they are playing verbal games “We don’t consider it a violation of law because it isn’t information till we run a query on it”, when some of us know it’s all just an illusion. Don’t you run a query to put it IN the database?

    I think these people (the LEOS, the politicians, the 3 letter agencies) have no belief that the system can be turned on them – a double edged blade; “I’m immune, I work for the agency that does that.” Yeah. And you are more likely to be watched than I am.

    My only answer to real privacy is darknetting and being off the grid. Say we get tens of millions of homes hosting a special router/NAS that can be used to bypass the PTB’s Net. Internet 2 as a friend says. People would have to choose wisely what they made public on their NAS (else we get Criminal charges for Civil violations) but given the numbers and amount of decryption these would require, you’d be beating them at their own game. The equivalent of taking your football and going home.

    Hmmm. Maybe a true ‘cloud’ – the data constantly moves, one day here, next day here…like a cloud roaming with the wind. Maybe we should have the document contain the reference info and the network device knows that it has documents with prefix 123456, but not the rest of it’s id (123456.987654.123789) and only when a call to it’s root id is sent will it even pull the second set for comparison. Whoops too much. Anyways they have the ‘average’ American lock stock and barrel. Only by asking what they have, will we even realize this so we can consider the true cost of our ‘freedom.’ The freedom that our forebearers and fighting men bled and died getting to us.

  • Anonymous December 20, 2014, 10:52 am

    Freedom doesn’t come like a bird on the wing.
    It doesn’t come down like the summer rain.
    You’ve got to work for it, fight for it, day and night for it.

    And every generation has to win it again.
    Pass it on to your children, mothers;
    Pass it on to your children, brothers.

    You’ve got to work for it, fight for it, every night for it;
    Pass it on to your children, pass it on.

    –this is an old hard-assed union organizing song from the ’50’s.

    So be aware that even the so-called left of yesterday were/are aware of the challenges of being – and staying – free. And the definitions of who is leftist and who isn’t kind of fades when you consider that we are all under massive attack, and most of us know it. It ain’t lefties who are the enemy. “They” are our very own authorities and the wealthiest, and they love to divide us. Look at the fact that we have more billionaires than ever, and more poor folks too. The fact is that all this is part and parcel of turning us all into serfs (done) and next, slaves.

    Slaves do the work and the benefits are put into the ledgers as profits for the masters. This isn’t getting rich by the sweat of your brow or the cleverness of a truly creative inventor, or the wealth that comes from a lifetime of work. The new wealth is all leveraging and mirrors.

    So it becomes that the battle is not between how we free people choose to live, as more social or more private, but between those who have everything, including YOUR freedoms and control over them – and those of us who are the 99.99%, who still have a bit of something left (we imagine). NSA? CIA? Government? Do you really imagine that there is any difference at all between Obama and Clinton and the Bushes and our Mormon friend? Really? They are all wealthy as all get-out, are all friends (the Bushes and the Clintons BARBECUE together, for god’s sake), and their issues are NOT ours.

    Those who would categorize us, would then control us; being shuffled into cattle chutes by those who consider themselves our masters is NOT left and right, it is control and it is the shaping of a destiny chosen by those who would believe they are superior to we the people (AKA masses) and thus have a divine right to make our decisions for us.

    And the NSA is the tool used to gather information on us and the Google is its public face. Ironic. And real.

    Personally, on some levels I hope that peak oil and its attendant problems come soon; I prefer to make my own choices rather than be shuttled, lied to and about, manipulated and then laughed at behind closed (rich) doors. And I would prefer that our leaders were on the same page as the rest of us.

    I would rather die free than live as a slave even wrapped in silk. I would rather that there were hope for the future rather than despair at the lack of hope and opportunity, and genuine community.

    And I suspect after the great fall that very few will brag about their wealth. Most of it is based on abundant and available oil. I suspect we are at peak oil, peak population, peak military and peak government. All follow, and fall, from peak oil; think it isn’t here? Look carefully. Think about the collapse likely to be very close. And after that, the NSA won’t have resources either.

    Keep that in mind when we hit the barricades, together.

  • Ted April 4, 2015, 11:09 pm

    In several currant formats, 500 megabytes per hour gives you better than VHS quality audio and video. A little back of the napkin math gives a very disturbing number. One yottabyte equals 500MB/hour for every man, woman, and child on earth, for over 31 years. While I have no idea what, specifically, the NSA has planned, they currently have the storage capacity to record every single moment of the lives of every single person alive.

  • B from CA August 20, 2015, 7:10 pm

    Acronyms: Did you know that “Pilots keep IT up longer”? What does “IT” stand for? Did you know that the GRID is an acronym for that Gay disease we now refer to with the acronym AIDS. The “G” in GRID stands for (you tell me). IT is the construction of The Tower Of Babylon; “And I will confound their language”. Whether this is by accident or by design, in the future, no one will be able to decipher our language. We simply have gone way too far with the employment of acronyms in our written and spoken communications. With that in mind I hope you will forgive me when I say in all sincerity “Hello NSA, I hope you enjoy reading my comment. I guess I have given you quite a lot of information about myself by falling for your trap and writing all these comments in so many sites. I’m Not Dangerous, just bored. If my friend happens to read this ‘I miss you and I still love you-I know the secret of my affection is no secret so with enormous respect and admiration I will end this one sided transmission (for the moment)’.”

  • B from CA August 20, 2015, 7:12 pm

    Acronyms: Did you know that “Pilots keep IT up longer”? What does “IT” stand for? Did you know that the GRID is an acronym for that Gay disease we now refer to with the acronym AIDS. The “G” in GRID stands for (you tell me). IT is the construction of The Tower Of Babel; “And I will confound their language”. Whether this is by accident or by design, in the future, no one will be able to decipher our language. We simply have gone way too far with the employment of acronyms in our written and spoken communications. With that in mind I hope you will forgive me when I say in all sincerity “Hello NSA, I hope you enjoy reading my comment. I guess I have given you quite a lot of information about myself by falling for your trap and writing all these comments in so many sites. I’m Not Dangerous, just bored. If my friend happens to read this ‘I miss you and I still love you-I know the secret of my affection is no secret so with enormous respect and admiration I will end this one sided transmission (for the moment)’.”


Leave a Comment