Summary: I wanted to publish a link to information about a nuclear facility located near my home. On the way I found out that an "anti-secrecy" organization has removed information from their site due to "security concerns". I also learnt how the Internet Archive works and found out that it is not reliable, datewise.

An official Secret
Last Saturday I was at the swimming pool with a friend and we heard the sound of a small airplane. Then we saw it flying over the hills and I said to him "you know, it's strange, it's not supposed to fly over there". We both knew what I was talking about, because, you see, According to Foreign Sources, there are caves in those hills, where missiles carrying nuclear warheads are stored.

I can see the hills from my porch. I see them when I am with the kids on the lawn. We all know what's there, they are always there and we do not talk about it. It's a secret. An Official Secret, so this is the reason why I have to write "According to Foreign Sources".

Even the Israeli Censorship is not unsensible. Should it be published anywhere, then it is no longer a secret. What the Israeli media do, therefor, when they want to write about an Official Secret is mention "According to Foreign Sources".

What happened to the information about Israel?
There is, however, an un-expected twist in this tale. For the purpose of publishing this story, I wanted to offer a link to a Foreign Source, specifically to a page on the website of the Federation of American Scientists. If you don't know FAS, here is what they write about themselves:

"The FAS conducts analysis and advocacy on science, technology and public policy, including national security, nuclear weapons, arms sales, biological hazards, secrecy, education technology, information technology, energy and the environment… FAS was founded as the Federation of Atomic Scientists in 1945 by members of the Manhattan Project who produced the first atomic bomb, to address the implications and dangers of the nuclear age. FAS is the oldest organization dedicated to ending the worldwide arms race, achieving complete nuclear disarmament, and avoiding the use of nuclear weapons, and much of its work has been in nuclear arms control and disarmament. FAS combines the scholarly resources of its scientists with a knowledge of practical politics…"

I remembered they had a link to a page describing and showing maps and satellite images of the nuclear base near my home. I looked for that page on their site but could not find it. Gone. All the links to the Israel facilities, nuclear and none-nuclear, had been removed from the site. Also, all the "Special Weapons News" about Israel from 2001 onwards have been removed.

Searching the Internet Archive I found the missing information.

www.fas.org/nuke/guide/israel/index.html

There were two more links on this page. One to a list of Israeli nuclear facilities and another one named "targets" with a list of the largest cities in Israel, their population and their geographical location (longitude & altitude).

According to the Internet Archive, the link was removed sometimes between May 21st 2001 and June 4th 2001. I was puzzled. Why did they remove the information from their site, and why then? Anyone could have argued that they did it because of the 9/11 attack, but why before that date?

To solve this mystery, I contacted FAS and asked them. Here is their answer:

"After September 11, 2001, links to nuclear facilities in Israel, India, Pakistan, and the United States were removed from our web pages. This was done in the interests of keeping information that could be used by terrorists off of our webpages."

Is Archive.org reliable?
I was not satisfied with this answer because I knew the information was removed before 9/11. I asked again and got this answer:

"Our records show that the Israel nuclear web page was modified in April 2001 and again in October 2001. I don't believe anything changed between April and October 2001. In October, we withdrew many pages concerning nuclear and intelligence facilities around the world because of security concerns."

I was baffled. The person answering my question is the director of the "FAS Project on Government Secrecy". At the top of the projects' homepage they write:

"Through research, advocacy, and public education, the FAS Project on Government Secrecy works to challenge excessive government secrecy and to promote public oversight."

I had no reason to suspect he was lying to me so I checked again.

When first checking the Internet Archive for the missing information, I kept the links to the pages with and without the information I was looking for. Indeed, one link had the information and another link didn't. I double-checked. I went back to the first page of fas.org on archive.org to make sure that the pages I was looking at were indeed from the dates I knew they were from. Lo and behold, they were not.

Now, when I write this, I am taking the same path again. Follow me to the page titled "Israel Special Weapons Guide" from May 21st 2001.

First, I clear my browsers' history and cache to make sure I am looking at the correct information.

I begin at web.archive.org/web/*/http://www.fas.org

I click May 21, 2001 * and arrive on the homepage of fas.org
web.archive.org/web/20010521214206/http://www.fas.org/

I click Special Weapons
web.archive.org/web/20010524205143/www.fas.org/nuke/index.html

I click Special Weapons News
web.archive.org/web/20010917033906/www.fas.org/news/index.html

I click Israel
web.archive.org/web/20011024162200/www.fas.org/news/israel/index.html

I click Israel Special Weapons Programs
web.archive.org/web/20011109042134/www.fas.org/nuke/guide/israel/index.html

I arrive at a page titled "Israel Special Weapons Guide" that doesn't have a link to "nuclear facilities". But it was supposed to be there. May 21st 2001 was before September 11th 2001, right?


Now I will take the same path but will start at the site when it was archived on June 4th 2001.

web.archive.org/web/20010604005326/http://www.fas.org/index.html
web.archive.org/web/20010603024613/www.fas.org/nuke/index.html
web.archive.org/web/20010917033906/www.fas.org/news/index.html
web.archive.org/web/20011024162200/www.fas.org/news/israel/index.html
web.archive.org/web/20011109042134/www.fas.org/nuke/guide/israel/index.html

Surprise! I reached the same URL as of May 21st 2001. And it also doesn't have a link to "Nuclear Facilities" either.

I reached the same URL starting at a different date.

How does the Web Archive work?
You would have thought that the Web Archive stores each page it archives and when you ask for this page you get back the exact same bits that were archived on any date.

We now know that it doesn't work that way. I assume that they store each particular page in order to save on disk space. If a page hasn't changed, they point to the last changed version of the page.

But there is a problem here. When I first begun writing this paper, I kept the URL of the page that I reached archived on May 21st 2001.

Here it is
web.archive.org/web/20010201173300/www.fas.org/nuke/guide/israel/index.html

It's not the same URL as the one I reach now when I begin at May 21st 2001 and this page has a link to "Nuclear Facilities".

There are two problems here:

1. The web archives' behavior is inconsistent


2. There is a contradiction between what I was told by FAS and what I saw on the Web Archive.

I think there is only one conclusion: The Web Archive is not reliable.

But there is more...

Beware of the robots

When I go to the page that has the link to "Nuclear Facilities" and click it, I get an error page saying:

Robots.txt Retrieval Exclusion.
We're sorry, access to http://www.fas.org/nuke/guide/israel/facility/index.html has been blocked by the site owner via robots.txt.

There is an explanation for this.

During my Email exchange with FAS, I wrote my contact there that the information they have removed from their site is still visible on the Web Archive.

My contact wrote me:

It was my intention to remove these pages from the Web Archive also - I will now do that.

I thought that when I will click this link I will find a message I have already seen on archive.org

web.archive.org/web/*/http://www.sigall.co.il

Blocked Site Error.
Per the request of the site owner, http://www.sigall.co.il is no longer available in the Wayback Machine. Try another request or click here to see if the page is available, live, on the Web.

This is strange.

The error page of archive.org has a link for showing the content of the robots.txt file that prevented the archiving of the requested page. Here it is:

User-agent: * 
Disallow: /eye/kosovo
User-agent: ia_archiver 
Disallow: /irp/overhead/
User-agent: ia_archiver
Disallow: /irp/facilities/

There is nothing in the archived robots.txt file to suggest blocking of information on Israel.

There is an explanation for this. In the Internet Archive FAQ, they write:

If a web site owner ever decides he / she prefers not to have a web crawler visiting his / her files and sets up robots.txt on the site, the Alexa crawlers will stop visiting those files and mark all files previously gathered as unavailable. [my emphasis]

The Internet Archive prevents viewing of archived pages according to the current robots.txt of a site but the error message links to the archived robots.txt.

The content of the current (June 27th, 2002) robots.txt is:

User-agent: * 
Disallow: /eye/kosovo
User-agent: ia_archiver 
Disallow: /irp/overhead/
User-agent: ia_archiver
Disallow: /irp/facility/
User-agent: ia_archiver
Disallow: /nuke/guide/israel/facility/
User-agent: ia_archiver
Disallow: /nuke/guide/israel/target/
User-agent: ia_archiver
Disallow: /nuke/guide/usa/facility/

User-agent: ia_archiver
Disallow: /nuke/hew/

I asked for archive.org's comments on what I wrote here and this is their answer:

hello,

As for the accuracy depicting the way wayback works, i'm sure what's written is true, but there are easier ways to say it.

i think wayback will consistently respond with the same results for a given search.

if wayback is looking for a file on a date and doesn't find it, it looks for the file on a later date. so you will sometimes change dates as you surf wayback.

wayback tells you when files have been excluded and gives you different messages based on the method of exclusion. there are 2 ways for site owners to exclude files or directories from wayback: automatic via a users robots.txt file, or to request a manual exclusion via email. it gives you the same message each time you look for a file.

will you come across links to pages that are not in wayback? yes. we try to have every page in our archive, but fall well short. we're learning and doing a better job as we learn.

No more secrets anymore
On June 24th, the police surrounded my hometown of Beit Shemesh. There were hot alerts that a Palestinian terrorist is heading towards the town carrying a bomb. On On June 26th, this was published on the website of Haaretz, one of Israels newspapers.

Report: A-bomb and missile storage dump was terror target

By Ha'aretz Service

According to a British media report,[my emphasis] the six Palestinians arrested this week near Beit Shemesh were en route to a suicide attack against an Israeli arms warehouse in which nuclear bombs and Jericho surface-to-surface missiles are stored, Army Radio reported Wednesday.

It quoted the British Foreign Report as saying that the planned attack was thwarted as a result of information received by the Shin Bet security service and by a large-scale manhunt that resulted in the capture of the six late on Monday.

"Within Israel's defense establishment, officials are denying this morning that security information of this nature was indeed received," the radio said.

Foreign news reports have said in the past that Jericho missiles and nuclear warheads and other devices were stored in underground bunkers near Moshav Zecharia, not far from Beit Shemesh.

I do not believe that the terrorist was heading towards the (heavily guarded) nuclear facility. It's more likely that he was on his way to the town itself. But the information on the nuclear facility near my town was openly published, relying on Foreign Sources, so I am allowed to write about it and now I also have the link I was looking for.

And another thing. Even if the Palestinian terrorists didn't know about the Nuclear Facility near Beit Shemesh because FAS has removed the information from their website and made it unavailable on the Internet Archive, now they know, so what's the point?

Update: the Bulletin of the Atomic Scientists, September/October 2002, a report on Israeli nuclear forces, 2002 (3 pages PDF).