The one truly pertinent bit of information relevant to this discussion:

NOTHING IS PERMANENT, on the internet or otherwise.

Who here remembers "newsgroups"? MySpace? AOL Chatrooms? YahooGroups? Geocities? The Internet Ring?

And rest assured that there will come a day when we laugh derisively at "Remember Facebook?" Possibly even Google.

Even the Internet Wayback Machine and various "archive" sites could die, victim of a hacker attack or China. (Okay, yeah, I'm being paranoid, but beats the alternative.)

If information like this is NOT backed up offline, either digitally or in hard copies, it only takes one person not updating his credit card info with the host to lose it all. We in effect went through this with all the microfilm preservation of newspaper archives--the papers were discarded, and the microfilm has started to prove its impermanence.

Resources like Google Docs are great for SHARING resources on a short-term basis, but NOT for "permanent archiving. Back up obsessively. Label the backup external hard drives.

Our one library has one member whose self-appointed job it has become to obsessively cruise the internet looking for relevant material on RailPictures, eBay, Facebook groups, etc. and save a digital copy, complete with credit and identification. He claims to be saving the raw information for research; we're little concerned about copyright or credit in instances like RPO marks, tickets, timetables, book pages a hundred years old, etc. He catalogues into folders as appropriate (by RR, usually), and I've taught him, to standardize abbreviations and the like so any properly-entered search will find what we seek.

There is no dodging. I apologize if I assume everyone knows how the internet works. I’m shutting down my web site and cancelling my hosting service which currently hosts these files so you can download them for free. This coincides with the closing of my business. If I don’t have a business, I don’t need a website. I’m donating the digital document(s) to the best possible long term space that wants to keep them online so they remain accessible to those in rail preservation. It’s not an index, it’s individual PDF files. Just like what Preston is doing with HRA except the cross reference was compiled and generated by me, not EMD.

However, I want to be able to to submit updates to the paint cross reference file for free as others send new info to me. I’m not looking for freebie web site hosting. I’m done with that. Just looking for some place that is willing to keep the file online. This is for the benefit of all. It is currently easy to find online by searching in your browser. If no one wants to host the file, it goes offline when I close my website thus can’t be found unless you know how to contact me directly and ask for a copy. So, if no one can find it or knows it it exists, why would I bother compiling all the data?

I’ve had too many people thank me for this project and the free research and consultation I do regarding color to have it just disappear.

Sorry if my response sounds a little snarky but that is the best way I can explain in.

Answer OP question yes such a place does exist. I have even use the internet archives to post locomotive manuals. I've also seen other railroad organizations use this website.

www.archive.org
https://archive.org/details/copy-of-p-42-dc

The problem with not setting robots.txt so the Internet Archive 'archives' your material is precisely that he wants to do periodic updates to the 'version of record' to which people would hardlink.

Internet Archive would be able to hold sequential 'copies of record' with particular dates, but you'd need to know the date of the 'latest' in order to find it with any authority. And that is what he does not want.

On the other hand, I see repeated instances where sites and people link to an archived page or even site where the links still operate, and the files themselves are perfectly accessible.

Robots.txt can only restrict a bot from indexing a folder or file and the restriction isn’t necessarily honored by all bots. As far as bots indexing a file for search engines go, as long as the file name doesn’t change, it’ll remain indexed no matter what the file date is. If you change the name you’ll have to wait until the site is crawled again for bots to pick it up and you can wind up with the dead links still in the index. Changing the name also breaks any hard outboard links to it. So keeping the name the same for any updates eliminates a lot of problems. I suppose you could write redirects in the .htaccess file for each update but that would eventually get to be larger than it should be. Not good. And Google hates redirects and will ding you in search ranking if you have more than a couple of the necessary redirects.

That’s probably a bit TMI but there ya have it.

Traditionally it fell upon institutions such as libraries to be the repository for printed knowledge.
Certain institutions always held specialized collections. Going to the digital age, it makes a lot of sense for HRA to be at the forefront of this. They are the specialized organization which best represents the field.

Right now the "technical library" section is pretty basic. If you think about all the technical documents which are held across all the railway museums, tourist railways, public transit agencies, etc. not to mention those in private collections, some kind of investment in cataloging is ulltimately going to be necessary. You can't scale a "google drive" model to hundreds of thousands of items and hope for it to be usable and maintainable.

And once you take a printed archive and start scanning it, the generated files and topics grow rapidly. Just one 4-inch binder of equipment information accumulated over 40 years by a Sales Engineer for a major locomotive builder generated over 600 individual charts, tables, notes, and correspondence files. Making that information useful and accessible depends on how well the file naming and indexing system performs.

Some organizations do a good job with the naming and indexing, some do not. You can take a document you consider well named and described in your own system and send it to an organization where the webmaster renames it in a way that makes it far less descriptive or useful when posted.

In doing training work in the locomotive building industry a required step in developing a new program was to define the presentation formats, styles, and document naming standard. In some cases this was a customer imposed contract requirement, although you might be surprised to see some of the very major railroad industry organizations that had no standard at all.

PC

TMI that didn't address the issue in question.

Internet Archive, which was the thing being discussed in context, respects robots.txt in determining what to archive. This is a very unfortunate reason many very significant Web sites are 'gone forever'.

And sometimes even a supposedly good indexing/filing system is inadequate for searching outside the primary area of interest. What is to my knowledge the only known detail on the Ingalls Shipbuilding 2000hp passenger unit is buried in the Bowes collection at the seaport museum in Philadelphia, and you have to 'walk the boxes' to realize what it is. Without a 'finding aid' I wouldn't even know to look there.

In good website design, you want your site to be able to be crawled and indexed. This is how search engines find and display search results. If you disallow your entire site, unless there’s a specific reason for doing so, there’s not much point in having one for the public to see if it can’t be found. Generally, the only things that you may not want indexed are members-only portions of your website, folders used testing or that hold scripts, or any area you don’t want easily found and have links displayed to the general public. Member areas and the like that are disallowed is where the internet archive will not pick up data and/or functionality if it’s crawler respects robots.txt. I don’t have an easy solution to that.

In those situations you include some logic for user agent in your authentication.

If it's a bot you like, you let them in.

Hello,

Just to add another organization to the list of those preserving materials...........

www.industrialarchives.org

They are worth investigating.

Respectfully,

Dave Griner

This thread switched over to the wrong siding belonging more in a beginner web development class.

Anyway, to clear up any confusion for those very few who care to know:
Put simply, bots, just like the web browser software on your device, use User Agent to identify what they are to the server. As a refinement, you can disallow specific bots access to your server via their User Agent in the robots.txt file versus the blanket disallow all. All others not restricted would get a pass by default. That’s how you pick and choose. It’s not a matter of defining which to allow, but which to exclude.

That's a dangerous inaccuracy in the way robots.txt works.
It only asks bots not to do whatever you tell it. They still can, there's nothing stopping them, it only works if they follow the protocol.

When you're dealing with well known above board ones, functionally there's no difference, but don't expect robots.txt to defend you from nefarious actors.

.