Just a note about redaction -- my personal experience with FBI redaction is that it is completely subjective based upon the person who happens to be doing the redaction. There isn't some giant conspiracy to omit facts -- sometimes the person redacts too much, other times they screw up, etc. I've had names redacted on one page, only to appear non-redacted a few pages later. Even in cases where I've sent in death certificates for certain names (I know for a fact who the redacted person is), the name is still redacted in the output. I'm sure it is a very laborious, human process and all such processes involve a lot of errors.
Yes, this is an excellent point to consider. A conspiracy seems sexier, but it comes down to the human potential for error and the bureaucratic fear of torpedoing your own career (or worse, someone else's life) just because of a mistake in a menial task. If you were the redactor and the redactor's supervising manager, if something is a gray area and your lawyers aren't giving this their full attention, why not redact?
Of course, any one who is interested enough to file for documents may already know what he/she wants and will call you out (with a lawsuit) if it seems you've redacted things unnecessarily. But for other situations, it seems unlikely that the requester will either not go through the trouble of suing or not even know if what was redacted is worth suing over.
An example of a high-profile amusing snafu occurred during the Rod Blagojevich trial when someone using Adobe Acrobat forgot to complete the redaction process, allowing redacted text to be copy-pasted into another text editor: http://capitolfax.com/2010/04/22/blagojevich-hurls-allegatio...
Yes, that was a technology-error rather than one of judgment...but technology errors are much easier to double-check for and yet it's not an uncommon error in the digital age of public documents.
Overuse of classification labels is a huge problem. The incentives are skewed as you suggest, so there is no real benefit to spending the time required to make decisions on each and every individual fact or statement. Entire swathes of boring information are classified for no real reason and a large number of things that someone might use as useful data are sequestered in random agencies. The cost of complying with FOIA is not insignificant and over-classification jacks up the cost. I don't know if there is a really effective way to "fix" the situation without somehow changing the incentives.
Having had a great deal of personal experience with the FBI and the channels and management they're required to use and satisfy, I can verify that it's entirely accurate that everything is down to the individual. It's exactly the same kind of bureaucracy people complain about in large companies, except that the very bare possibility of change we hold on to in large corporate environments is not present.
Working with that organization in particular will teach you very quickly about the kind of immense roadblocks that exist to even the more "moderate" of conspiracy theories. Not only are they beholden to absolutely everyone, they don't have time to make things up or bury facts. Obviously there are exceptions, but this and most other government agencies are just loose conglomerations of individuals doing a never-ending amount of thankless work with the reward of being scrutinized and accused of every possible crime at every possible turn.
I hold government and its agencies in no high regard, because people are people and we all suck in our own way, but suggesting conspiracy at certain levels is just laughable.
