2008-08-27

Detecting record structure in a file

As a signal processing kinda person, I'm well acquainted with the properties of the Fourier transform. It is usually thought of as a tool that has mostly to do with the discretized continuous time signals. But in the abstract, it's founded in shift-(in)variance. That makes it pretty much the ultimate tool for detecting cyclic structures, even if they're discrete at the origin.

I'm also a file/data format freak. A data representation enthusiast, if you can call anybody that. That makes me think, how about simply taking a long term FFT of a data file, in order to bring about any periodicity/record structure in it? Perhaps that sort of operation could serve as a reverse engineering tool when repetitive, record structures are present. And, looking into the Fourier statistics, perhaps it could even help calculate some sort of implicit measure of fractal dimension, which could help figure out the dimensionality of the record structure being looked at.

Virus conspiracy

Do you want a real-time, scifi, conspiracy theory? I think I can just about cook one up.

Why is it that commercial operating systems and software in general is so vulnerable to third party crackers? Usually I'd say it's because of stupidity and lacking incentives, but when I want to go fictional, I'd say it's because of the *right* incentives.

I mean, how do you hook your pristine generation of programmers to your system? By making it "secure"? I think not. You do so by making it vulnerable to every hack that a 15-year old wannabe programmer finds interesting. You go with the von Neumann architecture, in lieu of the Harvard one. You allow people to see the bytecode, and vice versa feed it to the interpreter. You let buffer leaks stay in place, *simply* to lure those script kiddies in, and so to build the programmatic ecosystem of five years from now.

Then on the other hand, open source software tends to be under the control of master hackers, acknowledged as such by their peers. It works by different rules altogether. A Linux guru would *not* want some random idiot to wrest away the power from hir. So those systems tend to be stabler and more secure; no virus to be seen in my installation of Debian, at least.

So perhaps we could argue that the long term sociological and economical effects that go along with commercial software production are the prime reason they're plagued with viruses and the like? Perhaps that mode of software production invites such a complacent attitude towards exploits that it could even be called a widespread "virus conspiracy"?