On the other hand, the original post we are commenting on specifically held up AIXI as an example of the kind of thing they are trying to create, so if you don't like that, then you probably will never like MIRI's research no matter how successful they are. :)

As for engaging the AI community, the most high-profile example I know of is that Stuart Russell is apparently now concerned about the MIRI-style "value alignment problem" (https://www.quantamagazine.org/20150421-concerns-of-an-artif...) and has some DARPA grant to work on it.

https://intelligence.org/all-publications/

    2015: 4
    2014: 6
    2013: 1
    2012: 5
    2011: 1
    2010: 4

MIRI publishes a lot more, but they mostly operate outside of traditional scientific journals, a lot of MIRI technical reports published on their website.

Staff: 1 out of 8 seem to have a PhD

Research Associates: 3 out of 14.

Also, AIXI is non-computable, but AIXItl is.

I just briefly tried to read up on AIXI. The concept is ...not without merit, but I think it totally sweeps the central problem (We have no idea how to program an AI) under the rug in two ways:

(1) The problem of programming ...programming anything at all, is swept under the rug by assuming a formalism in which every possible algorithm/program can be enumerated, and then just using brute force to go through, essentially search though all the character strings that represent implementations of any kind of algorithms, however bad they might be.

(2) The problem of trying to decide what we actually should want to do in complex situations is swept under the rug by just assuming that we have access to a reward function that tells the desirability of each outcome at each step.

So if this kind of helpful assumptions are allowed, the problem of, say, constructing a good chess program can be easily solved by the following meta-algorithm:

    1. List all chess programs
    2. Choose the best one

Consider that it took ~50 years to go from the inception of AI to a formal model like AIXI; or ~40 years from the definition of NP-completeness (ie. the realisation that scalability and resource usage are the real challenges for AI).

In that sense, AIXI provides two things: a hypothetical "gold standard" for AI builders to compare their research to; and a formal model which can be studied right now by those who aren't directly building AIs (like MIRI).

Consider an analogy to space flight. The engineering contains all kinds of resource constraints (eg. launch mass, strength-to-weight ratios, etc.), but it's still useful to ignore them temporarily and ask: what if we had as much fuel as we could ever want? What if we could keep the crew frozen during the journey? and so on. In other words, if we manage to overcome our current difficulties, what could we actually do with this tech?

No, not really. "If we ignore resource constraints" is ignoring most of the problem. Using Kolmogorov complexity in the Solomonoff Measure also constitutes ignoring the problem of generalization by assuming an optimal compressor into existence, which again is an issue of the cognitive resources of training data and processing power. Bayesian updating means it will achieve optimal expected reward, but also that AIXI can be "fooled" by the hierarchical nature of real environments' variance[1].

And the whole thing pays no attention to knowledge representation whatsoever.

It's basically a grand victory for the fields of AI and Machine Learning that still tells us basically nothing about how an actually existing, embodied mind has to function, except that statistical learning is most likely the core mechanism in some fashion (after all, neural networks show that the real thing isn't even necessarily Bayesian in any sense).

[1] Benjamin B. Machta, Ricky Chachra, Mark K. Transtrum, and James P. Sethna. Parameter space compression underlies emergent theories and predictive models. Science, 342(6158):604–607, 2013.

Special relativity tells us basically nothing about how an actually existing, physical spaceship has to function; but it does constrain our speculation about space travel (ie. no FTL, the fact that accelerating massive objects requires more and more energy, etc.).

It also provides some handy little suggestions that we may not have anticipated; eg. that mass can be converted into energy, which is certainly useful when trying to come up with practical designs.

But AIXI, I don't see how it constrains anything. It introduces a classification: AIXI type algorithms, and other algorithms. But we don't really know if this is a useful segmentation of the search space, or perhaps as useless as considering the merits of red spaceships vs. spaceships painted with other color.

I am not convinced. Also the reward function is assumed to magically be given. I think half of the difficulty in any real world problem would be how to design the reward function.

Also, even when assuming we have the reward function, do we really know that choosing the action that has the best weighted reward over the set on "world model algorithms" (hypothesis), produces actions that actually are intelligent? Yes, it sounds intuitively somewhat plausible, but do we have anything better than this hunch, that it sounds kinda good? Maybe it would actually turn out to produce really silly outcomes, who knows.

I am thinking, maybe the shortest "world model algorithms", which are given the largest weight, are just mostly stupid. And there is the No Free Lunch theorem, which states that averaging over all possibilities, while may sound clever, produces just garbage (i.e. no better than random guess).

That's exactly what MIRI's trying to do ;)

> do we really know that choosing the action that has the best weighted reward over the set on "world model algorithms" (hypothesis), produces actions that actually are intelligent?

As far as AIXI is concerned, this is the definition of an "intelligent action": that which leads to the largest expected utility over the agent's lifetime. It's fine to disagree with this definition, but one of the reasons to define AIXI at all is to have something concrete to point at, rather than spending decades debating these sorts of quasi-philosophical questions (note that I've carefully chosen words like "can be reasonably considered" rather than "is").

> I am thinking, maybe the shortest "world model algorithms", which are given the largest weight, are just mostly stupid.

The Solomonoff prior used by AIXI dominates all computable priors; in other words, even if these world models are stupid, no computable algorithm (including humans) can do better overall.

> And there is the No Free Lunch theorem, which states that averaging over all possibilities, while may sound clever, produces just garbage (i.e. no better than random guess).

The No Free Lunch theorem is a Mathematical curiosity with no particular relevance to the world. In particular, it completely ignores computational complexity: it gives equal weight to all (computationally) simple explanations (eg. "there is a star orbiting the Earth, it will keep orbiting" and "the Earth is spinning near to a star, and will keep spinning"), as well as to all (computationally) complicated explanations (eg. "the atmosphere has been bombarded by cosmic rays which, by sheer chance, have an effect which looks like a star, but it's unlikely for that coincidence to continue" and "the Earth is spinning near to a star, but tomorrow at 13:48 GMT the Martians, who have managed to elude all of our telescopes and probes, will attack the Earth with a weapon which switches the direction of rotation"), as well as all incomputable explanations. Trying to predict anything in such situations is clearly futile, which is basically what the NFL theorem says; yet such situations can never actually arise outside of though experiments.

Although AIXI itself is incomputable, it is specifically defined to interact with computable environments, so No Free Lunch doesn't apply.

No. Largest expected utility over the (weighted) set of all possible future timelines (i.e. hypotheses). AIXI chooses the action that gives the best average over the set of future timelines. But we only live in one timeline. Maybe an action that is very good, averaged over all possible futures, is very bad in our actual timeline?

Now we can think than an action that is good on average, maybe it is probably good in our real timeline, too. But the way AIXI gives weight to different future timelines is based on how short their MDL [1] is. Maybe this is not at all how real world works? Who knows.

A silly example: Maybe there are a lot of possible future timelines where things randomly explode. And maybe their MDL is actually shorter than for timelines where things stay stable. Then we produce a highly intelligent robot that does nothing else but seeks shelter in the nearest empty room. And this would be the "definition of intelligent action". (Defined as the maximized intelligent action over imagined future timelines where things mostly randomly explode.)

Btw, in NFL theorem, the averaging is over all possible datasets (and datasets usually describe something about past, not future), not over all possible explanations. (But yes, you can think that implicitly behind every dataset there is a multitude of world models which could have produced the dataset.)

[1] Minimum Description Length

From a superficial inspection, Hutter's back catalogue features 89 * @inproceedings, 3 * @book, 12 * @techreport, 51 * @article, and 1 * @compression prize

Over an elapsed period of '87 (from before he had completed his masters degree) to the present, on a published_artefacts/year basis, Hutter is more productive than MIRI.