Ray Solomonoff

Ray Solomonoff (July 25, 1926 – December 7, 2009) was an American mathematician who invented algorithmic probability, his General Theory of Inductive Inference (also known as Universal Inductive Inference), and was a founder of algorithmic information theory.

He was an originator of the branch of artificial intelligence based on machine learning, prediction and probability.

Ray Solomonoff was born on July 25, 1926, in Cleveland, Ohio, son of Jewish Russian immigrants Phillip Julius and Sarah Mashman Solomonoff.

At the age of 16, in 1942, he began to search for a general method to solve mathematical problems.

He attended Glenville High School, graduating in 1944.

In 1944 he joined the United States Navy as Instructor in Electronics.

From 1947–1951 he attended the University of Chicago, studying under Professors such as Rudolf Carnap and Enrico Fermi, and graduated with an M.S. in Physics in 1951.

From his earliest years he was motivated by the pure joy of mathematical discovery and by the desire to explore where no one had gone before.

He wrote three papers, two with Anatol Rapoport, in 1950–52, that are regarded as the earliest statistical analysis of networks.

In the late 1950s, he invented probabilistic languages and their associated grammars.

A probabilistic language assigns a probability value to every possible string.

In 1952 he met Marvin Minsky, John McCarthy and others interested in machine intelligence.

He circulated the first report on non-semantic machine learning in 1956.

In 1956 Minsky and McCarthy and others organized the Dartmouth Summer Research Conference on Artificial Intelligence, where Solomonoff was one of the original 10 invitees—he, McCarthy, and Minsky were the only ones to stay all summer.

It was for this group that Artificial Intelligence was first named as a science.

Computers at the time could solve very specific mathematical problems, but not much else.

Solomonoff wanted to pursue a bigger question, how to make machines more generally intelligent, and how computers could use probability for this purpose.

He was one of the 10 attendees at the 1956 Dartmouth Summer Research Project on Artificial Intelligence.

He wrote and circulated a report among the attendees: "An Inductive Inference Machine".

It viewed machine learning as probabilistic, with an emphasis on the importance of training sequences, and on the use of parts of previous solutions to problems in constructing trial solutions for new problems.

He published a version of his findings in 1957.

These were the first papers to be written on probabilistic machine learning.

Solomonoff first described algorithmic probability in 1960, publishing the theorem that launched Kolmogorov complexity and algorithmic information theory.

He first described these results at a conference at Caltech in 1960, and in a report, Feb. 1960, "A Preliminary Report on a General Theory of Inductive Inference."

Generalizing the concept of probabilistic grammars led him to his discovery in 1960 of Algorithmic Probability and General Theory of Inductive Inference.

Prior to the 1960s, the usual method of calculating probability was based on frequency: taking the ratio of favorable results to the total number of trials.

In his 1960 publication, and, more completely, in his 1964 publications, Solomonoff seriously revised this definition of probability.

He called this new form of probability "Algorithmic Probability" and showed how to use it for prediction in his theory of inductive inference.

As part of this work, he produced the philosophical foundation for the use of Bayes rule of causation for prediction.

The basic theorem of what was later called Kolmogorov Complexity was part of his General Theory.

Writing in 1960, he begins: "Consider a very long sequence of symbols ... We shall consider such a sequence of symbols to be 'simple' and have a high a priori probability, if there exists a very brief description of this sequence – using, of course, some sort of stipulated description method. More exactly, if we use only the symbols 0 and 1 to express our description, we will assign the probability 2−N to a sequence of symbols if its shortest possible binary description contains N digits."

The probability is with reference to a particular universal Turing machine.

He clarified these ideas more fully in his 1964 publications, "A Formal Theory of Inductive Inference," Part I and Part II.

Algorithmic probability is a mathematically formalized combination of Occam's razor, and the Principle of Multiple Explanations.

It is a machine independent method of assigning a probability value to each hypothesis (algorithm/program) that explains a given observation, with the simplest hypothesis (the shortest program) having the highest probability and the increasingly complex hypotheses receiving increasingly small probabilities.

Solomonoff founded the theory of universal inductive inference, which is based on solid philosophical foundations and has its root in Kolmogorov complexity and algorithmic information theory.

The theory uses algorithmic probability in a Bayesian framework.

The universal prior is taken over the class of all computable measures; no hypothesis will have a zero probability.

This enables Bayes' rule (of causation) to be used to predict the most likely next event in a series of events, and how likely it will be.

Although he is best known for algorithmic probability and his general theory of inductive inference, he made many other important discoveries throughout his life, most of them directed toward his goal in artificial intelligence: to develop a machine that could solve hard problems using probabilistic methods.