You almost certainly know that some problems are undecidable. Classic example of such a problem is a halting one. But how many problems are undecidable? Only this one? Or, may be, this one and six hundred other? Is the number of undecidable problems even countable? Well, it turns out that it’s not. While there exist countable number of algorithms (one can easily prove it by sorting programs written in some Turing-complete programming language in lexicographical order), set of possible problems has the cardinality of the continuum (proof of this fact is quite simple, it’s almost like the Georg Cantor’s proof that the set of the real numbers is not countable). So, decidable problems are like a drop in the ocean.

Other interesting fact can be proven. For any possible computational complexity (n^2, n^(n! log n) etc) there exists at least one problem for which algorithm with such complexity is the best one. That problem is “does given algorithm accept given input in no more than K steps” and it can’t be solved faster than in O(K) by actually running the algorithm.

Problem “does given statement have a proof of length N or less” is in NPC. So, if P=NP, we can choose reasonably big N and check for the existence of the proof (relatively) quickly. If there is no proof of reasonable length, we can conclude that problem is not of any interest. So, in fact, if P=NP, we should stop paying mathematicians for proofs because machines can probably do better. Unfortunately, nowadays almost everybody is quite certain that P is not equal to NP.

Another interesting topic is derandomization. Does every randomized algorithm have a corresponding deterministic algorithm? Recent success in the proof that PRIMES is in P is closely related to this question.

If the facts mentioned above make you excited, you should totally read the scribe notes. I’m out.

Here are some facts about Yandex, just in case:

• Yandex has its own computer science and data analysis school which is available for students for free. Really awesome people like Alexey Chervonenkis or Albert Shiryaev read lectures there. And Maxim Babenko’s course on effective algorithms and data structures is the best I’ve ever seen.
• Yandex hosts Internet Mathematics contest where interesting tasks somehow related to web-search are offered to participants. The goal of the contest this year is to predict the rate of traffic congestion based on previous observations. And last year contest was about learning a function that can predict the relevance of the document with respect to search query (it was just like the contest currently hosted by the Yahoo Labs). 
• Yandex is one of the two main sponsors of the TopCoder 2010 event (the other one is the U.S. National Security Agency).

 I’ll make a post with photos from the main Yandex office (which looks really great) quite soon. You stay connected :)

Here is plot from the “Elements of Statistical Learning” book comparing these two kinds of loss function on some synthetic data.

Imagine user has entered a sequence of symbols (k1,…,kn)=H (so called typing history) and then he enters new symbol by touching device screen at position l=(x, y). We’ll determine the intended symbol k by maximizing expression P(k | H) P(l | k) with respect to k. Here P(k | H) is the probability of observing symbol k given sequence of previously entered symbols (usually only last 6 symbols are considered). It can be estimated from any text corpus. P(l | k) is the probability of touching screen position l when symbol k is intended. In the paper it is modelled as a bivariate Gaussian distribution. It is also constrained in a special way to prevent total suppressing of very improbable keys (given the typing history).

Approach like that can dramatically reduce amount of typing errors on soft keyboards. Numbers, charts and comparison with the simple “static” keyboard and state-of-art dynamic keyboards with unconstrained P(l | k) can be found in the paper.

Suppose you have a 3d-tracking system that uses about 20 calibrated cameras installed in some crowded place like bank office or shopping mall. In such places calibrated cameras have a tendency to become uncalibrated sometimes, mostly due to vibrations caused by factors like construction work or cleaning. What should we do about that? Obviously, we should recalibrate cameras whose position or orientation has been changed (or change position and orientation back to original). But how can we find out what cameras should be calibrated again? Should we hire a man who’ll check image on every camera every day? Fortunately, we shouldn’t.

Camera decalibration can be detected automatically

What we have to do is to save snapshot from camera at the moment its extrinsic parameters were estimated. Then the following algorithm can be used to check for decalibration:

1. Extract something like SURF features from the saved snapshot and current camera image.
2. Match extracted features.
3. If not less than 10-15% of the matched points have the same pixel coords, camera is OK. Otherwise, it seems to be decalibrated.

As you can see, this algorithm is fully automatic and can be run, for example, every hour for every camera. It is also quite robust to presence of some temporary objects like humans in both images because it requires only small amount of matches to have the same pixel coordinates. It seems that it works bad (or doesn’t work at all) only in cases when there are very few good features available (for example, when camera looks at the white wall). But in those cases it’s very hard to detect camera decalibration even for human. One can also make this algorithm robust to illumination changes by using some lighting-invariant point descriptors or by image preprocessing.

OK, we have determined that camera has become decalibrated. And we also have old camera position, orientation and point matches between current and old camera images. It seems that we can automatically determine new camera position easily. Or can’t we?

Camera can not be recalibrated automatically

And here is why. Of course, there is a 3d reconstruction algorithm that takes a list of coordinates of matches together with camera intrinsic matrix and returns transform from old camera view space into view space of the new one (together with 3d coordinates of the points). Unfortunately, it can calculate transform and 3d coordinates only up to scale. Neither coordinates of the matched pixels nor intrinsic matrix contain enough information about the scale of you scene. Are you measuring everything in meters? Or in inches? You need more information (like real 3d coordinates of one of the matched points) to rescale algorithm results. But you don’t have them. So the reconstruction algorithm gives you pretty much nothing (well, it gives you correct camera orientation, but you can’t use it without rescaled translation).

It’s interesting that I’ve failed to find any papers covering the topic of decalibration and recalibration although it’s quite important for some practical computer vision applications. Should I write a short paper about it by myself?

That’s the very good explanation of what AdaBoost algorithm does, but to understand boosting better you should get deeper. There are another interpretations of boosting algorithm available, and some of them are really interesting.

Boosting as a forward stagewise additive modelling

Boosting can be viewed as a forward stagewise additive modelling process. Let’s assume we have some loss function L(y,y’) and want to find additive combination of weak classifiers that minimizes it. One way to do it is to use greedy approach. First, let’s select single weak classifier h1(x) that minimizes loss function L(y, h1(x)) over the training set. Then, let’s select weak classifier h2(x) that minimizes L(y, h1(x) + h2(x)). The next classifier, h3(x), should minimize L(y, h1(x) + h2(x) + h3(x)) and so on. This method has a great connection to boosting (explored, for example, in this paper or in this great book): if you choose exponential loss function L(y, y’) = exp(-y*y’), you’ll get exactly the AdaBoost algorithm.

Boosting as a gradient descent

At each step of boosting we’re selecting weak classifier that maximally reduces loss over the training set. So, we’re in fact moving in the strong classifier space in the direction of the loss function decrease. Each weak classifier selection here can be viewed as a gradient descent step. Approaches like gradient boosting are entirely based on this observation. In gradient boosting, gradient descent is performed semi-explicitly: strong classifier is represented by a vector of its values on the training set samples. Then, at each boosting step partial derivatives of the loss function (with respect to classifier already trained) are calculated, and regression tree is fitted to the gradient value in a least-squares manner. That regression tree is than selected as a weak classifier.

Boosting and game theory

Boosting has several connections with the game theory. Consider matrix Mij with number of rows equal to the number of samples in the training set and number of columns equal to the number of weak classifiers. Mij will be equal to 1 if j-th weak classifier classifies i-th sample correctly (otherwise it will have value of 0). Then it can be shown using von Neumann’s minimax theorem (applied to the game in mixed strategies with payoff matrix M) that if for every possible weight distribution there exists weak learner with error less than 1/2-eps, than there exists a convex combination of weak learners with a margin of at least 2*eps for every training sample. It means that AdaBoost has at least a potential for success. One can go further and discover that boosting can be viewed as a special case of an algorithm for approximately solving matrix games. Both topics are covered in this paper very well.

Connection to game theory also leads us to idea of using linear or convex programming techniques in boosting. Some approaches are based on it, but I’m not familiar with them very well.

1. Find a chessboard (bigger is better). Note that it should have distinct width and height (measured in chessboard squares) which should both be even (otherwise you will be unable to determine chessboard orientation given its picture).
2. Find a “calibration dude”.
3. Calibration dude takes a chessboard and waves it in front of the camera attached to a computer with calibration software installed. Calibration software takes about 20 images with distinct chessboard orientations (camera orientation remains the same, of course), finds inner corners of the chessboard on every image and then uses them together with information about real-world size of chessboard to determine intrinsic camera parameters.
4. Calibration dude puts chessboard on the floor in a way camera still can see it. Calibration software than takes one more image from camera, finds chessboard corners on it (again) and calculates camera position assuming that some predefined chessboard corner is located at the coordinate system origin and chessboard sides are oriented towards coordinate system axes. Of course, any other chessboard orientation can be specified in software, but this one is the most simple.

What problems do we have there? First of all, camera can see no floor at all, so we can’t just put chessboard on it during step 4. Instead we need to set it up somewhere else, not on the ground level. We should then carefully measure its position and orientation and pass them as an input to the calibration tool.

What if we have more than one camera seeing no floor, and those cameras are not overlapping? In this case we should repeat process described above for each camera, carefully measuring chessboard position in the world coordinate system every time. In fact, it’s a pain in the ass. Calibrating multiple cameras that way can be really slow and error-prone.

Much more interesting approach to multiple non-overlapping camera calibration was proposed in this paper. Its key idea is to fix chessboard position (put it at the origin) and move mirror instead. Cameras will see chessboard reflection in that mirror and use reflected image for calibration. Of course, some questions arise.

1. Is it legal to determine intrinsic camera parameters using reflected chessboard image? Answer is simple: yes. Authors prove that common calibration techniques give same result (except of coordinate system handedness) when applied to mirrored images.
2. Don’t we need to know position and orientation of the mirror when calibrating extrinsic parameters? No, we don’t. It turns out that every mirrored chessboard image imposes constraint on the position and orientation of the real camera. And if we have five (or more) such images, we can reconstruct position and orientation without any knowledge of mirror position.

This approach can save a lot of time and help to reduce part of the calibration error that arises from incorrect chessboard position and orientation determination. But it has it’s own drawbacks, of course. First of all, mirror is rather heavy. It’s not easy to manipulate it if your calibration dude is not a beefcake. Next, it’s hard to change orientation of the calibration pattern in frame from one snapshot to another when using mirror. It has to be in the field of view of the camera, and oriented such that the pattern’s image is reflected into the camera. These requirements may result in little variation in the pattern orientations as seen by the camera in the mirror and lead to solution degeneration.

Despite the drawbacks, this approach has the potential to increase speed of the multiple camera calibration process a lot. We will probably try it in 2010.

Let R be the event of giving the right answer to the question and M be the event that guy who answers is a maniac. From gathered statistics we know that P(R | M) = 0.98. Next, from Bayes theorem whe know that P(M | R) = P(R | M)P(M)/P(R). P(R) can be represented as P(R | M)P(M) + P(R | not M)P(not M). Next, assume that about 5 percent of usual people also gave the right answer to the question, so P(R | not M) = 0.05. Then, what’s the prior probability of M? I think we all agree that it’s quite small, about 1e-5 or even less. Now we are ready to calculate posterior probability of M:

P(M | R) = (0.98 * 1e-5) / (0.98 * 1e-5 + 0.05 * (1 - 1e-5)) = 0.0000098 / (0.0000098 + 0.0499995) ~ 0.0002.

Probability is very small, but why is that? That’s because my friend A was talking about the likelihood, but he didn’t take prior probabilities into account. And in this case prior probabilities are of great importance.

Btw, what’s if serial killers always answer right and normal people always give wrong answers? Then P(R | not M) = 0 and P(M | R) = 1, so our model works in extreme cases too.

First of all, let’s figure out factors influencing the successful completion of the trick. It looks reasonable that girl should search for your imaginary name in Google. Without it, trick wouldn’t work. It means that girl should have some internet device and area should be covered with internet (WiFi, 3G etc). Also, if your ugliness outweigh your imaginary wealth and fame, girl may not come with you (nevertheless, it’s highly improbable). Such suggestions allows us to build the following probabilistic model with discrete variables: 

In fact our model is a Bayesian network. It means that we should specify parent-conditional probability distribution for each variable it consists of. Let’s get started.

P(I) = 0.8 because almost all the modern phones have at least GPRS support.
P(C) = 0.9 because internet is available practically everywhere nowadays.
P(G | I, C) = 0.95 (wouldn’t you google for this strange man?)
P(G | not I, C) = 0.3 cause she can ask someone who has internet to google.
P(G | I, not C) = P(G | not I, not C) = 0 cause nobody can google without internet.
P(H) = 0.2 because not much guys are handsome.
P(S | G, H) = 0.99
P(S | not G, H) = 0.5 (it’s nice to be attractive, huh)
P(S | G, not H) = 0.9 cause wealth is more important than handsome look.
P(S | not G, not H) = 1e-5 (the worst case).

Computing probability of success is quite straightforward, so we will concentrate on two little more sophisticated questions:

1. What’s the probability of you being handsome if you’ve succeeded in the trick? 
2. If you haven’t succeeded in the trick, what’s the probability she had looked for your imaginary name in Google?

To compute those probabilities, we’ll use Bayes theorem together with the law of alternatives. This blog is not really a comfortable place for a lot of formulas, so the whole inference is available in the separate PDF. As we can see there, it’s not very hard to show that P(H | S) = 0.2449 and P(G | not S) = 0.2042. We can also write a small inference program using Infer.NET library which will give us exactly the same results.

Calculated probabilities agree with our common sense very well. Value of P(H | S) shows that it’s not necessary to be some handsome guy to perform a trick. And value of P(G | not S) tells us that most of the failures happen when girl refuses to google for some reason.

That was another example of probabilistic logic in action. Next time we’ll consider some continuos case like linear regression.

The most simple (but also the most hard) way is to add some kind of time bomb to the sources of every C++ compiler. That time-bomb should be activated a few years later, when everyone will be using compromised version of the compiler. After the activation of the time-bomb all the compiler executables will be destroyed on the first compilation. Nice scenario, isn’t it? The problem is to add time-bomb in a way that nobody in the team of compiler developers will notice it. An interesting problem for a great hacker, I guess. Man, if you do it, you’ll become a legend!

There is an alternative option. Not so powerful, but still possible. Every virus maker who cares should add search-for-compilers-and-delete-them-when-time-bomb-is-activated code to her (or his) brainchild. The problem here is that there are not much computer viruses for *nix systems, but most of the C++ compiler instances are concentrated there. Anyway, people, we should try!

I propose the following date for the time-bomb activation: December 12, 2012. Develop your viruses and keep it quiet.

Btw, it also can be a plot for a great action movie. Imagine some well-secured subterranean storage for the Last Compiler, a lot of men with big guns, and a bunch of heroes trying to fix the men’s biggest mistake: C++ language invention.

What’s the probability?

First of all, let’s see how do Bayesianists treat probability. In the Bayesian approach to probability theory, probability is “a measure of a state of knowledge”. It means that probability expresses one’s knowledge about value of some variable, but not the stochastic nature of the variable itself. In frequentist approach, instead, probability is an “objective uncertaincy” that arises from the nature of the experiment. Nevertheless, there aren’t much uncertain things in our world. The one who knows all the forces acting on the coin being flipped can accurately predict the outcome in experiment without involving any probabilities. The only “objective uncertaincy” known to me is the one that arises in the quantum mechanics, but, need to mention, we may discover some hidden factors influencing it someday.

Knowledge expression via random variables

Ok, probability is “a measure of a state of knowledge”. But how can we express our knowledge using it? Let’s consider a few simple examples.

Let’s assume we have some real number, A, and we don’t know it exact value, but somehow we know that it can take values only in range between 0 and 1. Then in Bayesian setting we can treat A as a random value with [0,1]-uniform distribution. Why is the distribution uniform? That’s because the only thing we know about A is its range and no other knowledge is available, so we don’t prefer some values in that range more than others.

Another example. Imagine that we have real number B, which exact value is almost known. We are pretty sure it equals to four, or at least is very close to it. How can we express that knowledge? Well, we can say that random value B has Gaussian distribution with mean=4 and very small standard deviation. In that case deviations of B from mean are possible, but highly improbable.

Prior and posterior knowledge

The most important formula in the Bayesian probability theory is, of course, the Bayes’ law. It shows us how our prior knowledge about A changes after observing B and becomes posterior knowledge. Bayes’ formula has rather simple form:

The first term in numerator is so-called likelihood function, which connects observation with unknown value. Actually, it shows how probable it is to observe given B for some value of A. The second term, P(A),  represents our prior knowledge about A, all the stuff we know about it regardless of any observations. Denominator does not depend on A, it’s just a constant for normalizing posterior knowledge cause we want it to be probability too. Result, P(A|B), is a conditional probability of A given some value of B. In terms of knowledge it is the answer to the question “What probability distribution represents the knowledge about A after we’ve learned the value of B?”

What can we do in Bayesian setting?

So, what’s it all about? Why do we need probabilities to encode our knowledge? What’s the profit? The profit is that we can infer knowledge using Bayes theorem. More formally, we can solve the following problem: “Given a set of observations B, related to unknown A with a likelihood function P(B|A), found A*, an estimate of A”. Common statistical approach there is using maximum likelihood approach: A* = argmax P(B|A), which simply finds the value of A for which the likelihood value is maximal. While this approach is asymptotically efficient, it works well only when B contains a lot of observations. Nevertheless, if we don’t have much obserations in B, but, instead, have some prior knowledge about A, we can estimate A* = argmax P(A|B) = argmax P(B|A) P(A) (MAP estimation, which is in fact maximum likelihood approach regularized with prior knowledge) or even A* = EA, where expectation is taken with respect to conditional distribution P(A|B) (Bayes estimator). When the size of B is small, Bayesian approach works much better then ML estimation. It works even when there are no observations at all (in that case, all the knowledge is just taken from prior distribution P(A))! In the same time, in cases when B contains a lot of information, Bayesian estimation works as good, as the ML approach and it’s also asymptotically efficient.

Bayesian approach and boolean logic

Bayesian approach can be viewed as an extension of a boolean logic for cases when we are not certain if some statement is true or not. More detailed explaination can be found, for example, in the “Probability Theory: The Logic of Science” book by Thompson Jaynes. We will consider, instead, a few intuitive examples:

Boolean logic: A&B is true, B is true. What’s the value of A? Well, A is also true.
Probabilistic logic: P(A, B) = 1. Then P(A | B) = P(A, B) / P(B) = P(A, B) / (P(A, B) + P(not A, B)) = 1 / (1 + 0) = 1.

Boolean logic: A->B. B is true. What’s the value of A? Oh, we don’t know. There is uncertaincy!
Probabilistic logic: P(B | A) = 1. P(A | B) = P(B | A) P(A) / P(B) = P(A) / P(B) = P(A) / (P(A) + P(not A) P(B | not A)).

In the first example probability theory just draws the same conclusions about value of A, as boolean logic do. Simple and obvious, isn’t it? Let’s then look at the second example. In it, classic boolean logic tells us nothing about value of A even if we know for sure that B is true. But in Bayesian setting it’s a just a posterior knowledge inference with a known likelihood. The answer we’ve got is “Value of B depends on prior distributions of A and B”. If we observe B=true four times more frequently than A=true, or, in Bayesian treatment, we are four times more sure that B is true than that A is true, then P(A | B) = 0.25 and we are almost sure A is false. It’s very natural because if we see B more often than A, it means that B does not always follow from A and, so, when we see B, A can still be false.

I want you to feel that probabilistic logic is much more powerfull than classical boolean logic, especially when reasoning about real world, where almost everything is not fully known, but a lot of prior knowledge is available.

Bayesian approach and Occam’s razor

One of the important philosophical principles, on which all the modern science rely, is the Occam’s razor. In latin it’s phrased as “entia non sunt multiplicanda praeter necessitatem”, which means “entities must not be multiplied beyond necessity”, and, so, the most simple explanation is always preferrable when alternatives are available. How is it related to the Bayesian approach?

First, let’s look at the Bayes’ formula again. Assume that M is some model of D, observed data. Then P(M|D) is a knowledge about what model is prefferable when D is observed. Using Bayes’ formula, P(M|D) can be decomposed into normalizing constant, likelihood term, which expresses how well model M fits to observed data, and also P(M), prior knowledge about M. If P(M) gives more probability to the more simple model, than among all the models having the same level of fitting to data, the most simple one will be selected.

But that’s not all. Assume that you have two models with equal prior probabilities, say, M1 and M2. When do we select M1 then? It’s quite reasonable to select M1 when P(D|M1) > P(D|M2) and to select M2 in all the other cases. Next, if M1 represents more simple model than M2, then M1 can explain less possible data than A2 can because A2 is more flexible. But it also means that P(D|A1)
 is greater for well-explained data. That’s why we’ll select more simple, constrained model even when prior probabilities of models are equal. Was it unclear? Look at the picture I’ve taken here:

Conclusion

I hope I’ve interested you in Bayesian approach even a little a bit. More posts will follow. What do you want to hear about? That’s what I have in plans:

• An example of a more sophisticated inference in some simple bayesian network, must be related to hot chicks somehow :)
• Linear regression in Bayesian setting. Regularization.
• Automatic relevance determination.
• Infer.NET examples for everything

Currently I’m thinking about series of posts related to Bayesian approach to probability theory and, especially, to statistical learning. Infer.NET examples will be involved for sure.