What exactly connects algorithms to the issues they are built to solve?
One way of answering this is to look at the No Free Lunch Theorem.
This set of ideas was developed by Wolpert and Macready in 1997 with the aim of shedding light on certain Machine Learning Algorithms and the related problems that they were built to solve. Before jumping into where the NFLT holds a special place in the AI industry now, however, it is important to understand its’ philosophical basis.
By all accounts, everything begins with David Hume, who is a philosopher that is celebrated for his scientist-like approach to the field. Reportedly, the No-Free Lunch Theorem was developed within the field of supervised learning based on some of the key ideas in his work. Apparently, one of Hume’s major conclusions in his work was that we should not make connections between things simply based on their continuous interactions or connections unless we have had some sort of personal experience with these things. Related to this, our primary source below claims that the No-free Lunch Theorem, serves to “formalize” Hume’s conclusion above and even, his overall work.
To better understand if this actually could be true and this theorem has really had that much of an impact already, it is important to look at the work of other professionals on the subject, however, before we get there, we can turn to Wolpert himself. Wolpert does clarify the assertion that his and Macready’s work “formalized Hume” by attempting to precisely explain the way this was done. Essentially, he states that in the end, their work effectively proved that science cannot predict the results of future experiments by using the results of past experiments. Inside of the theorem itself, it also appears that both researchers concluded that no system in existence at the time had the capability of predicting any sort of future experiment, related to any sort of future variable. They even appear to specify this statement by suggesting that no computer will be able to create a reliable “prediction algorithm.”
Given that a wide range of sources has provided evidence to the contrary over the past few years, we can then ask: what place does the No-Free Lunch Theorem hold today? Does it merely serve the purpose of reminding us that no prediction system will ever be completely accurate?
So where do we go from here?
For now, it seems reasonable to conclude that the No-Free Lunch Theorem serves as a mathematically based reminder that no prediction system can be perfect. In part two of this series, we’ll examine just how likely this is to be true in the context of current AI research.
Resources:
Primary Sources: http://www.no-free-lunch.org
http://www.no-free-lunch.org/coev.pdf
Further Reading: https://medium.com/@LeonFedden/the-no-free-lunch-theorem-62ae2c3ed10c
