jump to navigation

What Brought About our AI Revolution? July 22, 2017

Posted by Peter Varhol in Algorithms, Software development, Software platforms.
Tags: , , ,
add a comment

Circa 1990, I was a computer science graduate student, writing forward-chaining rules in Lisp for AI applications.  We had Symbolics Lisp workstations, but I did most of my coding on my Mac, using ExperList or the wonderful XLisp written by friend and colleague David Betz.

Lisp was convoluted to work with, and in general rules-based systems required that there was an expert available to develop the rules.  It turns out that it’s very difficult for any human expert to described in rules how they got a particular answer.  And those rules generally couldn’t take into account any data that might help it learn and refine over time.

As a result, most rules-based systems fell by the wayside.  While they could work for discrete problems where the steps to a conclusion were clearly defined, they weren’t very useful when the problem domain was ambiguous or there was no clear yes or no answer.

A couple of years later I moved on to working with neural networks.  Neural networks require data for training purposes.  These systems are made up of layered networks of equations (I used mostly fairly simple polynomial expressions, but sometimes the algorithms can get pretty sophisticated) that adapt based on known inputs and outputs.

Neural networks have the advantage of obtaining their expertise through the application of actual data.  However, due to the multiple layers of algorithms, it is usually impossible to determine how the system arrives at the answers it does.

Recently I presented on machine learning at the QUEST Conference in Chicago and at Expo:QA in Spain.  In interacting with the attendees, I realized something.  While some data scientists tend to use more complex algorithms today, the techniques involved in neural networks for machine learning are pretty much the same as they were when I was doing it, now 25 years ago.

So why are we having the explosion in machine learning, AI, and intelligent systems today?  When I was asked that question recently, I realized that there was only one possible answer.

Computing processing speeds continue to follow Moore’s Law (more or less), especially when we’re talking floating point SIMD/parallel processing operations.  Moore’s Law doesn’t directly relate to speed or performance, but there is a strong correlation.  And processors today are now fast enough to execute complex algorithms with data applied in parallel.  Some, like Nvidia, have wonderful GPUs that turn out to work very well with this type of problem.  Others, like Intel, have released an entire processor line dedicated to AI algorithms.

In other words, what has happened is that the hardware caught up to the software.  The software (and mathematical) techniques are fundamentally the same, but now the machine learning systems can run fast enough to actually be useful.

AI: Neural Nets Win, Functional Programming Loses October 4, 2016

Posted by Peter Varhol in Software development, Software platforms, Uncategorized.
Tags: , , ,
add a comment

Today, we might be considered to be in the heady early days of AI commercialization. We have pretty decent speech recognition, and pattern recognition in general.  We have engines that analyze big data and produce conclusions in real time.  We have recommendations engines; while not perfect, they seem to be to be profitable for ecommerce companies.  And we continue to hear the steady drumbeat of self-driving cars, if not today, then tomorrow.

I did graduate work in AI, in the late 1980s and early 1990s. In most universities at the time, this meant that you spent a lot of time writing Lisp code, that amazing language where everything is a function, and you could manipulate functions in strange and wonderful ways.  You might also play around a bit with Prolog, a streamlined logic language that made logic statements easy, and everything else hard.

Later, toward the end of my aborted pursuit of a doctorate, I discovered neural networks. These were not taught in most universities at the time.  If I were to hazard a guess as to why, I would say that they were both poorly understood and not worthy of serious research.  I used a commercial neural network package to build an algorithm for an electronic wind sensor, and it was actually not nearly as difficult as writing a program from scratch in Lisp.

I am long out of academia, so I can’t say what is happening there today. But in industry, it is clear that neural networks have become the AI approach of choice.  There are tradeoffs of course.  You will never understand the underlying logic of a neural network; ultimately, all you really know is that it works.

As for Lisp, although it is a beautiful language in many ways, I don’t know of anyone using it for commercial applications. Most neural network packages are in C/C++, or they generate C code.

I have a certain distrust of academia. I think it came into full bloom during my doctoral work, in the early 1990s, when a professor stated flatly to the class, “OSI will replace Ethernet in a few years, and when that happens, many of our network problems will be solved.”

Never happened, of course, and the problems were solved anyway, but this tells you what kind of bubble academics live in. We have a specification built by a committee of smart people, almost all academics, and of course it’s going to take over the world.  They failed to see the practical roadblocks involved.

And in AI, neural networks have clearly won the day, and while we can’t necessarily follow the exact chain of logic, they generally do a good job.

Update:  Rather than functional programming, I should have called the latter (traditional) AI technique rules-based.  We used Lisp to create rules that spelled up what to do with combinations of discrete rules.

No AI Cannot Solve the World’s Biggest Problems February 23, 2016

Posted by Peter Varhol in Software development, Software platforms.
Tags: ,
add a comment

‘AI can solve world’s biggest problems’ – Google Brain engineer.

The headline screamed for a response. The type of work being done with Google Brain is impressive.  It relies on neural networks.  Neural networks are cool things.  I was recently reminded by a new LinkedIn connection of the work and writing that I did on neural networks in the 1990s.

Neural networks are exceptionally good algorithms. They use multiple levels of nonlinear equations, running in parallel and serially, that can approximate a given response to a set of data inputs, then adjust the variables so that each successive pass can result in a slightly better approximation (or it may not; that’s just the way nonlinear algorithms work sometimes).

That may sound like just so much gobbledygook, but it’s not. In the 1990s, I used neural networks to design a set of algorithms to power an electronic wind sensor.  In my aborted doctoral research, I was using neural networks to help dynamically determine the most efficient routings through large scale networks (such as the nascent Internet at the time).

Let’s be clear. What I did had nothing to do with the world’s biggest problems.  The world’s biggest problems don’t already have mountains of data that point to the correct answer.  In fact, they rarely, if ever, have correct answers.  What they have is a combination of analysis, guesswork, and the many compromises made by dozens of competing interests.  And they will never result in even close to a best answer.  But sometimes they produce a workable one.

From my own experiences, I came to believe that having an instinctive feel for the nature and behavior of your data gave you a leg up in defining your neural network. And they tended to work best when you could carefully bound the problem domain.  That is hardly the stuff of something that can solve the world’s most difficult problems.

Neural networks can seemingly work magic, but only when you have plenty of data, and already know the right answers to different combinations of that data. And even though, you are unlikely to get the best possible solution.