How often does your algorithm learn? And other important questions to ask ML vendors

Vendors are quick to include AI in their sales pitch, but Principal Engineer and Chief Data Scientist recommend technology buyers ask three questions to help see through the hype to determine what they are actually getting.

Fralick chairs McAfee’s Analytic Center of Excellence and is responsible for the cybersecurity company’s technical analytic strategy that integrates into McAfee consumer and enterprise products.

Prior to Intel’s divestiture of McAfee, she was Chief Data Scientist in Intel’s Internet of Things Group where she developed machine learning and deep learning analytics for over eight different markets.

And while broadly applying the term AI makes it easier to market and talk about, for Fralick who has 40 years expertise in the field, each type of model under the AI umbrella has different levels of complexity and intelligence behind them.

“As a data scientist and an engineer, I look at AI to be very specific mathematically. I look at deep learning to be very different mathematically from machine learning.”

Speaking with Which-50 between sessions at MPower in Las Vegas last week, Fralick said most technology vendors claiming to do AI are really doing machine learning “and maybe a little bit of deep learning.”

Fralick shared her top three questions to help technology buyers better understand what they are getting from a machine learning solution, as well as what makes for a good or a bad answer.

1. How often does your Machine Learning algorithm actually “learn”?

The first thing Fralick recommends asking is, how often does your machine learning algorithm actually “learn”?

Models change over time, meaning when new information comes in the model is updating itself to be more relevant or more accurate. The process is outlined in the image below.

A correct response will indicate, “The algorithm learns at the rate that was determined at the time of model development and is updated periodically and applied to new signals.”

Some models may be updated continuously, while other models, they are updated based on statistically significant changes to inputs to the model, Fralick said.

Buyers should be wary of vague answers like “routinely”. Good follow up questions include “How often?”, “What signals does your company use to update the model?” Or even the very basic “How do you know the model is working?”

2. How accurate is your Machine Learning model?

Nothing is ever 100 per cent accurate.

“Accuracy is the measure of error,” Fralick explained. “So if your ML model is not as accurate as it could be, it means it has more error and giving you incorrect decisions.”

Error is measured differently for different models, such as false positives, false negatives, true positives and true negatives. But there are other ways to calculate error which require taking a look at the math. Fralick provided an example of a correct answer:

“The specific ML model for this purpose has a 95 per cent accuracy. We also measure other forms of accuracy, such as Root Mean Square Error, Generalized R^2, and additional ROC metrics such as Recall, F1 Score, and Matthew’s Correlation Coefficient.” […]

Read more – which-50.com