Ahmed El Deeb has a good post on ten tactics for dealing with small data.

Ahmed correctly points out that Enterprise AI projects are more likely to run into these constraints.

His post is a good example of how we need to think differently about tools and methods for Enterprise scope projects and that limited data is a driver of algo selection.

San Francisco Big Data Science hosted a talk on “AI in the Enterprise” the presenters were:  Polong Lin, Data Scientist, IBM;  Arno Candel, CTO, ‎H2O.ai; and Nir Kaldero, Head of Data Science, Galvanize.

Here are the top takeaways that stuck with me:

Enterprise AI projects are implemented by cross-functional teams

Many of the roles on these teams will be the same as on traditional software teams but ML drives critical changes to how those roles are executed.

Non-data scientists need role specific training

There are a range of non-data scientists working on Enterprise AI projects: product managers, program managers, UX designers, QA testers, devops, software engineers, data analysts, line managers, etc.   All of them will need different methods and tools for AI driven projects than they are used to with traditional software projects.

Some of the training they need overlaps with the sort of “Data Science 101” training you would give at the very start of a data scientists education.  However, other needed training is specific to their role.  A product manager will need “Data Science 202 for Product Managers” and so on for the various roles.

Biggest blocker is tech-biz gap

The biggest blocker to applying AI in the typical enterprise is the coordination gap between team members focused on data science and team members focus on business value.  Some organizations are defining a “middleman” role aimed at bridging this gap.  This role is so new that there is no consistent pattern for what title it is being given.  Sometimes it is a product manager, sometimes a data analyst, sometimes a data translator.

“Better models” blocked by black box concerns

Enterprises, such as financial services companies, have models in the lab with much better performance metrics than what they have in production however concerns about explainability and transparency mean those models don’t meet internal governance standards and external regulatory expectations.  Improving explainable AI (XAI) techniques can help unlock this potential.

Model management is lacking but is coming

“Model management” covers a range of capabilities: monitoring to trigger alerts when a model needs to be retrained; ability to characterize in human understandable terms the difference between current production model and new release candidate model; etc.

These abilities are generally lacking in current systems and when implemented are done in custom ad hoc way.  However, vendors (such as H20) have it on their roadmap to start addressing these needs.

Enterprise projects powered by machine learning will have cross-functional teams.  Which role will take the lead on integrating the team’s efforts will vary, most often in the enterprise it will be a product manager or program manager.  However, at times it will be a line manager or UX design lead or data scientist.  Whoever takes this role needs to make sure the project goes beyond being an interesting technical exercise and actually delivers real value to the organization.  To accomplish this we should take a human centered holistic approach to planning and managing the project.

Josh Lovejoy and Jess Holbrook from the Google UX team shared an interesting post recently on “Human-Centered Machine Learning” that directly addressed this point.  They covered:

• Don’t expect Machine learning to figure out what problems to solve
• Ask yourself if ML will address the problem in a unique way
• Fake it with personal examples and wizards
• Weigh the costs of false positives and false negatives
• Plan for co-learning and adaptation
• Teach your algorithm using the right labels
• Extend your UX family, ML is a creative process

They had some valuable detail on these topics.  Our top takeaways were:

ML is a team sport

Initial proof-of-concept projects executed by trophy data scientists have often been executed by a siloed and relatively isolated data science team.  Now as the investment and expectations for AI are ramping up, we can expect data scientists to be more integrated and enterprise projects to be driven by cross-functional teams.  Those players include PMs, UX designers, quality assurance, devops, line managers, etc.  Each of those folks needs to bring expectations to a machine learning project that are different from what they have ingrained in them from years of working on traditional software projects

Josh and Jess’ post touches on multiple implications of this trend.  For example they point out that a “big challenge with ML systems is prototyping”, emphasising how prototyping ML systems that adapt based on the data they encounter is fundamentally different from traditional systems with logic fixed by the development team.  This change in prototyping obviously impacts UX designers, PMs, functional managers, etc. who are involved in creating prototypes and acting on the results of user testing.

Be solution focused not technology focused

… product teams are jumping right into product strategies that start with ML as a solution and skip over focusing on a meaningful problem to solve …  you’ll want to assess whether ML can solve these needs in unique ways. There are plenty of legitimate problems that don’t require ML solutions.  A challenge at this point in product development is determining which experiences require ML, which are meaningfully enhanced by ML, and which do not benefit from ML or are even degraded by it. Plenty of products can feel “smart” or “personal” without ML. Don’t get pulled into thinking those are only possible with ML.  We’ve created a set of exercises to help teams understand the value of ML to their use cases. These exercises do so by digging into the details of what mental models and expectations people might bring when interacting with an ML system …
— Human-Centered Machine Learning

Focusing on solutions first is a consistent theme we hear.  Whether expressed as system utility over model performance or machine learning overkill or  thinking beyond performance metrics it comes back to focusing on delivering value and not just exercising the latest technology.

Understand how trade-offs impact users

It is normal for complicated systems to involve trade-offs.  With machine learning we are facing trade-offs that are different from what we are used to with traditional software engineering:  precision vs. recall;  bias vs. variance; model metrics vs. interpretability; etc.

In their post Josh and Jess dive into the precision-recall trade-off example:

Your ML system will make mistakes. It’s important to understand what these errors look like and how they might affect the user’s experience of the product. … In ML terms, you’ll need to make conscious trade-offs between the precision and recall of the system.
— Human-Centered Machine Learning

On an enterprise project those charged with representing the users point-of-view and generating organization value are typically the UX designers and PMs.  Those non-data scientist team members need to understand machine learning concepts well enough to actively and effectively work with data scientists to achieve the right balance in these trade-offs.

For example, if you are a PM and you don’t understand the illustration below than you are not ready to lead a machine learning project:

Consider end-to-end lifecycle

Josh and Jess make good points about how you the challenges in managing the long term lifecycle of ML projects are distinct from traditional systems and that these differences impact non-data scientists team members as much as the data scientists themselves.

The most valuable ML systems evolve over time in tandem with users’ mental models. When people interact with these systems, they’re influencing and adjusting the kinds of outputs they’ll see in the future. Those adjustments in turn will change how users interact with the system, which will change the models… and so on, in a feedback loop. … You want to guide users with clear mental models that encourage them to give feedback that is mutually beneficial to them and the model.

… [ML systems] adapt with new inputs in ways we often can’t predict before they happen. So we need to adapt our user research and feedback strategies accordingly. This means planning ahead in the product cycle for longitudinal, high-touch, as well as broad-reach research together. You’ll need to plan enough time to evaluate the performance of ML systems through quantitative measures of accuracy and errors as users and use cases increase, as well as sit with people while they use these systems to understand how mental models evolve with every success and failure.

… we need to think about how we can get in situ feedback from users over the entire product lifecycle to improve the ML systems. Designing interaction patterns that make giving feedback easy as well as showing the benefits of that feedback quickly, will start to differentiate good ML systems from great ones.

The typical enterprise machine learning project will be medium scope, cost conscious and specialized.  In those circumstances it will be common to have either a limited quantity of training data or a good quantity of training data with a limited quantity of that data being labeled.

Therefore the mass scale “supervised learning” that might be applied to a moonshot project won’t work.  The good news is that there are alternatives:

Semi-supervised techniques can use  a small amount of labeled data with a large amount of unlabeled data.  Some of these techniques can help identify latent features that might be useful in providing explainability and assessing generalizability.

Transfer learning can take advantage of a general purpose model trained on a large set of labeled public data to more efficiently create a specialized model to meet the needs of your specific enterprise.  Andrew Ng gives an example of  this video.

Unsupervised learning can be used to identify patterns and generate insights that are helpful for decision support purposes even when they don’t lead in one step to high accuracy fully automated systems.

In pedagogical learning the learning process is seeded with concepts provided by domain experts.  While this limits the solution space it enables much more efficient learning.  The Inkling language is an example of this approach.

Reinforcement learning is a learn as you go approach.  Think of it as on-the-job-learning for AI.

Tools and methodologies that work well for mass scale supervised learning may not work well with these other techniques.  This is part of the reason why enterprise projects need a distinct approach from the moonshots that we often envision as typical machine learning projects.