Top Tips For Testing Natural Language Processing Performance, By Shama Ugale;

Natural Language Processing (NLP) is a space that has evolved rapidly in recent years with endless possibilities for application in the real world. As a result, we have come a long way from command line interfaces to conversational interfaces, with Artificial Intelligence (AI) and Virtual Reality (VR) emerging as exciting technologies to bring NLP to the next level.

The rapid adoption of conversational interfaces among many sectors has resulted in significant progress in terms of the capabilities of chatbots – as we quickly moved from rule-based bots to AI-based bots, powered by machine and deep learning techniques, which use NLP to understand and process the human language.

As with any technology, it is vitally important that such software is evaluated in terms of performance before a decision is made regarding whether a model is ready to go to production. With that purpose in mind, here are a few tops tips for testing NLP performance…

Test early & regularly;

It might sound obvious but the more testing you do, the better your model will be. Like any app, it is good to have a couple of build stages and then test to validate whether the model is acceptable or requires more work. The build should only progress to the next stage when you are satisfied with the results of the testing. Furthermore, this process should be repeated after any amendments to the dataset or additions of new features to ensure that the algorithm is learning and predicting accurately in line with the changes, as well as the needs of the users.

Define KPIs

Formulating metrics for evaluation and analysing these results on an ongoing basis will enable you to create a kind of matrix which will produce a summary of all test results, including correct and incorrect predictions and classifications. You can then use these insights into the performance of the model to further train it and correct the outcomes. Such metrics might include accuracy, recall, confidence and precision.

Focus on training

Sometimes errors can arise from limitations in training which, again, is why repetition of testing is key. It can be useful to treat an NLP model like a toddler – for example, when learning how to identify objects, it is important to introduce variety so that children can identify objects which are the same but have different characteristics (so not just shape but colour, patterns, etc.). Algorithms need to be able to do this as well, which means you need to test thoroughly, with different situations or samples that require the same solution, otherwise issues can arise during the identification process.



Ensure data is right

Peter Gentsch (an expert in AI marketing, sales and service) once said: “To the user, chatbots seem to be “intelligent” due to their informative skills. However, chatbots are only as intelligent as the underlying database”. In other words, bots are only intelligent because of the data upon which they are based, so make sure you have the right dataset and the right insights into this data so you can train and develop the model effectively.

You can also use open source tools, such as Botium Coach, to test the performance of NLP models. However, the core approach should remain the same – test, test and test some more! The more tests and insights you can garner, the better your model will be in terms of accuracy and precision when it goes to production. As more personalised and sophisticated bots emerge, this element of the software development process will become even more crucial and valuable.


To access tons of resources, share topics/projects & join ongoing discussions on our Q4Q Knowledge hub, click here!

Want to discuss testing with like-minded individuals and tech experts? Attend Quest for Quality conference, to access tickets;

From Benjamin Talin’s Session – Business Leaders Need to Forget About Tech Trends & Focus On Company Requirements To Make Digital Transformation A Success;

We live in a digital world full of buzzwords. It’s all Artificial Intelligence (AI) here and blockchain there and business leaders fail to see the risk in buying into the hype of the latest technology trends. Companies are so busy embracing the latest technology, rather than determining whether they actually need it and identifying the issues it solves that are relevant to them.

In fact, this is the reason that 70-80% of digital transformation projects fail. These projects are failing not because the solution isn’t up to standard, but because the purpose of the organization has not been defined. In other words, businesses are buying into the hype and it is backfiring.

Organizations really need to educate themselves, change their approach and identify how they can enable digital transformation for themselves in a simpler way – because self-service analytics, quantum computing and serverless architecture aren’t for everyone.

Here are three things business leaders need to know if they are to be successful in their digital transformation journeys…

Digital Transformation is not the same as digitalization;

A common misconception is that the processes of digital transformation and digitalization are one and the same. This is not the case. Digitalization refers to taking what is offline and putting it into digital format, while digital transformation is, I would say, 5% about the digital or technical aspect and 95% about transformation. It is more about embracing change and involves a shift in strategy and culture (which may or may not involve new technological solutions).

Often, companies start their digital transformation journey by looking at the technology, before they actually look at the problems they need to solve, which is a big mistake. First, they should look at what they need to do and then develop or identify the most suitable solutions.

Furthermore, digital transformation involves numerous different elements. Companies must also build business models around their new approach, put strategies in place and train people to support this. Digital transformation is so much more than just technological solutions; it is a widespread shift in how the business operates.

The human element is more important than the solution itself

Whilst technology is certainly an enabler, there is so much more to consider than devices. People need to learn how to use those devices, ensure that these systems are working effectively and identify where such solutions can prove useful. Without humans, machines wouldn’t get anywhere.

The roles of people are also evolving and how we work is changing, in keeping with the rapid progression of technology. For example, it is predicted that the number of freelance workers will overtake the number of non-freelance workers in the US. Moreover, each department now has its own specialisms with dedicated roles. What was once a team of digital marketers has become multi-faceted with Facebook analysts, Instagram ad managers and LinkedIn specialists as well.

Every organisation is different, with its own different priorities and capabilities, but at the heart of each business is the human factor which needs to be considered by those making the decisions and should not be overlooked in favour of the latest products. As Patricia Fripp once said: “Technology does not run an enterprise, relationships do.”

A “value-first” approach is vital;

Theodore Levitt said: “People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.” This is how organizations should approach change management. The goal is not implementing AI or having the latest new technology, it is the outcome or value that AI delivers – in other words, how AI can help the company, staff or customers to achieve their goals.

For instance, I doubt Facebook started with the technology. It is more likely that they discovered that their users wanted to be able to share their stories, therefore they determined the technology that facilitates this. After all, the end user doesn’t care about the technology, but they do care about whether or not it enables them to do what they want to do.

By thinking about the value that such technologies can bring and focusing on the problem to be solved as a first step, businesses can decide exactly what technologies they need and increase the chances of digital transformation project success. Hence why organisations should always be asking “what are our goals and how does a particular technology enable us to our meet these?”

If organisations are going to be successful in their digital transformation journeys, they must first look at what they want to achieve, all the while keeping the human factor as a focus. By doing so, they can lay the building blocks upon which their company can succeed, innovate and thrive – enabled by technology solutions, not dictated by the latest technology trends.


To access tons of resources, share topics/projects & join ongoing discussions on our Q4Q Knowledge hub, click here!

If you are interested in discussing digital transformation, digitalization, change management or AI with like-minded individuals, check out this year′s Quest for Quality conference which will be hosted online. For more information and to register your interest;