Chatbots are conquering our world, as Angelika recently described in her blog article. Thanks to the technical advancement of artificial intelligence and machine learning, these bots can increasingly support us in our everyday lives.

Reason enough for coeno to take a closer look at the topic and set up a test case project: The conception and implementation of a chat offer with which the user can arrange a hair appointment. In an interview, Angelika Zerbe and Miriam Springer tell us how they proceeded in the project, what hurdles they encountered and how the results were received in the user test.

Miri, Angelika, you have been dealing with chatbots in the last weeks. But what exactly is a chatbot?

Angelika: Chat bots are text-based dialogue systems and are used for communication with the user. They therefore have an input mask for text or react to voice input. The reaction of a chatbot depends on its intelligence.

The simplest way is a completely predefined chatbot. It follows fixed rules and only gives the answers that have been programmed. Then there are the learning chatbots. They learn from previous conversations and thus increasingly improve their answers. The most intelligent type is called digital assistant. Here, neural networks are fed with very, very much data, so that the assistant can give answers to many topics. One example is Alexa from Amazon.

What are chatbots usually used for?

Angelika: Often they help the user to get information very quickly. So the user no longer has to find the information on a website himself, but the chatbot answers the question directly.

Miri: Alternatively, a chatbot can also help the user to quickly complete a certain task, for example making an appointment for a hairdresser or doctor.

You have developed a chatbot like this as part of a test case project. Could you briefly explain what exactly it was about?

Miri: We have found chatbots an exciting topic for quite some time. We have a client who offers a platform for making appointments. That's when we had the idea to create a chatbot for exactly this use case. To make things a bit more concrete, we decided to create a chatbot for a hairdressing salon.

How long did the project run?

Angelika: About from February to May, so a good 3 months. However, we only worked on it on a daily basis, whenever there was time. The project team consisted mainly of Josef as project manager, Miri and me.

Have you set goals in advance that you want to achieve?

Miri: The main focus was to expand our know-how on chatbots. We wanted to find out how best to approach the concept, what needs to be taken into account during implementation and whether users would use a chatbot at all to make appointments.we left the result largely open. A prototype was supposed to come out, which we could also test with users, but we didn't get more precise at first.

How did you proceed with the project?

Miri: First we collected questions, that is, everything that was unclear to us and where we saw possible problems. We then prioritised those that posed the greatest risk to the success of the chatbot. According to this order we did the first research and tasks.

Angelika: There were very many different tasks. But in general there were two different categories: On the one hand questions concerning the technical implementation. On the other hand questions concerning the design of the chatbot, i.e. what it is called, how it looks, how it talks and so on.

Miri: And then we already started to create a job map. This is a kind of process flow that contains all the questions the bot asks to achieve the goal. So we took a little tour of the office and asked everyone how they made an appointment with the hairdresser.  

What happened afterwards?

Miri: Angelika started programming first things to test what is technically possible at all.

Angelika: Exactly, we had decided to use the tool "Dialogflow" from Google. This supports both text and voice input. After a few tests, however, we decided to do without voice input in our prototype. This is relatively complex to maintain, as the bot cannot automatically pronounce every word correctly. At the beginning our bot spoke quite strangely before we realised that the language was still set to "English". When we changed it to "German", the pronunciation was largely correct, but some English words are now pronounced in German. This can be adjusted manually with phonetic transcription, but that was too much work for the prototype.

How does the building of a chatbot with Dialogflow work? Do you need programming skills?

Angelika: First of all Dialogflow already offers a click environment that is relatively intuitive to use. There you create different "intents". These are so to speak occasions to which the bot reacts. You assign training sentences to these intents, for example "Good morning" as a greeting.

But if the user later writes "Moin" instead, the bot doesn't know what to do with it. But I can see these inputs and add them to the intent "Greetings". So the bot learns little by little. For each intent, one or more answers are set for the bot.

Additionally you need programming skills. At the latest if the chatbot should remember something or process data somehow, you have to program this in the code. I have used Java Script for this.

‍

Miri, you were working on the concept in parallel, right?

Miri: Yes, exactly, I have extended the job map and we have used it to do a so-called Wizard of Oz test. That means we simulated a chatbot to get initial user feedback. The test subjects sat in a room in front of their computers and were given the task of making a hair appointment via Skype. In another room I sat and answered on behalf of the chatbot. I followed the job map as strictly as possible and only answered in the way the chatbot would have been able to. If the user wrote something that was not in the Job Map, I also returned an error message.

What lessons did you learn from the test?

Miri: It worked amazingly well. All five users managed to make an appointment. But the dialogue has stalled in the meantime and for some users it was a bit tedious. I also have to say that I found it very difficult to really only react to the pre-defined words. Sometimes I acted a bit more intelligent than the real chatbot :-). But this way we also learned what we have to add to make the bot react better. We also specified the fallback answers in case the bot does not understand something. This way we can bring the user back to a goal-oriented dialogue more quickly.

Angelika: It was also interesting to see how differently the users communicate. Some of them proceeded rather step by step, others wrote in the first sentence who they are, what they want and when they had time. Ideally, the chatbot should be able to deal with every case, of course. But programming this is not easy.

In addition, we asked the users for feedback on the type of language. This was positive throughout. The loose language and the "you" we used were well received.

What happened after the test?

Angelika: Miri reworked the job map again and I extended the programming. Nevertheless we were a bit disappointed with the result. The bot was not really intelligent. Of course much more would have been possible, but we would have needed much more time and deeper programming knowledge.

So we decided to concentrate more on the conception and to do another test, in which we tested the chatbot with free text input, a chatbot with choices and a form on a website against each other.  

What exactly is a chatbot with choices?

Angelika: That means that the bot does not only ask a question, but also offers direct answer options for clicking on. For example: "To what length do you want to have your hair cut?". The options would be: "short hair cut", "medium length" and "long hair cut". We have even illustrated this with pictures.

This gives the user less of a feeling of having a real conversation, but also reduces the likelihood of entering things that the bot cannot handle.

How exactly did this test go and what were the results?

Miri: We presented all 3 variants to the users one after the other, again with the task of arranging a hair appointment. The difference was that this time we used correctly programmed solutions and these were presented on the smartphone. This made a clear difference in that the users wrote texts on a mobile device in a completely different way than on the computer. The texts written were much shorter on the phone.

The chatbot with options was the most popular among users. This was probably also due to the fact that some users did not reach their destination with the bot with free text input or communication proved to be very cumbersome. In addition, the test persons mentioned as positive that the bot with options did not make them think about what he expected from them and that they could save themselves the tedious typing on their mobile phones.

‍

After these experiences, do you think it would be possible for us to design a chatbot for a customer without programming it?

Angelika: Yes, I think so. We would have to create a very detailed job map. It would still be a challenge to find a format for it so that it remains understandable and clear despite the large volume.

Miri: Wizard of Oz tests could give us clues during the design phase and we would need as many user tests as possible during the programming phase to gradually optimise the job map.

Angelika: In addition, it would be important to use the bot as often as possible so that it can collect training data and become more intelligent.

What were the most important lessons you learned from the project on chat bots?

Angelika: I found it interesting that programming with machine learning is going in a whole new direction. It is moving away from mathematical logic and towards training and feeding with data sets to make the machine smarter little by little. I am very curious to see how this develops.

I have also noticed how much a job map differs from a process flow for a UI. A UI only offers a limited number of interaction possibilities, but a bot allows the user to enter what he wants. The bot must be able to react to as much as possible in the most meaningful way.

Miri: That's right, especially with a free chatbot it is enormously complex and difficult to cover all functions. We have tried to limit ourselves to certain things, but this is very difficult to do. I think an MVP doesn't really make sense with a free chatbot, because the user gives up alla "I didn't understand you" after the third error message at the latest.

Angelika: Yes, you would at least have to put a lot of time into the conception of the fallback answers to pick up the user again and bring him closer to the solution. Here an extra error message concept is needed.

Miri: But every error message still means a diversion for the user and therefore not a good user experience.

Angelika: Another insight was once again how important the user tests are. We try to test all our UIs with real users, but for a chatbot this is even more important as language is maximally individual. It is impossible for a concept developer to think about what 80% of all users will enter.

Miri: But the good thing is that you can test a chatbot very early on using the Wizard of Oz test. So you don't have to design or program anything yet and you can find out a lot. I think that worked out very well in our project.

Chatbots are very hyped at the moment, but do you think they have a real future in our everyday life?

Miri: In principle yes, the question is only to what extent and for what occasions we will use the chatbots. When making an appointment, I think it would be very practical to have my hairdresser or doctor as a chatbot contact via Whatsapp or Facebook and make appointments.

But that's not so much different from arranging the appointment via their website, is it?

Angelika: Yes, that's true. But we could consider how the chatbot could be even smarter and contact the user proactively, for example. So the dentist chatbot could remind the user of the precaution and suggest a few appointments right away.

Miri: But of course it is true, how useful chatbots really are and will be, remains to be seen. At the moment it's just cool to use because we are all fascinated that we can communicate with a machine using normal speech. When this fascination is gone, it remains to be seen whether talking to a bot is still the fastest and most usable way to reach your goal.

Angelika: That will depend above all on what progress there is in the development of AI and machine learning. As long as the bots become smarter and smarter, I can well imagine that they will also support us more and more in our everyday lives.

Thank you both very much!