Chapter IV Interactive Experience Strategy

5.1 User And Scenario Oriented Process Design

5.1.1 Addition Of Exit Strategy

In the Task, we also need to allow users to exit the corresponding intent from the Task. At this time, there are two ways to realize the exit strategy: one is to add the exit related Trigger in the unintentional plan; the other is to add the silent slot filling unit in the intent, and use the exit slot to associate the exit related entity in the slot.

5.1.2 Enable Users To Participate Repeatedly

Often, in the process of building a Task, you will encounter the need to be able to repeatedly participate in the Task. As mentioned earlier, Task are often used for slot filling. If you want users to re participate in the Task, there are a lot to pay attention to for Request Block, because Request Block will only take effect when the slot has no value.

First, you can clear the corresponding slot in time after the Task is completed.

There are three ways to empty the slot:

1. Using the emptying mode of the slot operation unit, we usually use emptying in the front of the Task intent when multiple intent or slot are recycled. The advantage of the front is that if the later intent needs to cooperate with other intent, it is easy to know that all slot are empty before running, which is convenient for troubleshooting.
2. Use the slot value of the slot to clear the function after cell jump. This method can be used for some slot values that do not need to be recorded and used later, such as confirmation links, some question answering links that do not need to be recorded, and so on.
3. In advanced options, turn off reserved slot, so that the corresponding slot will be cleared after jump. However, there are many restrictions on the use of this function here. First of all, not reserving the slot is not applicable to the Task that needs to be triggered circularly within a single intent. In the case of not reserving the word slot, even if the silent slot filling unit is used, the Task cannot be restarted; Secondly, if the intent uses the same slot to ask the slot filling unit or silent slot filling unit after the jump, the slot will not be cleared.

Secondly, some simple single intent Process can use a silent slot filling unit in front of the Task when the reserved slot function is enabled. When the Task idle waiting time is not over, you can use the silent slot filling unit to repeatedly obtain the corresponding information and run the results according to the Process.

For example, when checking the weather, we can use the change date or city to query the weather of multiple days and cities.

5.1.3 Identify Old Users

In some marketing and questionnaire scenarios, you need to remember the user's characteristics and decide what the user can do according to these characteristics.

This requires the agent to have the ability of long-term memory. In my life, we can distinguish these features through attribute. Create a new attribute "complete questionnaire" (dialog building - personalized experience - user attribute - custom attribute) for the user.

When the user finishes the corresponding Task, it is OK to write "yes" to this attribute through the attribute writing unit to indicate that it has been completed.

Accordingly, before the formal start of the Task, we can use the attribute reading unit to read the "completed questionnaire" attribute and fill it into the "answered questionnaire" slot, and then use the slot operation unit to judge the results of this slot.

If the entity value in the "have you answered the questionnaire" slot is "yes", then the Inform Block is used to tell the user that "you have participated in filling in the questionnaire and cannot fill in it repeatedly.", If it has not been filled in, you can continue the Task, where you can easily use the collected values of other slot to jump to the default.

Note: we need formal channels when testing, such as experience pages, official account, nail, enterprise WeChat and so on. The debugging agent is a virtual user and has no way to write or read attribute.

5.1.4 Fill In Only Once In A Natural Week

It's easy to ask the user to have a 7-day interval between the current visit and the last visit. We just need to judge whether the time of the current day is 7 days less than that of the last visit. We have an implementation interface that can also support such requirements.

• Interface URL: get http://base-webhook-api.wul.ai/date/diff?start={start_time}&end={end_time}&unit= D
• Input parameters:

start	Slot with start date	Supported date format: yyyy MM DD or yyyymmdd supported system entity: Date
end	Slot with end date	Supported date format: yyyy MM DD or yyyymmdd supported system entity: Date
unit	Unit of difference	When unit = s, returns the second difference between the two times When unit = D, the number of days between two times is returned
range	Directly fill in the preset date range entity	2016-05-04 2020-01-01 if the range parameter is used, the start and end parameters cannot be configured

• Output parameters:

  • When the start time is later than the end time, the negative value of the number of days difference is returned, such as - 1
  • When the start time is earlier than the end time, return the positive value of the difference days, such as 3
  • Return 0 on the same day

Just pass the historical visit date and the current date through the single slot interface unit, and then you can get the return value. Finally, judge the relationship between the return value and 7.

Similarly, we first read the user's previous attribute for judgment. If the user has no historical visit date, he can directly start the formal Task. If there is a visit record, we can compare the difference between the historical visit date and the date of the current day. If it is greater than 7, we can enter the formal Task.

We can use the system slot to splice the dates. For example, after each Task, the organization can use the system slot to splice the date of the day in the slot operation unit using the assignment mode and write it into the user attribute.

How to judge that the questionnaire can only be filled in once in a natural week?

First of all, we need to figure out whether it is the same week. The only system slot related to the week we provide is the "sys\u weekday" system slot. The returned value is the day of the day.

Using this slot and the single slot interface unit that calculates the time difference, can we fill in the word slot only once a natural week at most?

The answer is yes, and the implementation is not complicated, but the logic should be clear first.

The algorithm is as follows:

First judgment: if the number of the day of the week of the day is less than the number of the day of the historical visit, then this Monday must be a new week, which can be filled in this time.

Second judgment: if the number of the day of the week is greater than or equal to the number of the historical visiting day of the week, continue to judge the difference between the date of the day and the visiting date. If the difference is greater than 7, it is a new week, otherwise it cannot be filled in.

The construction Process is as follows:

We can accomplish many Task with the help of simple interface units and their own functions. If the above Task does not have an interface, we can also derive the difference between dates according to the relationship between dates and months. Of course, having an easy-to-use interface can greatly reduce our workload.

Pay attention to special problems

Such personalized scenarios as how to deal with sensitive words and how to deal with mobile phone numbers as soon as the user comes up can be properly considered, provided that the basic Task dialogue can be completed and the sesame seeds should not be picked up and the watermelon lost.

How to analyze strange agent

There are two ideas:

From the whole to the part, observe what scenes and intent the agent has, what can be used, and what the Trigger is. Then, it analyzes the intent that can be triggered from front to back, and finds the slot and the entity values and branches inside.

From the local to the whole, directly use messages to debug the agent to view and observe the information that may be exposed by the agent. The debugging agent usually shows which intent is triggered and which entity are extracted. According to the associated entity, find the corresponding slot and the corresponding intent, and deduce the scene at that time.

5.2 Optimization For Manual Analysis

5.2.1 Answer Maintenance

We have talked about the basic maintenance methods of the answer in the construction part. We won't repeat them here. Let's emphasize the mistakes we often make.

Are The Concepts Involved In The Answer Unfamiliar

Taking the insurance industry as an example, the payment method of customers often has the situation of multiple payment and one-time payment, but there is a professional term in the insurance industry that is single payment. What is single payment? In short, it is one-time payment. However, if the payment method can be divided into multiple payment and single payment in the answer to ordinary customers, it is a test of customers' professional knowledge, and non professional customers will not know what to say, which will inevitably give rise to new questions. Therefore, the knowledge level of users should be fully considered in the process of writing answers.

In addition, common mistakes include the use of English, abbreviations, English abbreviations, pronouns, general terms and so on.

Whether It Is Complete And Whether The User Will Generate New Questions After Reading It

If users come and ask if you have any activities, how do you answer? Some service personnel may write that we have activities. But such a response is no different from no response. The best way to response is generally "we are holding a certain activity now. You can find it in the XX menu of wechat official account. You can also follow our microblog and wechat at ordinary times. The activities will be notified."

In this example, we should not only tell users that we have activities, but also tell users what activities they can participate in recently and where they can find ordinary activities. If there is specific information, we can also give users some guides to jump links. The concept of serving customers is not outdated at any time.

At the same time, it is also necessary to carefully add activity knowledge points in the FAQ dialogue, and timely disable or offline related knowledge points. Activity knowledge points are the most rigorous part.

In particular, we need to pay attention to resolutely avoid setting up rhetorical questions.

Let Users Know What Questions We Answer

A very common problem is that our knowledge points are often put in the knowledge base with such questions, such as "does the product have a certain function" and then "yes" when the answer is filled in by the corresponding personnel. Although this question and the answer are not wrong, it is definitely not a reasonable response of agent to customers, because agent is likely to make mistakes, So the answer is at least "We support x function", and the better answer should be "our products have X function, which can help XX, and this function can be found in XX".

It should be noted that although the last version of the answer contains the function and location of this function, it does not mean that the user can response to this knowledge point when asking where the XX function is. It depends on whether the frequency of this problem is high. If the frequency is high, we suggest that we should create a new knowledge point to specifically answer users' questions about where this function is. In the later stage, we can also consider pointing to this function to give some tips during product guidance. Of course, specific problems should be analyzed and solved in detail.

Associate More Referral Questions

When users ask questions about the concept, even if they get the answer of the concept, they may also want to know other knowledge related to the concept. At this time, they can skillfully set up relevant recommended knowledge points to help users ask questions better. Here, it is not a good practice to put all the answers under this concept, because it will generally lead to redundancy of answers.

Multimodal Interaction

Multimodal interaction can achieve more efficient dialogue and provide a better user experience. Multimodal interaction refers to

Other optimization and improvement points of the answer. You can consider whether there is a difference between the answer of the agent and the answer given by the manual customer service, how to optimize and improve if there is a difference, or whether the difference has been minimized.

5.2.2 Satisfaction Statistics And Optimization

We can also optimize our knowledge base according to user satisfaction. We can view it through [data analysis - user feedback - knowledge point satisfaction], and click the blue part to see the details of likes, dissatisfactions and mismatches.

Generally speaking, the reasons for the decrease of satisfaction can be analyzed through the previous and subsequent articles. The common problems are that the corresponding questions have no corresponding knowledge points, the answers are wrong, and the answers can not meet the needs of users.

The first two can be solved by optimizing the call rate. There may be many reasons why the answer can not meet the needs of users. For more common questions, please refer to our answer writing principles, which may be caused by the fact that the answer does not contain the information users want and the direction is not clear.

In addition, we should remember that when writing the answer, we should think from the perspective of users and see if the answer can solve the problem if we encounter this problem.

5.2.3Manual Design Optimization

According to the requirements of different manufacturers for manual transfer, there are also various designs for manual transfer. Generally speaking, mature agent can judge whether the question should be answered by agent or by manual access. Therefore, we recommend that the judgment of manual transfer be performed by agent.

The specific design performance is that there is no obvious manual conversion prompt or button on the page, but the manual conversion is realized through guidance and other methods.

Supplementary reading:

The Role Of Human Seats

Human seats have three main roles:

1. Provide more immediate and richer feedback for AI

Human beings can report or correct errors in different modules of speech recognition, intent recognition, entity extraction, action selection and other systems. In addition, humans can control injuries by actively retrieving imperfect action choices made by AI in advance, and allow AI to perform "Perfectly" after necessary human intervention. In this way, people can provide punishment and reward feedback as an intermediate step of regression learning.

2. Take action with the help of AI

People can choose the final action according to the potential action generated by AI, or generate the final action according to the suggestions made by AI.

3. Generate training data for AI

Last but not least, people can act as producers of knowledge annotation data. Every wrong example or correction method can be used as a negative sample, and every successful dialogue can be used as a positive sample. Humans can also label AI samples that can produce low confidence scores offline to improve the robustness of the system (also known as robustness).

5.3 Comprehensive Analysis Of Operation Cases

In 2019, we made a dialogue agent for famous shoe and clothing manufacturers. Let's take a look at how to deal with these problems in the actual scene.

background

Project introduction

Before our agent went online, the customer's wechat official account was a pure manual service, and the customer service workload was huge. Each customer service needed to undertake 70 users every day and process about 1400 messages; After the coming agent went online, the trainers continued to optimize the agent for five months, making the online accuracy of agent's non high frequency knowledge points reach 72%, and the completion rate of Task agent reach 91.9%, so that agent's resolution rate reaches 60%, and about 40% of user problems are solved by customers. Therefore, the work efficiency of customer service is greatly improved.

Deployment mode

I come to SaaS

User channel

There are two wechat official account. Lailai also provides dialogue agent services through the wlai SaaS platform.

Support recommendation questions, no more than 5 at most.

Support transfer to manual service: the session will be transferred to the customer's CSR supplier for subsequent manual service.

• Provide active labor transfer, such as: the user says, "I want to complain";

• Provide passive to manual, for example: the user says "I want manual".

• After the manual transfer, the agent assistance function is provided (the agent console displays the recommended answer for the agent to click send or edit and send).

Customers provide historical chat corpus to desensitize, clean, de duplicate and cluster them, and build and optimize the knowledge base.

To set up a Task agent according to the customer business scenarios: size query, size conversion, inventory query, order number query and return order number query.

Index requirements

• Effect index: Non high frequency knowledge points 60% accuracy

• Business indicators: agent Resolution rate Up to 43%, agent Satisfaction rate Up to 75%

Note: the accuracy mentioned in this article is the accuracy of non high frequency knowledge points. What is the accuracy of non high frequency knowledge points required by this customer? It is necessary to exclude and remove the recall part of TOP10 knowledge points. Generally, TOP10 recall is defined as high frequency knowledge points.

5.3.1 Initial Optimization Scheme Of Faq

Within one month (October) after the launch, the full traffic was not opened, and the daily average number of messages was about 2000, with an accuracy rate of 55.7%; On November 1, full traffic was opened. The average number of daily messages was about 5000. During the new product launch, there were many messages, up to about 10000-20000. The agent's resolution rate was about 20% and the accuracy rate was about 65.5%

Problems found in labeling

The accuracy of agent is low; The resolution rate of agent is low.

Key optimization indicators

Accuracy of FAQ agent

Optimization time

2019.10-2019.11

Cause analysis and solution of low accuracy

reason	Previous situation	Later measures
Labeling rules	Uncertain, random, deliberately delete some problems	Determine the rules and sample randomly according to the rules
Sampling quantity	100 per week	400-500 per week
Threshold setting	zero point eight	zero point seven
Problem handling	Random, no direction	Break one by one according to the key points

1. There is no unified standard accuracy rule in the early stage

The rules not uniformly marked in the early stage lead to random sampling, unreliable conclusions and failure to grasp the key points in optimization.

2. Unreasonable threshold setting

After that, we will sort out the indicators we really care about, what to do with the indicators, what parts to evaluate and what methods to use for scientific sampling.

3. The number of labels in the early stage is too small to form a data-based guidance

Each week, 100 items are tagged out, which are much less than 200 at the recommended weekly level, and then adjusted to 400 to 500 annotated data per week, which can better reflect the status of the knowledge base.

4. If the threshold is set high, it may not achieve better results

By clarifying the effect that users want, it is found that the original threshold of 0.8 can not achieve the optimal effect that customers want, but the threshold of 0.7 is more appropriate.

Specifically, we did these operations:

① 1000 user corpora are extracted for annotation

② The marked corpus is tested offline when the threshold is 0.7 and 0.8 respectively;

③ The accuracy and recall of agent under two thresholds are compared.

The test results show that the effect of agent is better when the threshold value is 0.7, and it can meet the needs of users. Therefore, the threshold value is adjusted to 0.7.

Handling of other problems:

Badcase type	resolvent
Loss of synonyms	New synonyms
Intersection of similar problems under certain knowledge points	Adjust similar problems to corresponding knowledge points
New business knowledge points not received	New knowledge points
Multiple intent (to be confirmed)	Stock in after disassembling into a single intent
Question method not covered	User mining, similar questions, online supplementary questions

The parts that were not correctly handled were selected and optimized respectively. It was found that there were many missing synonyms. After further understanding, it was found that users had many nicknames for the same shoes and clothes, which were often ignored in the early mining. Therefore, the customer's service personnel were concentrated to supplement the synonyms of shoes and clothes and add synonyms of several professional words.

Some users' question statements are too long and contain multiple intent. It is found that these different intent are often related to each other, and the corresponding recommended knowledge points are maintained in the system.

Optimization example:

1) Corpus crossing

The AI response is wrong because similar problems belong to wrong knowledge points:

The corpus crossing between the two knowledge points needs to be adjusted, such as:

[how to handle return refund] and [what is the return policy]

How to return and exchange goods - how to handle the return and exchange of goods

Return and exchange - what is the return and exchange policy

[how to apply for replacement] and [what if the product size is not suitable]

The sneakers I bought are too small. I want to change them to a larger size. - how to apply for replacement

My shoes are small. Can I change them to a larger size? - what if the size of the goods is not suitable

[how to customize shoes on the official website or app] and [customized service price on the official website]

How much does it cost to customize shoes on the official website

How much does it cost to customize a pair of shoes on your official website - the price of customized service on your official website

2) Uncovered question

Because there are few similar questions under non high frequency knowledge points and the questions covered are not enough, false recall usually occurs when users have new questions (the new questions usually contain some new words and are not learned by AI).

For such problems, AI trainers are required to select the new keyword of this part of knowledge points, and take this keyword as the core to mine similar problems and expand similar problems; At the same time, synonyms need to be added.

Will XX shoes be sold again

(1) Need to add a synonym: vendor = sell

(2) Take [vendor] and [vendor] as the core words to excavate similar problems

3) For the two knowledge points whose particles are too large and easy to recall each other by mistake, it is recommended to merge the knowledge points:

[how to issue invoice?] And [how to download electronic invoice?]

[how to make an appointment for pick-up?] And [courier pick-up time]

[which regions support "next day arrival" service?] And [how to select "next day arrival" service?]

Operation strategy adjustment - optimize recommended knowledge points

Customer trainers will judge other knowledge points highly related to users' questions according to their own service experience, so as to enrich the recommended questions of high-frequency knowledge points of online users and let users get richer and more comprehensive answers.

Through this recommended method of "guess what you want to ask", on the one hand, it gives users a direct access to solve problems, covering more daily, activities and other related contents. On the other hand, it also avoids the occurrence of users' non-standard questions.

5.3.2 Faq Mid And Late Stage Optimization Scheme

Performance effect

In the middle and late stage of the launch, the average number of daily messages was about 10000. Due to the hot centralized sale of new products in December, the maximum number of daily average messages reached 69000. The manual conversion rate of agent is about 60%, the resolution rate is about 40%, and the accuracy rate is about 65.5%.

Problems found in labeling

Low solving rate of agent and completion rate of Task agent

Key optimization indicators

FAQ resolution rate, Task agent completion rate, and labor conversion rate

Optimization time

2019.12-2020.02

Review the solution rate formula:

Solution rate = 1-labor rate

Therefore, reducing the labor conversion rate can improve the resolution rate. In view of this, we analyzed the messages before and after labor conversion in detail. About 100 knowledge points are extracted and analyzed, and the following conclusions are drawn:

Transfer to labor type	Proportion	Reason for transferring to labor	quantity
AI turns to manual after response error	2.97%	Mistakenly trigger FAQ to manual	three
User requests to transfer to labor	46.53%	Transfer to manual after	ten
		The first sentence requires manual transfer	thirty-seven
Switch to manual after triggering knowledge points	40. 59%	When will the replacement goods be delivered	three
		Member account cannot be logged in	one
		How to query order information	one
		How to handle return refund	two
		How to cancel an order	three
		When will the refund arrive	fifteen
		Why can't I place / pay	two
		When will my order be shipped	two
		Where is my order logistics	one
		What if you haven't refunded for more than 3-5 working days	five
		What should I do if I have paid but not paid	one
		Semantic ambiguity	five
The answer of knowledge points leads to manual transfer	9.90%	What about the lack of accessories or goods	one
		How to apply for replacement	one
		What if there is a suspected quality problem	one
		Return progress query	five
		Why can't you participate in the new product launch	one
		My order display error / incomplete / no	one
total	100%	/	one hundred and one

It can be seen from the analysis that the proportion of the user's request to switch to manual work and the proportion of switching to manual work after triggering the knowledge point accounts for nearly 86%. The optimization and improvement of the front call rate is still very effective, because the proportion of the agent's response error switching to manual work only accounts for 2.97%. In addition, 9.9% of the knowledge points are guided to manual.

Then we need to focus on the part that users require to switch to labor and the part that switches to labor after triggering knowledge points.

Optimization strategy: users require to transfer to labor

This situation belongs to the situation that AI does not provide consultation to users and users directly transfer to labor. For this part of users, you can analyze the interaction between users and manual customer service after transferring to labor to see what the user's high-frequency problems are. You can take this part of high-frequency problems as the recommended knowledge points associated with the two knowledge points [Hello] and [are you here].

The top 5 questions of users are:

1. Why is my order cancelled

2. Why can't I place an order

3. How to handle return refund

4. How to cancel an order

5. App cannot log in

In addition to manually screening these questions, we can also find the knowledge points most frequently asked by users through the system, which can be associated with recommended knowledge points after comprehensive consideration.

Note: this customer divides the greeting into two different knowledge points [Hello] and [are you there]. Different customers may deal with it differently, and we focus on the analysis logic and optimization methods, so we don't have to pay too much attention.

Optimization strategy: switch to manual after triggering knowledge points

In this case, after the user triggers these knowledge points, the AI response is correct, but the user still chooses to switch to manual. After specific analysis, it is found that the answer content set has the following problems, so the answer content needs to be optimized.

① The content is inconsistent and the answer is not what you asked

② The content does not cut into user problems

③ The content is complex and diverse

④ The content is too simple and optional

⑤ No recommendation questions

Optimization strategy: the answer of knowledge points leads to manual transfer

In this case, after the user triggers these knowledge points, the AI response is correct, and the answer in the response guides the transfer to manual work or recommends the transfer to manual work, and the user then selects the transfer to manual work. We need to analyze the knowledge points triggered by the user before transferring to labor, and then confirm whether it is necessary to guide the transfer to labor, and whether we can guide the customer to solve the problem by themselves through other ways.

This idea is actually quite novel. Since I have guided the transfer of labor, isn't it normal for users to transfer labor?

Not necessarily, because our time is different, and the change of solving ability may be that we thought it was necessary to switch to labor. Now we can reasonably answer the user's questions before guiding the user to switch to labor. In this way, some conversations will not be transferred to manual work because of satisfactory answers.

5.3.3 Task Dialogue Optimization Scheme

Taking a week in December as an example, agent will make the following analysis for the Task:

Trigger Task agent: 74 times, including 38 times of accurate triggering and 36 times of false triggering

Task completed successfully: 9 times

In the Task, the FAQ was triggered 66 times, and the accurate triggering rate was 72.73%

According to statistics, the accurate trigger rate is 51.35% and the completion rate is 23.68%.

Task completion rate = Times of reaching the last unit / times of accurately triggering Task

Proceed to the last unit: including the completion of taking the correct branch and the completion of taking the wrong branch

Optimization strategy: improve the accuracy of trigger conditions

There are two main situations in the false triggering of Task:

① Matching error of similar problems: adjust similar problems to correct knowledge points or intent;

② Trigger setting range problem: the Task is triggered when the user problem set in the original Trigger contains the style number or order number. The Task is triggered only when it is completely equal to the style number or order number.

Optimization strategy: reduce Task interruption and improve Task completion rate

Reason for interruption	Interrupt unit	frequency	Proportion
Ask about inventory	Ask intent	eight	27.59%
Direct interrupt	Query link	five	17.24%
The link is not recognized	Query link	two	6.90%
Ask if there is a female size	Ask intent	two	6.90%
Size chart required	Query link	one	3.45%
Ask for product details	Ask intent	one	3.45%
Ask about the quality of goods	Ask intent	one	3.45%
Link mismatch	Query link	one	3.45%
Intent unknown	Query link	one	3.45%
Ask whether the website is the official flag website	Query link	one	3.45%
Missed recall	Ask intent	one	3.45%
Direct to manual	Query link	one	3.45%
Ask for replenishment time	Ask intent	one	3.45%
Ask for size chart	Ask intent	one	3.45%
Ask for exclusive customization without style number	Query link	one	3.45%
The user does not know the link	Query link	one	3.45%
total		twenty-nine	100%

It can be seen from the analysis that many customers have asked whether there is inventory, but the agent did not design the corresponding Process originally, resulting in the customer's Task can not be carried out normally. Therefore, we have the corresponding solutions and cooperate with the customers to include the inventory query intent in our Task dialogue.

Optimization one

Users often ask whether there is inventory, but there is no such branch in the existing Process, which makes the agent unable to work normally

Solution: add an inventory query interface,

Optimization II

The link sent by the user belongs to a new product with limited sales or customized products. It is a new link type and does not conform to the originally set regex entity recognition rules, resulting in failure to recognize.

Solution: after discussing with the customer's business personnel, it is found that these two links also have certain characteristics. You can add a new entity to extract the links of these two special commodities according to the characteristics.

Optimization III

Unable to extract entity. When AI asked the user about the foot length, the user came up with a new idea: usually wear code x instead of telling the user the foot length

Solution: add a new branch to extract a new phrase entity

Optimization IV

The indicator of Task completion rate was adjusted and clarified. After meeting and discussion, the customer's project manager thought that the accurate triggering of FAQ in the middle should also help the user solve the problem (the user side did not know whether it was a Task or a FAQ service), so the numerator of Task completion rate added the number of times that the Task did not continue after the accurate triggering of FAQ in the middle.

Task completion rate = (times of reaching the last unit + times of not continuing Task after accurately triggering FAQ in the middle) / times of accurately triggering Task

Proceed to the last unit: including the completion of taking the correct branch and the completion of taking the wrong branch

5.3.4 Case Optimization Review

The accuracy of the agent is obviously improved due to centralized optimization at the initial stage; From November to December, due to the sale of new products, a large amount of consultation and more new knowledge points, there were two declines in accuracy; Since January, the trainers have focused on FAQ accuracy analysis and optimization scheme, and the accuracy has gradually improved and stabilized, reaching 72%.

The completion rate of the Task agent has been steadily increasing. At the initial stage, the completion rate of the Task was low due to the common inconsistency between the Task scene and the real online. After the analysis of the online corpus, the trigger conditions and branches of the Task were updated to enrich the Task scene, and the completion rate of the Task agent reached 91.9%.

With the improvement of agent effect: FAQ accuracy and Task completion rate, the solution rate and satisfaction rate of agent have been improved, and the acceptance index has been reached.

Final effect

(1) FAQ agent before optimization

Current situation of knowledge base: 613 knowledge points in total and 29101 similar problems

Model: general model

Threshold: 0.75

Online effect: the recall rate is 98%, and the accuracy rate of non high frequency knowledge points is 72%

(2) Situation before Task agent optimization

Status of agent: there are 5 intent in total, including size query, size conversion, inventory query, order number query and return order number query

Threshold: 0.80

Online effect: the completion rate is 91.9%

It can be seen that in a scene where agent has not been used before, through step-by-step efforts, we have found the right scene and understood the user's real intent. Finally, with the close cooperation of the customer, we have successfully defined the indicators that the user wants to improve, disassembled, summarized and optimized them. Finally, the accuracy of FAQ dialogue has increased from 55.7% to 72% (specific accuracy indicators), and the completion rate of Task has increased from 40% to 91.9%. The resolution rate of customer concern has also increased from 20% to 60%, which greatly reduces the burden of customer manual service.

5.4 Script Upgrade

In short, the script optimization that needs to be considered in the optimization stage can refer to these aspects.

5.4.1 Design According To The Number Of Conversations

Users hope that those who communicate with them for many times can get on the machine and remember us, so the design of agent dialogue should reflect this memory. The dialogue can be adjusted through the interpretation of the user attribute and the upper and lower attributes to make the machine more intelligent.

5.4.2 Confirmation Strategy

Including "explicit confirmation" of mandatory confirmation and "implicit confirmation" that users know but do not need confirmation;

Three levels of confidence: make the feedback of the machine (more flexible through the setting of confidence. General conditions: ⾼ - implicit, medium - explicit, low - reject / fallback);

Explicit confirmation: it is usually used when the agent wants to perform some important operations, for example, when the agent wants to place an order, transfer money, etc. for the user. Generally speaking, this step is an important matter for individual confirmation of users.

Implicit confirmation: usually the answer is the same as the response. When the confidence level is high and the dialogue is expected to be more smooth, it is generally not the whole sentence is to confirm the existing information.

⾮ language test confirmation: if there is a corresponding interactive interface, ⾳ frequency is used as the "landmark" prompt or the visual prompt of GUI. In the VUI scenario, explicit confirmation prompts such as "I know" are required for the delay (the user system has identified it);

General confirmation: it is suitable for open dialogue without specific feedback. Let the dialogue continue through "well, then" and other forms, which is helpful for users to share.

Visual confirmation: reduce the cognitive difficulty of users and more decision-making time through GUI. It needs to be confirmed for times, indicating that the user sends time.

5.4.3 Script Optimization Taking Capital Retention As An Example

Conversation type	Case enterprise a	Case enterprise B
Welcome words	Do you want to study abroad for junior high school, undergraduate and graduate students? Do you have a favorite country to study in? What do you want to know: study abroad planning, background improvement, examination and training, study abroad application, summer school, internship, international school selection? Immediately response [name + mobile number + consultation content] to give you the most professional study abroad suggestions + the latest authoritative study abroad materials!	Welcome to the study abroad consultation center to learn about study abroad, study abroad examination and study tour. What can I do for you? (in order to ensure that you can get the latest information faster, it is recommended that you leave "phone + current education + intended country / study abroad examination", and we will immediately arrange your exclusive consultant teacher to call back for detailed answers!) Hello, this is Miss Li. May I help you? Studying abroad, course training, study tour, short-term study abroad, internship promotion and so on can help you understand. If you need cooperation, please send an email directly to hezuo@laiye.com (the only designated contact information). If you need to apply, please send an email directly to qiuzhi@laiye.com， We will have a special person to give you a response. Thank you. Honey, are you busy looking for information online? We have carefully prepared a wealth of study abroad materials for each online consulting student. Please don't miss it. Let's arrange the corresponding professional study abroad teachers to prepare materials for you according to the country you are interested in, and answer and plan in detail. Which country do you want to get the study abroad strategy? We can all prepare it for you~
Country of study?	Which country would you like to visit? (buttons: USA, UK, Australia, Canada, New Zealand, Singapore, Ireland, Netherlands, Hong Kong, China)	Do you have any countries to study in? For example: the United States, Britain, Ireland, Canada, Australia, New Zealand, Europe and Asia can help you understand
User	U.S.A	U.S.A
Apply for a degree?	Do you plan to apply for junior high school, undergraduate or graduate?	The United States has the largest number of world famous schools and scholarships, and the quality of education is particularly good. Moreover, we have applied for more than 100000 admission to famous schools, with a success rate of 99%. What stage of study abroad courses do you want to go out to? For example: primary school, middle school, undergraduate, graduate, etc.
User	graduate student	graduate student
Apply for major?	What major do you plan to apply for in the postgraduate stage?	nothing
User	finance	nothing
Language achievement?	Have you got any results in the language test, TOEFL or IELTS? What is the GPA of undergraduate?	I see. Many students think so now. It's really a good choice.	Would it be convenient for you to give an overview of your personal situation? Such as: your current highest education, usual grades, language, etc.
User	Undergraduate gpa6 5 IELTS 7	Undergraduate gpa6 5 IELTS 7
Target institutions?	Do students have any target institutions at present?	nothing
User	NYU	nothing
Preparation phase?	nothing	According to the situation you told us, all aspects of you are actually very good. According to our past experience, we can help you understand and apply. As long as the teacher guides you to make careful planning and prepare the documents, you are sure to apply, and we will help you highlight your advantages and sprint to the top schools at that time. Have you made any preparation for studying abroad before? Our expert teachers can make a one-to-one study abroad plan for you free, so that you can have a bottom in your heart first.
User	nothing	Ready
Technical reservation 1	OK, you can leave a number that is convenient for receiving SMS and send you a list of colleges and universities to introduce relevant fees, including application conditions, score requirements, admission over the years, campus environment, etc. make your own planning. The information is free	It must be advantageous to prepare in advance. We can arrange local teachers to contact you first, make an assessment for you, and then give you corresponding study abroad suggestions, which will be helpful to your application. Or you can leave your phone number and name. The teacher will give you an introduction in this regard and give you relevant free study abroad planning and suggestions according to your personal situation. Our telephone consultation is completely free.
Capital retention script 2	If it's not convenient to leave your cell phone number, can you leave a wechat? Teachers add you to send information and communicate online through wechat.	May I have your name and contact number, please? I arranged for the teacher to call you at your convenience, which can explain various problems of studying abroad for you more conveniently and quickly
Capital retention script 3	Because I need contact information to help you arrange consultant teachers. Would you like to leave a contact information for your convenience? Phone or wechat is OK.	Well, please leave your contact number and name. We will help you arrange a teacher in charge of the United States. According to your current situation, we will first help you make a planning positioning, and introduce the time Process, detailed fees and admission requirements for studying abroad. Is it suitable for you to refer to
User	12345678901 Yang	12345678901 Yang
Confirm appointment	nothing	Thank you for making an appointment for our one-on-one study abroad consultation. The teacher will confirm the time and place of the appointment with you by telephone, and the teacher and will prepare relevant study abroad materials and plans for you in advance according to your situation.
Wechat / QQ?	Can I add your wechat to this mobile phone number? It is more convenient for wechat to send information and communicate.	Or you can leave your QQ / wechat, and we can arrange senior teachers from the headquarters to continue to give you more detailed answers.
User	Wechat is not a mobile number	well
Ask wechat / QQ?	What's your micro signal? It is more convenient for teachers to send information and communicate with you through wechat.	Uh huh ~ OK, now let's start to understand and prepare. The time is more appropriate and has advantages, especially considering applying for famous schools. Thank you for making an appointment for our one-on-one study abroad consultation. The teacher will confirm the time and place of the appointment with you by telephone, and the teacher and will prepare relevant study abroad materials and plans for you in advance according to your situation. It is recommended that you leave a QQ / wechat at the same time, so that you can have multiple contact information to facilitate communication.
User	XXXX	XXXX
Current city?	Which city are you currently in?	nothing
User	Beijing	nothing
full name?	What's the student's name?	May I have your name or address, please? (Mr. / Ms.)
User	Yang	Yang
Confirm appointment / conclusion	After going to work tomorrow, the consultant teacher will contact you as soon as possible to introduce and answer your questions in detail. Please be patient.	What else can I do for you? If not, the teacher of our Beijing customer service headquarters will contact you as soon as possible for a simple understanding. Please pay attention to our 020 phone number; Thank you for your attention. If you have any questions, please feel free to contact us (* ^ ▽ ^ *)
Conversion rate of retained capital	70% (example)	27% (example)
analysis	Good aspects: 1. the welcome words are brief and clear, clearly showing the format of retention, and guiding the strong willed users to complete the retention Process first. 2. The language style is capable and concise; 3. each round of dialogue only solves a problem with a clear purpose, which is clear at a glance. Users do not need to think too much about how to response; 4. The dialogue is highly instructive. Open problems such as "preparation stage" are disassembled into clear sub problems. There is little room for users to play freely and the probability of errors is reduced; 5. It conforms to the user's normal dialogue logic and has strong continuity, so it is difficult for the user to detect the difference; 6. Capital retention scripts are handled carefully, and reasonable recovery scripts are given according to different situations; 7. Use words such as "OK", "you see this", "mm-hmm" to help the dialogue connect more naturally and vividly. Worth optimizing: 1. Increase the consulting ability of some basic issues, such as tariff, explanation of labor demand and recovery script, to help consulting users improve their willingness to stay; 2. Let the user confirm the mobile phone name and other information once, which can effectively avoid the loss of clues caused by information errors; 3. For the identification of mobile phone number / micro signal, the confirmation is more accurate. At present, some foreign mobile phone numbers cannot be correctly identified, and users are repeatedly asked to input the correct mobile phone number; 4. The explicit confirmation of contact information is more flexible, "can you add your wechat to this mobile phone number?" The answer of "can wechat" is more general. At present, the response of "can wechat" can't fill the slot. Continue to ask wechat.	Good aspects: 1. for example, there are more detailed and different documents for different countries, degrees, grades, etc. response is personalized and feels more intelligent; 2. Rich knowledge, can answer different business questions, and feel quite professional; 3. achieve a certain degree of dialogue personalization by obtaining user attribute (geographical location, name). It can integrate the user information into the dialogue, indicating that the agent understands what the user says, and implicitly allows the user to confirm the information. Worth optimizing: 1. define the core business objectives, simplify the welcome language, highlight the key information, and make the user know the functions and services of the agent at a glance; 2. At the beginning of the dialogue, let users know the fastest way to retain funds, and help users with strong willingness to retain funds as soon as possible; 3. simplify the dialogue script. Each round of dialogue only transmits one message. Too many different types of messages in a round of dialogue will seriously interfere with the user's judgment and response efficiency; 4. Remove the information that is not highly relevant in the conversation and the user's last round of answers, and do not place too much information that "I think" the user needs; 5. Refine open-ended questions, clarify each information to be collected, and disassemble it into sub questions to help users better judge how to answer; 6. the Dialogue Flow should conform to the dialogue logic when communicating with people, pay attention to the continuity of the context, and do not insert problems or information that jump off the current scene and interrupt the rhythm of the dialogue; 7. Too many brands are exposed and the script has a strong "marketing flavor", so it is necessary to simulate the real situation and design a smoother dialogue; 8. Less is more.
summary	Minimalism: strong guidance, removing interference, pruning unnecessary branches and scripts. Advantages: 1. The cost of construction and maintenance is relatively low; 2. the Process is strongly guided to avoid the divergence of users' response, better avoid the problems that users cannot answer when triggering agent, and improve the effective interaction rate; 3. Direct and effective, with high conversion rate for users with strong investment intention. Defects: 1. the Process is linear, with a strong sense of "machinery" and "collection". For users with a strong sense of privacy protection, it is difficult to improve the success rate of capital retention; 2. the questions and feedback of users beyond the Process design are unnatural. It is difficult for users who judge whether to retain funds through basic consultation to improve the success rate of retaining funds. Key points of design: 1. Clarify the core business objectives and judge which "non essential information" needs to be removed; 2. try to truly simulate the way people talk with each other: the Dialogue Flow conforms to the context logic, with appropriate rhythm and feedback. Each round of dialogue only describes one question, and the expression is clear and concise; 3. Enrich the business knowledge base and provide better services to consulting users, so as to improve their willingness to stay and success rate; 4. continuously analyze and optimize to find typical problems that users care about but agent can't answer.	Humanism: allow users to ask and answer open questions, and reasonably feed back users' diversified statements by enriching the design of dialogue branches. Advantages: 1. Natural dialogue, more intelligent feeling, and improved user interaction stickiness and satisfaction; 2. Rich knowledge, feel more professional, and improve users' dependence and trust; 3. For users who judge whether to retain capital through consultation and users who are cautious about privacy, they can better meet their needs and improve their willingness to retain capital and success rate. Defects: 1. the agent has complex design and high construction cost; Encourage users to ask questions openly, and have a higher chance to trigger problems that agent cannot handle; 2. there are many branches of dialogue, and users are easy to get lost and deviate from the core business Process, resulting in a reduction in the success rate of capital retention. 3. the more complicated and open the Process is, the lower the success rate of retention will be. Key points of design: 1. define the core business objectives to avoid unclear main process due to complex Process design; 2. Avoid over design: put a large amount of information into the dialogue without details, interfere with users' understanding and judgment, and design too many detailed branches with low priority; 3. Only put the information related to the context in the dialogue, do not overestimate the needs of users, and put the information "I think" users need; 4. try to truly simulate the way people talk with each other: the Dialogue Flow conforms to the context logic, with appropriate rhythm and feedback. Each round of dialogue only describes one question, and the expression is clear and concise; 5. Design a more direct way to retain capital for users with strong willingness to reduce interaction steps as much as possible.

[1] Process optimization

The Process for studying abroad is abstracted into 2:

• Ask for key information for users who retain capital for the first time

Structure: Welcome (call to action for capital retention) + supplementary information + confirmation / conclusion

• For users who do not retain their capital at the first time, guide them to retain their capital

Structure: Welcome (call to action for capital retention) + Information Collection + set of electricity + supplementary information + confirmation / conclusion

Core module disassembly:

• Welcome: role positioning + call to action for capital retention + service content

• Information collection (trigger intent): target country + target degree + usual score and gpa+ language score + document preparation + current school, major and education background

• Set phone: leave mobile phone + confirm whether wechat is mobile phone + leave wechat / QQ

• Supplementary information: Name + call time + source

• Confirmation / CONCLUSION: implicit confirmation + conclusion (specify the call time)

• Fallback response: 2-3 kinds

Feedback:

• Country: US + UK + Canada, Australia + France, Spain, Italy + Europe + Japan, South Korea, Malaysia + Hong Kong, Macao + no intention + invalid

• Degree: valid + invalid

• Major and school: no feedback

• GPA: praise + comfort

• Language: praise + comfort

• Document preparation: valid + invalid

[2] Script optimization

Core module example dialog:

1. Welcome: role orientation + call to action for capital retention + service content

method	Speech skill
BOT role positioning + call to action for capital retention + service content	Hello, I'm your study abroad instructor, Mr. Li. Seeing that you pay attention to the study abroad situation in [country], you can leave your phone number and I will communicate with you at the first time. (10 seconds later) Hello, I can also help you answer the study abroad consultation questions such as study abroad and study tour. You can also communicate with me directly.

2. information collection (trigger intent): target country + target degree + target major and school + usual score and gpa+ language score + document preparation + current school, major and education background

method	Speech skill
BOT overtalk	Hello, classmate, + trigger unit dialogue
BOT target country	Which country do you want to know about studying abroad? (button to select country)
User	U.S.A
BOT target degree (judgment branch)	[the United States has the largest number of world famous schools and scholarships, and the quality of education is particularly good.] What stage of study abroad program are you going to apply for? (button to select degree)
User	graduate student
BOT target (specialty and school)	OK, is there a target (major and school)?
User	NYU Finance
BOT peacetime performance and GPA (non junior high school)	What's your current school average and GPA score?
User	Average score 81, gpa4 one
BOT language score	[your grades are very good in all aspects.] What are the scores of TOEFL, IELTS and other languages?
User	TOEFL 90, IELTS 6.5
BOT paperwork preparation	[your language score is very good. It will be very helpful to apply for a famous school.] Have you prepared documents, XXX and other materials in advance?
User	Not prepared
BOT current school, major and education	[it doesn't matter. Our teacher can help you prepare these.] Can you tell me your current college, major and education?
User	Three major financial institutions

3. Set phone: leave mobile phone + whether mobile phone is wechat + leave wechat / QQ

method	Speech skill
BOT mobile phone	OK, I see. I'll make an appointment for you as a tutor for [American] [graduate students]. Please leave your mobile phone. When will it be convenient for you tomorrow? I'll ask the teacher to call you according to the appointment and communicate in detail
User	12345678901 my mobile phone. Call me tomorrow afternoon
Is bot mobile wechat	Is this mobile phone your micro signal?
User	no
BOT leave wechat / QQ	You can leave wechat or QQ, and we will send you an internal study abroad information. You can take a look at the plan yourself, and then it is convenient for the teacher to send the information
User	Add me wechat, qwer_ one thousand nine hundred and ninety-six

4. Supplementary information: Name + call time + current city + source

method	Speech skill
BOT name	OK, may I have your name, please?
User	My name is Tian
BOT call time (when the slot is not filled)	Roger that. When will it be convenient to call [Tian] tomorrow?
User	Call back tomorrow afternoon
BOT current city (when no slot is filled)	After that, there will be more offline communication. Which city is suitable for me to make an appointment for you?
User	I am in Beijing
BOT source	OK, OK. Finally, please do me a favor. Where did you learn about Kai Tak? (button to select channel)
User	Classmate recommendation
BOT source feedback	Thank you for your patience,

5. Confirmation / CONCLUSION: information confirmation + call time confirmation + conclusion

method	Speech skill
BOT information confirmation + call time confirmation	[thank you for your patience,] [tomorrow afternoon] the instructor of our [Beijing] Headquarters [USA] [Graduate] will call [12345678901] this mobile phone to contact [Tian]
BOT conclusion 1	If there is no problem, please pay attention to our call starting from 020 [tomorrow afternoon]
BOT conclusion 2	Thank you for your attention. If you have more questions, please feel free to contact us (* ^ ▽ ^ *)

6. fallback response: 2-3 kinds

• BOT fallback response 1

• BOT fallback response 2

• BOT fallback response 3

Feedback example dialog:

1. Country: United States + Britain + Canada, Australia + France, Spain and Italy + other European countries + Japan, South Korea and Malaysia + Hong Kong, Macao + invalid

country	Speech skill
U.S.A	The United States has the largest number of world famous schools and scholarships, and the quality of education is particularly good.
britain	The quality of British education is first-class, the course time is short and the gold content is high.
Jiaao	Canada / Australia / New Zealand is a country with high cost performance for studying abroad. The quality of education is guaranteed and the policy of studying abroad and immigration is loose.
Faxiyi	France / Spain / Italy are tuition free at present, which is very cost-effective.
Other European countries	The quality of education in Europe is guaranteed, the welfare is high and the environment is good.
Japan and South Korea	Japan / South Korea / Malaysia has convenient communication, guaranteed education quality, low application difficulty and low cost.
Hong Kong and Macao	The quality of education in Hong Kong / Macao / Singapore is very good, and the living habits are similar and easy to adapt.
No intention	skip
invalid	We don't have business in [country] for the time being. Do you have any other alternative countries? (button to select country)

2. Degree: valid + invalid

situation	strategy
Effective	No feedback
invalid

3. Target major and school: no feedback

4. Usual performance and GPA: praise + appease

situation	Speech skill
fabulous	Your grades are very good in all aspects.
appease	The usual results are only a part of the school's reference. We will help you optimize and improve.

5. Language: praise + comfort

situation	Speech skill
fabulous	Your language score is very good. It