Review Labeled

This section will explain: how to perform quality control and optimization operations on the marked-up interface.

The annotated page is mainly used for viewing and reviewing the results after the annotation is completed, where the main action is quality control and adding the user results to the agent's intent trigger.

1 Introduction to interface operation

1. Interface fields:

   1. Marked query: the query at the time of marking, or if the original query was changed at the time of marking, the changed query is displayed.
   2. Marked intent: the markup intent selected on the to be marked up page, non-recalled and deleted intents will be displayed as "-".
   3. Labeling conclusion: Automatically discriminate the labeling results according to the labeling results, the rules for discriminating the labeling results are described in the following section, and the principles for issuing the labeling results.
   4. Mark time: mark time in the format of 2021-09-17 20:37:46.
   5. Depletion records: accession over intention trigger, no record at this time.
   6. Operations:
      1. Re-mark: Click to retype the pending mark, but not if it has already been triggered by an action intent.
      2. Delete: Delete the annotated record completely.

2. Filter fields and field values:

1. Log statement: i.e. ask for the time corresponding to the time period, for example, "2021-09-21 → 2021-09-27", you can click to select the date interval.

2. Log statement: i.e. ask for the time corresponding to the time period, for example, "2021-09-21 → 2021-09-27", you can click to select the date interval.

3. Mark conclusion: you can choose from a total of seven types: correct, recall error (in the library), recall error (out of the library), semantic unclear, ignore, recall omission, and belongs to new intent, and you can choose more than one.

4. Consumption records: you can choose to join over intent trigger, no record at the moment, and you can select more than one. Each of the above filters can co-exist. Click "OK" to filter according to the relevant logic, click "Put away" to put away the filtering interface, the last click to confirm the filtering options put away will not be lost.

5. Add intent triggers, you can multi-select annotation statements and add intent triggers in bulk.

Note: Modified intent-triggered validation requires clicking on Validate and "Train and Publish" to train and publish the agent.

So how do you apply these features? To know how to apply these features, you first have to understand what you can do with them and what the implications are.

2 Intent labeling and optimization

It can be simply considered that the very first part of the agent is to identify the user intent, and incorrect user intent identification is likely to result in problems such as wrong answers to replies and answers that deviate from the direction of the user's needs, so first you need to evaluate the agent coverage, recall and accuracy, and the evaluation method is called labeling.

At Laiye Technologies, the standardized and formalized labeling is called gold labeling, because this labeling is as valuable as gold.

Label principles

The first step is to distinguish whether the agent is recalled or not. Whether a agent is recalled or not is easy to assess, which is whether the user utterance can give a corresponding reply after matching with the agent's intent, as long as the confidence level exceeds a set threshold to get a reply from the agent, it is a recall.

The second step addresses the distinction between accuracy or inaccuracy, which can be divided into categories listed below.

1. Correct recall: the agent recalls with the correct intent, or replies to the tout if it recalls without intent.
2. Agent-wide error recall: the user message has a corresponding intent within the agent, but the agent replies with an answer of other intent (including no intent) or replies with a touted response.
3. Out-of-agent scope: a user message is within the scope of the business, but there is no corresponding intent within the agent, and the agent replies with an answer to an intent or a response to a tout response.
4. Unclear semantics: the user sends a message with unclear semantics, but the agent replies with an answer of some intent or replies with a touted response.

It can be easily understood what the correct intent is, that is, that the user question is semantically consistent with the recalled intent or that it is not recalled and should not be recalled, but here two new concepts are introduced, agent scope and semantic uncertainty.

• Agent-wide

  • the user message has a corresponding intent within the agent, i.e., to be within range of the agent.
  • messages that are relevant to the business scenario, but have no corresponding intent in the agent, are counted as out of agent scope. It is important to identify the relationship between the scope of the agent and the scope of the business here; the two are not equivalent; one is whether or not the attendant should answer, and the other is whether or not the relevant intent has been maintained within the agent.

• Unclear semantics

  • Pure numbers, letters, emoticons, symbols, sentences with unintelligible textual meaning and messages not related to business.

No intention V.S. Unclear semantics

No intent puts questions that the agent is not expected to reply to, in short, no intent is blacklisted; semantic unknown is not necessarily what the agent does not need to reply to, the agent can reply to the user by guiding some of their unrecognizable semantic intent into a complete question through certain guidance.

Usually, a user asks a question in the business domain, and if there is no corresponding configuration, it is considered semantically unclear. However, if corresponding configuration is available, the question can be answered by the agent.

For example, if a staffing scenario asks for "five insurance and one pension", if the intent is to configure "five insurance and one pension", then the answer to the keyword "five insurance and one pension" should be replied to. Generally speaking, this response can be an answer to the "5 premiums" keyword and an associated intent prompt. However, in this example, it is not appropriate to maintain "5 premiums" as no intent.

Combining with agent responses, the semantics of user messages can be classified into the following types:

1. correct: the agent recalled the correct intent, or tout, containing the tout formed by artificially setting no intent.
2. recall error (within the library): the user message has a corresponding intent within the agent, but the agent recalls other intents.
3. recall error (outside the library): a user message is within the scope of the operation, but there is no corresponding intent within the agent, and the agent recalls an intent.
4. unclear semantics: the user sends a message with unclear semantics, but the agent replies with an answer or tout of some intent.
5. recall omissions: user messages have a corresponding intent within the agent, but the agent replies to the tout.
6. belonging to a new intent: the user message falls within the scope of the business but has no corresponding intent within the agent, which replies to the tout.
7. Ignore: This message is ignored and not marked.

3 Intentional Accuracy and Recall Optimization Methods

1. Correct Ideal state, no optimization required, or optionally added to label the correct intent.

2. Recall error (in the library) may result from misplacement of similar questions, semantic crossover of different intentions, or unreasonable granularity of intentions. operations such as combing for similar issues within the corresponding intent, or adding new proprietary vocabulary to tune.

3. Recall error (out-of-library) Edit the question into the agent as a new intention, taking care to differentiate it from the semantics of the existing intention. See if there is a semantic crossover between the intent of the mistaken recall and the issue, and make adjustments.

4. Recall omissions Similar questioning may be missing from the counterpart intent and similar questions need to be added. Proprietary terms may also need to be added.

5. Belongs to a new intent The need to add the question as a new intention to the agent, taking care to distinguish it from the semantics of existing intentions.

6. Unclear semantics to see if the intent of the false recall intersects semantically with the question, which can also be addressed by adjusting the threshold. If the message is related to agent content, it can be guided by a welcome message and touting discourse, or by setting keywords to the way the user asks questions. If the message is specifically not intended to be responded to, it may be added to the no intent intent.

7. Ignore This message is ignored and not processed.

4 How to optimize according to the labeled results

Recall condition	user conclusion	label conclusion	operation	optimization methods
Recall	correct	correct	click the history to identify the correct intent; The correct button after the recommendation intent with the same recognition intent; The user searches for the same intent as the identified intent, labels it correctly, and clicks "in the library"	do not adjust or add into the intent
Recall	error, recall another intent	recall error (in the library)	click the correct button after the recommended intent different from the historical identification intent; Or search for the intent as different from the historical recognition intent, and click	in the library to add it to the correct intent or adjust the domain vocabulary
Recall	error, you should add an intent	recall error (out-of-library)	Click to search for other intents, label it as new or select the intent newly created after recall, and click "out-of-library"	to add it to the new intent
Recall	ambiguous	ambiguous	label as ambiguous	no need to operate
Recall	ignore	ignore	label ignore	no need to operate
Fallback	it belongs to out-of-scope or normal	correct	click the correct button after out-of-scope or "-" do not process or add to out-of-scope	If you find out the mistake, you should call back an intent, recall the omission, click the recommendation intent, or click "in the library" after searching the intent to add it to the correct intent
Fallback	error,belong to a intent	recall omission	The correct button after the recommendation intent with the same recognition intent; The user searches for the same intent as the identified intent, labels it correctly, and clicks "in the library"	to add them to the new intents
Fallback	add an intent	belong to a new intent	Click to search for other intents, label them as new intents or select the new intents after recall, and click "out-of-scope"	to add them to the new intents
Fallback	unclear meaning, no operation	unclear meaning	label as unclear meaning	no operation
Fallback	ignore	ignore	label ignore	no need to operate

5 Combined product operation

Accordingly, the results of recall intents can be marked on the "To be marked" page in the product, either with the corresponding intent (correct) or with another intent (incorrect); you can choose to mark the intent added later as out of the library; you can also mark the intent as semantically unknown or ignore.

This can then be combined with the product's historical recall to correspond out to the labeled conclusions, which are correct, recall error (in the library), recall error (out of the library), semantic unclear, recall omission, part of new intent, and ignore.

According to these situations, filter and check the corresponding parts that need to be added to the agent, and click "Add Intent Trigger" to quickly complete the agent education.

The part of the marking result that is not suitable, you can choose to call back to rework and re-mark; no need to mark can choose to delete.

Note: The semantics is unclear, ignoring because there is no corresponding intent, so it cannot be added to some intent, but will only form the operation record.