Bixby Developer Center

Bixby uses natural language (NL) from the user as input. You can improve Bixby's ability to understand NL input by training Bixby to understand real-world examples of natural language in Bixby Developer Studio (Bixby Studio). For example, in the Quick Start Guide, you train the dice game to recognize "roll 2 6-sided dice". This phrase is an utterance. NL training is based on utterances that humans might type or say when interacting within Bixby. Utterances don't have to be grammatical and can include slang or colloquial language.

For any capsule you create, you'll need to add training examples in Bixby Studio for each language supported. If you are planning to support various devices, you need to add training for those devices too. Training examples can also specifically target supported Bixby devices (such as mobile) and specific regions that speak a supported language (for instance, en-US for the United States and en-GB for Great Britain). Bixby converts unstructured natural language into a structured intent, which Bixby uses to create a plan.

In this introduction, you'll learn how to plan and add training. You'll also learn how to search through training entries effectively, and use aligned natural language to test utterances. Finally, you'll learn how to add vocabulary that can assist in training.

For more information on effective training guidelines and best practices, see Training Best Practices.

Video Tutorial: Training v2

The Training Editor

To use the training editor, select Training in the Files view of Bixby Developer Studio's sidebar. The editor opens to the Training Set List.

Training Set List

This screen presents a list of Training Sets and Capsule Targets for your capsule. A training set consists of one or more training entries. A training entry is a specific utterance Bixby should understand, like "book a space resort", annotated with the information Bixby needs to be able to interpret that utterance.

A training entry in Bixby can have the following targets:

• A specific language, for any locale or any device: en
• A specific language and locale, for any device: en-US
• A specific language and a specific device, for any locale: bixby-mobile-en
• A specific language, locale, and device: bixby-mobile-en-US

The list of training sets is derived from the targets block in your capsule.bxb file. For more information, read the Setting Up Your Capsule topic.

A target has a "training budget" of 2,000 training entries. This is computed as the sum of all entries that apply to that target. The training screen lists the budget available for each target as well as the parent targets that are part of that budget. For instance, if you had both bixby-mobile-en-US and bixby-mobile-en-UK targets, the en language set would be shown as applying to both of them. Training entries for patterns do not count towards this budget.

In addition, this screen shows status counts for each target, and whether your capsule's natural language (NL) model needs to be recompiled. You can recompile targets individually by clicking the "Compile" button that appears when you hover over a capsule target (this button will be dimmed if compiling that target is unnecessary), and compile all targets by clicking "Compile All Targets".

Training Entry List

Clicking on a target will open up the list of training entries for that target.

For each training entry, this page shows:

• The entry's goal and, if applicable, specialization
• The utterance the entry is trained on, with its values highlighted
• The entry's training status

A target can have dozens, hundreds, or even thousands of training entries. You can search and filter the list in several ways:

• Select "Language", "Locale", "Device", or "All" to show only items in that training set (for example, just en entries in the bixby-mobile-en-US list)
• Use the search field to show only entries that contain specific text, as well as search filter keywords
• Use the right-hand sidebar to select and deselect search filters

From the Training Entry List, you can add a new training entry as well as recompile the natural language model.

Adding New Training

After you've modeled your capsule's domain, defining the concepts, the actions, and the action implementations, the training is what pulls it all together. The user's utterance provides the natural language (NL) inputs to Bixby's program, and the goal associated with that utterance tells Bixby what the output of the program needs to be. A training example consists of a sample utterance annotated to connect values to your capsule's concepts and actions. Bixby uses the utterance and annotations to produce an intent.

With good training examples, Bixby will do more than simply memorize the words and their meaning. It will learn the most important aspects of those examples and apply that knowledge to new words and sentences. Natural language training teaches the platform about sentence meaning.

Add new training from the Training Entry List screen by clicking the Add button.

For each training entry, you'll:

1. Set a goal for the utterance
2. Identify values in the example's utterance
3. Verify the plan that Bixby creates for the example

Let's look at each step in more detail.

Set a Goal

Goals are usually concepts. If your capsule includes a Weather concept that contains all the information for a weather report, then when the user asks "what's the weather", the goal is the Weather concept.

Sometimes, though, goals might be actions. This usually happens when the user is purchasing a product or a service in a transaction. For example, a capsule that books airline flights would have training examples that use a BookFlight action as a goal.

Identify Values

An utterance is a phrase or sentence, appropriate to the training example's language and locale, that directs Bixby toward the training example's goal. To reach a goal, Bixby often needs input values: the restaurant, date, and time of a dinner reservation; the location and day for a weather report. The user's utterances often contain some of the necessary values:

• "Find me a Hawaiian restaurant"
• "Book a table for 4 on Saturday at 7pm"
• "Find a resort on Mars for next weekend"

To help Bixby identify values in an utterance, you annotate the values, matching them to concepts in your capsule or in Bixby library capsules.

• "Hawaiian" is example.food.CuisineStyle
• "4" is example.reservation.Size
• "Saturday at 7pm" is viv.time.DateTimeExpression
• "next weekend" is viv.time.DateTimeExpression
• "Mars" is example.spaceResorts.Planet

To annotate a value in an utterance, highlight the value in the utterance and click the v:value button to add a value signal. Then, fill in the appropriate information in the form that appears in Bixby Studio's right-hand sidebar.

You can also use keyboard shortcuts to annotate utterances:

• Option/Alt+V for values
• Option/Alt+R for routes

For enumerated values, also set the Form field to a valid symbol that matches the annotated word or phrase; if the phrase is "extra large" and this matches a Size of ExtraLarge, the form should be ExtraLarge.

The values you annotate in an utterance should match primitive concepts, not structure concepts. If you need a structure concept as an input for an action, create a Constructor that takes the primitive concepts that match the properties of the structure concept as inputs, and outputs the structure concept. See the Constructor documentation for an example.

You don't need to explicitly tell Bixby to use the constructor action; the planner uses the action in the execution graph when it needs to construct a concept.

In addition to annotating utterances with values, you can also annotate them with routes. Read the Routes section for information on when you might use routes (and when you shouldn't).

Check the Plan

Once you have identified all the values in the utterance, confirm that the plan looks like it should. Here's the plan for "Find hotels near Mars" from the Space Resorts capsule:

This is a simple plan, and you can see how the inputs feed into actions that resolve the goal. Make sure the goal is correct, and the inputs have the right types. If you aren't sure, go ahead and run it in the device simulator by clicking the Run on Simulator button. If anything in the plan looks odd, executing it often exposes the problem.

Specializations

Some utterances require more information in your training example for Bixby to process them correctly. A user might say something that requires the context of the previous utterance, or Bixby might need to prompt the user for information needed to complete the initial request. These cases are handled by specializations.

Continuation Of

After a user says something like "buy a medium shirt", they could follow it up with "change the size to large". When a user follows up their previous utterance with a related request or refinement, this is a continuation. A continuation is an utterance that only makes sense in the context of the previous utterance.

A continuation needs to specify two nodes:

• The goal the continuation should reach. This is specified in the "Goal" field, just like any other training example.
• The goal of the utterance the continuation is refining. This specifies what this example is a "continuation of".

To train "change the size to large" in the shirt store sample capsule:

• The goal of this training example is the CommitOrder action.
• The utterance is a continuation of a previous utterance whose goal is also the CommitOrder action, so we specify CommitOrder again in the "Continuation of" field.

The program generated by the previous utterance doesn't have to reached its conclusion before the continuation is acted upon. In the Shirt capsule above, the user is presented with a confirmation view before the CommitOrder action finishes executing, and that's the moment the user might change the order's details with a continuation utterance. In other capsules, the first utterance's execution might have fully completed before a continuation. For example, a weather capsule could support the utterance "give me the weather in London" followed by the continuation "what about Paris".

Continuations are a common use case for routes; in the example above, the Flags list for this training entry shows a route through the UpdateOrder action, which modifies the existing saved order. Read about Routes for more information (and alternatives to using routes). In addition, "Large" has been given a Role of ChangedItem; read about Adding Context with Roles for more information.

At Prompt For

When there isn't enough information to reach a goal, Bixby prompts the user for further information. For example, suppose in the previous example, instead of "change the size to large" the user had only said "change the size". Bixby needs to know what to change the size to, and asks the user for this information. As the developer, you can account for these prompts by adding examples of how the user would respond to a prompt.

To make a training example for a prompt, add an example of a response to a prompt. Enter small in the Add Training field. This is a response to a prompt for example.shirt.Size, so enter Size as the goal. From the No specialization drop-down menu, choose At prompt for:

In this case, Size is both the goal of this training example and the specialization node. The text small in the utterance is annotated as the node Size and the form Small.

Pattern For

Patterns provide vocabulary for structured concepts. They are not real training examples, but are rather treated like vocabulary entries in a training-like format.

Consider DateTime training. The viv.time DateTime library capsule uses patterns so that developers who import it only have to train one main type: viv.time.DateTimeExpression. Inside viv.time, training consists of patterns that map to the DateTimeExpression concept. The patterns are all variations of ways people could refer to DateTime objects. Patterns do not use machine learning, but are simpler templates whose pieces are matched explicitly.

Consider these training examples:

Example 1:

NL Utterance: "2:30 p.m. this Friday"

Aligned NL:

[g:DateTime:pattern] (2)[v:Hour]:(30)[v:Minute] (p.m.)[v:AmPm:Pm]
(this)[v:ExplicitOffsetFromNow:This] (friday)[v:DayOfWeek:Friday]

The pattern being matched here is:

[hour]:[minute] [AM or PM] this [day of week]

Example 2:

NL Utterance: "July 23rd"

Aligned NL:

[g:viv.time.Date:pattern] (July)[v:viv.time.MonthName:July] (23rd)[v:viv.time.Day:23]

This matches a simpler pattern:

[month] [day]

Here is the training entry in the editor for Example 2:

Note above that both goals (the left goal and the right Specialization Node... goal) are the same: viv.Time.Date.

The pattern in Example 2 will not match an utterance of "the 23rd of July"; that will need to be matched by a different pattern. Multiple patterns can exist for the same goal.

Another feature of patterns that's distinct from normal training is that patterns can be reused in other training examples, and even in other patterns. The date patterns above could be included in a pattern for DateInterval, allowing Bixby to understand ranges like "January 1st through January 7th".

All these different DateTime intent signatures should be interchangeable, for the most part, without having to train all of the possible combinations of DateTime inputs in your capsule.

Patterns are designed for matching language elements that are regular and structured; if an utterance exactly matches a pattern, Bixby can always understand it. Regular training entries allow for less regular speech patterns and variations to be understood through machine learning, where it's not practical to define every possible variation.

Advanced Topics

Aligned NL

You've already likely seen examples of Aligned NL, a plain text representation of utterances that include the annotations for goals, values, and routes. When you annotate an utterance in the training editor, Bixby creates aligned NL. Bixby also generates aligned NL when processing a real query.

You can use aligned NL in the Device Simulator to give Bixby queries even when you haven't compiled the current NL model. The training editor can switch between displaying and editing annotated NL and the aligned NL for utterances by toggling the Show Aligned NL checkbox:

Here are some other examples of aligned NL utterances:

[g:viv.twc.Weather] weather on (saturday)[v:viv.time.DateTimeExpression] in
(san jose)[v:viv.geo.LocalityName]

[g:flightBooking.TripOption] I need a flight from {[g:air.DepartureAirport]
(sjc)[v:air.AirportCodeIATA:SJC]} to (lax)[v:air.AirportCodeIATA:LAX]
(tonight)[v:time.DateTimeInterval]

Annotations are denoted in brackets. The first character indicates a signal, the kind of annotation, followed by a colon and the concept the annotation refers to. Words or phrases might follow the annotations, grouped in parentheses like "san jose" above. Subplans are enclosed in braces. There are four kinds of annotation types:

• g: goal signal
• v: value signal
• r: route signal
• f: flag signal

To process a request, Aligned NL is translated by Bixby into an intent. You can see the intent for a request in the Debug Console. The intent for the "research space resorts on Mercury" request above is:

intent {
  goal {
    1.0.0-example.spaceResorts.SpaceResort#all
    @context (Outer)
  }
  value {
    1.0.0-example.spaceResorts.Planet (Mercury)
  }
}

Adding Context with Roles

Sometimes, Bixby needs more context to fully understand values. For instance, the following utterance includes two airport codes: "find flights from SFO to YUL leaving this December 22nd". The codes are both the same AirportCode concept, but one has a role of DepartureAirport and one has a role of ArrivalAirport.

To add roles to utterances, highlight the parts of the utterances that need roles and click the Role button. You can add the node for the role in the right-hand sidebar.

Now that you've added these roles to the training example, the Bixby planner can distinguish that the second airport is using a type that is a role-of the parent type. You can learn more about role assignment.

Another place roles are used is when two or more values can be used together to specify an item. For instance, consider the utterance "Remove the XL shirt from my cart" in the shirt example capsule: "XL" is a Size and "shirt" is a SearchTerm; together, though, they have the role of RemovedItem.

Training on Properties

There are instances when a user utterance targets a property of a goal rather than the goal itself. This is called property projection. For example, when a user asks the space resorts capsule "Where is The Mercurial", the intent of the utterance is asking for the location of a specific space resort, not information about the space resort as a whole.

In cases like this, where the utterance asks for a specific part or "property" of the goal concept, you can add specific training entries to target properties like this.

To add a property to a target goal, specify the property with a hash (#) notation: in this case, SpaceResort#planet. The structured concept SpaceResort has a property for the total (planet). By making this the goal, you assert that the utterance's intent and goal is the planet (planet) in the Space Resort model (SpaceResort).

Flags

When adding training, there are some utterances in which it is difficult to precisely annotate values.

Take, for example, the following utterance: "reserve a table for 3 people for dinner next Friday". We can annotate the number of people the reservation is for ("3") and the requested time ("dinner next Friday"). But, we need to add an extra criteria to this search query: we should filter out restaurants that are not currently accepting reservations.

In cases like this, when you cannot clarify the intent by annotating specific words or adding additional vocabulary, you can use flags to give clues to Bixby. You can think of flags as a way to annotate the entire utterance rather than a specific word or phrase within it. Just like annotations, flags can provide values or routes.

In this example from the shirt capsule, "Remove the XL shirt from my cart" demonstrates several training concepts, including a route flag:

• "XL" is annotated as a Size
• "t-shirt" is annotated as a SearchTerm
• "XL t-shirt" is together given a role of RemovedItem
• The entire utterance has a flag of UpdateOrder, specifying an action to route through on the way to the goal of CommitOrder

To add a flag, click the Flag button, then select "Create Value Flag" or "Create Route Flag" from the dropdown. You will be prompted in the sidebar for the necessary information, such as the route's node, or the value's node and form.

Routes

A route tells the planner to go through a "waypoint" on the way to the goal, so it can factor in more information or prompt for more information.

For instance, take the utterance "Is it raining near me?" To train this, you could add a flag signaling that the weather condition is rain: set the node weather.WeatherCondition to the value rain. Then you would add a route of geo.CurrentLocation to let Bixby know to factor in the current user location.

Another example of using a route is a continuation that needs to prompt for more information. For example, in the restaurant reservation example, the user might give Bixby the utterance, "change the party size"; Bixby will need to prompt for the new size:

[g:ChangeReservation:continue:Reserve,r:UpdatePartySize] Change the party size

The PromptForPartySize route lets Bixby know that it needs to prompt the user.

Routes can be attached to individual values, but are more commonly associated with flags on a whole utterance. In either case, you still need to annotate specific words in utterances as values. Additionally, when you can substitute specific words within a sentence (such as "cheap", "inexpensive", and "affordable"), normal training still offers the benefit of vocabulary.

Routes can make your capsule's logic harder to follow, and in many cases you can design your capsule so they're unnecessary. The continuation example above could be implemented so that UpdatePartySize is a valid goal, for instance, or party size could itself be a value supplied to ChangeReservation. Read Training Best Practices for more guidance.

Batch Actions

You can select multiple training entries for certain batch operations:

• Click the "more actions" menu (the dropdown menu accessed with the "•••" button at the right-hand side of the search bar) to choose Select All or Select None.
• Click the checkmark at the top left corner of each individual training entry to select or deselect that entry.

The available batch actions when one or more entries are selected include:

• Edit: Make changes that apply to all of the selected entries, such as setting new goals, flags, or specialization types.
• Copy: Copy the selected entries to another target. You can choose to copy them to a different locale, language, or device target. Copying entries duplicates them: the entries remain in the current target, and are also created in the new target.
• Move: Move the selected entries to another target. You can choose to move them to a different locale, language, or device target. Moving entries removes them from the current target.
• Verify: Verify the plans of the selected entries.
• Delete: Delete the selected entries.

Search Filters

The training editor offers several ways to search through your trained utterances.

• Use the sidebar on the right to quickly filter by status, issues, goals, and specializations.
• Use the search bar at the top of the screen to search for words and phrases within your training.
  • The search matches your entered phrase exactly. If you search for the phrase weekend in december it will match all entries with that exact phrase, but will not find "weekend in January".
  • Search is case-insensitive: Samsung, samsung and SAMSUNG are all equivalent.
  • Exclude a word from a search query by prefixing it with the - character: Show me the weather in san jose -bernadino will match show me the weather in san jose and show me the weather in san francisco but not show me the weather in san bernadino.
• You can use scope prefixes in search text to refine searches, using keywords in the following table.

Valid fields for the has: scope are value, route, role, continuation, pattern, prompt, tag, enabled, disabled, and flag. This scope searches for entries that have any field of that type, rather than specific contents in that field.

Valid learning statuses are learned, not-learned, and not-compiled.

You can combine search scopes to filter more narrowly. To search for training with a goal of weather.Weather related to humidity tomorrow, you could use goal:weather.Weather humid tomorrow.

Troubleshooting

Bixby Studio informs you if your training examples have issues. However, if you have a large number of training entries, it can sometimes be difficult to quickly find a training entry causing an issue. This section discusses how to find any issues and what certain training entry statuses mean.

Training Entry Statuses

Training entries usually exist in one of three states:

• Learned: Bixby has learned, and can execute, this training entry.
• Not Learned: Bixby does not have enough information to learn from this training entry.
• Not Compiled: The NL model needs to be recompiled before this entry can be learned.

When you add training entries, they will usually start as Not Compiled. This status applies as long as Bixby hasn't generated NL models for those entries.

To generate new NL models when you add new examples or update old ones, click Compile NL Model. When the models are generated, Bixby changes the label of the learned utterances to Learned. If your training entry is similar enough to previous training examples, Bixby might label it as Learned even before you compile the model.

Changes in other training or vocabulary can cause an utterance to revert to a Not Learned or Not Compiled state. If Bixby can't learn a specific example, consider adding similar training examples, reviewing existing training entries for conflicting utterances, or updating vocabulary.

There are four other possible status values for training entries:

• Disabled: The training entry is not scanned for errors and not trained.
• Illegal Plan: The training entry has unsatisfied input constraints or unconsumed signals. This means that the intent of what you have trained does not match with the way the models have been designed to work. See Illegal Plans.
• Unverified Plan: The training entry's current plan differs in a nontrivial way from the plan at the time this entry was saved. This can happen when model changes make the training entry's plan behave differently from the last time it was saved. You can resolve this by ensuring the plan for the entry is correct and recompiling the NL model, or by running Verify All Plans.
• Incomplete: These are generally semantic, such as the lack of a goal or referencing Node IDs that do not exist.

Finding Training Entry Issues

You can quickly find issues using the search sidebar on the Training Sets Screen. Check the Issues subheading for categories of warnings and errors.

If you are submitting a capsule and one of your training entries is throwing an error, you can search for that specific error using the id: search scope.

1. Open the submission page and select your failed example.

2. Select "Capsule Interpreter Training".

3. On the log page, look for the entries causing the issue:

    --------------------------------------------------------------------
    (4) Entries in state NOT_LEARNED (ERROR: actively causing rejection)
    --------------------------------------------------------------------
    View all items in category (copy the search between the markers into the training tool)
    >>>> BEGIN COPY <<<<
    id:t-jcvrft523b1j3iwzti7wx2zq7,t-q1b5nsdjb2clwy8fkhre80ecg,t-wczwvczrezonkmp2djlaj9wcj,t-v7s9drvf7whsbfxup9n9rw2j8
    >>>> END COPY <<<<

4. Copy the failure ID(s), as directed in the training window.

5. Search for your training example with the ID by pasting the ID into the search bar. This will find the entry blocking the submission.

Illegal Plans

An illegal plan occurs when the graph that Bixby's planner has created for a specific training entry cannot be run. There are several possible reasons why a plan might be illegal:

• The models have input constraints which cannot be satisfied.
• The graph has an unused signal, which appears as a detached node.
• The plan contains an uninstantiated plan-behavior (Always) input.
• The plan contains three or more consecutive lossy action nodes chained together with no intervening nodes.

A "lossy" activity is one that has the potential of losing user-provided signals, such as a search action, fetch action, or transactions. You can often resolve the illegal plan error by adding a route, or making one of the actions a different non-lossy type such as Calculate or Resolver, though certain actions types should be used for specific situations. An illegal plan normally indicates a problem with your capsule's modeling; in general, you don't want to force Bixby to create long chains of actions in order to fulfill user requests.

Verifying Plans

You can clear "Unverified Plan" statuses by selecting the Verify All Plans command in the "more actions" menu (the dropdown menu accessed with the "•••" button at the right-hand side of the search bar). This command asserts that the plans for all complete, legal, and enabled entries are correct. It will create a new plan from the Aligned NL for each entry, based on the current state of your capsule's models, vocabulary, and other data.

The training status of each entry might change after running this command. Disabled plans will not be verified.

Training Limitations

Number of Training Entries

A capsule can have up to 2,000 training entries per target. A target is a specific combination of device, language, and region:

• en: All devices, English language, all regions
• bixby-mobile-en: Mobile devices, English language, all regions
• bixby-mobile-en-US: Mobile devices, English language, US region

The 2,000-entry limit is shared across parent targets. For example, the en target is a parent of all English-speaking device targets such as the bixby-mobile-en, bixby-mobile-en-US, and bixby-other-en-US; if en had 1,000 entries in it, then bixby-mobile-en would inherit those entries, and could have up to 1,000 more. If en has 1,000 entries in it and bixby-mobile-en had 500 more, then bixby-mobile-en-US would inherit from both of its parents, and could add up to 500 more.

No Named Entities

Bixby does not have platform-level support for recognizing named entities. That means it won't automatically be able to distinguish between people, places, movie titles, etc. It is up to you to distinguish these entities in a domain-specific way, through training examples and vocabulary.

Reserved Utterances

There are several utterances that have specific meanings in particular situations. For example, "Next" while navigating a list will read the next item or page of items. When possible, create complete training examples to help distinguish between these reserved meta-commands. For example, train "Cancel my reservation" instead of just "Cancel" for English because "Cancel" clears conversation context in Bixby.

For the full list of meta-commands, see the Reserved Utterances topic reference.

Automatic Speech Recognition and Text-to-Speech

Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) are useful Bixby capabilities for communicating with users. However, the natural language utterances you train and the vocabulary you add are only used in their specific capsule (although they might be used over time to help improve ASR and TTS performance).