Bixby Developer Center

Bixby uses natural language (NL) from the user as input. You can improve Bixby's ability to understand NL input by training Bixby to understand real-world examples of natural language in Bixby Developer Studio (Bixby Studio). For example, in the Quick Start Guide, you train the dice game to recognize "roll 2 6-sided dice". This phrase is an utterance. NL training is based on utterances that humans might type or say when interacting within Bixby. Utterances don't have to be grammatical and can include slang or colloquial language.

For any capsule you create, you'll need to add training examples in Bixby Studio for each language supported. If you are planning to support various devices, you need to add training for those devices too. Training examples can also specifically target supported Bixby devices (such as mobile) and specific regions that speak a supported language (for instance, en-US for the United States and en-GB for Great Britain). Bixby converts unstructured natural language into a structured intent, which Bixby uses to create a plan.

In this introduction, you'll learn how to plan and add training. You'll also learn how to search through training entries effectively, and use aligned natural language to test utterances. Finally, you'll learn how to add vocabulary that can assist in training.

For more information on effective training guidelines and best practices, see Training Best Practices.

The Training Editor

To use the training editor, select Training in the Files view of Bixby Developer Studio's sidebar. The editor opens to the Training Summary.

Training Summary

The Training Summary page shows a list of training entries for a given training source. You can add new entries, edit existing entries, and compile a Natural Language model for a specific capsule target. You can also see Statistics, which shows an overview of each defined capsule target and the remaining "budget" available for training entries within each one. (See Training Limitations for a discussion of budgets.)

The training source is a resources folder specific to the language and optionally locale and device, such as en, bixby-mobile-en-US, or ko-KR. You can both select existing sources and add new ones with the Training Source dropdown on this screen. The compile target is the combination of language, locale and device, such as bixby-mobile-en-US.

When a training source is selected, the summary page shows the existing training entries for that source. For each training entry, you'll see:

• The utterance the entry is trained on, with its values highlighted. Values correspond to concepts you've modeled within your capsule, such as a reservation date or hotel name.
• The entry's goal. Goals also usually correspond to concepts, such as a completed reservation, but can correspond to an action.
• If applicable, the entry's specialization.
• The entry's training status.

A target can have dozens, hundreds, or even thousands of training entries. You can search and filter the list in two ways:

• Use the search field to show only entries that contain specific text, as well as search filter keywords
• Use the right-hand sidebar to select and deselect search filters

From the Training Entry List, you can add a new training entry as well as recompile the natural language model.

Adding New Training Entries

After you've modeled your capsule's domain, defining the concepts, the actions, and the action implementations, the training is what pulls it all together. The user's utterance provides the natural language (NL) inputs to Bixby's program, and the goal associated with that utterance tells Bixby what the output of the program needs to be. A training example consists of a sample utterance annotated to connect values to your capsule's concepts and actions. Bixby uses the utterance and annotations to produce an intent.

With good training examples, Bixby will do more than simply memorize the words and their meaning. It will learn the most important aspects of those examples and apply that knowledge to new words and sentences. Natural language training teaches the platform about sentence meaning.

Add new training from the Training Summary screen by clicking the Add button.

You will first be prompted to add the utterance you wish to train on. An utterance is a phrase or sentence, appropriate to the training example's language and locale, that directs Bixby toward the training example's goal:

• "Find me a Hawaiian restaurant"
• "Book a table for 4 on Saturday at 7pm"
• "Find a resort on Mars for next weekend"

After entering the utterance, you'll need to:

1. Set a goal for the utterance
2. Identify values in the example's utterance
3. Verify the plan that Bixby creates for the example

Let's look at each step in more detail.

Set a Goal

Goals are usually concepts. If your capsule includes a Weather concept that contains all the information for a weather report, then when the user asks "what's the weather", the goal is the Weather concept.

Sometimes, though, goals might be actions. This usually happens when the user is purchasing a product or a service in a transaction. For example, a capsule that books airline flights would have training examples that use a BookFlight action as a goal.

Identify Values

To reach a goal, Bixby often needs input values: the restaurant, date, and time of a dinner reservation; the location and day for a weather report. The user's utterances often contain some of the necessary values:

• "Find me a Hawaiian restaurant"
• "Book a table for 4 on Saturday at 7pm"
• "Find a resort on Mars for next weekend"

To help Bixby identify values in an utterance, you annotate the values, matching them to concepts in your capsule or in Bixby library capsules.

• "Hawaiian" is example.food.CuisineStyle
• "4" is example.reservation.Size
• "Saturday at 7pm" is viv.time.DateTimeExpression
• "next weekend" is viv.time.DateTimeExpression
• "Mars" is example.spaceResorts.Planet

To annotate a value in an utterance, click or highlight the value in the utterance and click the Value tab in the dropdown that appears. Then, enter the concept this value is associated with in the Node field. For enumerated values, you'll also need to enter a valid symbol in the Form field; if the value is "extra large" and this matches a Size of ExtraLarge, the Form should be ExtraLarge. The autocompletion for the Node field will provide a list of valid forms delimited by a colon, such as Size:ExtraLarge or, in the following example, Planet:Mars.

The values you annotate in an utterance should match primitive concepts, not structure concepts. If you need a structure concept as an input for an action, create a Constructor that takes the primitive concepts that match the properties of the structure concept as inputs, and outputs the structure concept. See the Constructor documentation for an example.

You don't need to explicitly tell Bixby to use the constructor action; the planner uses the action in the execution graph when it needs to construct a concept.

In addition to annotating utterances with values, you can also annotate them with routes. Read the Routes section for information on when you might use routes (and when you shouldn't).

Check the Plan

Once you have identified all the values in the utterance, confirm that the plan looks like it should. Here's the plan for "Find hotels near Mars" from the Space Resorts capsule:

This is a simple plan, and you can see how the inputs feed into actions that resolve the goal. Make sure the goal is correct, and the inputs have the right types. If you aren't sure, go ahead and run it in the device simulator by clicking the Run on Simulator button. If anything in the plan looks odd, executing it often exposes the problem.

Draft Entries

When you are creating a new entry, clicking the back arrow in the upper left corner will return you to the summary screen, but save your work as a draft.

When a draft is saved, it will appear as an option on the summary screen.

Clicking the Cancel button will return to the summary screen without saving the new entry as a draft, and will discard any existing draft.

Specializations

Some utterances require more information in your training example for Bixby to process them correctly. A user might say something that requires the context of the previous utterance, or Bixby might need to prompt the user for information needed to complete the initial request. These cases are handled by specializations.

Continuation Of

After a user says something like "buy a medium shirt", they could follow it up with "change the size to large". When a user follows up their previous utterance with a related request or refinement, this is a continuation. A continuation is an utterance that only makes sense in the context of the previous utterance.

A continuation needs to specify two nodes:

• The goal the continuation should reach. This is specified in the "Goal" field, just like any other training example.
• The goal of the utterance the continuation is refining. This specifies what this example is a "continuation of".

To train "change the size to large" in the shirt store sample capsule:

• The goal of this training example is the CommitOrder action.
• The utterance is a continuation of a previous utterance whose goal is also the CommitOrder action, so we specify CommitOrder again in the "Continuation of" field.

The program generated by the previous utterance doesn't have to reached its conclusion before the continuation is acted upon. In the Shirt capsule above, the user is presented with a confirmation view before the CommitOrder action finishes executing, and that's the moment the user might change the order's details with a continuation utterance. In other capsules, the first utterance's execution might have fully completed before a continuation. For example, a weather capsule could support the utterance "give me the weather in London" followed by the continuation "what about Paris".

Continuations are a common use case for routes; in the example above, the Flags list for this training entry shows a route through the UpdateOrder action, which modifies the existing saved order. Read about Routes for more information (and alternatives to using routes). In addition, "Large" has been given a Role of ChangedItem; read about Adding Context with Roles for more information.

For more examples, see the Continuation for Training sample capsule.

At Prompt For

When there isn't enough information to reach a goal, Bixby prompts the user for further information. For example, suppose in the previous example, instead of "change the size to large" the user had only said "change the size". Bixby needs to know what to change the size to, and asks the user for this information. As the developer, you can account for these prompts by adding examples of how the user would respond to a prompt.

To make a training example for a prompt, add an example of a response to a prompt. Click Add, enter small in the training example field, and click Add. This is a response to a prompt for example.shirt.Size, so enter Size as the goal. From the No specialization drop-down menu, choose At prompt for:

In this case, Size is both the goal of this training example and the specialization node. The text small in the utterance is annotated as the node Size and the form Small.

Pattern For

Patterns provide vocabulary for structured concepts. They are not real training examples, but are rather treated like vocabulary entries in a training-like format.

Consider DateTime training. The viv.time DateTime library capsule uses patterns so that developers who import it only have to train one main type: viv.time.DateTimeExpression. Inside viv.time, training consists of patterns that map to the DateTimeExpression concept. The patterns are all variations of ways people could refer to DateTime objects. Patterns do not use machine learning, but are simpler templates whose pieces are matched explicitly.

Consider these training examples:

Example 1:

NL Utterance: "2:30 p.m. this Friday"

Aligned NL:

[g:DateTime:pattern] (2)[v:Hour]:(30)[v:Minute] (p.m.)[v:AmPm:Pm]
(this)[v:ExplicitOffsetFromNow:This] (friday)[v:DayOfWeek:Friday]

The pattern being matched here is:

[hour]:[minute] [AM or PM] this [day of week]

Example 2:

NL Utterance: "July 23rd"

Aligned NL:

[g:viv.time.Date:pattern] (July)[v:viv.time.MonthName:July] (23rd)[v:viv.time.Day:23]

This matches a simpler pattern:

[month] [day]

Here is the training entry in the editor for Example 2:

Note above that both goals (the left goal and the right Specialization Node... goal) are the same: viv.Time.Date.

The pattern in Example 2 will not match an utterance of "the 23rd of July"; that will need to be matched by a different pattern. Multiple patterns can exist for the same goal.

Another feature of patterns that's distinct from normal training is that patterns can be reused in other training examples, and even in other patterns. The date patterns above could be included in a pattern for DateInterval, allowing Bixby to understand ranges like "January 1st through January 7th".

All these different DateTime intent signatures should be interchangeable, for the most part, without having to train all of the possible combinations of DateTime inputs in your capsule.

Patterns are designed for matching language elements that are regular and structured; if an utterance exactly matches a pattern, Bixby can always understand it. Regular training entries allow for less regular speech patterns and variations to be understood through machine learning, where it's not practical to define every possible variation.

Advanced Topics

Aligned NL

You've already likely seen examples of Aligned NL, a plain text representation of utterances that include the annotations for goals, values, and routes. When you annotate an utterance in the training editor, Bixby creates aligned NL. Bixby also generates aligned NL when processing a real query.

You can use aligned NL in the Device Simulator to give Bixby queries even when you haven't compiled the current NL model. The training editor can switch between displaying and editing annotated NL and the aligned NL for utterances by toggling the Show Aligned NL checkbox:

Here are some other examples of aligned NL utterances:

[g:viv.twc.Weather] weather on (saturday)[v:viv.time.DateTimeExpression] in
(san jose)[v:viv.geo.LocalityName]

[g:flightBooking.TripOption] I need a flight from {[g:air.DepartureAirport]
(sjc)[v:air.AirportCodeIATA:SJC]} to (lax)[v:air.AirportCodeIATA:LAX]
(tonight)[v:time.DateTimeInterval]

Annotations are denoted in brackets. The first character indicates a signal, the kind of annotation, followed by a colon and the concept the annotation refers to. Words or phrases might follow the annotations, grouped in parentheses like "san jose" above. Subplans are enclosed in braces. There are four kinds of annotation types:

• g: goal signal
• v: value signal
• r: route signal
• f: flag signal

To process a request, Aligned NL is translated by Bixby into an intent. You can see the intent for a request in the Debug Console. The intent for the "research space resorts on Mercury" request above is:

intent {
  goal {
    1.0.0-example.spaceResorts.SpaceResort#all
    @context (Outer)
  }
  value {
    1.0.0-example.spaceResorts.Planet (Mercury)
  }
}

Adding Context with Roles

Sometimes, Bixby needs more context to fully understand values. For instance, the following utterance includes two airport codes: "find flights from SFO to YUL leaving this December 22nd". The codes are both the same AirportCode concept, but one has a role of DepartureAirport and one has a role of ArrivalAirport.

To add roles to utterances, highlight the relevant parts of the utterances and click + Add Role. You can add the node for the role in the right-hand sidebar. Once added, the role's annotation shows up as a purple outline.

Now that you've added these roles to the training example, the Bixby planner can distinguish that the second airport is using a type that is a role-of the parent type. You can learn more about role assignment.

Another place roles are used is when two or more values can be used together to specify an item. For instance, consider the utterance "Remove the XL t-shirt from my cart" in the shirt example capsule: "XL" is a Size and "t-shirt" is a SearchTerm; together, though, they have the role of RemovedItem.

With two already annotated values such as this example, you will not be able to highlight both values at once to add the role. Instead, click on either value, then click + Add Role. The outline that appears around the selected value can be moved with the mouse to highlight the entire role phrase.

Training on Properties

There are instances when a user utterance targets a property of a goal rather than the goal itself. This is called property projection. For example, when a user asks the space resorts capsule "Where is The Mercurial", the intent of the utterance is asking for the location of a specific space resort, not information about the space resort as a whole.

In cases like this, where the utterance asks for a specific part or "property" of the goal concept, you can add specific training entries to target properties like this.

To add a property to a target goal, specify the property with a hash (#) notation: in this case, SpaceResort#planet. The structured concept SpaceResort has a property for the total (planet). By making this the goal, you assert that the utterance's intent and goal is the planet (planet) in the Space Resort model (SpaceResort).

Flags

When adding training, there are some utterances in which it is difficult to precisely annotate values.

Take, for example, the following utterance: "reserve a table for 3 people for dinner next Friday". We can annotate the number of people the reservation is for ("3") and the requested time ("dinner next Friday"). But, we need to add an extra criteria to this search query: we should filter out restaurants that are not currently accepting reservations.

In cases like this, when you cannot clarify the intent by annotating specific words or adding additional vocabulary, you can use flags to give clues to Bixby. Flags are a way to annotate the entire utterance rather than a specific word or phrase within it. Just like annotations, flags can provide values or routes.

In this example from the shirt capsule, "Remove the XL shirt from my cart" demonstrates several training concepts, including a flag:

• "XL" is annotated as a Size
• "t-shirt" is annotated as a SearchTerm
• "XL t-shirt" is together given a role of RemovedItem
• The entire utterance has a route flag of UpdateOrder, specifying an action to route through on the way to the goal of CommitOrder

To add a flag, click the bar to the left of the NL field. (The entire field will be highlighted when you are over it.) Then select the "Value" or "Route" tab. You will be prompted in the sidebar for the necessary information, such as the route's node, or the value's node and form.

Routes

A route tells the planner to go through a "waypoint" on the way to the goal, so it can factor in more information or prompt for more information.

For instance, take the utterance "Is it raining near me?" To train this, you could add a flag signaling that the weather condition is rain: set the node weather.WeatherCondition to the value rain. Then you would add a route of geo.CurrentLocation to let Bixby know to factor in the current user location.

Another example of using a route is a continuation that needs to prompt for more information. For example, in the restaurant reservation example, the user might give Bixby the utterance, "change the party size"; Bixby will need to prompt for the new size:

[g:ChangeReservation:continue:Reserve,r:UpdatePartySize] Change the party size

The PromptForPartySize route lets Bixby know that it needs to prompt the user.

Routes can be attached to individual values, but are more commonly associated with flags on a whole utterance. In either case, you still need to annotate specific words in utterances as values. Additionally, when you can substitute specific words within a sentence (such as "cheap", "inexpensive", and "affordable"), normal training still offers the benefit of vocabulary.

Routes can make your capsule's logic harder to follow, and in many cases you can design your capsule so they're unnecessary. The continuation example above could be implemented so that UpdatePartySize is a valid goal, for instance, or party size could itself be a value supplied to ChangeReservation. Read Training Best Practices for more guidance.

Batch Actions

You can select multiple training entries for certain batch operations:

• Click the "more actions" menu (the dropdown menu accessed with the "•••" button at the right-hand side of the search bar) to choose Select All or Select None.
• Click the checkmark that appears when you hover over the top left corner of each individual training entry to select or deselect that entry.

The available batch actions when one or more entries are selected include:

• Edit: Make changes that apply to all of the selected entries, such as setting new goals, flags, or specialization types.
• Copy: Copy the selected entries to another target. You can choose to copy them to a different locale, language, or device target. Copying entries duplicates them: the entries remain in the current target, and are also created in the new target.
• Move: Move the selected entries to another target. You can choose to move them to a different locale, language, or device target. Moving entries removes them from the current target.
• Verify: Verify the plans of the selected entries.
• Delete: Delete the selected entries.

Search Filters

The training editor offers several ways to search through your trained utterances.

• Use the sidebar on the right to quickly filter by status, issues, goals, and specializations.
• Use the search bar at the top of the screen to search for words and phrases within your training.
  • The search matches your entered phrase exactly. If you search for the phrase weekend in december it will match all entries with that exact phrase, but will not find "weekend in January".
  • Search is case-insensitive: Samsung, samsung and SAMSUNG are all equivalent.
  • Exclude a word from a search query by prefixing it with the - character: Show me the weather in san jose -bernadino will match show me the weather in san jose and show me the weather in san francisco but not show me the weather in san bernadino.
• You can use scope prefixes in search text to refine searches, using keywords in the following table.

Valid fields for the has: scope are value, route, role, continuation, pattern, prompt, tag, enabled, disabled, and flag. This scope searches for entries that have any field of that type, rather than specific contents in that field.

Valid learning statuses are learned, not-learned, and not-compiled.

You can combine search scopes to filter more narrowly. To search for training with a goal of weather.Weather related to humidity tomorrow, you could use goal:weather.Weather humid tomorrow.

Troubleshooting

Bixby Studio informs you if your training examples have issues. However, if you have a large number of training entries, it can sometimes be difficult to quickly find a training entry causing an issue. This section discusses how to find any issues and what certain training entry statuses mean.

Training Entry Statuses

Training entries usually exist in one of three states:

• Learned: Bixby has learned, and can execute, this training entry.
• Not Learned: Bixby does not have enough information to learn from this training entry.
• Not Compiled: The NL model needs to be recompiled before this entry can be learned.

When you add training entries, they will usually start as Not Compiled. This status applies as long as Bixby hasn't generated NL models for those entries.

To generate new NL models when you add new examples or update old ones, click Compile NL Model. When the models are generated, Bixby changes the label of the learned utterances to Learned. If your training entry is similar enough to previous training examples, Bixby might label it as Learned even before you compile the model.

Changes in other training or vocabulary can cause an utterance to revert to a Not Learned or Not Compiled state. If Bixby can't learn a specific example, consider adding similar training examples, reviewing existing training entries for conflicting utterances, or updating vocabulary.

There are four other possible status values for training entries:

• Disabled: The training entry is not scanned for errors and not trained.
• Illegal Plan: The training entry has unsatisfied input constraints or unconsumed signals. This means that the intent of what you have trained does not match with the way the models have been designed to work. See Illegal Plans.
• Unverified Plan: The training entry's current plan differs in a nontrivial way from the plan at the time this entry was saved. This can happen when model changes make the training entry's plan behave differently from the last time it was saved. You can resolve this by ensuring the plan for the entry is correct and recompiling the NL model, or by running Verify All Plans.
• Incomplete: These are generally semantic, such as the lack of a goal or referencing Node IDs that do not exist.

Finding Training Entry Issues

You can quickly find issues using the search sidebar on the Training Sets Screen. Check the Issues subheading for categories of warnings and errors.

If you are submitting a capsule and one of your training entries is throwing an error, you can search for that specific error using the id: search scope.

1. Open the submission page and select your failed example.

2. Select "Capsule Interpreter Training".

3. On the log page, look for the entries causing the issue:

    --------------------------------------------------------------------
    (4) Entries in state NOT_LEARNED (ERROR: actively causing rejection)
    --------------------------------------------------------------------
    View all items in category (copy the search between the markers into the training tool)
    >>>> BEGIN COPY <<<<
    id:t-jcvrft523b1j3iwzti7wx2zq7,t-q1b5nsdjb2clwy8fkhre80ecg,t-wczwvczrezonkmp2djlaj9wcj,t-v7s9drvf7whsbfxup9n9rw2j8
    >>>> END COPY <<<<

4. Copy the failure ID(s), as directed in the training window.

5. Search for your training example with the ID by pasting the ID into the search bar. This will find the entry blocking the submission.

Illegal Plans

An illegal plan occurs when the graph that Bixby's planner has created for a specific training entry cannot be run. There are several possible reasons why a plan might be illegal:

• The models have input constraints which cannot be satisfied.
• The graph has an unused signal, which appears as a detached node.
• The plan contains an uninstantiated plan-behavior (Always) input.
• The plan contains three or more consecutive lossy action nodes chained together with no intervening nodes.

A "lossy" activity is one that has the potential of losing user-provided signals, such as a search action, fetch action, or transactions. You can often resolve the illegal plan error by adding a route, or making one of the actions a different non-lossy type such as Calculate or Resolver, though certain actions types should be used for specific situations. An illegal plan normally indicates a problem with your capsule's modeling; in general, you don't want to force Bixby to create long chains of actions in order to fulfill user requests.

Verifying Plans

You can clear "Unverified Plan" statuses by selecting the Verify All Plans command in the "more actions" menu (the dropdown menu accessed with the "•••" button at the right-hand side of the search bar). This command asserts that the plans for all complete, legal, and enabled entries are correct. It will create a new plan from the Aligned NL for each entry, based on the current state of your capsule's models, vocabulary, and other data.

The training status of each entry might change after running this command. Disabled plans will not be verified.

Statistics

The statistics screen presents a list of capsule targets for your capsule in overview form, showing you how many entries are applicable to each target and the source folders those examples are contained in. To navigate to the statistics screen, click the button with the statistics icon near the top of the training summary page.

A target has a "training budget" of 2,000 training entries. This is computed as the sum of all entries that apply to that target. The statistics screen lists the budget available for each target as well as the parent targets that are part of that budget. For instance, if you had both bixby-mobile-en-US and bixby-mobile-en-GB targets, the en language set would be shown as applying to both of them. Training entries for patterns do not count towards this budget.

In addition, this screen shows status counts for each target, and whether your capsule's natural language (NL) model needs to be recompiled. You can recompile targets individually by clicking the "Compile" button that appears when you hover over a capsule target (this button will be dimmed if compiling that target is unnecessary), and compile all targets by clicking "Compile All Targets".

Training Limitations

Number of Training Entries

A capsule can have a "budget" of up to 2,000 training entries per target. A target is a specific combination of device, language, and region:

• en: All devices, English language, all regions
• bixby-mobile-en: Mobile devices, English language, all regions
• bixby-mobile-en-US: Mobile devices, English language, US region

The 2,000-entry limit is shared across parent targets. For example, the en target is a parent of all English-speaking device targets such as the bixby-mobile-en, bixby-mobile-en-US, and bixby-other-en-US; if en had 1,000 entries in it, then bixby-mobile-en would inherit those entries, and could have up to 1,000 more. If en has 1,000 entries in it and bixby-mobile-en had 500 more, then bixby-mobile-en-US would inherit from both of its parents, and could add up to 500 more.

No Named Entities

Bixby does not have platform-level support for recognizing named entities. That means it won't automatically be able to distinguish between people, places, movie titles, etc. It is up to you to distinguish these entities in a domain-specific way, through training examples and vocabulary.

Reserved Utterances

There are several utterances that have specific meanings in particular situations. For example, "Next" while navigating a list will read the next item or page of items. When possible, create complete training examples to help distinguish between these reserved meta-commands. For example, train "Cancel my reservation" instead of just "Cancel" for English because "Cancel" clears conversation context in Bixby.

For the full list of meta-commands, see the Reserved Utterances reference.

Automatic Speech Recognition and Text-to-Speech

Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) are useful Bixby capabilities for communicating with users. However, the natural language utterances you train and the vocabulary you add are only used in their specific capsule (although they might be used over time to help improve ASR and TTS performance).