Bixby Developer Center

Bixby Developer Studio includes a built-in device simulator that lets you test both natural language (NL) and Aligned NL queries on a variety of simulated Bixby-compatible devices. It lets you set the user's location and timezone, manage OAuth sessions, and apply learning models, as well as letting you launch the Debug Console to analyze the execution graph of the last-run query.

Launch the Simulator with the View > Open Simulator menu command, or by clicking the Simulator button at the top of the sidebar in the Files view.

The Simulator Window

The Simulator has its own set of application tabs running down its left side, similar to the main editor window. Like the main window, these affect the view in the Simulator's sidebar. From the top, going down:

• The Input view lets you set the capsule and target device being tested and enter queries. Depending on the device, you can also enable or disable Bixby's hands-free mode.

• The User view lets you configure a variety of data passed to the capsule, including profile information, overriding GPS and clock information, learning and execution behavior, capsule permissions, and open OAuth sessions.

• The Settings view lets you choose which Bixby device to test against. This list changes depending on what targets are available for your capsule.

Note that the sidebar in the Simulator can be collapsed, either by clicking the "X" icon in the sidebar's top right corner or by clicking the application tab icon for the currently-open view. When the sidebar is collapsed, click any of the application tab icons to open it again.

Using the Simulator

To test a query, use the Input view.

1. Check to make sure the Simulator is set to run the capsule and target you plan to test, for example: "viv.dice". You can click on the capsule and target name to change them.
2. Enter a query in the input window. The query can be one of the following:
   • A Natural Language query (available when the NL model has been compiled)
   • An Aligned NL query (always available)
   • An intent (always available)
3. Click Run to execute the query.

You can load a previous query and result screen by clicking the "˿" button. Click Reset to reset the Simulator and erase the query history stack.

If Bixby's NL model is out of date because you have edited models or code in your capsule that might affect the execution plans for your training, you can still enter an Aligned NL query or an intent. If you enter an NL query and the model needs to be compiled, the Run button will be disabled and a Compile button will appear in the Simulator window next to the capsule name:

Once a query is run, the Debug button will be enabled:

Click the Debug button to open the Debug Console with the current query preloaded.

Keyboard Shortcuts

The Device Simulator supports keyboard shortcuts for many of its commands and views. For a full list, refer to the Simulator Keyboard Shortcuts cheatsheet.

Testing Speech

The Simulator supports both text-to-speech (TTS) and automatic speech recognition (ASR) for testing purposes. You can access these with the speech buttons in the lower right-hand corner of the input window:

• To turn on text-to-speech, click the speaker icon, or type Ctrl+T (Windows/Linux) or Cmd+T (Mac). The icon is dimmed when TTS is disabled and bright when TTS is enabled. It is off by default. When TTS is on, Bixby's dialog will be spoken aloud.

• To use automatic speech recognition, click and hold on the microphone icon or press and hold the spacebar while you are speaking into your computer's microphone. Release the button when you finish speaking. Your speech is transcribed in real-time into the input window, and the query is run when you release the mouse button or spacebar.

  ASR requires access to your computer's microphone; your operating system might prompt you for this access. Also, you must have compiled an interpreter for the ASR feature to be available in the Simulator.

Note that there are limitations to ASR and TTS.

Simulating Other Devices

Use the drop-down in the Settings view to select a device to simulate. The devices available are dependent on the targets your capsule declares in its capsule.bxb file.

In addition to selecting screen resolution, other elements of the Simulator can change based on the selected device. For instance, not all devices support keyboard input, although the simulator's input window will always let you both type and speak utterances. Some devices might have more limited input controls, such as TV remotes.

Here are some differences to be aware of between the Simulator and real devices:

• Not all mobile devices support portrait and landscape in the Simulator.
• The lightbox feature on the Simulator is available only for TV and watch devices.
• Hands-free mode in the Simulator does not turn the microphone on automatically; you must still hold down the space bar or the Simulator's microphone icon while speaking.
• Many options that appear in various setting screens on a real device appear in the "User" settings tab in the Simulator.
• The Simulator has some abilities that real devices do not (for instance, clearing Selection Learning).
• The Simulator provides a Wallpaper option in the Settings that mimics a very light, light, dark, and very dark wallpaper that a user might have on their phone. These are meant to be a reference for you when testing images and icons in your capsule.

Testing Hands-Free Mode

The Simulator can be put in hands-free mode by checking the box labeled Hands-free in the input window. This works for all devices that support hands-free mode.

Simulating a TV Remote

When you are simulating a TV target, you can use the keyboard shortcuts listed in the following table to simulate a TV remote:

You can also use the mouse and click on buttons.

Testing Audio

This section discusses how to test your capsule if you use the bixby.audioPlayer library or if you use the audio-control component.

With the AudioPlayer Library

When testing a capsule that includes audio playback via the AudioPlayer library, the Simulator will show audio controls along with informational fields from the relevant AudioInfo and AudioItem concepts.

The buttons in the player are fully functional. From left to right, these buttons do the following:

• toggle random shuffle mode
• move to previous track
• toggle play/pause
• move to next track
• cycle through repeat modes: no repeat, repeat a single AudioItem, repeat the whole playlist

With the Audio Control Component

If you are testing a capsule that uses the audio-control component in a View, the Simulator will automatically include the controls for playing the audio and display any information specified in that component. These controls are fully usable in the Simulator, just as if a user was controlling the audio on a mobile device. This includes the controls for the following:

• Playing/Pausing the audio
• Scrolling to a specific time in the track with the seek bar, if enabled
• Skipping forward/backward by a specified amount of time, if enabled
• Skipping tracks forward and back, if enabled

Additionally, below the simulated screen is a bar that shows the same media controls that a user would see if they were on their lock screen or in the notifications view while swiping down fom their screen, if they were playing audio from a capsule with the audio-control component. This matches the media control keys of the client.

Simulating app-launch

The app-launch key lets result views open external applications if they are supported on the client device. As these applications are not available in the Simulator, when app-launch is executed in your capsule, the following occurs:

• The Simulator displays a message indicating that the user has exited Bixby at that point. If the application name is available (through the app-name key), it will also be displayed here.
• Data relating to the app launch (the application ID and name, payload information sent from Bixby, etc.) is recorded to the Simulator log; click the View Log button to display it on this screen.

Step History

As you enter NL and Aligned NL queries and respond to input views, the steps you take while interacting with the capsule will be recorded in the Step History along the right-hand side of the Simulator window.

• Click the Hide steps button to hide the step list, and Show steps to reveal it.
• Click Export story to use the current step list to build a Story for testing your capsule. See Stories and Assertions in Testing Capsules for more information.

User View

The User view lets you change data returned to your capsule about the user by simulating different locations and time zones. This view also lets you control learning and execution settings.

When you create a dataset, it will be placed in the user-mocks/ folder. Add a new mock dataset file, or edit an existing one, with the New File command, in either the File menu or on Bixby Studio's toolbar. You can also right-click in the file tree and select New > Other File…. Choose the Mock file type and the appropriate file subtype.

Mock data files are JSON format, either a single object or an array of objects. Unlike standard JSON, mock data files can include JavaScript-style comments. This can be useful to comment out mock data you wish to temporarily deactivate.

• Single line comments should be prefixed with //.
• Multi-line comments start with /* and end with */.

Client Function Calls

If your capsule defines a client-endpoint, the Bixby server can use it to request client-side data that isn't present on a simulated device. You can handle these cases in the simulator by preparing mock datasets. These sets are stored as JSON files, and are named after a corresponding action model in your capsule.

To add a dataset, use the New File command in Bixby Studio, selecting the Mock file type and the Client Function Call subtype. Select the action the dataset should be sent in response to in the Action Name menu that appears; the file will be given the appropriate name.

The editor will open with the dataset's JSON file and allow you to enter your mocked data. The data should be a representation of the concept the action returns. For example, the FindContact action in your capsule might output a Contact concept.

You can open, disable/enable, or delete a dataset by right-clicking on its name.

Mock Installed Apps

For client endpoints that specify companion applications with client-app-support, you can create the special Installed Apps dataset to specify a list of application IDs and versions to mock this behavior in the simulator.

The user interface for mocking installed apps is identical to the one for mocking datasets. Create it with the New File command, the Mock file type, and the Installed Apps subtype. The editor will open with a file named apps.json, allowing you to specify a list of simulated apps. This file is a list of applications, specified by their Android appId and appVersion values.

[
  // music app
  {
    "appId": "com.player.music",
    "appVersion": "49482743"
  },
  // another app
  {
    "appId": "com.another.app",
    "appVersion": "16737289"
  }
]

The list of mock installed apps is simulator-wide, not specific to individual capsules or simulated devices.

Mock Device Context

If you need to mock device context information available in $deviceContext in the Simulator, create it with the New File command, the Mock file type, and the Device Context subtype. This will create and open a device-context.json file. Add device information your capsule needs to the JSON object in the file.

For example, if you need to mock the $deviceContext.$user.is24HourFormat property, you can add this information in the device-context.json file:

{
  "$user": {
    "is24HourFormat": true
  }
}

The information available through the $deviceContext variable on a real device depends on the client. For specific client information, see the Device Context Mobile Wiki page.

Mock Device Metadata

Some user context information available in $vivContext can be mocked in the Simulator using the device-metadata.json file. Create it with the New File command, the Mock file type, and the Device Metadata subtype. This will open a template with the available settings:

{
  "storeCountry": "US",
  "deviceModel": "SM-G965N",
  "is24HourFormat": false
}

Mock App Context

If you need to mock application-specific context information in the Simulator, you can add an app-context.json file by creating it with the New File command, the Mock file type, and the App Context subtype. This will create and open a app-context.json file with a simple skeleton template:

{
  "appId": "example.app.id",
  "appVersion": "1",
  "context": {}
}

Change the appId and appVersion keys to match that of an app in the installed apps file, and add information to be returned in the context key.

Mock PDSS Data

If you need to mock PDSS data for testing your capsule's training when combined with dynamic vocabulary supplied by PDSS, you can add a pdss-data-mocks.json file by creating it with the New File command, the Mock file type, and the PDSS Data Mocks subtype. This will create and open a pdss-data-mocks.json file with a skeleton template that has fields for common PDSS data types.

To use this file, edit it to include the data that your capsule needs for testing. For instance, if you add "gas valve" to the iot.device list, then when Bixby processes the user utterance "turn on gas valve", then "gas valve" will be identified as a DeviceName.

GPS/Clock Override

Use this section to change the location and date/time reported to the capsule in the Simulator.

Under Location, enter a specific latitude and longitude or enter a city in the Location field. Recently-used cities will be remembered in the list. You can also specify "My Current Location" to have the Simulator geolocate you, and specify "No GPS Available" to test your capsule's performance on a device where GPS does not exist or is turned off.

Under Time, specify the time zone the Simulator reports to the capsule, as well as the time and date. If you have specified a custom time and date, click Clear to return to reporting the current time.

// set myDate to the real current time or the override in the Simulator
myDate = new Date($vivContext.testToday ?? Date.now())

Learning and Execution

This section has the Deterministic Mode checkbox, which affects execution, and allows you to see bootstrapped preferences.

The Deterministic Mode checkbox controls several aspects of how the capsule behaves at execution.

• Selection Learning - If enabled, selection learning is turned off, but capsules can still use selection rules. Disable this option to allow capsules to use learned selection preferences stored for the user. Use the Reset Learned Behavior button to clear selection learning data saved in the Simulator.
• Randomized Dialog - If enabled, dialog in capsules will behave as if choose (First) is specified and ignore choose (Random). If disabled, Dialog templates that use choose (Random) behaves normally. For more information, see the choose reference.
• Permission Prompts - If enabled, permissions are always prompted for in every conversation, even if permissions were previously granted.
• Date with Local Endpoints - If enabled and your local endpoint uses DateTime, the Date and Time are automatically initialized to the conversation start time.
• Stories - If enabled, allows you to export stories. Deterministic Mode must be enabled for every step in a story, or you will have to start the story over.

You can also view bootstrapped preferences here. These are a subset of learned preferences whose values are specified in the capsule to accelerate or bypass the learning process for those values.

OAuth

Current OAuth sections are displayed here, if any exist. You can clear existing sections from this tab.

Permissions

If the simulated capsule has requested permissions from the user, such as access to location information or library-specific permissions, those permissions will appear here, and will be checked if access has been granted by the user during simulation. You can clear (and set) requested permissions from this tab. Just like on real devices, permission settings are remembered across invocations in the simulator.

Settings View

Settings lets you specify the device being simulated and switch between UI themes.

When you change devices, the primary effect in the Simulator is changing the display resolution. If you choose a device that does not have a display, hands-free mode will automatically be enabled.

On mobile device targets, three UI themes are available:

• Default (Bixby's dark Navy theme)
• Dark (OneUI dark mode)
• Light (OneUI light mode)

When the Dark theme is selected, images specified with dark-theme will be used; when the Light theme is selected, images specified with light-theme will be used. When the Default theme is selected, or when an image for the currently selected theme is not specified, images will use the image specified in the parent url key.

Troubleshooting

If you're seeing issues in the Simulator, such as compiling issues or getting unexpected results, check that your capsule is properly synced in the workspace. Then try switching or re-selecting the target and recompiling the interpreter.

If you are still experiencing issues, you can file a report for the Support Team