Introduction to Breeze Runtime development
Breeze uses the Breeze Runtime to control cameras and acquire hyperspectral data, and to apply the analysis tree workflow including all models to the data.
A third party application can use the Breeze Runtime API to act on new data acquired from the cameras or files, and use the API to act on objects classified by Breeze, and receive the output data from Breeze's algorithms.
This article outlines how to use the Runtime API from a third party application.
A Breeze Runtime license is needed to use the API from a third party application.
Introduction
A third party application is software outside Breeze that is developed to obtain data from workflow executing in the Breeze Runtime. Typically the application also controls the Runtime to load workflows, take references and start and stop prediction.
It is the responsibility of you, the user of the Runtime, to write the code needed to interact with the Breeze Runtime. Prediktera, unfortunately, cannot develop or take ownership of your code.
What the Runtime provides
Specifically, the Runtime API consists of
a JSON-based command API with commands and results in JSON format. Used to control cameras, load new images and execute workflows in real-time or on existing images.
an JSON-based Event stream with data on objects such as classification and quantification data and meta data such as size, shape, position, etc. A client application can use this to take action based on the data. Learn more on object segmentation in Segmentation guide. Note: an alternative to using the event stream can be to use Actions in Breeze to have it trigger other software based on the workflow.
a TCP socket Data stream with binary data for each pixel for the output of Breeze's classification and quantification algorithms used in the workflow. This can be used for real-time visualization of the process.
The different streams are optional and be used independently.
The TCP ports used by the API is configurable in Settings.
The Runtime API does not provide raw spectral data, but this can be captured to disk on demand programmatically using the Runtime command StartCaptureOnPredict which saves data to /Data/Runtime/Measurements/{date}.
If white and dark references are taken by the Runtime, those are stored in the workspace under /Data/Runtime/References.
The Runtime API cannot be used to replicate the full functionality in Breeze, because it uses file-system access to the workspace and internal APIs.
Target audience
To develop an industrial solution using the Breeze Runtime, the team needs
knowledge how Breeze works for data analysis, and the ability to create a workflow for the target data analysis, which can be classification of objects, for example.
software development experience including some experience in JSON and TCP socket communication.
industrial application know-how on how to interact with other equipment in the system, such as nozzles for air jets to perform sorting.
to learn and use the the Breeze Runtime API,
What Prediktera provides
Prediktera provides guides like the one your reading now, tutorials, reference documentation, and sample code in Python and C# to help you learn how to program the Breeze Runtime. Technical support is also offered.
To develop and maintain a Breeze integration solution we recommend that customers invest the time to learn the product inhouse, and don’t rely on outside contractors to simplify long term maintenance of the solution.
Learning the Runtime API
To learn to use the Runtime API in your code we recommend these steps:
First you need to install Breeze on your development computer, and connect it to the camera that you want to use. For the purpose of learning the Runtime API you can use a Breeze Trial license and the built-in Simulator camera that is used in the tutorials. Learn more in Installation and getting started and Licensing.
Next, unless you already know Breeze and have analysis tree workflow in Breeze that you will use, you should work through the Nut classification tutorial in Breeze including all three parts. This creates a projects with example data that you can use with Breeze.
Finally, export the workflow to Runtime and control it programmatically from your application. To learn this step, read the rest of this page and look at the examples it links to. After that you might want to follow the Runtime Classification of nutstutorial which builds on the Nets classification project you created above.
Using Breeze Runtime Demo and Breeze Client to learn
A good way to learn the API is use the Breeze Runtime Demo application which enables API exploration at the high level without any programming. You click buttons too call different commands, and what the results on screen. Learn more in Breeze Runtime Demo and the Runtime tutorial.
Use the Breeze Client to visualize data from a running Runtime workflow. The data you see in real-time in Breeze Client is the same that you can programmatically access using the Runtime’s Data stream and Event stream. Learn more in Breeze Client.
Towards the end of the tutorial Runtime Classification of nutsyou can see how the Breeze Runtime Demo and Breeze Client applications can be used in detail.
Designing a workflow
The first step is developing an analysis tree workflow in a project in desktop Breeze, and working on it interactively until it achieves the design goals for the data analysis:
You record images to use in modelling, and then work with the models and analysis tree until you're happy. You can also try your workflow in real-time directly in Breeze in Recorder in the Continuous (Play) mode with data from your camera.
One important design goal for a workflow is performance so that the workflow at the target speed. Too many advanced models can impact performance negatively. You can evaluate performance by looking at the analysis speed measured in FPS (frames per second, where a frame in this context is a single line of pixels) in the Runtime log when you apply your workflow. For example: if you see that your workflow works at 200 FPS then you would know that if you require 0.5 cm tall pixels, this workflow can run at 1 m/s on a conveyor belt.
When the workflow runs nicely in Breeze, the next step is to try it out in Runtime:
Preparing a workflow for the Runtime
To use a workflow with the Breeze Runtime, there are a few steps that need to be performed:
If you are going to use the optional Data stream, and want pixel predictions for all output data from the workflow, you need to change the Runtime Prediction mode to Pixel prediction lines. The default setting for Prediction mode is RGB which corresponds to the same visualizations you can see in Breeze Recorder. Learn more under the next heading below.
Before any project's workflow can be used with Runtime, you must export a copy of it to the Runtime. In desktop Breeze, click Export > Export to Runtime and provide the necessary settings:
a runtime workflow name name
the segmentation level which defines the output data available in the workflow.
To learn how the selected segmentation level affects the output, go the Breeze Table tab and switch segmentation at the bottom of the screen. The same columns you see in the Table will be available in the Runtime output.
the object processing setting (previously called Type): Parallel or Sequential. Parallel identifies multiple objects in parallel. This efficient and objects are identified and send to clients as soon as they have been fully seen in real time. Sequential analyses the camera stream line by line and processes objects once no object is seen anymore, or when the project option
Setting/measure/maxlineshas been reached. For overlapping objects this means that there will be a delay until information about the objects are sent in the Event stream, because the Runtime is waiting until there is no object visible any more.
To help understand the Object processing setting, you can compare with the Segmentation settings in Breeze Recorder. Note that Crop available in Breeze Recorder, does not exist as a Runtime option.
Exporting to Runtime puts a copy of the workflow in the \Data\Runtime folder in the workspace as an xml file with the name you provided. All models and necessary files are prepared by the export to work stand-alone in the Runtime.
This copy is a disconnected snapshot of the project's workflow. If you make changes in the project or in the project option for Prediction mode, you have to re-export to Runtime. You can export many variants of a workflow like this, each with a unique name.
After the workflow has been exported, close desktop Breeze which also closes the Runtime executable.
Data stream prediction modes: RGB or Pixel prediction lines
You can configure the Runtime’s optional Data stream to return different data for a workflow using the Prediction mode option for the project. This option also controls how the camera data is visualized in Breeze Client (note that Breeze Recorder always uses RGB mode).
Find this option by first opening the project in Breeze, and then click Options and the bottom left. This will open the project-specific options (not to be confused with the global Settings). Then expand the tree of options to the left to locate Setting > runtime.
The options for Prediction mode are:
RGB - the output sent in the data stream are three channel pseudo RGB values visualizing the classification and quantification data from the workflow. This corresponds what is visualized in Breeze Recorder. This mode is recommended for visualization and results in smaller volumes of data compared to the other options. You switch variable to see using the
SetPropertycommand as described in Developers reference guide.The option Blend background controls if the pseudo RGB images should be blended into the image so that it shows the classification or quantification data on top of a pseudo RGB visualization of the raw data. You can control this dynamically using the
SetPropertycommand forVisualizationBlend.
Pixel Prediction Lines - the output in the data stream are predictions lines where each pixel gets a prediction for each Analyse Tree descriptor used in the exported workflow. These data are either classification or quantification vectors. This option should be selected if you want all prediction information for predictions for each pixel at the same time. The next heading provides guidance how you can programmatically understand what data the Pixel Prediction Line contains.
If you want to visualize Pixel Prediction line data as RGB, you have to interpret the data and decide yourself how this should be turned into three channel RGB.
Object to Pixels Pipeline - advanced use case which is rarely used, where predictions for N frames are buffered in the runtime, and postprocessed before they are streamed, frame by frame to the client.
When you change Prediction mode in Options, you need to re-export to Runtime to use the new setting.
Starting the Runtime
Before the Runtime workflow can be used by your application, you need to start the Runtime manually or from your code (it is only started automatically by Breeze Runtime Demo and Breeze Client).
Command line switches can be used to tell the Runtime the workspace folder and any workflow that should be loaded on startup as you can see in Developers reference guide. For advanced uses the Runtime can also be used as a service: Breeze Runtime as a Windows Service.
Programmatically using a Runtime workflow
Finally, connect to the Runtime from your code and issue commands and listen for data as desired:
The first step is usually loading a workflow with the LoadWorkflow command. The reply to this command contains detailed information on the workflow in the returned Message JSON object. Specifically this information is useful to understand the data returned in the Runtime’s Event stream and Data stream. Learn more in Developers reference guide for the LoadWorkflow command.
In your code, when loading a workflow, always load a workflow from the Data\Runtime folder which has been exported as described above.
Also, remember to re-export the workflow if you make changes to the project in Breeze.
Next, you probably want to issue commands to start predictions, connect to the Event stream to learn about objects identified by the workflow’s segmentation, and use the Data stream to obtain detailed data for each pixel, that can be used for visualization.
Detailed example code for this is available in the Code examples, and a step-by-step guide are found in Runtime Classification of nuts. The Python Apply Changes exampleshows a different application of the Runtime where the workflow is applied to new images on disk.
While you interact with the Runtime, you can monitor the Runtime logs to see how it responds to the commands. The logs are also where you learn about warnings, errors and other information how the Runtime performs. Learn more inBreeze log files and troubleshooting.
If you have performance issues, the RuntimeDiagnostics can be useful for performance troubleshooting.
Interpreting the binary data in the Data stream
The Data stream provides data for each pixel. The data is an encoded binary stream that need interpretation by the client code. The contents in the stream is controlled by the Data stream Prediction mode setting described above.
For Pixel Prediction Lines the binary output will contains data for all descriptors in the workflow. To learn which these are, you use the returned Message from the LoadWorkflow command which contains the definitions of descriptors. Consult the documentation for LoadWorkflow in the Developers reference guide for details.
One way to see this information using only tools that come with Breeze is to use Breeze Runtime Demo to load a workflow:
Then copy Message contents from the the Reply text (as highlighted in the screenshot) and paste it to a text editor such as Visual Studio Code where you can easily re-format the JSON and look at it:
The Python client example is fully functional Python code that relatively easily can be extended to parse the binary data in the Data stream, using the information you learned above.