Scoring Images Using Autoscoring Models¶
ROCK MAKER comes with an auto scoring infrastructure that supports multiple auto scoring models. The infrastructure comes equipped with FORMULATRIX’s in-house auto scoring model, Sherlock. With improved accuracy compared to MARCO, Sherlock raises the bar in identifying crystals with exceptional reliability.
Our auto scorers will score images from 0% - 100% (or 0 - 1). The higher the score, the higher the chances of having a crystal.
Sherlock¶
Trained in-house, Sherlock is a model that improves the prediction results of crystals in drops. These improvements result from process changes at FORMULATRIX, using auto scoring model networks and training with additional and diversified customer data and feedback.
Sherlock classifies images into 5 classes: crystal else, crystal, others, precipitate, and clear. Crystal else is a new class introduced, denoting an image with a combination of crystals and other classes. This provides users with more detailed information about the images and helps to ensure more accurate classifications. The crystal else is the class when you have crystals along with another class. This helps the model in picking crystals that are embedded in precipitates.
The Difference between the Crystal and Crystal Else Classes*
*The crystal class has a score of 0.8-1.0 and only consists of crystals, as shown in the image on the left-hand side. The crystal else class has a score of 0.6-0.8 and has other classes alongside the crystal class. In the image on the right-hand side, the crystal else has crystals and phase separation.
Crystals > Crystal Else > Others > Precipitate > Clear
Crystal Class |
Score |
Color Code |
|---|---|---|
Crystal |
0.8-1.0 |
|
Crystal Else |
0.6-0.8 |
|
Others |
0.4-0.6 |
|
Precipitates |
0.2-0.4 |
|
Clear |
0.0-0.2 |
|
Sherlock: An Improved Crystal Detection¶
Sherlock provides you with improved accuracy and recall in identifying crystals when compared to MARCO (find out more about the Auto Scoring Framework).
Images where Sherlock Outperformed MARCO*
*Crystals identified by Sherlock but missed by MARCO.
As we can see on the chart below, Sherlock 2 gains more accuracy and recall than MARCO and Sherlock 1 with our data set. Your mileage may vary according to your data. We rely on your feedback, so please make sure to let us know either directly via the dispute functionality described below or directly via e-mail. Sherlock is a bit biased towards finding crystals reducing the false negatives, but it may yield additional false positives.
Results Compiled from Data Provided by 6 Different Customers
*Overall Accuracy measures the proportion of correct predictions when compared to human crystallographers.
*Crystal Recall measures the percentage of actual positive cases the model identifies correctly.
*Crystal Precision measures the true positive rate in the model. Higher crystal precision indicates higher true positive rates and lower false positive rates.
Auto Scoring Framework¶
Our auto scoring infrastructure includes a self-updatable virtual machine (implemented in the Virtual Box) and within it all auto scoring model scorers, including Sherlock v2, Sherlock v1, and MARCO. The virtual machine is often deployed on the ROCK IMAGER controlling PC. Our framework will ensure all newly captured images get scored with all enabled auto scoring models. It is important to note that running more than two models together comes with a time cost.
Auto Upgradable-Sherlock Framework
Switching Auto Scoring Models¶
Important
Only administrators can execute this task.
Considering the time it takes to score the images, it’s best to have only 2 models enabled simultaneously, preferably MARCO and a Sherlock model of your choice. Once the latest model is available, you will get a notification email and can enable it from Global Preferences.
To enable or disable an auto scoring model on your ROCK MAKER:
Open your ROCK MAKER and go to the Application Menu Button
on the top left bar of your ROCK MAKER window.Hover your cursor over the Setup menu and select Global Preferences.
Setup Menu and Global Preferences
In the Global Preferences window, the General tab will open by default. Then, click Scoring.
Scoring Properties in General Tab
In the Scoring properties, locate the auto scoring model(s) you want to enable or disable and click the corresponding checkbox(es).
Auto Scoring Model Fields
If everything is set, click Apply.
How Does Autoscoring Work?¶
Once service is installed on your computer and enabled in ROCK MAKER, all upcoming inspections imaged using visible light path will automatically be scored by all selected auto scoring models. Each drop will be scored into five categories: clear, precipitate, other, crystal, and crystal else, and will have respective probabilities associated with them where 1 (or 100%) is most likely to be a crystal and 0% is least likely to be a crystal. The probabilities are then converted to a final score and labeled by the ROCK MAKER. The details of the scores can be viewed from the MARCO Panel, while you can see the final score in either the canvas or the drop view.
MARCO Panel*
*The first row in the image above displays the MARCO score while the second row displays the Sherlock score.
Dispute Auto Score¶
If you disagree with any of the auto scores, click the Dispute button 
, and the Dispute Auto Score dialog box will appear. You can provide suggestions that will be sent to FORMULATRIX to improve auto scoring accuracy in the future.
Dispute Auto Score Dialog Box
The red bar and the Dispute icon on top of the drop image in the Canvas or Drop view indicate that the auto score result has been disputed.
Disputed Auto Score
Related Topics