Scoring Images Using Auto Scoring Models¶
ROCK MAKER comes with an auto scoring infrastructure that supports multiple auto scoring models. The infrastructure comes equipped with FORMULATRIX’s in-house auto scoring model, Sherlock. With improved accuracy compared to MARCO, Sherlock raises the bar in identifying crystals with exceptional reliability.
Our auto scorers will score images from 0% - 100% (or 0 - 1). The higher the score, the higher the chances of having a crystal.
Sherlock¶
Trained in-house, Sherlock is a model that improves the prediction results of crystals in drops. These improvements result from process changes at FORMULATRIX, using auto scoring model networks and training with additional and diversified customer data and feedback.
Sherlock classifies images into 5 classes: crystal else, crystal, others, precipitate, and clear. Crystal else is a new class introduced, denoting an image with a combination of crystals and other classes. This provides users with more detailed information about the images and helps to ensure more accurate classifications. The crystal else is the class when you have crystals along with another class. This helps the model in picking crystals that are embedded in precipitates.
![The Difference between the Crystal and Crystal Else Classes*](../../../_images/Crystal_and_Crystal_Else1.png)
The Difference between the Crystal and Crystal Else Classes*
*The crystal class has a score of 0.8-1.0 and only consists of crystals, as shown in the image on the left-hand side. The crystal else class has a score of 0.6-0.8 and has other classes alongside the crystal class. In the image on the right-hand side, the crystal else has crystals and phase separation.
Crystals > Crystal Else > Others > Precipitate > Clear
Crystal Class |
Score |
Color Code |
---|---|---|
Crystal |
0.8-1.0 |
|
Crystal Else |
0.6-0.8 |
|
Others |
0.4-0.6 |
|
Precipitates |
0.2-0.4 |
|
Clear |
0.0-0.2 |
Sherlock: An Improved Crystal Detection¶
Sherlock provides you with improved accuracy and recall in identifying crystals when compared to MARCO (find out more about the Auto Scoring Framework).
![Images where Sherlock Outperformed MARCO*](../../../_images/sherlock_outperformed_marco1.png)
Images where Sherlock Outperformed MARCO*
*Crystals identified by Sherlock but missed by MARCO.
As we can see on the chart below, Sherlock 2 gains more accuracy and recall than MARCO and Sherlock 1 with our data set. Your mileage may vary according to your data. We rely on your feedback, so please make sure to let us know either directly via the dispute functionality described below or directly via e-mail. Sherlock is a bit biased towards finding crystals reducing the false negatives, but it may yield additional false positives.
![Results Compiled from Data Provided by 6 Different Customers](../../../_images/results_compiled1.png)
Results Compiled from Data Provided by 6 Different Customers
*Overall Accuracy measures the proportion of correct predictions when compared to human crystallographers.
*Crystal Recall measures the percentage of actual positive cases the model identifies correctly.
*Crystal Precision measures the true positive rate in the model. Higher crystal precision indicates higher true positive rates and lower false positive rates.
Auto Scoring Framework¶
Our auto scoring infrastructure includes a self-updatable virtual machine (implemented in the Virtual Box) and within it all auto scoring model scorers, including Sherlock v2, Sherlock v1, and MARCO. The virtual machine is often deployed on the ROCK IMAGER controlling PC. Our framework will ensure all newly captured images get scored with all enabled auto scoring models. It is important to note that running more than two models together comes with a time cost.
![Auto Upgradable-Sherlock Framework](../../../_images/auto_upgradable_framework1.png)
Auto Upgradable-Sherlock Framework
Switching Auto Scoring Models¶
Important
Only administrators can execute this task.
Considering the time it takes to score the images, it’s best to have only 2 models enabled simultaneously, preferably MARCO and a Sherlock model of your choice. Once the latest model is available, you will get a notification email and can enable it from the Global Preferences submenu in the Settings menu.
To enable or disable an auto scoring model on your ROCK MAKER:
Click the Global Preferences submenu. Then, click the Scoring tab.
Global Preferences Submenu and Scoring Tab
Locate the auto scoring model(s) you want to enable or disable and click the radio button
.
Locating Auto Scoring Models
Click Save on the top right bar of the Global Preferences submenu.
Save Button
How Does Auto Scoring Work?¶
Once service is installed on your computer and enabled in ROCK MAKER, all upcoming inspections imaged using visible light path will automatically be scored by all selected auto scoring models. Each drop will be scored into five categories: clear, precipitate, other, crystal, and crystal else, and will have respective probabilities associated with them where 1 (or 100%) is most likely to be a crystal and 0% is least likely to be a crystal. The probabilities are then converted to a final score and labeled by the ROCK MAKER. The details of the scores can be viewed from the Auto Score tab in the Score panel, while you can see the final score in either the canvas or the drop view.
![Auto Score Panel*](../../../_images/auto_score_panel1.png)
Auto Score Panel*
*The first row in the image above displays the MARCO score while the second row displays the Sherlock score.
Dispute Auto Score¶
Although Sherlock makes great promises, it might not be perfect for all situations. Using your feedback, we will continue to improve the models and release them as we see the improvements. If you disagree with any of the auto scores, we encourage you to send us feedback: click the Dispute button and the
Dispute Auto Score dialog box will appear. Fill in the form, and your suggestions and images will be sent to us upon clicking the OK button. It will not share the metadata of the images.
![Dispute Auto Score Dialog Box](../../../_images/dispute_auto_score_dialog_box1.png)
Dispute Auto Score Dialog Box
The red bar and the Dispute icon on top of the drop image in the Canvas view below indicate that the auto score result has been disputed.
![Disputed Auto Score](../../../_images/disputed_auto_score1.png)
Disputed Auto Score