Scoring Images Using Auto Scoring Models

ROCK MAKER comes with an auto scoring infrastructure that supports multiple auto scoring models. The infrastructure comes equipped with FORMULATRIX’s in-house auto scoring model, Sherlock. With improved accuracy compared to MARCO, Sherlock raises the bar in identifying crystals with exceptional reliability.

Our auto scorers will score images from 0% - 100% (or 0 - 1). The higher the score, the higher the chances of having a crystal.

Sherlock

Trained in-house, Sherlock is a model that improves the prediction results of crystals in drops. These improvements result from process changes at FORMULATRIX, using auto scoring model networks and training with additional and diversified customer data and feedback.

Sherlock classifies images into 5 classes: crystal-else, crystal, others, precipitate, and clear. Crystal-else is a new class introduced, denoting an image with a combination of crystals and other classes. This provides users with more detailed information about the images and helps to ensure more accurate classifications. When drops are scored as crystal-else, it is considered to be just behind pure crystals in the scoring system.

The difference between the crystal and crystal else classes

Crystal Class	Score	Color Code
Crystal	0.8-1.0
Crystal-else	0.6-0.8
Others	0.4-0.6
Precipitates	0.2-0.4
Clear	0.0-0.2

Sherlock: An Improved Crystal Detection

Sherlock provides you with improved accuracy and recall in identifying crystals when compared to MARCO (find out more about the Auto Scoring Framework).

Images where Sherlock Outperformed MARCO*

*Crystals identified by Sherlock but missed by MARCO.

As we can see on the chart below, Sherlock gains more accuracy and recall than MARCO with our data set. Your mileage may vary according to your data. We rely on your feedback, so please make sure to let us know either directly via the dispute functionality described below or directly via e-mail. Sherlock is a bit biased towards finding crystals reducing the false negatives, but it may yield additional false positives.

Results Compiled from Data Provided by 6 Different Customers

*Accuracy measures the proportion of correct predictions when compared to human crystallographers.

*Crystal Recall measures the percentage of actual positive cases the model identifies correctly.

Auto Scoring Framework

Our auto scoring infrastructure includes a self-updatable virtual machine (implemented in the Virtual Box) and within it all auto scoring model scorers, including Sherlock v1 and MARCO. The virtual machine is often deployed on the ROCK IMAGER controlling PC. Our framework will ensure all newly captured images get scored with all enabled auto scoring models. It is important to note that running more than two models together comes with a time cost.

Auto Upgradable-Sherlock Framework

Switching Auto Scoring Models

Important

Only administrators can execute this task.

Considering the time it takes to score the images, it’s best to have only 2 models enabled simultaneously, preferably MARCO and a Sherlock model of your choice. The current Sherlock model available is Sherlock 1, but more models will be coming up. Once the latest model is available, you will get a notification email and can enable it from the Global Preferences submenu in the Settings menu.

To enable or disable an auto scoring model on your ROCK MAKER:

1. Open your ROCK MAKER and go to the Settings menu .

2. Click the Global Preferences submenu. Then, click the Scoring tab.

   

   Global Preferences Submenu and Scoring Tab

3. Locate the auto scoring model(s) you want to enable or disable and click the radio button .

   

   Locating Auto Scoring Models

4. Click Save on the top right bar of the Global Preferences submenu.

   

   Save Button

How Does Auto Scoring Work?

Once service is installed on your computer and enabled in ROCK MAKER, all upcoming inspections imaged using visible light path will automatically be scored by all selected auto-scoring models. Each drop will be scored into five categories: clear, precipitate, other, crystal, and crystal-else, and will have respective probabilities associated with them where 1 (or 100%) is most likely to be a crystal and 0% is least likely to be a crystal. The probabilities are then converted to a final score and labeled by the ROCK MAKER. The details of the scores can be viewed from the Auto Score tab in the Score panel, while you can see the final score in either the canvas or the drop view.

Auto Score Panel*

*The first row in the image above displays the MARCO score while the second row displays the Sherlock score.

Dispute Auto Score

Although Sherlock makes great promises, it might not be perfect for all situations. Using your feedback, we will continue to improve the models and release them as we see the improvements. If you disagree with any of the auto scores, we encourage you to send us feedback: click the Dispute button and the Dispute Auto Score dialog box will appear. Fill in the form, and your suggestions and images will be sent to us upon clicking the OK button. It will not share the metadata of the images.

Dispute Auto Score Dialog Box

The red bar and the Dispute icon on top of the drop image in the Canvas view below indicate that the auto score result has been disputed.

Disputed Auto Score