Scoring Images Using AI-Based Auto Scoring Models¶

ROCK MAKER comes with an auto scoring infrastructure that supports multiple auto scoring models. The infrastructure comes equipped with FORMULATRIX’s in-house auto scoring model, Sherlock. With improved accuracy compared to MARCO, Sherlock raises the bar in identifying crystals with exceptional reliability.

Our auto scorers will score images from 0% - 100% (or 0 - 1). The higher the score, the higher the chances of having a crystal.

All AI-based auto scoring models in ROCK MAKER are built following Convolutional Neural Network-based image classification techniques, as detailed below.

Auto Scoring Models	Dataset Sources	Architecture Models	New Features
MARCO	https://marco.ccr.buffalo.edu/download	InceptionV3	Basic CPOX drop classification
Sherlock 1	In-house Scored FSS	EfficientNetB3	Image tiling Crystal-else classification
Sherlock 2	In-house Scored FSS	EfficientNetB3	Improved image tiling
Sherlock 3.1	In-house Scored FSS	EfficientNetB3	Improved image tiling Contaminants classification Crystal Morphology
Sherlock 4	In-house Scored FSS	EfficientNetB3	Drop detection tiling Ultraviolet image support

Sherlock¶

Trained in-house, Sherlock is a model that improves the prediction results of crystals in drops. These improvements result from process changes at FORMULATRIX, using auto scoring model networks and training with additional and diversified customer data and feedback.

Sherlock 1 and 2 classify images into 5 classes: crystal-else, crystal, others, precipitate, and clear. Crystal-else is a new class introduced, denoting an image with a combination of crystals and other classes. This provides users with more detailed information about the images and helps to ensure more accurate classifications. The crystal-else class represents a state where crystals have formed in a drop, but the presence of non-crystal elements (such as precipitate or phase separation) lowers its rank compared to the crystal class.

The Difference Between the Crystal and Crystal-Else Classes*

*The crystal class has a score of 0.8-1.0 and only consists of crystals, as shown in the image on the left-hand side. The crystal else class has a score of 0.6-0.8 and has other classes alongside the crystal class. In the image on the right-hand side, the crystal else has crystals and phase separation.

Sherlock Classes	Score	Color Code
Crystal	0.8-1.0
Crystal-Else	0.6-0.8
Others	0.4-0.6
Precipitates	0.2-0.4
Clear	0.0-0.2

Sherlock 3, the latest Sherlock version, classifies crystals based on morphology into six classes from good diffraction to poor quality crystals: 3D, rods, plates, needles, microcrystals, and spherulites. This classification helps identify high-quality crystals suitable for X-ray diffraction analysis.

Morphology Classes	Score	Image Example
3D	0.9-1
Rods	0.84-0.9
Plates	0.78-0.84
Needles	0.72-0.78
Microcrystals	0.66- 0.72
Spherulites	0.60- 0.66

Sherlock 3.1 includes the previous crystal classes—crystals, crystal-else, phase separation (previously named “others”), precipitates, clear, and contaminants—enhanced with morphology classifications ranging from 3D, rods, plates, needles, microcrystals, and spherulites.

Sherlock 4 uses the same classes as Sherlock 3.1, with some enhancements. It can identify positive or negative hits in ultraviolet images and includes drop detection that improves crystal recall and overall accuracy. Sherlock 4 also features retrained contaminants, reducing the number of false positives for visible images.

Ultraviolet Hit Class	Image Example
Positive
Negative

All Sherlock versions work best with images captured using polarized and visible lightpaths at zoom levels around 70% to avoid images being too small or cropped.

Sample Images That Work Best with Sherlock

The following sample images are considered not ideal for auto scoring accuracy. These are still supported, but may not give the most accurate scoring results.

Sample Images Considered Not Ideal for Auto Scoring Accuracy

Below is a comparison of classes among auto scoring models in ROCK MAKER.

Auto Scoring Models	Classes
MARCO	Clear, Precipitate, Other, Crystal
Sherlock 1	Clear, Precipitate, Other, Crystal, Crystal-else
Sherlock 2	Clear, Precipitate, Other, Crystal, Crystal-else
Sherlock 3.1	Clear, Precipitate, Phase-Separation, Crystal, Crystal-else, Contaminants Morphology: 3D, Needle, Spherulite, Plate, Micro, Rods
Sherlock 4	Clear, Precipitate, Other, Crystal, Crystal-else, Contaminants Morphology: 3D, Needle, Spherulite, Plate, Micro, Rods Ultraviolet Image: Negative (salt crystal), Positive (protein crystal)

Sherlock: An Improved Crystal Detection¶

Sherlock provides you with improved accuracy and recall in identifying crystals when compared to MARCO (find out more about AI-Based Auto Scoring Framework).

Images where Sherlock Outperformed MARCO*

*Crystals identified by Sherlock but missed by MARCO.

As we can see on the chart below, Sherlock 2 gains more accuracy and recall than MARCO and Sherlock 1 with our data set. With the new Sherlock 4, you also get UV Hit classification when the model detects a crystal in the drop. This further classifies crystal drops as Positive (Protein) or Negative (Salt) crystals. Provided you choose to send images for training through ROCK MAKER, our AI models will continue to improve in overall accuracy and recall, while reducing false results.

Results Compiled from Data Provided by 6 Different Customers

*Overall Accuracy measures the proportion of correct predictions when compared to human crystallographers.

*Crystal Recall measures the percentage of actual positive cases the model identifies correctly.

*Crystal Precision measures the true positive rate in the model. Higher crystal precision indicates higher true positive rates and lower false positive rates.

The following table presents detailed classification performance comparisons of the auto scoring models in ROCK MAKER.

Auto Scoring Models	Accuracy	Precision	Recall
MARCO	67.05%	78.28%	74.29%
Sherlock 1	73.63%	82.38%	82.64%
Sherlock 2	77.59%	88.86%	85.67%
Sherlock 3.1	81.07%	87.80%	89.90%
Sherlock 4	82.72%	90.30%	90.63%

AI-Based Auto Scoring Framework¶

Our auto-scoring infrastructure includes an independent RM Scorer application that supports all auto-scoring models: MARCO, Sherlock 1, Sherlock 2, Sherlock 3.1, and Sherlock 4. Our new framework is decoupled from the Virtual-Machine requirement and ensures that all newly captured images are reliably scored by all enabled auto-scoring models.

Auto Upgradable-Sherlock Framework

Switching AI-Based Auto Scoring Models¶

Important

Only administrators can execute this task.

Considering the time it takes to score the images, it is best to have up to 3 models enabled simultaneously, preferably MARCO and two Sherlock models of your choice. Once the latest model is available, you will get a notification email and can configure it from the Global Preferences submenu in the Settings menu.

To enable or disable any AI-based auto scoring models on your ROCK MAKER:

Open your ROCK MAKER and go to the Settings menu .
Click the Global Preferences submenu. Then, click the Scoring tab.

Global Preferences Submenu and Scoring Tab
Locate the auto scoring model(s) you want to enable or disable and click the toggle button .

Locating AI-Based Auto Scoring Model
Click Save on the top right bar of the Global Preferences submenu.

How Does AI-Based Auto Scoring Work?¶

Once service is installed on your computer and enabled in ROCK MAKER, all upcoming inspections imaged using visible light path will automatically be scored by all selected auto scoring models.

Previous Sherlock versions score each drop into five classes: clear, precipitate, other, crystal, and crystal else, and will have respective probabilities associated with them where 1 (or 100%) is most likely to be a crystal and 0% is least likely to be a crystal. The probabilities are then converted to a final score and labeled by the ROCK MAKER.

In addition to the above scoring system, Sherlock version 3 introduces morphology-based scoring, categorizing each drop into six classes ranging from good diffraction to poor quality crystals: 3D, rods, plates, needles, microcrystals, and spherulites.

Sherlock 4 adds UV score information to these classifications based on whether or not there is fluorescence detected in the drop. In the Canvas menu of the experiment page, the top left corner of each drop image displays a colored + or − icon to indicate the result of the ultraviolet (UV) analysis:

+ Green: The drop is classified as crystal or crystal-else and has a positive UV score.
+ Orange: The drop is classified as precipitate and has a positive UV score.
− Red: The drop has a negative UV score.

You can view the details of the scores from the Auto Score tab in the Score panel and the final score in either the canvas or the drop view.

Sherlock Version 2 Detail

Sherlock Version 3.1 Detail

Sherlock Version 4 Detail

The composite score is calculated by combining the probabilities and a weight factor. The probabilities represent the confidence level of the scoring model for the corresponding classification. A drop marked 97% as a crystal (as displayed in the image above) would mean that the model is 97% confident over the existence of a crystal in that drop.

Auto Scores Status Dialog Box¶

Accessible from the Integrations section on the main page, the Auto Scores Status dialog box displays the auto scoring model’s status: reachable or not reachable.

Accessing Auto Scores Status Dialog Box

This dialog box allows you to:

Select the log you want to download in the Log File section: Auto Score, Dispute Score, or Retrain Images.
Save the scoring log(s) as TXT files on your local drive using the DOWNLOAD button.
Update the information in the dialog box upon changes using the REFRESH button.
Exit the dialog box using the CLOSE button at the bottom right or on the top right corner.

Dispute AI-Based Auto Score¶

Although Sherlock makes great promises, it might not be perfect for all situations. Using your feedback, we will continue to improve the models and release them as we see the improvements. If you disagree with any of the auto scores, we encourage you to send us feedback: click the Dispute button and the Dispute Auto Score dialog box will appear. Fill in the form, and your suggestions and images will be sent to us upon clicking the OK button. It will not share the metadata of the images.

Dispute Auto Score Dialog Box

The Dispute icon on the top right corner of the drop image indicates that the auto score result has been disputed.

Disputed Auto Score

Related Topics: