Segment Anything for Microscopy

  1. Anwai Archit
  2. Luca Freckmann
  3. Sushmita Nair
  4. Nabeel Khalid
  5. Paul Hilt
  6. Vikas Rajashekar
  7. Marei Freitag
  8. Carolin Teuber
  9. Melanie Spitzner
  10. Constanza Tapia Contreras
  11. Genevieve Buckley
  12. Sebastian von Haaren
  13. Sagnik Gupta
  14. Marian Grade
  15. Matthias Wirth
  16. Günter Schneider
  17. Andreas Dengel
  18. Sheraz Ahmed
  19. Constantin Pape

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Accurate segmentation of objects in microscopy images remains a bottleneck for many researchers despite the number of tools developed for this purpose. Here, we present Segment Anything for Microscopy (μSAM), a tool for segmentation and tracking in multidimensional microscopy data. It is based on Segment Anything, a vision foundation model for image segmentation. We extend it by fine-tuning generalist models for light and electron microscopy that clearly improve segmentation quality for a wide range of imaging conditions. We also implement interactive and automatic segmentation in a napari plugin that can speed up diverse segmentation tasks and provides a unified solution for microscopy annotation across different microscopy modalities. Our work constitutes the application of vision foundation models in microscopy, laying the groundwork for solving image analysis tasks in this domain with a small set of powerful deep learning models.

Article activity feed

  1. Version published to 10.1038/s41592-024-02580-4
  3. Arcadia Science

    Thanks for the positive feedback! And especially good to hear that the installation and segmentation worked well for you! For tracking: overall this is still the most experimental workflow, and I agree that we need to document it better, especially how to adapt the parameters for fast moving cells. (See more on future plans below). It would also be helpful for us to see the example where you struggled with it. If you can share this data publicly than it would be best if you can open a topic on https://forum.image.sc/ with the tag 'micro-sam' and we could discuss it there. If you can't share it publicly feel free to reach out via mail (constantin.pape@informatik.uni-goettingen.de). Regarding the overall plans: I want to extend the user tutorials and usability for tracking (and other functionality) soon. However, we are currently working on fully implementing napari plugins for micro-sam, which will take a couple more weeks / few months. After this I will update and extend the tutorials.

  4. Arcadia Science

    The ability to segment cells from images enables unbiased and high-throughput phenotyping for biological studies. I was excited to come across this preprint on Segment Anything for Microscopy, which builds upon the promising Segment Anything Model developed by a team at Facebook. Your model particularly excels in expanding the utility of the Segment Anything Model to cellular images and their associated biological features.

    Having tried Segment Anything for Microscopy, I'd like to offer some comments and questions:

    1. The software was straightforward to install, and I commend the clarity of the instructions provided on the Github repository. I was up and running within 30 minutes, which is excellent.

    2. I was impressed with the automatic segmentation feature in 2D mode, which successfully segmented an algal cell in my test image without additional training.

    3. For tracking mode, however, I encountered some difficulties when testing an image stack of an algal cell swimming in a microchamber. I allotted 30 minutes for this test, expecting this to be sufficient based on my experience with similar tools. The algorithm seemed to struggle with tracking the cell when it moved significantly between frames. Could you include a section in the documentation that describes how to adjust tracking parameters to account for different velocities of cell movement? This would be a useful addition for users like me who deal with fast-moving cells.

    4. I couldn't find any documentation regarding batch processing of images. Is this feature available, or do you have plans to implement it?

    Thank you for your contribution to the field. I look forward to future updates and improvements to Segment Anything for Microscopy. I used ChatGPT to revise an initial draft of my comments for clarity and accurate word choice, but I verify that all of the text accurately reflects my review of the software.

    Galo Garcia, Scientist at Arcadia Science

  5. Version published to 10.1101/2023.08.21.554208v1 on bioRxiv