Google Git
Sign in
chromium / chromium / src / a3dafb8 / . / chrome / browser / screen_ai
tree: 59748acf69b4e75fca5a8f5dfe12a8faf85beb32 [path history] [tgz]
  1. public/
  2. BUILD.gn
  3. DEPS
  4. DIR_METADATA
  5. main_content_extraction_browsertest.cc
  6. optical_character_recognizer_browsertest.cc
  7. OWNERS
  8. pref_names.cc
  9. pref_names.h
  10. README.md
  11. screen_ai_dlc_installer.cc
  12. screen_ai_dlc_installer.h
  13. screen_ai_dlc_installer_unittest.cc
  14. screen_ai_downloader_chromeos.cc
  15. screen_ai_downloader_chromeos.h
  16. screen_ai_downloader_non_chromeos.cc
  17. screen_ai_downloader_non_chromeos.h
  18. screen_ai_install_state.cc
  19. screen_ai_install_state.h
  20. screen_ai_install_state_unittest.cc
  21. screen_ai_service_router.cc
  22. screen_ai_service_router.h
  23. screen_ai_service_router_browsertest.cc
  24. screen_ai_service_router_factory.cc
  25. screen_ai_service_router_factory.h
  26. screen_ai_service_router_unittest.cc
chrome/browser/screen_ai/README.md

Chrome Screen AI Library

Purpose

ScreenAI service provides accessibility helpers, is downloaded and initialized on demand, and stays on disk for 30 days after the last use.
The service is created per profile and will stay alive as long as the profile lives.
See services/screen_ai/README.md for more.

How to Use for OCR

Depending on your use case restrictions, choose one of the following approaches.

  1. If you are adding a new client for OCR, add a new enum value to screen_ai::mojom::OcrClientType, otherwise choose an appropriate one for it in the next steps.
  2. Join chrome-ocr-clients@ group to get notifications on major updates.
  3. Using OpticalCharacterRecognizer:CreateWithStatusCallback, create an OCR object, and wait until the callback is called. This will trigger download and startup of the service (if needed) and reports the result.
    Once the callback is called with true value, use OpticalCharacterRecognizer:PerformOCR.
    Creation of the object can only be done in the UI thread.
  4. If you cannot use the callback, create the object using OpticalCharacterRecognizer:Create and keep calling OpticalCharacterRecognizer:is_ready until it tells you that the service is ready.
    Then use OpticalCharacterRecognizer:PerformOCR as above.
    Creation of the object can only be done in the UI thread.
  5. If neither of the above work, in the browser process call screen_ai:ScreenAIServiceRouterFactory:GetForBrowserContext:GetServiceStateAsync to trigger library download and service initialization and receive the result in a callback.
    Once you know the service is ready, trigger connection to it in your process by connecting to screen_ai:mojom:ScreenAIAnnotator interface.
    Before calling any of the PerformOCR functions, call SetClientType once to set the client type.
    For an example see components/pdf/renderer/pdf_ocr_helper.cc.

How to use Main Content Extraction

If you are adding a new client for MCE, add a new enum value to screen_ai::mojom::MceClientType. In the browser process call screen_ai:ScreenAIServiceRouterFactory:GetForBrowserContext:GetServiceStateAsync to trigger library download and service initialization and receive the result in a callback.
Once you know the service is ready, trigger connection to it in your process by connecting to screen_ai:mojom:Screen2xMainContentExtractor interface.
Call SetClientType once to set the client type.
For an example see chrome/renderer/accessibility/ax_tree_distiller.cc.

Caution

ScreenAI service has a large memory footprint and should be purged from memory when it‘s not needed. To do so, it monitors connections to itself and if all clients have disconnected, it shuts down.
To help with this process, please close your connections to the service as soon as you don’t need them and reconnect again when needed.
Have support code for possible disconnecting form the service and reconnect if needed. This can happen due to a service crash, and also to ensure clients are disconnecting when they don‘t need the service, the service may in future shut down when it’s idle without considering open connections.

Bugs Component

Chromium > UI > Accessibility > MachineIntelligence (component id: 1457124)