How to locate and click image on screen with Pyautogui?

How to locate an image on-screen and click using Pyautogui?

Get the full guide to using Pyautogui here.

One of the important aspects of the Pyautogui module is Image recognition.

That is, we can find the coordinates of any image on the screen, provided if it is displayed on the screen(understandably). Let’s see how.

Required Tools

  • Jupyter Notebook
  • Screenpresso/ Snipping Tool

Pyautogui installation

Open the command prompt by typing “cmd” in the windows search. Then run the below lines.

pip install pyautogui

Once, pyautogui library is installed, proceed further.

Locate an image on-screen and click on it

To locate an image available on-screen and to click on it, follow the below steps.

Step 1:

Take a screenshot of the image using tools such as Snipping Tool or Screenpresso(recommended).

I want to recognize the windows icon at the bottom left corner of our screen. Hence, let’s take a screenshot of it.

locate on image and click windows icon

Step 2:

Create a folder and save the image in the created folder.

Here, I have created a folder called Image_recognition and saved the image under the name “screenshot.png“.

 

Step 3:

Open Jupyter Notebook –> Browse to above folder –> Click New and save the file in the same folder.

As you can see in the below image, both the .ipynb(Pyautogui_code.ipynb) file and the image to be recognized(screenshot.png) are available in the same folder.

locate on screen and click Pyautogui

Step 4:

Use the below code in the Jupyter notebook.

from pyautogui import *
click(center(locateOnScreen("screenshot.png")))

If the image is on display while the above code is executed, then you can find your mouse pointer clicking on the image.

pyautogui

Or else, the program will throw an error.

ImageNotFoundException: Could not locate the image.

Tip:

You are usually on your Jupyter notebook when executing the code.

Sometimes, you want to locate an image that is not displayed on the screen at the time of executing the above code. In such a case, you need to provide proper waiting time for your code.

So, the code waits for you to change screens and display the image which your code wants to recognize.

For that, use sleep function.

If I want my code to wait for 5 seconds, so that I can change screens to display the image which needs to be clicked, then…

from pyautogui import *
import time
time.sleep(5)
click(center(locateOnScreen("screenshot.png")))

I will execute the above code by using Shift+Enter in the Jupyter notebook. Then, within 5 seconds I need to display the image that needs to be recognized by our code, by switching screens.

Locate an image and move the mouse pointer to it

Alternatively, to locate an image and move your mouse pointer to the image and not click on it, then just replace click in the above code to moveTo.

from pyautogui import *
moveTo(center(locateOnScreen("screenshot.png")))

That is all you need to know about the Image recognition part. If you need the official documentation for the above information, then here you go.

If the above idea is interesting and new for you, then I urge you to read the full article about the Pyautogui library provided below.

Screen Automation using Pyautogui

2 thoughts to “How to locate and click image on screen with Pyautogui?”

Leave a Comment