How to locate an image on-screen and click using Pyautogui?
One of the important aspects of the Pyautogui module is Image recognition.
That is, we can find the coordinates of any image on the screen, provided if it is displayed on the screen(understandably). Let’s see how.
- Jupyter Notebook
- Screenpresso/ Snipping Tool
Open the command prompt by typing “cmd” in the windows search. Then run the below lines.
pip install pyautogui
Once, pyautogui library is installed, proceed further.
Locate an image on-screen and click on it
To locate an image available on-screen and to click on it, follow the below steps.
Take a screenshot of the image using tools such as Snipping Tool or Screenpresso(recommended).
I want to recognize the windows icon at the bottom left corner of our screen. Hence, let’s take a screenshot of it.
Create a folder and save the image in the created folder.
Here, I have created a folder called Image_recognition and saved the image under the name “screenshot.png“.
Open Jupyter Notebook –> Browse to above folder –> Click New and save the file in the same folder.
As you can see in the below image, both the .ipynb(Pyautogui_code.ipynb) file and the image to be recognized(screenshot.png) are available in the same folder.
Use the below code in the Jupyter notebook.
from pyautogui import * click(center(locateOnScreen("screenshot.png")))
If the image is on display while the above code is executed, then you can find your mouse pointer clicking on the image.
Or else, the program will throw an error.
ImageNotFoundException: Could not locate the image.
You are usually on your Jupyter notebook when executing the code.
Sometimes, you want to locate an image that is not displayed on the screen at the time of executing the above code. In such a case, you need to provide proper waiting time for your code.
So, the code waits for you to change screens and display the image which your code wants to recognize.
For that, use sleep function.
If I want my code to wait for 5 seconds, so that I can change screens to display the image which needs to be clicked, then…
from pyautogui import * import time time.sleep(5) click(center(locateOnScreen("screenshot.png")))
I will execute the above code by using Shift+Enter in the Jupyter notebook. Then, within 5 seconds I need to display the image that needs to be recognized by our code, by switching screens.
Locate an image and move the mouse pointer to it
Alternatively, to locate an image and move your mouse pointer to the image and not click on it, then just replace click in the above code to moveTo.
from pyautogui import * moveTo(center(locateOnScreen("screenshot.png")))
That is all you need to know about the Image recognition part. If you need the official documentation for the above information, then here you go.
If the above idea is interesting and new for you, then I urge you to read the full article about the Pyautogui library provided below.