• This screencast is for storeText: also the Web Scraping with Selenium IDE tutorial at https://ui.vision/.
• Dec 08, 2020 Selenium Selenium is an open-source web-based automation tool. Selenium primarily used for testing in the industry but It can also be used for web scraping. We’ll use the Chrome browser but you can try on any browser, It’s almost the same.

In this article I will show you how it iseasy to scrape a web siteusingSelenium WebDriver. I will guide you through a sample project which is written inC#and usesWebDriverin conjunction with theChromebrowser to login on thetesting pageand scrape the text from the private area of the website.

Jul 15, 2020 Selenium is a great tool for web scraping, especially when learning the basics. But, depending on your goals, it is sometimes easier to choose an already-built tool that does web scraping for you. Building your own scraper is a long and resource-costly procedure that might not be worth the time and effort.

Downloading the WebDriver

First of all we need to get the latest version ofSelenium Client & WebDriver Language Bindings and theChrome Driver. Of course, you can download WebDriver bindings for any language (Java, C#, Python, Ruby), but within the scope of this sample project I will use the C# binding only. In the same manner, you can use any browser driver, but here I will use Chrome.

After downloading the libraries and the browser driver we need to include them in our Visual Studio solution:

Creating the scraping program

In order to use the WebDriver in our program we need to add its namespaces:

Then, in the main function, we need to initialize the Chrome Driver:

This piece of code searches for thechromedriver.exefile. If this file is located in a directory different from the directory where our program is executed, then we need to specify explicitly its path in theChromeDriverconstructor.

When an instance of ChromeDriver is created, a new Chrome browser will be started. Now we can control this browser via thedrivervariable. Let’s navigate to the target URL first:

Then we can find the web page elements needed for us to login in the private area of the website:

Here we search for user name and password fields and the login button and put them into the corresponding variables. After we have found them, we can type in the user name and the password and press the login button:

At this point the new page will be loaded into the browser, and after it’s done we can scrape the text we need and save it into the file:

That’s it! At the end, I’d like to give you a bonus – saving a screenshot of the current page into a file:

The complete program listing

Get the whole project.

Conclusion

Selenium Ide Web Scraping Examples

I hope you are impressed with how easy it is to scrape web pages using the WebDriver. You can naturally press keys and click buttons as you would in working with the browser. You don’t even need to understand what kind of HTTP requests are sent and what cookies are stored; the browser does all this for you. This makes theWebDrivera wonderful tool in the hands of a web scraping specialist.