vxUI070xSUq1R5qoKnETxcDi+rmjMk6ernMZ657hMus=

Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.

Home

+fQFnIt2m7zOhUxAXCo9PeQRtnla4zeiD+GrBd1qars=

Start learning about web scraping, web crawling, data collection, and popular tools to start developing your own scraper.

Introduction

rDjJgGtIQNdrL2Gg8R5+1jG8IKD/sDM36V9vHsybE4E=

Learn about browser DevTools, a valuable tool in the world of web scraping , and how you can use them to collect data from a website.

Browser DevTools

odqIipF+n/S+p7DExJ1Kn8JB4t/lH9ttsNXQKTknzS8=

Learn how to use browser DevTools, CSS selectors, and JavaScript via the DevTools console to collect data from a website.

Using DevTools

lC8uKRMnVcCNPYlR7coR9PRW+XkDzgpaTxf7gmN7xR8=

Continue learning how to collect data from a website using browser DevTools, CSS selectors, and JavaScript via the DevTools console.

DevTools (continued)

3hVW+b9Pv0mYeIyc++t0aubec9DrbG51BEFDeHoj/ZQ=

Set up your computer to be able to code scrapers with Node.js and JavaScript. Download Node.js and NPM and run a Hello World script.

Computer preparation

4/0ezVxOMA2av2S/aBpMykBhOjqkPl8TEw2CIngtOls=

Create a new project with NPM and Node.js. Install necessary libraries, and test that everything works before starting the next modules.

Project Setup

j2oofsUUa/nZhi7DngMobi46aP4VIu2JijIbLAM+658=

Learn how to use JavaScript and Node.js to create a web scraper, plus take advantage of the cheerio and got-scraping libraries to make your job easier.

Node.js scraper

+xHKooOfQp6h5N062xspcl9UqZukglKkmwmsrI1y+cs=

Continue learning how to create a web scraper with Node.js and Cheerio. Learn how to parse HTML and print the results of the data your scraper has collected.

Node.js (continued)

rVDYkR4OmnJ/+qDyfqsELl4VLNsFDZFcKBHGDuxMfjg=

Learn how to save the results of your scraper's collected data to a CSV file that can be opened in Excel, Google Sheets, or any other spreadsheets program.

Saving results to CSV

No85DDiwoeN/0dN87IXoyU9N1F/D8GGOOH2BW8nr7Po=

Learn about HTML, CSS, and JavaScript, the basic building blocks of a website, and how to use them in web scraping and data collection.

Basics of data collection

rw5y4x08/nzrZb9L3Gl3DGJtIxqfuUGZfkGzKcRgMQw=

Review our e-commerce website scraper and refresh our memory about its code and the programming techniques we used to collect and save the data.

Recap of data collection

kLK4R2uKdG9G0EVDDF4P81fArGNT6veIUt7YTLcIvt8=

Learn what a link looks like in HTML and how to find and collect their URLs when web scraping using both DevTools and Node.js.

Finding links

Fj3xG2tULe+2E2tGIKo8qjodQtaJnax0xJmt0gQLUhg=

When you collect links from a web page, you often end up with a lot of irrelevant URLs. Learn how to filter the links to only keep the ones you need.

Filtering links

8kV474wQUPOPzNyiTD9LTOVrGisCNVm4iwDRnvcBI8A=

Learn about absolute and relative URLs used on web pages and how to work with them when parsing HTML with Cheerio in your scraper.

Relative URLs

2QEzd42TL70IHGxfWFSRRhh/M5LvzsMoeCw2sK/BtZ0=

Learn how to crawl the web using Node.js, Cheerio and an HTTP client. Collect URLs from pages and use them to visit more websites.

First crawl

zWNO5YBQIcooXu0T7upYtJI5Qlw86jZ6RP1OoIEbfQs=

Learn how to add data collection logic to your crawler, which will allow you to extract data from all the websites you crawled.

Scraping the Data

IyhH8y8XSIcUv6K0IEuSeDQZWcJxsUyrRgh41JQGtXY=

Learn how to build scrapers quicker and get better and more robust results by using the Apify SDK, an open-source library for scraping in Node.js.

Professional scraping

B1s1QyDy1GwcSV96v2jsanda0VpRyZlWrkt7f1EnMXE=

Learn how to scrape the web with a headless browser using only a few lines of code. Chrome, Firefox, Safari, Edge - all are supported.

Headless browsers

7hO8edbc8rY7IH6mpFyHCsvnz1y4neGi1j/XH9wLpAk=

Learn how to process the data you scraped using the Apify SDK and how to convert JSON to Excel files using the Apify API.

Processing data

c5wHa7kdwzwXzUtQhO5SD5TP2MO82QBJQ/27PHYyyj0=

Learn how to crawl the web with your scraper. How to extract links and URLs from web pages and how to manage the collected links to crawl the web.

Basics of crawling

+pfHqlIp/ffknXspPrveRjV+iky1H95VdmD+meR7uhY=

Understand the standards and best practices that we here at Apify abide by to write readable, scalable, and maintainable code.

Best practices

2l3JEYI8r6NIzkPYAdSXBt97iMVdKFBR3lQdpo8oXeQ=

Learn how to develop web scrapers with this comprehensive and practical course. Go from beginner to expert, all in one place.

Web scraping for beginners

5CWlAvqNeuAMzYqRnanMsdwYvGhM4XkDVnhiJZfsCtc=

Learn how to effectively locate a website's API endpoints, and learn how to use them to get the data you want faster and more reliably.

Locating API endpoints

hnMAo1VepbuvRd8yHr0DO5sO6/lKwIcDmZ8MY2O+mE4=

Learn about how some APIs require certain cookies, headers, and/or tokens to be present in a request in order for data to be received.

Cookies, headers, and tokens

d2pRNL2WITcm23+3LGP5JC9DkgexidGpU1MvLHl2wUQ=

Learn the benefits and drawbacks of API scraping, how to locate an API, how to utilize its features, and how to work around common roadblocks.

General API scraping

XqAjhm71yzx67juNOIj4sFtNNRM3+2txW7SYb4ZGVw4=

Learn how to modify the variables of a JSON format GraphQL query to use the API without needing to write any GraphQL language or create custom queries.

Modifying variables

P73NR6evGor0zWCY2SGNF70hyLpurL23IwFNsDMOvFQ=

Understand what introspection is, and how it can help you understand a GraphQL API to take advantage the features it has to offer before writing any code.

Introspection

3kaZI5NyPMWItULpgUcOEQzZvee/2O1dECtcKG5LyP0=

Learn how to write custom GraphQL queries, how to pass input values into GraphQL requests as variables, and how to retrieve and output the data from a scraper.

Custom queries

Ru4eyeqe5ndURwbU55jbO486eHyizwusO8T1s4WCWKA=

Dig into the topic of scraping APIs which use the latest and greatest API technology - GraphQL. GraphQL APIs are very different from regular REST APIs.

GraphQL scraping

r0Hudvp5agpAGWVSxRNNiGp9gX2+vLjnYNJ78pfGCMc=

Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements.

API scraping

QjSRX96KtvI09d9rW6OcPDSrxB7aPjHauICsglleK9M=

Learn about "hidden" data found within the JavaScript of certain pages, which can increase the scraper reliability and improve your development experience.

JavaScript objects within HTML

QyMd4xvbXcMqmIK3wJgFWKztpRy/M7o5LotkNPdoffM=

Learn about dynamic pages and dynamic content. How can we find out if a page is dynamic? How do we programmatically scrape dynamic content?

JS rendering and dynamic pages

8mzLcmZiICGB96pJQarQKEIkZ2ItgK+vci5v0WyGhgs=

Dynamic pages and SPAs

fwBr5kgR/l3yx0frRr9rOlan77qNyv/tT3cU1zfeUJw=

Learn how to use and automagically rotate proxies in your scrapers by using the Apify SDK, and a bit about how to easily obtain pools of proxies.

Using proxies

qWMzLWP7OUdKU/bchF1CsLiHugD4n6JCd+la4qPSYzA=

Learn all about proxies, how they work, and how they can be leveraged in a scraper to avoid blocking and other anti-scraping tactics.

Proxies

8gVIksYWxdUHE2uCfzY9OiH2ur++wu3hPiu3zv0nH7E=

Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues.

Anti-scraping protections

q99e+KuQvAlnhCjIHcTJ+9pQ/ReWpCYCtyWenkDGNyA=

What is an actor? How do we create them? Learn the basics of what actors are, how they work, and try out an actor yourself right on the Apify platform!

Actors

wYYrW+5kr3CvtiwGAQl96G1NaJo4/yTSxuOkaLuJ14Q=

Build and run your very first actor right on the Apify platform from a template. This lesson provides a hands-on experience with building and running an actor.

Creating actors

g2iW/PN7HevOQAttw5Ym7dPDUf8Nx7q8PcI2FoP+1Gs=

Create an actor from scratch which takes an input, processes that input, then outputs a result that can be used elsewhere.

Inputs & outputs

evJcqSDX/zpJ5+SVHgXgE1KbOBDF2A+xopwngHYlF04=

Learn how to use the Apify API to programmatically call your actors, retrieve data stored on the platform, view actor logs, and more!

Apify API

j6z08hnI4rYQPSM0M4dmG9EtHWLCtAObA5qC8TsnXlE=

Interact with the Apify API in your code by using the apify-client package, which is available for both JavaScript and Python.

Apify client

odHz6nngHcid/qM6lGPe5aXOkND71X+m6LKfKpilmwo=

Get started with the Apify platform by creating an account and learning about the Apify Console, which is where all Apify actors are born!

Getting started

J7p4Zx6/syfIla/aFtJpY1U3jvskvji+qFnxeE0+YJ8=

Learn to accept input into your actor, do something with it, then return output. Actors can be written in any language, so this concept is language agnostic.

TZzlYEqcpazwzpHIZm0G1W0w+IGwFVCliwggPs3JCgU=

Learn how to generate a user interface on the platform for your actor's input with a single file - the INPUT_SCHEMA.json file.

Input schema

HJuw8pTeTaQ/qCYgzvUICM1m3Q/fK1FSo/Si4flkv68=

Understand how to write a Dockerfile (Docker image blueprint) for your project so that it can be run within a Docker container on the Apify platform.

Dockerfile

9TRnvkhUP0aVdI5S+Nsny4bZDVaEe8ncP9hvjy8xAwg=

Push local code to the platform, or create a new actor on the console and integrate it with a Git repo to optionally automatically rebuild any new changes.

Deploying

ObhXg497A2RID956J3MJ8zh7L0lC5+82UMESBcSvWVU=

In this course learn how to take an existing project of yours and deploy it to the Apify platform as an actor in just a few minutes!

Deploying your code

IhKukePaWbCKaE7m89rUISzS2gYRUHykCcPL187WBEQ=

Learn all about the Apify platform, all of the tools it offers, and how it can improve your overall developement experience.

Apify platform

jkV0Q/4SsgAsymvWNOSWM0vcoii2ZEbpUB4+bM9ZdE8=

Learn about the Apify SDK, as well as some semi-advanced topics which will be important throughout the next lessons in this section of the course.

I - Apify SDK introduction

Xr+wGt+O4o3OhrDXjDCYOjpSyE70MfiOiryFiws58EE=

Learn more advanced details about actors, how they work, and the default configurations they can take. Also learn how to integrate your actor with webhooks.

II - Webhooks & advanced actor overview

ZHoV7AnxWmlxy0SulzGm9/q54Pd7Azk06eo0C3PpMug=

Learn how to manage your actor's source code more efficiently by integrating it with a Github repository. This is the standard on the Apify platform.

III - Managing source code

kAMSt79IIW1cJxvm4CzJOA6Kbs0jEyj8f4VAhM9dJc4=

Understand how to save the configurations for actors with actor tasks. Also, learn about storage and the different types Apify offers.

IV - Tasks & storage

j/oIXB8+ycEPbHnOKvFxQJp/IxVR0q90deI6DlCcjx4=

Gain an in-depth understanding of the two main ways of programmatically interacting with the Apify platform - throught the API, and through a client.

V - Apify API & client

URW7ORIM+vgSHFVPF2bfwKNdl9XKRDfMD2VpPgm5we0=

Learn about what actor migrations are and how to handle them properly so that state is not lost and it can safely be resurrected.

VI - Migrations & maintaining state

BP2PwubgoPxzygmD9bJ4MNwqfrB7FMO6zwYZ9QVlKqA=

Build a pro Apify actor. With all of the code provided right within the lesson, you can learn advanced scraping techniques by coding along!

I - Actor building

obBCGaxiiv9vjqYvF6K24Cv6NrmJpij5KpTgRpl88jo=

Learn how to integrate webhooks into your actors. Webhooks are a super powerful tool, and can be used to do almost anything!

II - Integrating webooks

qfkOtVykwz7G7ZqWRt7LdvY+bhbazp5QCe4n6Vxhw1I=

View in-depth answers for all three of the quiz questions that were provided in the corresponding lesson about managing source code.

III - Managing source

aqquTuiixMmRxSIDqhHxYiQQAzvriCZ/im+sWimtsbg=

Follow along with step-by-step instructions on how to complete the task outlined in the previous lesson. Use different storage types, and create a task.

IV - Using storage & creating tasks

ree9uxUVfHXNQ289TmnL5uAycWT+0hpUHCwmi3IataM=

Learn how to interact with the Apify API directly through the well-documented RESTful routes, or by using the proprietary Apify JavaScript client.

V - Using the Apify API & JavaScript client

nTCvQekQewerdpXTuRFiCknyTy70nEz8RpN13xkmv6c=

Get real-world experience of maintaining a stateful object stored in memory, which will be persisted through migrations and even graceful aborts.

VI - Handling migrations

GjELEenb1AK47k1MfEQySAr0mPKAGMeLocxYUbYPPHQ=

View all of the solutions for all of the activities and tasks of this course. Please try to complete each task on your own before reading the solution!

Solutions

JgEO6yURZG+ojws+GxSgnAFhqIYiKh2tEgkcj/MKwQA=

After learning the basics of actors, learn to develop pro-level scrapers on the Apify platform with this advanced course.

Expert scraping with Apify

TGAtPSzyuQgSoOe4lcWgOK3/CfvT/iqrGqhFReVL5To=

Understand what the Browser object is in Puppeteer/Playwright, how to create one, and a bit about how to interact with one.

I - Launching a browser

yOsoTm5AB34JM+TwReFhsJWqqrhjyjXRnYIe5UqKiR4=

Learn how to programmatically do actions on a page such as clicking, typing, and pressing keys. Also, discover a common roadblock that comes up when automating.

Interacting with a page

xDdmwhUrrXNAWTb1WnQl6fbEMTE5XYBN7hspcZhjvjg=

Learn the importance of waiting for content and events before running interaction/collection code, as well as the best practices for doing so.

Waiting for content & events

yYnvr3cKzntqAt04oNJqFrM+f7CAEgsK+lZVvrFT6IU=

Understand that the Page object has many different methods to offer, and learn how to use two of them to capture a page's title and take a screenshot.

Page methods

AigblrteDWz3GsLLR+0TMIPNjcD06eCx8Xb+lGdlCq0=

Learn how to create and open a Page with a Browser, and how to use it to visit and programmatically interact with a website.

II - Opening & controlling a page

RsZQNaBENRZoRDT6i4AjOQDLv+szIFGlzP+Lv13M2Rg=

Learn in-depth how to use two of the most popular Node.js libraries for controlling a headless browser - Puppeteer and Playwright.