In one of my previous posts, I talked about using Zenscrape to extract data at scale. Their API makes it incredibly easy for you to make a lot of requests to scrape webpages from different websites. Zenscrape took care of a lot of things for us, like localization and JavaScript rendering. All this makes it possible for you to extract content from any webpage that you need.
What if you want to scrape search engine result pages? You can certainly do that with Zenscrape, but there is a more specialized service created by the same team of developers called Zenserp that will help you do it more efficiently.
Advantages of Using Zenserp
I will begin the tutorial by listing some of the advantages offered by Zenserp when it comes to extracting real-time data from search engine result pages.
Coverage of All Available Snippets
I am sure you have noticed during your own Google searches that not all search pages show the same kinds of search results. Users get a mixed set of content, depending on the search terms they used.
Besides including regular organic search results, these search engine result pages (SERPs) can contain snippets for other types of content such as news results, paid results including all kinds of Google ads, related questions, knowledge graphs, and some image or video results.
Filtering through all that content to find what you are looking for can be difficult. However, Zenserp takes care of sorting through all this data for you automatically. It also regularly updates the API to include any new type of content added to the SERPs.
Support for Multiple Platforms
It is no secret that Google dominates the search engine market. Keeping this in mind, the Zenserp API provides multiple endpoints specific to Google-based search results. For example, you can send a request for a regular SERP in Google or you can be more specific and get search results from the Images, Videos, Maps, News, or Shopping tabs.
The Zenserp API also has endpoints to give you search results from YouTube, Bing, Yandex and DuckDuckGo. They also intend to keep adding new endpoints to the API when the need arises.
These and many other features help you save valuable time to focus on your core business idea instead of spending time figuring out how to extract information from SERPs.
Scraping SERPs Using the Zenserp API
Now, it's time to learn how we can take advantage of the Zenserp API to extract valuable SERP data in real time.
You should start by creating a free account on Zenserp. Signing up for a free plan will give you a quota of 50 API requests per month. This will make it easier for you to follow this tutorial while allowing you to run your own tests with the API. Zenserp also personalizes the code snippets in the documentation by filling in your API keys once you have logged in successfully.
The API keys are also useful for executing all your requests in the Zenserp playground. You can simply select the API endpoint from the dropdown and fill out any other blank fields that you like before executing the request.
Getting SERP Data From a Regular Google Search
Let's begin with one of the most common search scenarios: people type something in the search bar and get the search results.
You can get the search results by making a request to /api/v2/search in the API with the appropriate parameters. We will use the search term "Gift Ideas" in our case. As I mentioned earlier, you can either run this request through the Zenserp playground or get back some data from the API endpoint programmatically. Both these will cost you one credit each.
The only required query parameter here is the search term, and it is denoted by q. You are also likely to get different search results depending on your location and the type of device you are using. However, those are optional parameters.
The following PHP code with the Zenserp API will help us get Google SERP data as if we were running a search query from New York. Please note that we have decoded the response as an associative array using the json_decode() function to navigate through key-value pairs.
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
$data = [
"q" => "Gift Ideas",
"location" => "New York,New York,United States",
];
curl_setopt($ch, CURLOPT_URL, "https://app.zenserp.com/api/v2/search?" . http_build_query($data));
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Content-Type: application/json",
"apikey: YOUR_API_KEY",
));
$response = curl_exec($ch);
curl_close($ch);
$json = json_decode($response, true);
?>
Zenserp also shows you what the SERP for the above request actually looks like. You can see that the JSON text response included below matches whatever we see visually.
Here is part of the JSON response that we get back while executing the above API request. We can now write some code to extract the data we need by taking advantage of this neatly structured JSON.
{
"query": {
"q": "Gift Ideas",
"location": "New York,New York,United States",
"apikey": "011951b0-177b-11ec-9490-872985f6948b",
"url": "https://www.google.com/search?q=Gift+Ideas&oq=Gift+Ideas&uule=w+CAIQICIfTmV3IFlvcmssTmV3IFlvcmssVW5pdGVkIFN0YXRlcw&hl=en&gl=us&sourceid=chrome&ie=UTF-8"
},
"organic": [
{
"position": 1,
"title": "Gift Ideas - Target",
"url": "https://www.target.com/c/gift-ideas/-/N-96d2i",
"destination": "https://www.target.com › gift-ideas",
"description": "Target/Gift Ideas (4149). Find the perfect gift. for every person, budget and occasion (even if it's just because). Who are you shopping for?",
"isAmp": false
},
{
"position": 2,
"questions": [
{
"question": "What are some thoughtful gifts?"
},
{
"question": "What should I buy for Christmas 2020?"
},
{
"question": "What is the best gift to give anyone?"
},
{
"question": "What to give a person who has everything?"
}
]
},
{
"position": 3,
"title": "Amazon.com Gift Ideas: The most popular items ordered as ...",
"url": "https://www.amazon.com/gp/most-gifted",
"destination": "https://www.amazon.com › most-gifted",
"description": "Discover birthday gifts ideas, anniversary gift ideas and wedding gifts. Save time by browsing only the best gifts, including tablets, books, music, video games ...",
"isAmp": false
},
... MORE SUCH JSON DATA
Let's say we want to know the questions that people usually ask when they are looking for "Gift Ideas". This means that we will be searching for the questions key in the organic results. We can then loop through the organic results, assuming that we don't know where the questions will be located and don't want to provide a hard-coded value.
Here's the code that will give us a list of all the "People also ask" questions:
<?php
$organic_results = $json['organic'];
foreach($organic_results as $result) {
if(isset($result['questions'])) {
$questions_asked = $result['questions'];
break;
}
}
foreach($questions_asked as $question) {
echo $question['question']."\n";
}
/*
Output
What are some thoughtful gifts?
What should I buy for Christmas 2020?
What is the best gift to give anyone?
What to give a person who has everything?
*/
?>
Using More Specific Google Search Endpoints
As I mentioned earlier, the Zenserp API also provides some more specific endpoints if you want to find just images, videos, shopping results, etc., for a particular query. This is done with the help of the tbm parameter. It can have one of the following values:
| Parameter Value | Search Type |
|---|---|
| isch | Image Search |
| vid | Video Search |
| nws | News Search |
| shop | Shopping Search |
Since we are interested in limiting our search to image results, I will set it to isch for our example. The search query this time will be "Apollo 11". Here is the PHP code to execute the Zenserp API request and get the image search data.
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
$data = [
"q" => "Apollo 11",
"tbm" => "isch",
"location" => "New York,New York,United States",
];
curl_setopt($ch, CURLOPT_URL, "https://app.zenserp.com/api/v2/search?" . http_build_query($data));
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Content-Type: application/json",
"apikey: YOUR_API_KEY",
));
$response = curl_exec($ch);
curl_close($ch);
$json = json_decode($response, true);
?>
As you can see, we didn't have to change much to turn our regular Google search query into an image search for Zenserp. All we did was add a tbm parameter and update the query parameter q.
The JSON response that you get back from the API will look similar to the one below.
{
"query": {
"q": "Apollo 11",
"tbm": "isch",
"location": "New York,New York,United States",
"apikey": "011951b0-177b-11ec-9490-872985f6948b",
"url": "https://www.google.com/search?q=Apollo+11&oq=Apollo+11&uule=w+CAIQICIfTmV3IFlvcmssTmV3IFlvcmssVW5pdGVkIFN0YXRlcw&tbm=isch&ijn=0&asearch=ichunk&async=_id%3Arg_s%2C_pms%3As%2C_fmt%3Apc&hl=en&gl=us&sourceid=chrome&ie=UTF-8"
},
"related_searches": [],
"image_results": [
{
"position": 1,
"thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRVlm-i3gu5nFHus9kJJgDsno6u1oL40g_G46kZT7Qa_WeuoAvC&s",
"sourceUrl": "https://cdn.britannica.com/92/118692-050-194A0468/Neil-Armstrong-Apollo-11-Michael-Collins-Edwin.jpg",
"title": "Apollo 11 | History, Mission, Landing, Astronauts, Pictures ...",
"link": "https://www.britannica.com/topic/Apollo-11",
"source": "www.britannica.com"
},
{
"position": 2,
"thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQyYKlOtsEOKWggUXrUkjjVChdxprUU5WJcqTUeZbUeE-UmRCpk&s",
"sourceUrl": "https://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Aldrin_Apollo_11_original.jpg/1200px-Aldrin_Apollo_11_original.jpg",
"title": "Apollo 11 - Wikipedia",
"link": "https://en.wikipedia.org/wiki/Apollo_11",
"source": "en.wikipedia.org"
},
...MORE SUCH JSON DATA
We can use the following PHP code to iterate through all the image results returned by Zenserp and easily extract the required information from the data. In our case, we just want the sourceUrl, title, and link.
<?php
$image_results = $json['image_results'];
foreach($image_results as $result) {
$source = $result['sourceUrl'];
$title = $result['title'];
$link = $result['link'];
echo '<figure>
<img src="'.$source.'">
<figcaption><a href="'.$link.'">'.$title.'</a></figcaption>
</figure>';
}
?>
You can use it to do many things, like downloading all those images or determining how many of the top 100 images are shown from your or some other website. For this example, we output all the information as part of figure tags on a webpage.
Get Bing SERP Data Using Zenserp
The Zenserp API also has endpoints to help you get data from the search results of other popular search engines like Bing, Yandex, and DuckDuckGo. The code to get back a JSON response for a query is very similar to what we have been doing so far.
The PHP code below will give you the search results for the query "Mangoes" on Bing. I should mention that Zenserp generates all these snippets for you automatically in its playground.
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
$data = [
"q" => "Mangoes",
"search_engine" => "bing.com",
];
curl_setopt($ch, CURLOPT_URL, "https://app.zenserp.com/api/v2/search?" . http_build_query($data));
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Content-Type: application/json",
"apikey: YOUR_API_KEY",
));
$response = curl_exec($ch);
curl_close($ch);
$json = json_decode($response, true);
?>
Running the above code will give you a JSON response that looks similar to the one below.
{
"query": {
"q": "Mangoes",
"search_engine": "bing.com",
"apikey": "011951b0-177b-11ec-9490-872985f6948b",
"url": "https://www.bing.com/search?q=Mangoes"
},
"navigation_tabs": [
{
"link": "https://www.bing.com/search?q=mango+fruit&filters=dtbk:\"MCFvdmVydmlldyFvdmVydmlldyE1MTEyNWI1Yi0xNTNlLWJlM2YtZTRhYS0xYTQ1MzFkMzE4N2M%3d\"+sid:\"51125b5b-153e-be3f-e4aa-1a4531d3187c\"+tphint:\"f\"&FORM=DEPNAV",
"title": "Mango",
"subtitle": "Stone Fruit",
"is_primary": true,
"is_active": false,
"image": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAYGBgYHBgcICAcKCwoLCg8ODAwODxYQERAREBYiFRkVFRkVIh4kHhweJB42KiYmKjY+NDI0PkxERExfWl98fKcBBgYGBgcGBwgIBwoLCgsKDw4MDA4PFhAREBEQFiIVGRUVGRUiHiQeHB4kHjYqJiYqNj40MjQ+TERETF9aX3x8p//CABEIACgAKAMBIgACEQEDEQH/xAAYAAEBAQEBAAAAAAAAAAAAAAAHBggFA//aAAgBAQAAAAAGtb89neMjaZEi3waWLu5UDmZjv85i1GsUwR//xAAZAQEAAwEBAAAAAAAAAAAAAAAGAgQFAAP/2gAKAgIQAxAAAAA/0bftWyrJHffk9KQVEnNf/8QAKhAAAgIBBAAGAgEFAAAAAAAAAQIDBAUABhESExQhMUJRMkEHIiNxgaH/2gAIAQEAAT8AOXs2q8OOktuuPSVXdF4PHp05A+wPQDWfGSn3GtCLG9LUVZVSoOPD6og6uWB5LcerawUFvblVLctcZCe1JG7xx8EI03IkKhvgg4BOs/LQfOOlPC3IOUnNihPw6LGV5EyFfbWPStDdxyCM27clPuqlPC6ScE9OT6OCuo8ZDAiPJHZMrAlWhKsB9cKeDr+LMXJezM9pDZRE5WezKR35PwQcHk/fOo9pYKesI3a2SsbxpJ5hu6K7Bz1/2ARrd/8AEV+hehv1c3ZnpSyCGZpB2ngV/T1K/kh0EzWNzcFqa1WnWhMFWOWdSeg/WsXvKrj4GSXb1SzKwBErMQRraGb85Vq3jUhrG4ZGdIhwvcP15/yQNY607ALpkjtwz1p17QyxtG6/auODrHbIrVtw2MaMQ0yxyslmzaJ8IIjfkoHuToRr5lVhH5yAIHI/Z9A2tjSt5W7Qdh/anYxAHnqfmAf+jW3clLIGjlflkbjnVGwWcDt7nW+942sbmMnDistA5tyTib5eF2PXprFS4uKPIpZpy2JpYOtXr8GPyOsBvSfC458YaMfpN4nc89+R8dYHdGKmkWeCynSQf1xMwWSM/TKdZPfNGjXkSjIbFtkPURDnwh7Fzz7kfoa27Qwj7tdZrTtR6MS06CN5G9+pGv/EACQRAQACAAUDBQEAAAAAAAAAAAECAwAEERIhBTFBBhMVIlGx/9oACAECAQE/AMzRWQbqZ1BCE9Iy4Gck5UxXWRk3FpsKkQV868Bi2NNm1lGE3b4l2/mPUHTssZSVtUWM2Qyd0n69uy46f1a/KZj2LpKswHxo4hU3xLGSa/qmLMvVbKyFgyjM1RXxxjM9FyXyEh3pW8C99P3GYfuDzpA5cf/EACERAAIBBAEFAQAAAAAAAAAAAAECAAMEERIFFSEiMVFx/9oACAEDAQE/ALWmajaUiGB9D8+y42WmqaYCwAThVoW9/uEGXUrL21By6jsRHqaOVyBiJlSGBIOZdchX6YKgCbEfIj5B8RP/2Q==",
"image_alt": "Mango",
"image_base64": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAYGBgYHBgcICAcKCwoLCg8ODAwODxYQERAREBYiFRkVFRkVIh4kHhweJB42KiYmKjY+NDI0PkxERExfWl98fKcBBgYGBgcGBwgIBwoLCgsKDw4MDA4PFhAREBEQFiIVGRUVGRUiHiQeHB4kHjYqJiYqNj40MjQ+TERETF9aX3x8p//CABEIACgAKAMBIgACEQEDEQH/xAAYAAEBAQEBAAAAAAAAAAAAAAAHBggFA//aAAgBAQAAAAAGtb89neMjaZEi3waWLu5UDmZjv85i1GsUwR//xAAZAQEAAwEBAAAAAAAAAAAAAAAGAgQFAAP/2gAKAgIQAxAAAAA/0bftWyrJHffk9KQVEnNf/8QAKhAAAgIBBAAGAgEFAAAAAAAAAQIDBAUABhESExQhMUJRMkEHIiNxgaH/2gAIAQEAAT8AOXs2q8OOktuuPSVXdF4PHp05A+wPQDWfGSn3GtCLG9LUVZVSoOPD6og6uWB5LcerawUFvblVLctcZCe1JG7xx8EI03IkKhvgg4BOs/LQfOOlPC3IOUnNihPw6LGV5EyFfbWPStDdxyCM27clPuqlPC6ScE9OT6OCuo8ZDAiPJHZMrAlWhKsB9cKeDr+LMXJezM9pDZRE5WezKR35PwQcHk/fOo9pYKesI3a2SsbxpJ5hu6K7Bz1/2ARrd/8AEV+hehv1c3ZnpSyCGZpB2ngV/T1K/kh0EzWNzcFqa1WnWhMFWOWdSeg/WsXvKrj4GSXb1SzKwBErMQRraGb85Vq3jUhrG4ZGdIhwvcP15/yQNY607ALpkjtwz1p17QyxtG6/auODrHbIrVtw2MaMQ0yxyslmzaJ8IIjfkoHuToRr5lVhH5yAIHI/Z9A2tjSt5W7Qdh/anYxAHnqfmAf+jW3clLIGjlflkbjnVGwWcDt7nW+942sbmMnDistA5tyTib5eF2PXprFS4uKPIpZpy2JpYOtXr8GPyOsBvSfC458YaMfpN4nc89+R8dYHdGKmkWeCynSQf1xMwWSM/TKdZPfNGjXkSjIbFtkPURDnwh7Fzz7kfoa27Qwj7tdZrTtR6MS06CN5G9+pGv/EACQRAQACAAUDBQEAAAAAAAAAAAECAwAEERIhBTFBBhMVIlGx/9oACAECAQE/AMzRWQbqZ1BCE9Iy4Gck5UxXWRk3FpsKkQV868Bi2NNm1lGE3b4l2/mPUHTssZSVtUWM2Qyd0n69uy46f1a/KZj2LpKswHxo4hU3xLGSa/qmLMvVbKyFgyjM1RXxxjM9FyXyEh3pW8C99P3GYfuDzpA5cf/EACERAAIBBAEFAQAAAAAAAAAAAAECAAMEERIFFSEiMVFx/9oACAEDAQE/ALWmajaUiGB9D8+y42WmqaYCwAThVoW9/uEGXUrL21By6jsRHqaOVyBiJlSGBIOZdchX6YKgCbEfIj5B8RP/2Q==",
"rank": 1,
"global_rank": 1
},
{
"link": "https://www.bing.com/search?q=mango+fruit+nutrition+facts&filters=dtbk:\"MCFvdmVydmlldyFrZ192M19udXRyaXRpb25fZmFjdHMhNTExMjViNWItMTUzZS1iZTNmLWU0YWEtMWE0NTMxZDMxODdj\"&FORM=DEPNAV",
"title": "Nutrition facts",
"is_primary": false,
"is_active": false,
"rank": 2,
"global_rank": 2
},
{
"link": "https://www.bing.com/search?q=mango+fruit+uses&filters=dtbk:\"MCFvdmVydmlldyFrZ192M191c2VzITUxMTI1YjViLTE1M2UtYmUzZi1lNGFhLTFhNDUzMWQzMTg3Yw%3d%3d\"&FORM=DEPNAV",
"title": "Uses",
"is_primary": false,
"is_active": false,
"rank": 3,
"global_rank": 3
},
... MORE SUCH JSON DATA
Just like the Google SERP in our first example, Zenserp provides a visual representation of the results for direct comparison.
We can now extract whatever information we like from the JSON response. In this example, I chose to create a list of organic search results as well as the related searches. Here is my code to do that.
<?php
$organic_results = $json['organic'];
$related_searches = $json['related'];
echo '<h2>Organic Search Results</h2>';
foreach($organic_results as $result) {
$link = $result['link'];
$title = $result['title'];
echo '<h3><a href="'.$link.'">'.$title.'</a></h3>';
}
echo '<h2>Related Searches</h2>';
echo '<ol>';
foreach($related_searches as $related) {
echo '<li>'.$related['text'].'</li>';
}
echo '</ol>';
?>
The following image shows the end result after using a little bit of CSS.
Final Thoughts
Zenserp is a great tool for extracting useful information from the result pages of different search engines. If you have ever tried to create a tool to scrape information from SERPs, you will start to truly appreciate the amount of time and effort it saves by giving access to all that data with a very simple API.
You no longer need to worry about creating sophisticated scrapers and then constantly updating your parser for the ever-changing markup of those search result pages to get back some useful data. The best part is that you don't have to pay anything upfront to test Zenserp. You can create a free account in a couple of minutes and get 50 API requests free each month to see if it meets your project needs.
No comments:
Post a Comment