AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Powershell website monitor script12/26/2023 ![]() # make a request to the site and return the status code. $client.Credentials = new-object ($fromAddress, $monitorPassword) $message = new-object ($fromAddress, $toAddress, $subject, $body) # send an email message via gmail function SendMessage($body) If you’re curious, I ended up getting a good deal on an Acer XFA240 and the picture looks amazing with 144hz! Let me know in the comments below if you’ve created or plan on creating a web scraping tool.$fromAddress = "One or more websites are not responding" Feel free to use my script in the demo if you want to configure your own notifications for PC part deals. Like I said previously, it is an art, a lot of the difficulty depends on how the web site is formatted and what information you are looking for. It opens up so many possibilities to create useful scripts that one might think were not possible. We wait for a sale for a good monitor to pop up and see our end result:Īs you can see web scrapping tools can be incredibly powerful for parsing useful web pages. $password = Get-Content "C:\Scripts\aespw.txt" | ConvertTo-SecureString #obtain password for gmail account from encrypted text file $sale.link | out-file C:\scripts\SaleDb.txt -Append #Save link to text file so we don't send the same deal twice If ((Get-Content C:\scripts\SaleDb.txt) -notcontains $sale.link) So we’ll use this to narrow our search: $("a") | Where Inspecting the web page further, I can see that each sale hyperlink has the class name “b5szba-0 fbxLDD”. ![]() But this would give us all hyperlinks on this page, we need to narrow down our search more to only pull the links that are for sales. So we want to run a search for all HTML objects tagged as an “a” and we’ll want to output the “href” for these instead of the “innertext” as we did in the example above. Looks like these are tagged as “a” which defines a hyperlink in HTML. When using the inspection tool in Firefox (CTL + SHIFT + C) and clicking on one of the sale links, I can see the HTML snippet for that post: Let’s go back and look at the webpage formatting and see what else we can scrape from it to get the links. I like what we have so far, but I don’t only want the post headings, but also the links for each sale. Yay, it worked! We are able to collect all the deals posted: Then we will run through each object and display the “innertext” content which is the text content of the tag we are searching for: $("h2") | ForEach-Object -MemberName innertext Now we are going to take our new variable and parse through the HTML data to look for any items tagged as “h2”. We will use the Invoke-WebRequest PowerShell cmdlet and the URL to the Reddit webpage and save it as a variable in order to collect the HTML information for parsing: $data = invoke-webrequest -uri "" Let’s try scraping all of our “h2” tags and see what we come up with. I can see that each Post is tagged in HTML with the “h2” tag : Now, let’s use the Web Developer tool in our browser to further inspect the HTML portion of these posts. We can see that each Sale is displayed with a big header which contains all the info we want to know, the item and the price: Let’s take a look at the structure of BuildAPCSales. Web Scraping is an art since many websites are structured differently, we will need to look at the way the HTML is structured and use PowerShell to parse through the HTML to gather the info we are looking for. Web Scraping with Invoke-WebRequestįirst, we need to take a look at how the website is structured. I know there is a Reddit API available that we could use to interface with, but for the purpose of demonstrating making a web scraping tool we are not going to use it. Also, because of the limited amount of stock for some of these sales, it would be extremely beneficial to know about these deals as soon as they are posted. As an avid gamer such as myself this would be extremely useful to check routinely and report back on any deals for the PC parts I’m looking for. This is an extremely useful web page as many users contribute to posting the latest deals on PC parts. We are going to scrape the BuildAPCSales subreddit. Today I’m going to show you how to build your own Web Scraping tool using PowerShell. This can be a huge time saver for instances where collecting and reporting on data from a web page can save employees or clients hundreds of hours. However, there are other tricks we can use with PowerShell to automate the collection and processing of a web pages contents. Sometimes there isn’t always an API or PowerShell cmdlet available for interfacing with a web page. Building a web scraping tool can be incredibly useful for MSPs.
0 Comments
Read More
Leave a Reply. |