Using Django with Scrapy to index web content

Using Django with Scrapy to index web content

A few months ago, I found myself with some spare time, I decided to use it to take a dive into Django, a python based web framework. I wasn’t really sure what I wanted to do at first, so I followed a few tutorials from around the web and created some simple example sites, I noticed right away, having some prior python experience, that Django was pretty easy to use. I started to brainstorm things that I could do with Django in line with creating an actual functional website. After some consideration, ammochamp.com was born, an ammunition price indexer. The idea was to gather ammunition price data from around the web, and display the results to the end-user in a fashion that made it easy to find the cheapest pricing available. After some research, I settled on using scrapy to gather the data from the web and store the data in a postgresql database, and django to display the information that has been scraped. This was my first time using Scrapy to gather data from the web, and it ended up being quite a joy to work with. I started by following the basic Scrapy tutorial from their website. While I did have to do additional research to complete this project, answers to almost every question I had were found at scrapy.org. In this next piece of code, I’m going to show you an example of the scrapy script I created: scraper.py Python from __future__ import division from scraper.items import AmmoDeals from scrapy.selector import HtmlXPathSelector from scrapy.contrib.loader import XPathItemLoader from scrapy.contrib.loader.processor import Join, MapCompose from scrapy import Selector import... read more
A Simple HTML Contact Form Using PHP/Ajax/JS

A Simple HTML Contact Form Using PHP/Ajax/JS

As a web developer, I’m constantly using contact forms. If you talk to a handful of developers, they will all have most likely written several throughout their career. I usually use straight PHP to handle these types of forms, however, recently, I decided it would be nice to have something that was a little more versatile, incorporating some newer programming techniques and languages. We will be using the following javascript/jQuery components: jQuery jquery-migrate – To handle compatibility with code for older versions Twitter Bootstrap – Handle theming and visual elements jquery.validate – Validate form input jquery.easing – Apply animations to objects wow.js – Handles some visual elements of the form contact.js – All our code Let’s take a look at our index.html: index.html XHTML <!DOCTYPE html> <html> <head> <!--[if lt IE 9]> <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script><![endif]--> <title>Simple HTML Contact Form using Ajax/PHP/JS</title> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <link href='http://fonts.googleapis.com/css?family=Open+Sans:300,700,800,400,600' rel='stylesheet' type='text/css'> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link rel="stylesheet" href="css/animate.css"> <link rel="stylesheet" href="css/font-awesome/css/font-awesome.min.css"> <link rel="stylesheet" href="css/stylesheet.css"> </head> <body> <div id="preloader"> <div id="status">&nbsp;</div> <noscript>JavaScript is off. Please enable to view full site.</noscript> </div> <div class="wrapper"> <div class="container"> <section id="info"> <div class="topText"> <div class="content"> <h1>Simple HTML Contact Form</h1> <div class="short-tag"> using Ajax/PHP/JS </div> </div> </div> </section> <div class="contact-form-holder"> <div class="message-box"></div> <form id="contact-form" class="contact-form" method="post" > <div class="col-xs-12 col-sm-6 no-margin"> <div class="control-group"> <div class="controls"> <input id="cname" data-placeholder="Your Name*" name="name" size="25" class="le-input required col-xs-12" /> </div> </div> </div> <div class="col-xs-12 col-sm-6 no-margin"> <div class="control-group"> <div class="controls"> <input id="cemail" data-placeholder="Your Email*" name="email" class="le-input required email col-xs-12" /> </div> </div> </div> <div class="col-xs-12 no-margin"> <div class="control-group"> <div class="controls"> <input id="subject" data-placeholder="Subject" name="subject" size="25" class="le-input col-xs-12" /> </div> </div> </div> <div... read more
How to Install vCenter Server Appliance 6

How to Install vCenter Server Appliance 6

I have always used, and loved the vSphere Client for ESXi host management. Unfortunately, in recent years, vmWare, in their infinite wisdom, have opted to disable new features of ESXi hosts when using the vSphere Client, forcing us to use the purely web-based vCenter Server. The server comes in two flavors The “plain” vCenter Server, it is installed on top of an existing installation of Linux or Windows The vCenter Server Appliance, which runs on SUSE Linux and is installed by automatic deployment via a web browser and the VMWare Client Integration Plugin. For the purposes of this article, we will be discussing implementing vCenter Server using the vCenter Server Appliance. First, mount the iso using your preferred method, if you are using Windows Server 2012 or Windows 10, mounting should be integrated into the OS, if using an older OS, you can try magicIso. Open the drive where you have mounted the ISO and navigate into the vcsa directory, and install the VMWare Client Integration Plugin. When installation is complete, navigate up one directory to the root directory of the ISO, and open vcsa-setup.html. This will launch a web browser. it’s important to note that if you are using a server based environment, such as Windows Server 2012, depending on your local security policies you may need to add an exception to allow the browser to open the vmware-csd:// link, this is outside of the scope of this article, but should be extremely easy for an admin. Grant approval for launching of the VMWare Client Integration Plugin by clicking allow on the popup. You should now see the... read more
How to Enable the SSD option on a drive that is not detected as SSD in ESXi

How to Enable the SSD option on a drive that is not detected as SSD in ESXi

I recently had a drive fail in my RAID array on one of my home servers, a HP Proliant DL360 G6. The failed drive was a 300GB SAS drive. I did not have a spare available, and needed the server back up ASAP. I had a 500GB SSD that I was not using at the time, and I decided to use that to substitute until I could get a proper replacement shipped. Getting the replacement drive installed was easy enough. I didn’t have any caddies, so I removed the 4 screws that held the “imminent failure” SAS drive in place, and replaced the drive with the new 500GB SSD. Since the 500GB drive was not going to be part of an array, I just had to create a couple of single drive arrays in the SAS controller’s interface. Once I booted up the host with the array’s created, I did have to do some finagling with esxi via the shell to get it to recognize the drives. The problem for me was that both drives had been formatted before, one was part of a RAID array, and the SSD I had just formatted to EXT4 before installation. The fix for these couple issues is outside of the scope of this article, but I will be writing another post related to this topic soon. Once I had the server fully booted up, I headed over to the vSphere Client and formatted the drive as VMFS. I then noticed that the drive was being detected as NON-SSD. The following commands are from vmware.com, and they will “force” a drive to be... read more
Side-load Kodi on Sony XBR-55X850C

Side-load Kodi on Sony XBR-55X850C

I recently became the owner of a new Sony XBR-55X850C. I have to say, the picture quality in 4K is absolutely awesome, IF you can find content. Before purchasing the TV, I did some research and found that there were quite a few problems with app availability, and several customers felt that the firmware would barely qualify as a beta. I brought my concerns to Best Buy, and the employees insured me that there has been an update this past week (between 6-8-15 and 6-14-15) that fixed most of the issues. So I grabbed one and brought it home. First impressions were awesome, the TV is very slim, light, and fit the mount I already had. After hooking up the TV, the first thing I wanted to see was some 4K footage. I found some on the YouTube app, which was really nice, but I can only stand so much nature footage. Next, I tried netflix, as there is supposed to be some 4K content there, keep in mind the netflix app had been missing entirely before the last update. Since there is no was of identifying 4K content on netflix, I simply searched “4K” and looked at what was returned. I don’t think everything returned is 4K content. I would expect to see a “4K” or “UHD” logo in the area where the HD logos etc. are presented, but there is not one. Streaming content from netflix is very disappointing, it seems there is still some software issues, as I can only watch a few minutes of a movie before being dumped to a buffering screen that usually... read more
Creating A Working Buildroot for SB6120/6141 Komodo

Creating A Working Buildroot for SB6120/6141 Komodo

Here is a tutorial I wrote a while back about creating a working buildroot to compile firmware for the SB6120/6141 (and possibly other models of cable modems). This is all done using open source items. First thing, install ubuntu 9.04 using the iso with this name: ubuntu-9.04-desktop-amd64.iso, while this tutorial may work on 32 bit linux with some adjustments, it will only work with a 64 bit system in it’s current state. The repositories will be down, because this is an old release of ubuntu. Run this command to correct our /etc/apt/sources.list: Shell sudo sed -i ’s/us.archive.ubuntu.com/old-releases.ubuntu.com/g’ /etc/apt/sources.list 1 sudo sed -i ’s/us.archive.ubuntu.com/old-releases.ubuntu.com/g’ /etc/apt/sources.list Run this command to update the repositories and upgrade all packages: Shell sudo apt-get update && sudo apt-get -y upgrade 1 sudo apt-get update && sudo apt-get -y upgrade restart, wait until update manager pops up, then let it update all the remaining packages. Restart again. Run the following commands: Shell sudo su root 1 sudo su root Install virtual box tools: Click devices>Insert Guest Additions CD Image Run terminal, and type the following command: Shell sh /cdrom/VBoxLinuxAdditions.run 1 sh /cdrom/VBoxLinuxAdditions.run Logout and log back in and you should now have fullscreen support etc. Run these commands to install the necessary tools to build: Shell sudo su root apt-get autoremove apt-get install kernel-package libncurses5-dev fakeroot wget bzip2 build-essentials bison flex unifdef build-essential texinfo libc6-dev-i386 quilt ia32-libs g++-multilib wget http://downloads.sourceforge.net/project/sb6141.arris/SB6141_1_0_6_1_SCM00/SB_KOMODO-1.0.6.1-SCM00.tar.bz2?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsb6141.arris%2Ffiles%2FSB6141_1_0_6_1_SCM00%2F&ts=1408062579&use_mirror=iweb 1234567 sudo su root apt-get autoremove apt-get install kernel-package libncurses5-dev fakeroot wget bzip2 build-essentials bison flex unifdef build-essential texinfo libc6-dev-i386 quilt ia32-libs g++-multilib wget http://downloads.sourceforge.net/project/sb6141.arris/SB6141_1_0_6_1_SCM00/SB_KOMODO-1.0.6.1-SCM00.tar.bz2?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsb6141.arris%2Ffiles%2FSB6141_1_0_6_1_SCM00%2F&ts=1408062579&use_mirror=iweb Then, change the name of the file, just for ease, to the original name,... read more
SNMP Daemon crashing on Esxi 5.5 – PowerEdge 6800

SNMP Daemon crashing on Esxi 5.5 – PowerEdge 6800

I have a Dell Poweredge 6800 that I run Esxi 5.5 on, even though it’s not fully compatible. I recently began to monitor this box using Observium. Every other device on my network had SNMP working fine, but, for osme reason, SNMPD would crash all the time on my Esxi Box. After several hours of research, I was unable to determine the source of the crash. I ended up working around it by adding a cron job to restart the daemon every five minutes, using the following code: [code language=”bash”] # chmod u+w /var/spool/cron/crontabs/root # echo ‘*/5 * * * * /etc/init.d/snmpd restart > /dev/null 2>&1’ >> /var/spool/cron/crontabs/root # chmod u-w /var/spool/cron/crontabs/root # kill $(cat /var/run/crond.pid) # crond [/code] While it’s not the most eloquent workaround, it works for me and allows me to get consistent monitoring results from Observium. I hope to eventually upgrade to a server that is on the Esxi 5.5 HCL, but for now, this looks like the best option. If anyone comes up with a better solution, please let me know! So, if you’re having problems, try this to fix SNMP Daemon crashing on Esxi 5.5 – PowerEdge 6800... read more
Skip to toolbar