Web apps with Node and Express

I have to write some PHP from time to time, but I have always primarily been a JavaScript developer. It was the first language I learned and it is where I am most comfortable. This is why I got so excited about Node.js. Node can be very empowering for a JavaScript developer. It takes your skillset outside of the browser and opens up a whole new world of possibilities. It is now possible to build command line tools, full web apps, and even desktop applications in good old familiar JavaScript. In this post I am going to cover building a super simple app with Node and the Express famework. It is going to be pretty basic but it also gives a good overview of the possibilities.

 

Setting up Our App

The first step is going to be installing node. I use Homebrew for this on OSX but you can also go to the official Node.js website. Once you have Node installed you will also get npm. Npm is a package manager for nope modules; the Express framework we are going to be using is a node module.

 

Using Node

With node insalled you should be able to open up a terminal window and type node -v to see the version of node you have installed. You can now run JavaScript files from your terminal by typing node name-of-file.js . By itself, this is not that usefull, so we are going to need to install some additional modules through npm. Type the following into your terminal (you may need sudo).


npm install express jade body-parser cheerio request -g

These are all of the modules we are going to be using. I will explain each one more thoroughly but for now just get them loaded onto your machine.

 

The App

Were going to build a simple web app that lets us search for definitions. We will be able to enter a term into a search bar and the app will query wikipedia to show the definition for that term. Its pretty basic and not particularly useful but it makes a good demo. We’ll be using Express to handle the server, Request to query Wikipedia, Jade as out template engine, and cheerio to extract the data we want from Wikipedia.

 

Node App Screen

Above is our basic interface showing the results for node.js.

 

Get Started

First create a new JavaScript file in your current directory called server.js, then create a views sub directory. Inside the views sub directory create a file called index.jade. This will be the template for our page.

 

Next, in server.js we will define our modules


var express = require('express')
var bodyParser = require('body-parser')
var request = require('request')
var cheerio = require('cheerio')
var app = express()

 

this pulls in all the required dependencies. We also create an app variable to hold express. Next we define two additional variables. One is the url on Wikipedia we are going to pass queries to. The other is an empty object to hold our query term and the returned definition.


var url = 'http://en.wikipedia.org/wiki/Special:Search'
var search = {}

 

We had to include the body-parser module so Express can parse the body of HTTP request. We will need this to get the POST data. So we’ll go ahead and tell express to use it.



app.use(bodyParser.urlencoded())

 

Handling HTTP Requests

Express has methods for each of the major HTTP verbs GET, POST, PUT, and DELETE. In this tutorial we will only be using GET and POST. We can chain these methods together like so.



app
   .get('/',function(req,res){
			

   })
   .post('/', function(req,res){
				

   })
   .listen(3000)


 

We now have a server that will listen for GET and POST requests on port 3000. If you run node server.js from the terminal, then navigate to localhost:3000, you’ll see – nothing. Because the server doesn’t do anything yet. But it is working. We need to give it a page to send back. This is where the jade templating language comes in.

 

Working with Jade Templates

Open the index.jade file we made earlier and add the following



.wrap(style='width:80%;margin:50px auto;font-family:arial')
	h1 Search Wikipedia
	div
		h2= term
		p= definition

	h3 Search for a Term
	form(method='POST')
		input(name='searchTerm')
		button Submit

 

Jade is basically a shorthand for HTML. It compiles to regular HTML but lets you write it without all the brackets and closing tags. Also, notice the equals signs after the h2 and p tag. These are template tags so the values after them will be JavaScript variables. HTML attributes are defined in parentheses, we use this to set up the form and to add some very basic inline styling.

Defining a Route

Lets go back to out server.js file and add the following code to the app.get method.



app
   .get('/',function(req,res){
        // pass the search object to the jade template
	res.render('index.jade',search)
	search = {}
   })

 

The app.get method takes two paramaters. First a route; since our app has only one page we just use the root ‘/’ route. The second parameter is a callback function that takes the request, and response objects. By calling render on the res object we are telling Express to serve the index.jade template and use the search variable as the data. Since search is still an empty object the page will only render a search bar. We also clear the data in search for later when we do add data to it.
Next we need to handle post requests. This is where most of our apps logic will live. This is also were we will introduce the request , and cheerio modules.

Making Requests with Node

The request module helps make it easy to make HTTP requests from our javascript code. It takes two parameters, a options object, and a callback. In the options you can configure things like the url, headers, timeout, and query string. For our app our request looks like this.



app

    //get method ..

   .post('/', function(req,res){
	// use url defined above, build query string from POST data
	request({ 
	   url: url,
	   qs:{search: req.body.searchTerm } 

	}, function(err, res, body){

	   // check that the request was successful 
	   if (!err && res.statusCode == 200) {
						
							
	   }
	})
				
				
   })

 

So what were doing here is taking the input from the search bar in out template and using it as the query string to make a request from wikipedia. The returned data will be available as the body inside the callback function. A great write up on the request module, Cheerio and scraping pages with Node in general can be found here, Easy web scrapping with Node. This article covers these topics in depth. We are only taking a quick glance at them here.

Screen Scraping with Cheerio

The cheerio module will seem very familiar to you if you’re used to using jQuery for front end work. Cheerio is basicly jQuery for the server. Since there is no DOM on a server you can’t use traditional DOM methods like querySelector or getElementById. The body in our response is just a string of all the HTML for that particular Wikipedia page. We only want some of the text content of the page so we need to parse this string to get only what we need. Cheerio gives us a clean simple api to do this so we dont need to write regular expressions, we can just use CSS selectors. Like so.



app

    //get method ..

   .post('/', function(req,res){
	// use url defined above, build query string from POST data
	request({ 
	   url: url,
	   qs:{search: req.body.searchTerm } 

	}, function(err, res, body){

	   // check that the request was successful 
	   if (!err && res.statusCode == 200) {
		// load response into cheerio, parse as jQuery				
		var $ = cheerio.load(body)
	        search['definition'] = $('#mw-content-text').find('p').slice(0,3).text()					
	   }
	})
	// save the search term so it can be displayed
        search['term'] = req.body.searchTerm			
				
   })

 

Here we are taking the text content of the first three paragraph tags within a div with the id of mw-content-text. This gives us a nice short snippet of a definition. It is essentially the top section of a Wikipedia article. To figure out how I decided this, simply go to a page on wikipedia and view the source in the developer tools.
Now that we have the text we want we save it as the description property of the search object we defined earlier. We also save the search term.
Next we just need to use Express’s redirect function res.redirect('/') . This function will automatically make a get request to the main route, which will serve our page with the data populated. However, we can’t just add the line to the bottom of the function like you might expect. Due to the asyncrounous nature of most node functions; including request, the code will finish executing before the request is complete. This means if we simply do this.



app

    //get method ..

   .post('/', function(req,res){
	// use url defined above, build query string from POST data
	request({ 
	   url: url,
	   qs:{search: req.body.searchTerm } 

	}, function(err, res, body){

	   // check that the request was successful 
	   if (!err && res.statusCode == 200) {
		// load response into cheerio, parse as jQuery				
		var $ = cheerio.load(body)
	        search['definition'] = $('#mw-content-text').find('p').slice(0,3).text()					
	   }
	})
	// save the search term so it can be displayed
        search['term'] = req.body.searchTerm	
        res.redirect('/')		
				
   })

 

The redirect function will be called before we have our data from the request. Instead we have to listen for the request functions end event and then call the redirect.



   .post('/', function(req,res){
	// use url defined above, build query string from POST data
	request({ 
	   url: url,
	   qs:{search: req.body.searchTerm } 

	}, function(err, res, body){

	   // check that the request was successful 
	   if (!err && res.statusCode == 200) {
		// load response into cheerio, parse as jQuery				
		var $ = cheerio.load(body)
	        search['definition'] = $('#mw-content-text').find('p').slice(0,3).text()					
	   }
	})
	.on('end', function(){
	    res.redirect('/')
	})
        search['term'] = req.body.searchTerm	
        
				
   })

 

In this example its important to note that the code search['term'] = req.body.searchTerm is actually being run before the redirect. The redirect function only runs once the request is complete and the response has been parsed. And now the app should be working. You should now be able to enter a term in the input feild and have the definition for that term show up on your screen. The completed code for our server.js file is as follows.


var express = require('express')
var bodyParser = require('body-parser')
var request = require('request')
var cheerio = require('cheerio')
var app = express()


var url = 'http://en.wikipedia.org/wiki/Special:Search'
var search = {}

app.use(bodyParser.urlencoded())

  app
	.get('/',function(req,res){
		res.render('index.jade',search)
		search = {}
	})

	.post('/', function(req,res,next){
			request({ 
				url: url,
			 	qs:{search: req.body.searchTerm } 

			}, function(err, res, body){

				if (!err && res.statusCode == 200) {
					
					var $ = cheerio.load(body)
					search['definition'] = $('#mw-content-text').find('p').slice(0,3).text()
						
				}
			})
			.on('end', function(){
				res.redirect('/')
			})
			search['term'] = req.body.searchTerm			
		})

	.listen(3000)

 

This app is strictly for demo purposes. And I realise it is not particularly useful itself, but it does show the potential of what can be done with Node. It is also the exact kind of application that node is good for. You dont want to be doing heave server side computations with node. It is not meant for that. Node is first and foremost for networking. Heavy computations should be handed off to other middleware while node goes on handling requests. It excels at building large scalable interconnected networking applications that depend on a large number of third party services.

Awwwards