Problem Set 8 - Documentation & Retrospective

So far in this course we have built things and in the process understood various aspects of how to build web applications.

If we want users to actually use our application, we must provide them with a manual or at the simplest a "how it works" page.

The process of coding is the best way to learn software engineering, and very often we learn just as much by making mistakes as we do by getting things right.

pset: Create a help page for your application. Then take a step back and write about what you learned, what mistakes you made, and a future roadmap for your application.

Learning Objectives:

• Understand how to create a simple and effective help page
• Understand how to learn from your mistakes
• Understand how to communicate your ideas (technical or otherwise) in writing
• Understand how to create a product roadmap

Problem Set 7 - Social

In the previous problem set, we learned how to secure our web application. In this problem set, we will integrate our application to the social web.

pset: take the application you made in problem set 6, and integrate it with one or more social applications, such as Twitter, Facebook, Google+, etc.

Learning Objectives:

• Understand the meaning and importance of the social web
• Understand the meaning of web api's and how to integrate with them
• Understand how we can create value for our users by utilizing the social web

Problem Set 6 - Security

In the previous problem set, we learned about scalability. In this problem set, we are going to learn about security. Since our application will be deployed on the open Internet, it is vulnerable to all sorts of security exploits. In this problem set, we will learn about common security exploits and how we can protect our application against them.

pset: Take the application you created in problem set 5 and add user management and authentication to it. Then try to make commin security exploits on it. Now understand what you can do to protect your application against such exploits.

Learning Objectives:

• Understand user management, authentication, user roles and other related concepts
• Understand the common security exploits possible for web applications
• Understand how to protect your application against them

Problem Set 5 - Scalability

In the previous problem set, we took the application we had developed in problem set 3, and made it more usable. In this problem set we will learn how to create scalable web applications

pset: Take the web application you have with you in problem set 4, and make some requests to it. Measure the time it takes for the application to provide responses. Now seed it with huge data, and again measure the time it took to make responses. Did the response time suffer? Try to make it faster by tweaking your queries. Now try to make it faster by adding a cache. Next simulate thousands of users accessing your application at the same time. Does the response time suffer? Can you make it faster?

Learning Objectives:

• Understand the notion of scalability
• Understand how data and queries affect the response time of an application
• Understand how we can make the application faster by using a cache
• Understand how the number of simultaneous hits can affect the response time of an application
• Understand how we can use clustering to improve the response time
• Find out if using a no-sql database will improve the scalability of your application

Problem Set 4 - Usability

In the previous problem set, we created a simple multi-user web application in which we learned about concurrency and how to avoid problems related to concurrency.

A good web application is one which is not only technically sound but also aesthetically pleasing, and offers a good user experience.

pset: In this problem set, we will take the application made in problem set 3, and understand how to make it more usable

Learning Objectives:

• Understand the notion of usability
• Understand some common usability design patterns
• Learn how to make your web design aesthetically pleasing
• Learn about the importance of clear error messages
• Learn about the importance of internationalization and how to accomplish it in your web framework

Problem Set 3 - Simple multi-user application

In the previous problem set, we created a simple database backed application. We learned, why dynamic applications are better than static ones, and how databases can help us build dynamic web applications.

Now that you have a basic understanding of dynamic web applications, let us start tackling some complex issues. Most web applications are made to be used by multiple users simultaneously. This creates all sorts of concurrency related issues. Think of an airline reservation system. Imagine user A goes to the website to make a reservation and is looking at the available seats. By the time user A decides which seat they want to reserve, user B comes about and reserves that seat. However user A may not yet know of this. It is important that the system does not allow user A to re-reserve that same seat.

pset: In this problem set you can make any kind of a multi-user reservation system. For this exercise, you have to select a web framework on the platform of your choice.

Learning Objectives:

• Understand concurrency related issues in multi-user web applications
• Understand how the database and application server handle concurrency
• Understand how web frameworks help us with the flow of a web application
• Understand how web frameworks help us with templating
• Understand what are database transactions, and why we need them
• Understand how to design a complex data model, which consists of relationships between tables
• Understand different types of SQL joins and the pros and cons of using them

Problem Set 2 - Simple two page database backed application

In the previous problem set (pset 1), we created a simple single page web application. This was a static application, because the data could not be changed.

Real world apps are dynamic where the data changes depending on user activity, or external systems. The question is where will we store this changing data. We could use the file system, but a database is a far better place to store the data.

pset 2: In this problem set, we will create a simple 2 page application.

1. contact book

Learning objectives:

• Understand why we need a dynamic web application
• Understand why a database is the best way to store the data
• Understand the features a database provides (ACID)
• Understand how we connect to the database from our application
• Understand the concepts of UI templates where we will fill in parts with the dynamic data
• Understand page flow
• Understand how to create a simple data model
• Posting data (HTTP GET & POST)
• Deploying a database backed application

Problem Set 1 - One page application

Most of you already know about the Internet and the world wide web (if you are not sure, check out this page, and this page to understand how they differ). The Internet is helping us collaborate in ways which were either impossible or extremely difficult in the past. Even though it may not single handedly eradicate world hunger, poverty, and war, I believe that the Internet can and is playing a role in helping people understand each other, earn money, and live in peace.

In this course you will learn how to make web applications, so you can create a company in your garage and become a millionaire :-)

Now that you understand the difference between the Internet, and the world wide web, it may be a good idea to understand from a user perspective, what makes a good web site. Head over to this page written by Phillip Greenspun to understand more of what makes a great website, and what makes a website a ripe candidate for suck.com. After you are done, check out this page, to understand the financial potential of building Internet applications.

Now that you have finished reading the pages liked to above, you can start preparing for the first activity. In this activity, you have to create a simple one page static website. To do this, you will need a webserver on which you will deploy your website. You can choose any webserver you wish to work with. We have listed a few popular web servers below. Don't worry too much about making the right choice for the web server, since you can always change it for future activities, if you do not like it. Select something and start working.

Web Servers:

• Apache web server
• Lighttpd
• Tomcat (Use this if you intend to use for Java programming the web application)
• Jetty (Use this if you intend to use Java for programming the web application)
• Microsoft IIS (Use this if you intend to use ASP.NET for programming the web application)

Learning Objectives for this section:

1. What is the web
2. What is a web server and a web browser
3. What is the HTTP protocol
4. What is HTML
5. How to install, setup, and start a web server
6. How to point the web server to an application
7. How to connect to the application from a web browser
8. Basic HTML designing and best practices
9. Basic HTTP request / response (also include headers, mime types, error codes)
10. usability 101

Reference:

• Static site development
• Learn HTML in 21 minutes
• HTML Dog

Activity 1: Install the web server of your choice

Create a blog post explaining all that you have done, along with your notes. Submit a link to your blog post in the 'activity submission' form below.

Activity 2: In this activity you have to create a single page website. This page should contain multiple paragraphs, images, and ideally some video embeds (perhaps YOUTube) from the Internet. What follows is a guideline for how you may go about doing this activity.

• Install a web server on your system
• Create a simple web page 'index.html' for your static web page. Use well formed HTML and make it clean and readable
• Start your browser and load the url which will display your page. For most web servers it should be http://localhost:8080 , but it might be different for your server. Check the manual if you cannot view the page
• Create a simple stylesheet style.css and use it for styling your web page. Understand why using a stylesheet is a good idea

Create a blog post explaining all that you have done, along with your notes. Submit a link to your blog post in the 'activity submission' form below.

Activity 3: Install a plugin for your web browser which will let you examine HTTP requests and responses. Look at the request and response when you view your web page. Try and understand what the request means and also understand the request and response headers

Create a blog post explaining all that you have done, along with your notes. Submit a link to your blog post in the 'activity submission' form below.

---

My Notes

Activity1:

Downloaded Apache for Windows (32-bit) from http://httpd.apache.org/download.cgi#apache22

Checked the MD5 sum, and installed it using 'typical' installation on a Win 32 box.

Since this was an msi installer, Apache was installed as a system service. I re-started the service and pointed my browser to localhost port 8080... no luck... tried port 9090... no luck... checked the configuration file and realized that Apache had started on the default port 80. Pointed my browser to the default port and saw the "It Works!" message.

Activity 2:

I found the document root by looking at the configuration file. There is a line which states:

DocumentRoot "C:/Program Files/Apache Software Foundation/Apache2.2/htdocs"

So the document root is in the htdocs directory. Opening this directory revealed a file called index.html, which contained the "It Works!" message. This is the file I need to modify for this activity.

I modified index.html, and added some content to it. For the styling part, I had read about less css sometime back, and was keen to try it out. I implemented the style sheet using less css.

---

Activity 3

Since I am using Firefox (ver 6x), I installed the Firebug to examine request and response headers.

I can see the following request headers (below I mention the headers and what they mean):

Host - The domain name of the server (I wonder if it will include the port if server is on a non-standard port)

User-Agent - This is the user agent string. It identifies the browser which made the request.

Accept - Content types that are acceptable

Accept-Language - Acceptable languages

Accept-Encoding - Acceptable encodings

Accept-Charset - Acceptable charsets

Connection - Specifies if the connection will be closed after the response is sent, or whether it will remain persistent

If-Modified-Since - Allows 304 not modified to be returned if the requested content has not been changed.

If-None-Match - Allows 304 not modified to be returned if the requested content has not been changed.

Cache-Control - Can specify to all caching mechanisms from the client to the server, whether this object can be cached or not

I can see the following response headers (below I mention the headers and what they mean):

Date - The date when the response was generated

Server - The server

Last-Modified - When was the last time the response was modified

Etag - Identifier for a specific version of a resource

Accept-Ranges - What partial content range types this server supports

Content-Length - The length of the response body in octets

Content-Type - The mime type of this content

Overview of problem sets

Before designing the problem sets, we first need to outline what concepts the students need to learn in the course.

• Basic Web, HTTP, and HTML concepts (what is the web, what is a web server, what is HTTP, what is HTML & CSS, what do we mean by good HTML design, basics of the request / response cycle)
• Database backed apps 101 (why do we need database backed applications, setting up a database server, designing a small data model, designing a small 2 page web app, page flow, accepting data from the user, displaying data to the user based on the state of the database)
• Concurrency (what are the concurrency issues faced when creating a multi-user application and how to resolve them, database transactions)
• Database backed apps 102 (design a larger web application and understand page flow, usability, designing a more complex data model, and issues of scale, user managemnet, authentication)
• Security (take the app created in the previous problem set and understand the security issues faced by the application, and try to make it more secure)
• Make your own app and learn other aspects like social networking, mashups, etc.

Retrospective

Phillip has very kindly posted a retrospective of all the things they learned after teaching SEIA for 4 semesters. I think it is important to understand what worked and what did not before attempting to design my own course.

There are two strategies to teach software engineering. It can be taught as a survey course, where we expose students to the entire engineering lifecycle and a lot of concepts. The hope is that the students will understand enough to be able to start working on a actual project, and then learn the ropes over the next 6 - 10 years. Another approach is to teach each individual part of the software engineering process in depth as a separate course. I am not sure if the latter approach will work well. I think software is fun when we are actually writing code. That is also the way in which most learning happens. I can say this from personal experience. If we taught every aspect of software engineering separately and in depth, then there is a chance that the student would forget the concepts when they actually have to use them. For all these reasons, I personally prefer the survey course approach. Evidence also seems to support my theory. At MIT EE students were earlier taught a course in which they learned networks in great detail. Even though students might become experts in that topic, at the end of the semester, they forgot why they wanted to learn EE. That is not a very encouraging state of mind to be in. Similarly, one of the most popular courses at MIT is the 'unified engineering' course offered by their aeronautical department.

What are the concepts that we want to teach the students ?

• object-oriented design (modularization) where each object is a Web service (distributed computing)
• about concurrency and transactions
• how to build a stateful user experience on top of stateless protocols
• about the relational database management system
• basics of usability
• basics of security

MIT focuses on teaching students principles rather than skills. The students learn the skills while applying the principles.

Next Phillip talks about how the classic 6.916 course was conducted. The students had done one course on software engineering, so they all had decent debugging skills. Time wise, this course consisted of 3 hours of classroom time, and 6 hours of supervised laboratory time. Besides this I am sure the students would have had to spend at least 1 hour doing research on their own. So this is approximately a 10 hour / week course for MIT students.

Quoting:

The fundamental characteristics of 6.916 Classic were the following:

• problem sets provided the core knowledge
• teaching assistants, oftentimes alumni, worked side-by-side with students in a supervised twice weekly lab to help with design issues and questions of taste
• projects ran the entire length of term, starting with application design and ending with intense software development
• by the end of the semester, each student had built four or five database-backed Internet applications

Using this format, by the end of the semester, a student would have build about 4 - 5 database backed Internet applications.

Quoting Phillip: There is research which seems to suggest that people do a lot of things in a course, but not very accurately, have better recall than students who do a few things but very accurately. My own experience agrees with this theory. This theory presents yet another argument for building survey courses.

The course has various problem sets which expose a student incrementally to new concepts in building web based applications. The problem sets are followed by an actual project with an actual client. Even though an actual project with a client is a great idea (in theory), in practice it has a lot of flaws. An actual client may be finicky and change their mind often about what they want. This is not good for a college course. I am more in favor of allowing the student to build a project of their choice, or maybe to do away with this stage totally, and let them simply start working in the real world and make their web application after working on the problem sets in the course.

At this point of time, I am convinced that a survey course with problem sets which will expose students incrementally to new concepts is the right way to design a course in building applications for the Internet. Now the next challenge is to create the problem sets. I think here is where my course is going to differ from MIT's 6.171

Creating a Software Engineering for Internet Applications course

Some time back I came across an excellent course called Software Engineering for Internet Applications. The course is taught by Phillip Greenspun at MIT, and has also been replicated by several other universities.

From some time, I have been thinking of creating an Applied Course on Internet Applications on diycomputerscience.com. I wanted it to be a course which would teach the most important concepts of Internet applications like understanding the HTTP protocol, web servers, databases, concurrency, scalability, security, usability, etc in a hands on approach.

The SEIA course provides a perfect guideline for such a course. I will probably make a few changes to the SEIA course, but my course will probably re-use a large part of the SEIA structure, but will probably have different problem sets.