Project A

Released

16 September 2021

Submission

Deadline	30 September 2021 (for Feedback)
	21 October 2021 @ 5 PM EDT (Extended)
Where	Web Submit
Name	A
Files	A.zip or A.war

Version

1.9	20 October 2021	Fixed a bunch of errors in the example responses.
1.8	20 October 2021	Add example requests and responses.
1.7	27 September 2021	Deadline extension until 21 Oct (just before Test 1).
1.6	27 September 2021	Minor revision to Section D. Bullets 12-16.
1.5	27 September 2021	Minor revision to Section C.4. Added this.
1.4	23 September 2021	Minor revision to Section C.3. Added this.
1.3	23 September 2021	Minor revision to Section G. Added this.
1.2	23 September 2021	Minor revision to Section D. Bullet 6.
1.1	16 September 2021	Minor revision to Section B.3, Step 3.
1.0	16 September 2021	Initial release

Important Notice

Based on the number of current submissions and feedback from the lab sessions, I am giving everyone a deadline extension until 21 October 2021 at 5 pm to submit this Project for the 1% grading component. Consider it a 1% bonus to your Test 1 if you submit your project beforehand.

Despite this extension, there is one major advantage for you to submit your project (completed or incomplete) by 30 September 2021. If you submit by 30 September 2021, I will provide you feedback regarding your submission and whether or not there are issues that need to be fixed. The feedback should help you fix any issues and aid you in Test 1. Feedback will be provided via a FEEDBACK.txt file that will be added to your submission folder and accessible via the Web Submit portal.

Executive Summary

In this project, you will create several multithreaded TCP services as building blocks of a microservice architecture. Using a custom (non-HTTP) protocol. These services will expose several foundational concepts, such as basic networking, JSON/XML marshaling, database access, inter-service calls, stateful versus RESTful, and cloud computing. You will later use these services to build a shopping cart web application. In addition, this project will address scalability issues, such as multithreading, connection pooling, and throttling, as well as certain security concerns, such as vulnerabilities and authentication.

The Environment

The RemoteLab Workstations

All development work in this course must be done within your home directory on EECS systems; i.e. not on your own computer. You can access your home directory by visiting the remoteLab URL and then selecting an EECS system to work on:

• During the scheduled lab hours, select any available workstation under Linux Remote Desktop (EDU) (ea). Each workstation can take up to two users at the same time.
• Outside the scheduled lab hours, if there are no available workstations under Linux Remote Desktop (EDU) (ea), select red1 or red2 under Linux Remote Desktop (EDU) (red) or any of the workstations under Linux Remote Desktop (EDU) (wsc105) or Linux Remote Desktop (EDU) (wsc106).

These connections provide full GUI access; i.e. as if you are physically sitting at a workstation in the lab. Note that it is also possible to access your home directory at any time by connecting to red.eecs.yorku.ca via ssh, but this gives you CLI access only.

Setting up the Course Environment

To set up your environment for this course, issue the command:

$ 4413Go

It will create a folder named 4413 under your home directory, and it contains several libraries, drivers, softwares, databases, and tools. For development, you can use any editor (nano, vim, …) or Eclipse to build your classes, but whatever you use, do not store your own files (e.g. your workspace) in the 4413 folder. Store them anywhere else under your home directory and leave 4413 alone.

Developer Tips

There are several key scripts and programs within the ~/4413/bin directory that you need. These include:

derbyOn
derbyOff
ij
tomcatOn
tomcatOff
ngNew

To make it more convenient to access them, you should add the ~/4413/bin/ directory to your shell’s PATH environment variable by appending the follow:

For tcsh, append to .cshrc:

setenv PATH "$HOME/4413/bin:$PATH"

For bash, append to .bashrc:

export PATH="$HOME/4413/bin:$PATH"

To determine which shell is your default shell, run:

$ echo $SHELL

Alternatively, this prints out your current shell:

$ echo $0

Creating a Eclipse Project

1. Open Eclipse and select a directory (not the 4413 folder) to be your workspace.

2. Navigate via the menu:

   File > New > Dynamic Web Project
   

3. Enter a project name and ensure the Target runtime is Apache Tomcat v9.0 and leave the rest of the fields and configurations as their default values. If the Target runtime is empty or None:

   1. Click New Runtime....
   2. Select Apache Tomcat v9.0 and then click Next.

   3. Set the Tomcat installation directory to:

      /eecs/home/<user>/4413/pkg/tomcat
      

      Navigate there with the Browse... dialog and click Open when you have found and selected it.

   4. Then click Finish and the Apache Tomcat v9.0 Runtime should be added as the Target runtime.

   ---

4. Click Finish and create the project.

5. Now you can right-click your project, navigate the menu to:

   New > Class
   

6. Enter the Name of class and its Package. Then click Finish.

Adding External Java Libraries

1. In Eclipse, right-click your project and navigate the menu to:

   Build Path > Configure Build Path
   

2. Select the Libraries tab in the Properties window.
3. Click Add External JARs.

4. In the popup window, navigate to your Home directory and then the 4413 folder. Go to:

   /eecs/home/<user>/4413/lib
   

5. Select all of the JARs files and click Open.
6. Now you will see that the JARs have been populated into the JARs and class folders on the build path in the dialog window. Click Apply and then Apply and Close. Under your project, you will see the libraries are listed under Referenced Libraries.

The Services

This Project asks that you develop and test five microservices with the following functional specifications:

Geo — A Computational Service

Given two points on Earth, this service returns the geodesic distance between them. This service will be used later by a shopping cart application to estimate the drone delivery time. Each point is specified using its latitude (t) and longitude (n) expressed as signed decimal degrees with East of Greenwich being a negative longitude and south of the Equator being a negative latitude. The distance (in kilometres) between two such points is given by:

Note that the four coordinates must be in radians so start by multiplying each by $\pi / 180$. Test your code by computing a known distance (you can for example use a map app and pick two points connected by a straight road). You can use this handy website to verify your calculations: here.

Note that we are estimating the drone delivery time here, but in a later project, we will estimate the driving time by taking the road network and the current traffic volume into account.

Geo - Example 1

Request:

43.77535348921289 -79.5015111512824 43.843195533568355 -79.5394410357609

Response:

8.134553593941805

Geo - Example 2

Request:

1 2 3 4

Response:

314.4029510236249

Geo - Example 3 (Missing Arguments)

Request:

1 2 3

Response:

Don't understand: 1 2 3

Geo - Example 4 (Non-numeric Arguments)

Request:

1 2 3 A

Response:

Don't understand: 1 2 3 A

Developer Tips

Use Telnet

$ telnet <host> <port>

Use telnet to test your server. Login into red.eecs.yorku.ca via SSH and send a request to the server with telnet from there. Remember, you need to give it the IP address of the computer the server is running on as seen from the network and not its loopback address (not 127.0.0.1).

Quote — A Database Service

Given the ID of a product and a return format (json or xml), this service looks up the ID in the Product table in the hr schema of the Derby database and returns the product’s ID, name, and price in the desired format. If the ID is not found, the return should have “<ID> not found” as ID, an empty name, and 0.0 as price. Use GSON and JAXB to serialize and marshal, and use “id”, “name”, and “price” as identifiers for the JSON elements and XML nodes. Create a “Product” bean to facilitate the transformation (more on this in lecture).

Quote - Example 1

Request:

S10_1678 json

Response:

{"id":"S10_1678","name":"1969 Harley Davidson Ultimate Chopper","price":48.81}

Quote - Example 2

Request:

S10_1678 xml

Response:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<product>
    <id>S10_1678</id>
    <name>1969 Harley Davidson Ultimate Chopper</name>
    <price>48.81</price>
</product>

Quote - Example 3

Request:

S00_0000 json

Response:

{"id":"S00_0000 not found","name":"","price":0.0}

Quote - Example 4

Request:

S00_0000 xml

Response:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<product>
    <id>S00_0000 not found</id>
    <name></name>
    <price>0.0</price>
</product>

Quote - Example 5 (Missing Arguments)

Request:

S10_1678

Response:

Don't understand: S10_1678

Developer Tips

The Derby Database

To start and stop the Derby database, run these commands in the terminal:

$ derbyOn
$ derbyOff

Developer Tips

Inspecting the Database

You can use the ij commmand (include in ~/4413/bin/) to inspect the Derby database.

1. Start the Derby database.

   $ derbyOn
   

2. Run the command:

   $ ij
   > connect 'jdbc:derby://localhost:64413/EECS';
   > show tables;
   > select * from hr.product;
   

3. For additional commands, run:

   $ ij
   > help;
   

4. Remember when you are done, to stop the database server, by running:

   $ derbyOff
   

Auth — An API & Database Service

Given a username and password, this service authenticates these credentials and returns “OK” or “FAILURE” accordingly. This service will be used later by the shopping cart application to authenticate users. Authentication is done by adding a long salt to the password; using a cryptographic function to hash the result; and then repeating the process count times. More on this in lecture. The CLIENT table in the Sqlite3 database stores the salt, count, and hash of each user. The table adopts PBKDF2 (Password-Based Key Derivation Function 2) to perform the hash, which is the current best practice. An API for computing PBKDF2 is provided in the hr4413 library (in 4413/lib) through the following method in the g.Util class:

public static String hash(String password, String salt, int count) throws Exception

If the username is not found in the table, or if found but the computed hash differs from the stored one, return “FAILURE”; otherwise, return “OK”.

Auth - Example 1

Request:

Mike@Miranda.ca magic2Go

Response:

OK

Auth - Example 2

Request:

Mike@Miranda.ca magic2NOO

Response:

FAILURE

Auth - Example 3

Request:

DoesNot@Exist.ca magic2GO

Response:

FAILURE

Auth - Example 4 (Missing Arguments)

Request:

Mike@Miranda.ca

Response:

FAILURE

Developer Tips

The SQLite Database

You can inspect the SQLite database with the sqlite3 command.

1. Navigate in your terminal to:

   /eecs/home/<user>/4413/pkg/sqlite/
   

   There you will find the Models_R_US.db SQLite database file.

2. To inspect it, run:

   $ sqlite3 Models_R_US.db
   > .tables
   > .fullschema
   > select * from client;
   

Testing your Code

Below is a table of the username and password pairs that you can use to test against your Auth service.

username	password
Mike@Miranda.ca	magic2Go
Sierra@Samosa.com	mySecret
Echo@Erable.org	Two4One
Nancy@November.me	Far2Away

Geo2 — A Stateful, Delegating Service

This “pedagogical” service delegates to the RESTful Geo service in a stateful manner to shed light on session management. It receives the coordinates of the first point in one request and the coordinates of the second point in another. It needs to somehow “link” the two request (despite the multithreading nature of the service) and then supply all four real numbers to Geo and return its response (i.e. no computation is involved).

Geo2 - Usage & Input Arguments

Usage:

$ java Geo &        # run in the background, & is not an argument
Server listening on ea78/130.63.96.34:36378

$ java Geo2 130.63.96.34 36378 &
Server listening on ea78/130.63.96.34:36128

$ telnet 130.63.96.34 36128

Geo2 - Example

Each request is a separate TCP connection.

Request & Response 1:

43.77535348921289 -79.5015111512824

Cookie: 0

Request & Response 2:

1 2

Cookie: 1

Request & Response 3 (Cookie = 0):

43.843195533568355 -79.5394410357609 0

8.134600613408246

Request & Response 4 (Cookie = 1):

3 4 1

314.4029510236249

Request & Response 5 (Cookie = 0, Cookie deleted after 2nd request):

10 10 0

Invalid Cookie: 0

Request & Response 6 (Missing Arguments):

10

Don't understand: 10

Hint

If only two parameters are received then treat this as a fresh request; store the parameters in some data structure and return a pointer to them as a cookie. If three parameters are received then the third as a cookie that allows you to link this request with a prior one and thus obtain all four coordinates.

Hint

Maintaining the previous parameters requires a shared object and data structure. While working in a multi-threaded environment, there are some situations when multiple threads in a single program try to access the same resource at the same time. Due to concurrency, this situation may result in unexpected and erroneous results. For this, Java provides a mechanism of synchronization that helps to avoid such race conditions by providing synchronized thread access to the shared resource or data.

More information, see:

• Synchronized Methods
• Synchronized in Java Syntax and Example

Gateway — An HTTP API Gateway

This “pedagogical” service sheds light on the challenges involved in building an API Gateway, such as service registration, discovery, and orchestration. It is also our first foray into HTTP services. Its client is a browser and its URL is: http://host:port/SRV?p1=v1&p2=v2... where SRV is either Geo or Auth or Quote and where p1, p2, ... are the names of parameters of the SRV service requires and v1, v2, ... are the values corresponding to each of those parameters.

The job of this service is to discover the needed service (extract its name from the request and find its IP and port if alive); perform inter-protocol transformation; invoke the service; and then return its response.

Gateway - Usage & Input Arguments

Usage:

$ java Geo &      # run in the background, & is not an argument
Server listening on ea78/130.63.96.34:36378

$ java Auth &
Server listening on ea78/130.63.96.34:41883

$ java Quote &
Server listening on ea78/130.63.96.34:45321

$ java Gateway 130.63.96.34 36378 130.63.96.34 41883 130.63.96.34 45321 &   # Geo Auth Quote
Server listening on ea78/130.63.96.34:36128

$ telnet 130.63.96.34 36128

• Each request is a separate TCP connection.
• Each request, i.e. GET /service HTTP/1.1, must be followed by a blank line, i.e.: two \r\n newlines (carriage return and linefeed).

• You can use a web browser to connect the service.

  http://130.63.96.34:36128/Geo?t1=1&n1=2&t2=3&n2=4
  

• You may have additional response headers. Like this:

  HTTP/1.1 200 OK
  Server: Java HTTP Server : 1.0
  Date: Wed Oct 20 13:18:41 EDT 2021
  Content-type: text/plain
  Content-length: 18
  
  314.4029510236249
  

  But the bare minimum is:

  HTTP/1.1 200 OK
  Content-type: text/plain
  
  314.4029510236249
  

Gateway - Example 1

Request (Geo):

GET /Geo?t1=1&n1=2&t2=3&n2=4 HTTP/1.1

Response:

HTTP/1.1 200 OK
Content-type: text/plain

314.4029510236249

Gateway - Example 2

Request (Auth):

GET /Auth?username=Mike%40Miranda.ca&password=magic2Go HTTP/1.1

Response:

HTTP/1.1 200 OK
Content-type: text/plain

OK

Gateway - Example 3

Request (Auth - Failed):

GET /Auth?username=Mike%40Miranda.ca&password=magic2NOO HTTP/1.1

Response:

HTTP/1.1 200 OK
Content-type: text/plain

FAILURE

Gateway - Example 4

Request (Quote as JSON):

GET /Quote?id=S10_4698&format=json HTTP/1.1

Response:

HTTP/1.1 200 OK
Content-type: text/plain

{"id":"S10_4698","name":"2003 Harley-Davidson Eagle Drag Bike","price":91.02}

Gateway - Example 5

Request (Quote as XML):

GET /Quote?id=S10_4698&format=xml HTTP/1.1

Response:

HTTP/1.1 200 OK
Content-type: text/plain

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<product>
    <id>S10_4698</id>
    <name>2003 Harley-Davidson Eagle Drag Bike</name>
    <price>91.02</price>
</product>

Gateway - Example 6

Request (Incorrect Resource Path):

GET /doesnotexist HTTP/1.1

Response:

HTTP/1.1 404 NOT FOUND
Content-type: text/plain

NOT FOUND

Gateway - Example 7

Request (Missing Parameters):

GET /Geo?t1=43.77535348921289&n1=-79.5015111512824 HTTP/1.1

Response:

HTTP/1.1 400 BAD REQUEST
Content-type: text/plain

BAD REQUEST

All five services use TCP for transport; they are all RESTful, except for Geo2, which is stateful; and they all use a custom protocol except for Gateway, which uses HTTP.

Service Implementation

Use the design patterns, methodologies, and hints demonstrated in lecture in order to speed up development and learn best-practice approaches.

• Develop all services in Java and include 4413/lib in your classpath. If you use Eclipse, configure the project’s build path so that its libraries include the needed files in 4413/lib as external jars.
• Each of the 5 microservices must have its own class and its own port. This way, they can be deployed in one container, in different containers, in different VMs, or even on different physical hosts.
• Name the service classes Geo, Quote, Auth, Geo2, and Gateway.
• The service class must extend Thread and must have a main method.
• The service class must act as a server with a TCP transport and must bind to localhost.
• The server’s port should be set to zero to indicate any available port.
• The server must be multithreaded with a new thread per connection.
• The custom protocol for the first 4 services is text-based, case-sensitive, and line oriented. The request must be a single line (EOL-terminated) with its tokens whitespace delimited. The response is also text-based and EOL-terminated.

• For the Gateway service, the protocol is HTTP. Hence, the request is multi-line and begins with the line:

  GET /SRV?p1=v1&p2=v2 HTTP/1.x
  

  followed by HTTP headers that end with an empty line. The response begins with the two lines:

  HTTP/1.1 200 OK
  Content-Type: text/plain
  

  followed by an empty line and then the return of the service.

• All services return must be Don't understand: <request> if the incoming request is malformed.
• Use standard out for logging.
• Log Server listening on <ip:port> upon server start.
• Log Connection from <ip:port> upon receiving a request.
• Log Disconnected from <ip:port> upon closing a connection.
• Log Connected to <service> at <ip:port> when connecting to another service or database.
• Log Disconnected from <service> at <ip:port> when disconnecting to another service or database.
• Close the connection with the client after issuing the response.
• Handle exceptions per client by logging the exception message and closing the connection with that client.

Testing & Deployment

Do all your tests on the same host; i.e. launch the service in the background (using &) and test it in the foreground. This way, you will not be blocked by the York firewall which plugs all non-standard ports. To test, use

telnet <ip> <port>

or one of the tools discussed in lecture. Once everything is working, start the the service on a workstation, copy down the host IP address and port number, log into red (rather than a workstation) by simply ssh‘ing to it, and connect to the service on the workstation as a client from red. Please DO NOT run your service directly on red as the non-standard ports are blocked and red is a shared environment.

Clean-Up

Do not leave services running unnecessarily in the background on time-shared servers — they will consume system resources. Once you are done with a service, terminate it. Use the fg/bg commands or ps -u and kill to list and terminate all you services on red, red1, and red2.

Submitting Your Work

Locate all your files (i.e. java, class, shell, and other files that you created to build and test this project. Add also a README.txt that briefly describes what the files are and how to launch them. Upload all these files to this project’s section in the course cloud (Web Submit). You can also submit your work via the command-line, by first zip‘ing your files into a ZIP archive:

$ cd /path/to/A         # change directory to your project A
$ zip -r A.zip .        # zip everything in A directory into an archive
$ submit 4413 A A.zip   # submit the archive

If you make changes, you can overwrite your submissions.

Later, you can re-download your submitted files from Web Submit and extract the files, by placing it in an empty (destination) directory and running:

$ unzip A.zip

This ensures the files are available to you when you need them as well as submitted for the 1% Lab assessment component. You can of course also save your work to your favorite git, Google Drive, DropBox, S3, or some other storage.

Submission Helper

You can export your Eclipse project as a ZIP or WAR archive file.

To export as ZIP:

1. In Eclipse, navigate via the menu:

   File > Export...
   

2. In the Export dialog, select:

   General > Archive File
   

3. Check the project, folders, and files you want to archive.
4. Specify a To Archive File: destination location of the archive file.
5. Select the options: Save in zip format and Create directory structure for files, and check Compress the contents of the file.
6. Then, click Finish.

To later, re-import the project:

1. Navigate via the menu:

   File > Import...
   

2. In the Import dialog, select:

   General > Projects from Folder or Archive
   

3. Specify the Import source by clicking Archive... and navigating to it in the filesystem.
4. Check the folders, including the project you want to import.
5. Then, click Finish.

Alternatively, you can export your project as a WAR file. To do this:

1. Navigate via the menu:

   File > Export...
   

2. In the Export dialog, select:

   Web > WAR File
   

3. Select the Web Project you want to export.
4. Select the Destination to put the WAR file.
5. Important! Check Export source files.
6. Then, click Finish.

To later, re-import the project:

1. Navigate via the menu:

   File > Import...
   

2. In the Import dialog, select:

   Web > WAR File
   

3. Specify the WAR file and the Web Project you want to import.
4. Select the Target Runtime as Apache Tomcat v9.0.
5. Then, click Finish.