During the summer vacation, I joined the Orbital project in SoC. This is a module-bearing project for SoC freshmen to get some experience in developing real-life applications(mainly web app or mobile app).
The final product for my team is now hosted on Digital Ocean (initially on Heroku): https://viz.paradite.com/
This a team-based project so I partnered up with Dongyan. I was in charge of backend, routing and the visualization and she is in charge of UI and UX.
Ideation
I got the rough idea for Graphpedia long time ago. When I was in year 1, I realized that there are a lot of terms and technologies to know and understand in order to start my own programming projects, and the number of terms are forever expanding. As someone who is new to all these terms, it is very hard to get a good understanding of them by just looking at the wikipedia page or the official site, because these places are normally filled with a lot of texts and even more sophisticated terms. And the other problem with new programmer is that they lack the context in which the terms are used, the basic understanding of the frameworks.
Therefore I got this idea to make a web app to help people understand the terms very quickly by providing a visualization of the terms using their relationships with other terms.
Progress & Learning
I learnt a lot of things during this project. The basic framework for the project is node.js and express, with neo4j as the graph database. I was lucky enough to find a template for working with node.js and neo4j so I do not need to do the low level API calls to neo4j database from neo4j. I enhanced the backend routing by providing a search function with partial search support using regex in neo4j.
The key element of this project is visualization. After looking at several web visualization tools, I decided on using d3.js, as it has the tree layout which is suitable for visualizing the relationships. The most difficult part of this project is probably parsing the data from neo4j database into json and pass it to front-end for d3.js to render the visualization. For parsing, I used array operations and construct the json objects/arrays. For passing the json object, I simply turned it into a string and passed it as one of the parameters for rendering in jade. From there, I am able to use the parse the json string back and use it in d3.js scripts.
The template has no abstraction for relationships and only one type of relationship was allowed. Hence, I have to do abstraction for adding, displaying and deleting relationships. This turns out to be extremely difficult as the types of relationships are used across different places: backend neo4j database, router, front-end svg for visualization and front-end html. So in the end at the later stage of development, I added a relationship model just to properly manage the relationship types. In this way, I do not need to modify other parts of code whenever I want to add a new relationship.
The other interesting part of this app is the path function which visualizes the relationship between two terms, the relationship can be indirect and connected via some middle terms. This function is possible because of the nature of neo4j as a graph database. There are built-in graph algorithms, including get the shortest path from one node to another via a number of intermediate terms.
Towards the end of the project, we had some spare time so I decided to do some basic data-mining. during my research, I found out that Stack Exchange has its own API, which allows me to get a list of popular tags on stackoverflow. Hence, I managed to crawl the most popular 100 tags from stackoverflow into my database along with their synonyms(however I did not have enough time to do merging of synonym terms so I just put them in the description field).
In order to compensate for the lack of relationships in our database, I also added a page dedicated to adding relationships. This page displays random suggestions and allows adding of relationships with a few clicks.
For the completeness of the project, I also added a second database – mongoDB for managing user accounts. The users who are logged in as considered as administrators and are given the power to delete a relationship or a term.
Peer Evaluation & Mentor
The Orbital project consists of 3 milestones. Teams are required to propose functions to be implemented in each milestone and at the end of each milestone we are supposed to evaluate each other’s progress on the projects and give feedbacks. This is helpful as we are motivated to complete the proposed functions and we will help each other test out the app.
The other part of the Orbital project is the mentorship. For the teams aiming for highest level of achievement (Apollo 11), we get to choose a mentor for us. For my team, we are very lucky to get a good mentor who has a lot of experience in web development. It is through him that we knew about the neo4j graph database and purecss which was used to do the sidebar.
Splash down & Conclusion
We had our splash down event yesterday(22 Aug Friday) where all the teams present their projects to other people including other SoC students and guests outside SoC. It was very nice of the organizer(and the sponsor) to give us free printing service for our A1-sized posters!
Overall, this was a great experience with a lot of learning points. The Orbital project is the best platform for the freshmen to try developing something on their own and learn about teamwork.