BrainGate Data Visualization

An interactive data visualization for a 
neuroscience research group.

Data Visulization Preview

Background

Overview: BrainGate, a research institution at Brown University, is working to restore the independence of people affected by neurologic disease by developing technologies that allow patients to control a computer interface with their brains. While patients use this technology, the BrainGate researchers collect data on the patients’ cursor movements and clicks. In order to improve their technology, the researchers needed an intuitive way to visualize and analyze the data they collect. I worked with a team of Computer Science students at Brown University to design and develop an interactive data visualization tool for the BrainGate researchers.

Scope: This project was done over the course of a semester as part of a Computer Science class at Brown titled “Creating Modern Web Applications.” Each student group was assigned a client in the Brown or Providence community, in addition to class assignments. During this project we: Interviewed our users (the BrainGate researchers), wrote user stories, wireframed different design solutions, created high fidelity mockups, wrote specifications and requirements, researched frontend and backend technologies that would meet our needs, developed a working prototype, presented to our class and our client, had weekly meetings with the BrainGate engineering team (our client) and some of the researchers to understand their needs and get their feedback on ideas, wireframes, mockups and prototypes.

Tools: Google drive, Github, Sketch, Illustrator, pens and paper, whiteboards, Sublime, Google Chrome

Team

Jean

Jean

Front End Developer

Marcin

Marcin

Back End Developer

Zhengyi

Zhengyi

Front End Developer

Sophia

Sophia (me!)

Experience Designer

Original Problem Statement

Our client, the BrainGate engineering team, needed a way for the BrainGate researchers to better visualize their data.

Users

The BrainGate Researchers

Interviews: We began this project by interviewing a few of our users to learn about their jobs, the tools they use and their day to day experiences. This project is very unique in that we have a very small and specific user group that could quickly give us feedback and who can continue to improve their program as they use it.

researchers

A summary of what we learned: The researchers are responsible for running sessions where a patient can use the BrainGate technology. While the researchers observe the patients using the system, they collect a lot of numerical data. For example, they collect data on the movement of the cursor over time and how accurate the participant was with the cursor. They take qualitative notes on their sessions with the patients in Word documents. The researchers would like to be able to see this data in a visual format, instead of trying to compare or analyze pure numbers. The researchers are interested in troubleshooting the BrainGate system by comparing the results of different settings, and visualizing the data collected after each session. Specifically, they are interested in visualizing the position and trajectory of the cursor over time, and being able to create filters for the data collected.

New Problem Statement

After conducting user interviews and learning more about the problem we redefined our problem statement:

The BrainGate researchers need a way to organize their data, visualize patient cursor movements overtime and annotate data based on their observations.

Before this project, the BrainGate researchers had no real way to visualize the data they collect about the patients’ cursor movement overtime other than just data coordinates, time stamps and what as provided to them in matlab. They also collect notes about their sessions with the patients using this technology in Word documents and record things like “at 0:45 seconds someone made a loud noise in the background and caused a distraction.” The notes were disconnected from the data points. All of their data was stored in MatLab files and not in a database.

Ideation and Wireframing

After our initial conversations with the BrainGate engineering team and interviews with some of the researchers I sketched some ideas.

sketch sketch sketch

Each time I met with the BrainGate team I got some feedback on the wireframes and continued to iterate. See the mockups section for descriptions of design decisions and the functions of specific parts of this data visualization.

sketch wireframe wireframe wireframe wireframe wireframe
sketch
wireframe wireframe wireframe wireframe wireframe wireframe wireframe wireframe wireframe

Mockups

This mockup has many parts to it. I will explain each section below.

mockup mockup diagram

Timeline: One of the biggest challenges was visualizing X and Y coordinates over time. I used inspiration from YouTube as well as other video and music players to design a timeline that the researchers could use to play through or scrub through the “recording” of each session. The timeline includes markers to show that a note was added at a specific time.

timeline

If the user hovers over a timeline marker then the note is lightly highlighted and if the user clicks on the timeline marker then the note is highlighted. They can also see a preview of the graph at that time by hovering.

timeline preview timeline select note

Graph: This is where the researchers can view the X and Y coordinates of the patients cursor. When the researchers are calibrating the system for the patient and testing different parameters, the researchers have the patients play a series of “games. ” In our mockups and prototypes we decided to focus on analyzing the data that was collected in this game. However, in the future, any cursor data (not just game data) could be analyzed using this program. In this game, the patient is aiming to move the cursor to the light blue target.

graph single view

While the position of the patient’s cursor at any given moment is important, where the cursor was and where it will go in relation to its current position is also very important. To visualize this information, I included a line that shows the future of the cursors pathway as well as the past of the cursors pathway. There is also a “full view” where the researchers can view all of the cursor movements for the entire session with the patient.

graph full view

The red dot is the “current” location of the patient’s cursor and the blue dot(s) is the place where the patient “clicked”.

Notes: The user can add notes at specific times. For example, if a researcher sees a cursor jump at 0:20 seconds, they can add a note at that time with the tag “jump”. The user can refer back to these notes later or another researcher can review the notes to better understand the session.

add note

If a user hovers over one of the notes they will see a preview on the timeline at the time that the note was taken.

preview note

If a user clicks on one of the notes it takes them to the time the note was taken on the timeline and graph.

select note

Filters: In order to load the data into the program, the users can select specific intervals of data based on filters. For example, some of the filters include things like the date the data was recorded, the name of the patient and the type of task the patient was doing. As the researchers analyze the data, they will add notes and tags such as “distraction”, “jitter”, “jump” or other tags that they decide are necessary. They will then be able to filter for and open data based on these tags.

filters filter date filter patient

Three Views: There is a “full view” where the researchers can view all of the cursor movements for the entire session with the patient. There is a one session view where the researcher can view only one trial at a time with lines indicating the pathway of the cursor. There is a parameters view where the researchers can view all of the specific parameters that were set during the session. A few of the most important parameters can be viewed in the full view and in the one session view, but a more extensive list of all the parameters can be seen in the parameters view.

Prototype

We built a working prototype of this program. It was a great learning experience to work with a team of engineers to actually build something that I designed. We used a local MongoDB database (with our data converted to JSON from the Matlab files), svg.js library for the graph and timeline and RESTful API to load the data into the program.

Read more about the design of the program here.

You can also check out our code and documentation here.

Reflection

The clients and researchers gave very positive feedback to our final presentation of our prototype. It was very rewarding to see that something I worked on would be used to help hardworking researchers make a positive impact in patients’ lives. It was a great learning experience to work with a client, a group of users as well as a team of engineers. I learned about understanding users’ needs, asking good questions, designing a creative experience, communicating my design decisions to clients and engineers and web app technologies. This project will be a memorable one because of the awesome people that I had the opportunity to work with and because the project will help in BrainGate’s mission to give patients with neurological disorders more independence.