© 2017 NIJEL

Building the 2014 One D Scorecard

Jul 22, 2014 12:25pm | 0 Comment
D3 Landing Page

[This original post, from the Data Driven Detroit blog, was a conversation between us at NiJeL and the One D Scorecard's Project Lead Jessica McInchak. Many thanks to Ms. McInchak and the D3 team for putting together this post, and for again being such excellent partners on this project!]

Data Driven Detroit (D3) launched the 2014 One D Scorecard in May (read more about that here).  Today, we’re writing to share more about our process for making this interactive data tool through a Q & A with NiJeL, a team of data scientists and developers that D3 collaborated with to build out the Scorecard.  But first, a little context about the project:

In its third iteration, the new Scorecard makes exciting strides in data management and presentation.  Working with NiJeL, we focused our resources and energy on two key components of development: an administrative tool for data management on the back end; and an interface powered by interactive data visualizations.  We also revamped our data by updating and curating indicators, creating indices for each of the five Priority Areas, and incorporating a data deep-dive using original Opportunity Mapping research from the Kirwan Institute.

Let’s dig into four questions with Lela Prashad and JD Godchaux of the NiJeL team.

D3: Way back when we first started working together, we warned you that we were managing the One D Scorecard data through Excel workbooks, a single workbook per indicator (over 50 workbooks at one time!).   It was a bit of a data nightmare in a few ways, especially when it came time for annual updates or when we needed to compare individual indicators across geographies.  How did you sift through our data and come to the new centralized solution we’re using today?

NiJeL: Good question! We realized early on that we needed an easy way for D3 staff to update indicators as these datasets are updated, rather than all at once, say on an annual basis. We also understood that we wanted an automated way to add these new data to the One D Scorecard website as soon as an indicator is updated, and the only reasonable way to make this happen was to build a database to house all these indicators. So, we built a MySQL database and modified Xataface, an open-source software designed to add a simple admin interface for a MySQL database.

Once we had honed in on using these tools, we went through each Excel workbook and added each indicator to the MySQL database, slightly reorienting the data from the Excel sheets to make it easier to use. We then wrote two scripts, one to simply pull all of the relevant indicators for each region and package them up all together so the website could create the visualizations that it does, and another to calculate the five Priority Area index values and the overall One D Index score for each region.

Now, the staff at D3 can update any specific indicator by uploading a CSV (comma-separated value) file with any new data they would like to add. Once these new data are added, the web app will update the site visualizations once there is a critical mass of the data from each year to make the index calculations meaningful. We’re hoping this will be a big improvement!

D3: While older iterations of the Scorecard ranked regions based on their performance in a single indicator, we took that a few steps further this year using indices.  An index lets us roll up the individual indicators that comprise a Priority Area into a single summary score, and then roll up each of those five Priority Area scores to create a One D Index Score for each region, making comparisons comprehensive and straightforward.  Our favorite feature is how an index calculates on the fly and smartly recognizes when too few indicators for a given index have been updated to update the index itself.  Can you share a bit about the process for programming in these analytical features to the highly visual front end?

NiJeL: Of course! As you mentioned, we want to be smart about how we’re calculating the index scores for each of the five Priority Areas and the overall One D Index, and we want to do this in the context of new data being continually added to the database. To accomplish this goal, we programmatically look at the group of indicators in each Priority Area and determine if more than 50% of these indicators have data for the year in question. For instance, the Economy Priority Area has 7 indicators, so if 4 or more of these indicators have data available for 2012, then we would calculate an Economy Index for 2012.

However, for us to go ahead and calculate a One D Index and include the year in our visualizations, each Priority Area would need to surmount the 50% threshold. Once that occurs, the indicators and Priority Area indices are added to the visualizations and the data become available for download.

D3:  The design of the new One D Scorecard comes from Kat Hartman who drew inspiration from a JavaScript library for interactive data visuals aptly named Data-Driven Documents, D3.js.  What are some of the challenges of taking static wireframe designs to interactive visuals?  And mostly, which chart was your favorite to build and why?

NiJeL: Well, we had the distinct advantage of working with two individuals, one being Ms. Hartman and the other being D3 Project Manager Jessica McInchak, who both were interested in web interactive design and building interactive visualizations. Both actually contributed to the codebase for the One D Scorecard, which is an extremely rare thing for a designer and a project manager to want to do, but both Ms. Hartman and Ms. McInchak were excited to have the opportunity which made for a fantastic working relationship.

Ms. Hartman’s design for many of the elements in the One D Scorecard were inspired by other designs live on other websites, and so when it came time to build these visualizations, we did have some examples to view, though most were written using other tools like Raphael. We decided to use D3.js mainly because of its flexibility — it allowed us to be able to build the visualizations as closely as possible to what Ms. Hartman designed. The most challenging aspect of building to static wireframes, like the ones Ms. Hartman designed, is understanding the intended interactions and transitions between states within a particular visualization. It’s challenging as a designer to draw out intended interactions and as a developer to follow through on those intentions, but our close collaboration with Ms. Hartman and Ms. McInchak, minimized any differences we had on building the interactions as intended.

It’s tough to pick our favorite chart to build, since they all had their challenges and rewards, but we’d have to say the “array of pinwheels’ visualization (where viewers can see each region’s pinwheel in an array of rows and columns) was our favorite to build.




In this visualization, the pinwheels load such that the region with the highest One D Index value is placed in the upper left corner and the remaining regions are placed in descending order from left to right and top to bottom. Visitors can reorder the pinwheels by selecting a specific priority area to view from the “Organize By” drop down menu. Building this chart required extensive use of D3.js transitions, which allowed us to be creative in how we moved from one state to another. When visitors select a different Priority Area (or a different year of data) to view, we effectively run three separate transitions on each regional pinwheel. First, we change the color of each pinwheel slice, setting the Priority Area selected to its full opacity and setting the opacity of the other slices to almost fully transparent. At the same time, we change the size of the pie slice if a visitor has selected to view a different year with the time slider. Finally, we reorder the pinwheels in descending order based on the index chosen, but that transition only occurs after a 1 second delay to allow the first set of transitions to complete. Building these transitions in an attempt to clearly compare the differences across regions and indices was the most challenging and fun part of the development.

D3: The 2014 One D Scorecard presented a lot of opportunities to collaborate.  Not only did we work with your team around development, but we also partnered with the Kirwan Institute to integrate their Southeast Michigan Regional Opportunity Mapping initiative.  Kirwan’s original research was presented through static maps of the overall index scores.  What was your motivation and method for interactively mapping both the index and individual indicators?

NiJeL: The work that the Kirwan Institute contributed to mapping the Regional Opportunity Index for the Detroit region was fantastic, and fairly straightforward to map. However, we wanted to build a way for visitors to view the Opportunity Index in the context of the component variables that were used to calculate the index. To this end, we modified an open-source Javascript library, Crosslet, that combines the power of D3.js, Crossfilter.js, a Javascript library that allows for multi-dimensional filtering of large datasets in a web browser, and Leaflet.js, an open-source mapping library.

Crosslet is particularly designed to allow visitors to explore one or multiple variables to see how each is connected, and to see representations of those connections on a map and a simple frequency distribution bar chart. For instance, if a visitor selected the median household income variable, and then selected the income range of $0 – $50,000, they would see only the geographies (census tracts) that have median household incomes below $50,001. They would also see the frequency distribution of the Opportunity Index, high school completion rates, and vacant property rates. Clearly, the distribution of each of the other variable is skewed toward the negative end of the spectrum when we select that income range. However, if we click on the selected range and drag it toward a higher income range, we can see the frequency distributions of the other variables shift toward the more positive end of the spectrum along with income, and on the map we can see which census tracts specifically fit these new criteria. One can also select a range of any other variable on the map to further filter these data. We think that gives visitors a great entry point to exploring these data and drawing their own conclusions about the drivers behind opportunity in the Detroit metro region.

Thanks, NiJeL!

The new One D Scorecard is a powerful tool for its users to access data through visualizations, but it’s also a powerful data management system for D3 to maintain and scale these datasets into the future.  And we couldn’t have built it without the awesome team at NiJeL.  We’re already counting down the months until the newest annual data indicators are released so we can do our first update!

Check out the 2014 One D Scorecard at onedscorecard.datadrivendetroit.org, and NiJeL’s GitHub repository to explore the code driving the interactive data visualizations.

If you’re interested in talking more about code and collaboration for the One D Scorecard or beyond, connect with D3’s Project Lead Jessica McInchak at jessica@datadrivendetroit.org.

2014 Data Driven Detroit One D Scorecard Launches Today

May 21, 2014 9:10am | 0 Comment

We are proud to announce the launch of Data Driven Detroit's (D3) One D Scorecard! NiJeL partnered with D3 to develop interactive charts, graphs, and maps from their long running Scorecard initiative measuring metro Detroit’s performance in comparison to over 50 metropolitan regions throughout the U.S. across five priority areas: Economic Prosperity, Educational Preparedness, Quality of Life, Social Equity, and Regional Transit.

One D Scorecard

From our D3 partners:

Among its many new features, the 2014 One D Scorecard uses interactive data visuals to learn about our region.  For example, we find that our region – metro Detroit – scores higher on the Social Equity Index than on any of the other four priority area indices in 2011.  Using the bar chart, we unpack the index and individually explore the indicators that drive the Social Equity priority area.  There, we see that metro Detroit leads all other regions included in the Scorecard in “percent of owner-occupied housing” for Hispanics and ranks second in the same category for White households; therefore, positively boosting our overall Social Equity Index score.  And this is only one short chapter of the many data stories to be discovered!

D3 will be able to keep these indicators updated as new data come in now with their shiny new database and administrative tools.

We invite all to explore the Scorecard's interactive visualizations and see how your city ranks at http://onedscorecard.datadrivendetroit.org

You can also check out the One D methodology and datasets that make up the indicators and download the data.



NiJeL Pitches In on the Ocean Health Index

Oct 16, 2013 8:04pm | 0 Comment
We're pleased to announce the release of the updated 2013 Ocean Health Index, a project commissioned by Conservation International and other environmental and ocean advocacy groups. The Ocean Health Index evaluates the condition of marine ecosystems according to 10 human goals, which represent the key ecological, social, and economic benefits that a healthy ocean provides. A healthy ocean is one that can sustainably deliver a range of benefits to people now and in the future. A goal scores highest when the maximum sustainable benefit is achieved through methods that do not compromise the ocean’s ability to deliver that benefit in the future. The Index score is the average of the 10 goal scores. 

NiJeL updated the work of the talented GIS guru, Lee Altman, and the fantastic design team at Radical Media, mainly focusing on the updating the online maps like the one below:

Ocean Health Index Overall Index Score, 2013

The website and mapping interface show the goal score for each Exclusive Economic Zone (EEZ) for each of the ten goals, and the main mapping interface allows you to switch between each of the goals to get a sense of regional trends. You can also mouse over any country to see all scores for that country's EEZ, or click on the country to see a page visualizing that country's data.

For a full synopsis of the project, please visit the news release from Conservation International. Many thanks to Laura March and the team at Radical Media!

Riverkeeper Releases New Water Quality Pages Developed by NiJeL

Jul 3, 2013 2:52pm | 0 Comment
Riverkeeper website

Today, Riverkeeper announced our work with mWater on their newly updated water quality testing locations section of their website. We produced a series of dynamic maps, tables and charts for all of their water quality testing locations along the Hudson River from New York Harbor to the confluence with the Mowhawk River north of Albany.

Riverkeeper's online database includes water quality measurements for enterococcus, a marker for fecal contamination, and other water chemistry variables, like pH and salinity, in some locations back to 2006. Our work allows visitors to view these data over time, and to view more detailed information on any single sampling location.

You can see the results of our collaboration with mWater and Riverkeeper here. Also, you can read Riverkeeper's press release on the new site here.

New York State Sewage Discharge Map

Jun 30, 2013 1:27pm | 0 Comment
NY Sewage Map

On a webinar last Wednesday to demonstrate some of our new work with Riverkeeper, our colleague Tracy Brown used part of the session to update the community on New York State's implementation of the Sewage Pollution Right to Know Law that the helped pass last year. This law requires the NYS Department of Environmental Conservation (DEC) to notify the public of a raw or partially treated sewage discharge within 4 hours of the event, but the implementation of the law has been uneven at best.

Tracy then demonstrated just how uneven implementation has been. She directed her screen to the DEC Sewage Discharge Reports page, the only place the DEC is disseminating sewage discharge reports. On that page are two links -- one to to their Combined Sewer Overflow (CSO) Wet Weather Advisory web page where one can get the locations of CSO outfalls, but not information on actual discharges, and the other to an Excel spreadsheet. That file contains sewage discharge reports from "publicly owned treatment works (POTWs) and publicly owned sewer systems (POSSs)." One look at that file made us indignant.

First, it appeared to us that DEC compliance with the law to them meant pointing the public to a partially completed Excel file up on a webpage. We don't consider that adequate. How is the public notified that your spreadsheet has been updated? Next, the actual file is rife with spelling errors and missing data. The addresses inputed are often names of businesses without an address, or are altogether incomplete. How is the public supposed to use these data if they can't locate the discharge? Honestly, this dataset should be an embarrassment for the State of New York.

This experience got us thinking -- how could we make this situation better? It's unlikely that DEC is going to improve it's data dissemination practices, so we began thinking about how we could use these data build something the pubic might actually be able to use. Being a data visualization and mapping company, we decided to play to our strengths and build the NY Sewage map.

On this map you can view where sewage has been discharged, and get a quick idea of the volume of each discharge and how it was treated. Each discharge has an associated popup with other information from the DEC spreadsheet. In the upper right hand corner, visitors can also use the links to view data for the latest week or month of reports, or view all of the data since May 1, 2013, the day the law took effect. We also have links to the original Excel spreadsheet and a CSV that we created to populate the map.

If you're interested in the nuts and bolts of how we built this map, we'll be writing our process up shortly and posting to our blog, as well as posting the code up on GitHub. We'll share those links here in a future edit.

We would very much appreciate any feedback you may have, especially if you have ideas on how to integrate the CSO discharge data or other data that we maybe missing. We'd also like to hear from you if you're using the map and have ideas to make it better. Thanks!!