Designing Service Metrics for Public Facilities at the NYC Department of City Planning
Written by Axel Bax
As a Siegel PiTech PhD Impact Fellow, I spent the summer working with the Capital Planning and Support team at the New York City Department of City Planning (DCP). My project was to design service metrics for facilities in the city to determine how well they are serving their local population.
The Capital Planning and Support team supports NYC agencies and the public by making data and community needs related to city facilities more transparent and useful for various planning processes. Because each facility type is overseen by a different agency, the available data vary, limiting the applicability of service metrics across facilities. Therefore, my goal was to create a methodology for evaluating service of a type of facility that could be generalized to other facility types, even when underlying data vary.
I explored the available data to see which facilities have the most information available, both about the physical features of the facility and the ways in which people interacted with it. I chose to focus on public libraries because Capital Planning and Support already has a plethora of very recent data on the Brooklyn public libraries for me to explore, and as someone who studies literature, I have a soft spot for libraries.
Method
My task was to determine useful service metrics for Brooklyn Public Libraries; the service metrics should quantitatively determine whether an individual library is accessible and meeting the needs of the surrounding community, or if investments are needed to improve it. These service metrics could be fixed, where a certain threshold must be met by each library, or they could be relative, such that libraries are compared to each other to determine what underperformance is. When determining a service metric, it is helpful to think about what population a library serves. There are a variety of ways to define the population: it could be everyone within a 15-minute walk of a library, everyone within a buffer of a ½ mile radius, or the average number of people that use the subway stop near the library. I tried each of these definitions to determine the population that lived in proximity to a library.
Another challenge to designing a service metric is deciding which of the many statistics (or combinations thereof) indicates that a facility is operating effectively. Libraries serve a variety of purposes in their communities, such as lending books or providing computers with wifi for those without access. A library in an area with a larger teenage population might have more programming and events aimed at teenagers or might be open later in the evenings or weekends so students can utilize its resources. A library might be trying to do all these things, but the actual facility may be outdated and in need of repairs.
Even if we know which statistic to use in designing a service metric, we still need to define what constitutes an appropriate ‘quality threshold’ for that metric. For example, if we chose the number of books checked out each month (i.e. circulation) as an indication of how well a library is serving its community, how do we decide what circulation levels determine that a library is doing a ‘good’ job? We could normalize each library based on the population nearby, so that a library with higher population density around it is expected to have higher circulation. More people implies more resources should be available, and since we know the population density of Census tracts in New York City, we can estimate how many resources a library should have based on its location and the number of people living nearby.
I did this comparison between the number of people living within proximity of a library, and a number of library metrics available: circulation, holdings, new card applications, number of events hosted, attendance at these events, visits (total number of people entering the library), number of meeting rooms available, computers available, interior area, age of the building, and number of years since last renovation.
However, it turns out that there is little correlation between any of these library usage metrics and the number of people that live within proximity of the library. This lack of correlation holds true, even if library event attendance and local population are broken down into different age demographics. This suggests that people often visit libraries other than the one closest to where they live, either because it offers better access to certain programs or amenities, or maybe the use of a library depends on factors of the library or the population that we were unable to measure. This makes it difficult to evaluate how well a library serves its population since we cannot geographically define who goes to a particular library.
To overcome this challenge, I compared the libraries to each other without concern for the population in proximity, since it seems that people are willing to travel to any library. For each metric, I looked at the distribution of scores across all of the libraries, determined the mean and the standard deviation, and identified underperforming libraries as ones which fell more than 1 standard deviation below the mean. However, this approach is not very interpretable: what if a library performs average across all metrics, but underperforms in terms of holdings? It seems that this library still has people visiting, even if it has less books. Is this really something that needs to be addressed?
To get a better understanding of which metrics are the most useful, I chose to include a qualitative approach to choosing metrics. To do this, I examined the Community Board Budget Requests (CBBRs), which reflect the funding needs that communities identify for themselves (as opposed to agencies identifying for the communities). Each budget request contains text explanations about why they think a particular facility needs more funding. I filtered for budget requests related to libraries, then used a topic modeling approach, which identifies words and phrases that are thematically similar, then groups entries together based on its themes. This approach helped me identify that most library budget requests tended to either be related to the physical state of the building (e.g. renovations) or the programming offered (e.g. events for children or later hours for students). I grouped together metrics about the physical state of the building (building age, years since last renovation, number of meeting rooms available, number of computers available, and interior area) and programming metrics (circulation, holdings, new card applications, number of events hosted, and attendance at events), leaving number of visits as its own general metric. For each group, all of the metrics were averaged together before identifying outliers as before.
The final service metric that I created identified libraries that are underperforming in terms of physical building metrics, programming metrics, and general visits. Therefore, libraries are scored on a 0-3 scale based on how much they are underperforming relative to each other. This methodology can be used to then identify where to prioritize or target improvements.
Axel Bax
Ph.D. Student, Information Science, Cornell University
Impact
While I focused on creating service metrics for just the public libraries of a single borough, my task was to create a proof of concept that can be reapplied to other boroughs and, ideally, to other types of facilities in New York City. My finding that population does not always correlate to the service metrics of the library is interesting and suggests that the inter-connectedness of New York City means that people may travel anywhere in the city to visit a library, or any other city facility that they might use. Additionally, my use of Community Board Budget Requests to identify the needs of a community are easy to replicate for many other types of facilities. Capital Planning and Support can utilize these techniques and expand upon them with other types of facilities in New York City, so that there is a consistent way of evaluating the needs of communities with regards to city facilities in a consistent way.