The U.S. Recreation Information Database (RIDB) API has been substantially upgraded, providing programmatic access to an authoritative information source on the U.S. government’s parklands, historic sites and recreational attractions.

While the RIDB aims to be a single point of information access spanning data managed by multiple federal agencies, including the Department of the Interior, the National Park Service and the Smithsonian Institution, the underlying data set and API are both still in development, so some data may not yet be current or may be missing attributes.

The idea is that through wide distribution, an ecosystem can grow up around the API, both contributing more accurate information and demonstrating demand for the resource.

RIDB API Overview

Data endpoints available through the RIDB include:

• Organizations: The government agencies that contributed data to the RIDB.
• Recreation areas: All recreation areas managed by the U.S. government. There are also endpoints for media, links, addresses, activities and events held at each of the recreation areas (also for facilities and campsites).
• Facilities: Facilities at each of the recreation areas.
• Campsites: Data on campsites attached to each of the facilities.
• Permit entrances: Permit entrance information associated with each of the facilities.

An entity relationship diagram is included in the API docs to visually represent how all the data items are connected.

Data can be returned in JSON or XML format (JSON is the default), while a CSV dump of the entire data set is also available.

API calls are free, and while there is no license, documentation clearly stipulates that the data can be used and published in any manner without informing any government agency, although users are encouraged to acknowledge the RIDB API as the source of the data.

Additionally, the privacy policy states that API registration and downloaded data requests are stored solely for the purpose of monitoring developer needs, and anonymous cookie information is collected in order to improve site usability.

Those who are looking for a reliable data source for their commercial products or for citizen advocacy and use cases may still be disappointed by any commitment to a service-level-type agreement in making the data available. The API is covered by public laws governing federal data quality that note that while the data set is meant to be a general reference and will be updated whenever possible, it does not make any commitments to the data’s reliability, accuracy or timeliness.

Challenges in Creating an API Built on Aggregated Government Data

The RIDB API is fairly unusual among U.S. government APIs, as it relies on data sets owned and shared by more than 30 agencies. It is also an essential aggregated data set for many nonprofits and third-sector organizations working in partnership with government agencies to maintain and manage national parklands.

Dan Parker, one of the architects at software company Nsite responsible for managing the updated version of the API, explains the collaborative approach across government sectors:

We met with RIDB stakeholders during the requirements process to better understand their data input and export needs. This included both data consumers and data providers. We also opened our development server to the public for beta testing and feedback. Based on this feedback, we were able to make several improvements to the API before its initial release.

Because the RIDB API had been released as a SOAP API in version 1, the revamped version did not have to revisit issues like the data entity nomenclature being used to define agencies across government. There are maintained lists of all government agencies with APIs and developer portals, and Project Open Data also manages a list of all government agencies and their progress toward opening their data. But what is less clear is whether all data sets and APIs across government define and describe their own government agencies in the data sets in the same way. So it is uncertain whether other aggregated data set API projects similar to the RIDB would have problems matching data entities around something as seemingly simple as how government agencies refer to themselves.

Jereme Monteau, co-founder and CTO of Trailhead Labs, has been part of a team spearheading the open source, open standard project OpenTrails. This project encourages recreational parkland walking and hiking trails to use a common data standard so that as the ecosystem around digitizing parks and recreation support services grows, new products can quickly get on with creating value for end users rather than managing disparate nomenclature and data formats across data sets.

Monteau describes how something as simple as different ways to describe agencies can impact on innovation opportunities:

If you look at the OpenTrails and the RIDB data sets, there is a little bit of overlap (for example, at the organization level of who oversees parklands, etc.). How we link them together is an interesting question.

In the RIDB API, if you look at the organization, there is a parent ID so you can see the hierarchy of agencies, so that is the way that hierarchy is described. It is the same issue you see in how organizations are described in, say, the APIs from AngelList, LinkedIn, CrunchBase. … You could be dealing with five different ways for companies to be described.

We have been trying to standardize that within the parks and recreation domain because for us, government agencies are just one of the stakeholders. For example, the National Park Service has regional not-for-profit partners. We have been doing some work to map that landscape out, because maintaining the data set of outdoor information crosses all these different groups.

The API breaks with federal open data policy by not including a metadata schema in data.json format. And unlike some government agency providers such as the Department of Commerce, the RIDB has not chosen to also create an apis.json metadata schema.

"The API does not use apis.json or data.json to document the metadata or schema," Parker says. "We used Slate as the base for the template to document these items and published the documentation to our GitHub presence. Each resource is explained along with a data dictionary and an entity relationship diagram."

While the lack of a metadata schema is a worry, one of the advantages of this approach is that developers can make pull requests directly from the API documentation/GitHub portal.

Initial Feedback on the RIDB API

Monteau discusses his experiences providing feedback on early releases of the new API version:

Following the response from [the] tech community around the lack of an API component in the RFP, the project team asked us to take a look at the beta release. They sent us the links, and I started to poke around with it and sent them my feedback. I had just come off developing the first version of our API, so I was pleasantly surprised at their first pass at it.

It is exciting because it is at the federal level and involves 31 agencies and massive public land holdings. It’s an impressive data set.

The new version feels like a more modern API: The documentation there is good, basic authentication, and it is easy to get an API key. From a developer perspective, it is much easier to get started with than it was before. The previous API was an XML schema-based approach and felt outdated. The data is well-organized, it has a nice hierarchy and includes all of the things you would expect to see in a national camping data set.

The data they are publishing does not make use of the OpenTrails standard, but the new version does make it easier for someone like us to aggregate it with all the OpenTrails data we have coming in. So it means for us, we can provide a one-stop source for recreation at any level (federal, county or municipal).

It is still early for the RIDB. We have talked to them about adding more geo information. There are no trails information, but there is info on permits that you might need. So there is potential there to use something like OpenTrails to link it to the permits. The data set is pretty light on geospatial info; only a few of the entities have latitude and longitude, and there are no park boundaries.

Monteau sees that these are the sorts of improvements that can be made by an active developer ecosystem around the RIDB: “That's what’s exciting about this API being available in this way,” he says.

Where Does the RIDB API Sit in Recreation.gov’s Mission Statement?

Ongoing management of the RIDB API faces a number of particular challenges. The RIDB API is part of a $150 million 10-year request for proposal tender managed via external contractor. The U.S. Department of Agriculture is preparing to release a new RFP tender for the next 10 years of management.

In October, an early version of the RFP documentation outlined the vision for a provider to maintain Recreation.gov’s assets, which would include management of the RIDB API. The documentation at that time did include diagrams highlighting the central role of an RIDB API in drawing together disparate data sets and enabling and ecosystem of partners to be generated, as well as transactional APIs to deliver services like permit approvals and park entries via API:

But outside of that diagram, the wording of the documentation significantly played down the management of the API.

Despite community advocacy led by API and recreation industry influencers, including API evangelist Kin Lane and Alyssa Ravasio, CEO of Hipcamp and chairperson of the open data advocacy group Access Land, the latest draft of the proposed RFP does not substantially reframe the RIDB API as the central tool enabling Recreation.gov to fulfill its mission of being a one-stop support service.

Moving Into General Use: The Missing SLA Question

One issue that the RIDB, as an experimental, early public release product, has been able to sidestep for now is how to resolve the reliability of government-provided open data released via API. The RIDB documentation specifically recognizes both economic development and citizen customer service as two of the key goals in making the API available.

For economic development, the hope is that the RIDB API will jump-start new industry opportunity and spark a wave of products that are built on the underlying data set.

For citizen customer service, the hope is that the data set makes the United States’ natural parklands more accessible to visitors and their owners, the American public. The RIDB API could even be the key technology that enables government service delivery in parklands, for example, for booking campsites or paying for permit entry fees directly via the API, thus improving the efficiency of government service delivery to people.

But in order for public products to be commercially viable, or for citizens to best participate in the enjoyment and maintenance of public lands, the RIDB API will need some sort of commitment to accessibility and reliability.

This speaks to a question of how governments will provide service-level agreements (SLAs) around the opening up of government data sets.

In a recent post on Government Technology, Ravasio describes this as a new paradigm of wholesale and retail service delivery. In her view, there needs to be government partnerships with private companies to provide SLAs that ensure accessibility, quality and robustness of open data sets, in much the same way that governments sign service contracts with private utility companies and transport providers to ensure that reliable energy, water and transport services are available to the public. Then those private partnerships create ecosystems with other providers that create retail products (either direct to consumer or to industry markets) that are built on those data sources.

The current RFP blurs these two functions into the one project, which Ravasio says “ideally” would be procured separately.

She wrote:

Wholesale is foundational infrastructure: databases, hosting, point-of-sale, and APIs. Retail is the public-facing Recreation.gov website, mobile application and potential third-party applications. …

Wholesale and retail provide a framework for how the government should structure the business model for the contractor and third parties. When a customer books a campsite using a third-party app, a set portion of the booking fee goes [to] the contractor for the infrastructure that supports this transaction (wholesale), and the other part of the fee goes to the third-party application (retail). When a customer books through the official Recreation.gov website or mobile application, the contractor receives the entire user fee.

Meanwhile, Monteau acknowledges the work of the Access Land coalition in moving this conversation forward. He cites the work that Trailhead Labs’ OuterSpatial platform initiative is doing as an example of the wholesale approach to using government open data via API, not unlike how Transport API in the U.K. has agreements with a range of transport providers across London and then packages up each of those data sources into the one API that developers can use to build hyperlocal products and services:

Our vision is that we are a platform. We want to see more players in that space (that is, the retail side) that have a more API approach to things.

We’re excited by this opportunity because it’s hard. When you look at the landscape, there are good parallels between transport and parks. It is very similar. There are a bunch of agencies within jurisdictions; it is very difficult for them to provide a comprehensive service like this. It’s very hard to collaborate one on one, so there needs to be something else. We are excited to be that platform and provide that function.

Four Initial Use Cases

A myAmerica Developer Summit held recently in Washington, D.C., shows how developers are already building new products off the API.

Twelve projects were initiated over the two-day event, with projects from community groups (Topophi), private companies (Booz Allen Hamilton, IBM and NIC), U.S. government teams, and startups like Stamen Design, Hipcamp and Trailhead Labs.

The projects included:

Trail Assist

With more than 3,000 parks listed with text attributes in the RIDB, the Trail Assist project, led by Stamen Design and Trailhead Labs, aims to enhance these listings by adding geographic and media data to the information returned by the RIDB API. By harvesting Instagram pictures, OpenStreetMap contributions and CartoDB API integrations, the team was able to start building tools to enable crowdsourcing of trails data so that new visitors can better plan their hikes, while also encouraging park visitors not to use unofficial trails that add to the erosion of parkland. (Interestingly, it was decided that an emergency response version of the data that did include these "bootleg trails" to aid in evacuation would also be created, but in the versions shared among the community, these tracks would be left out.)

Side Stream API

E-government transactions processing provider NIC saw several staffers volunteer in teams at the developer summit to build several tools, most notably the Side Stream API. This API aims to allow developers to supplement RIDB API data with additional Internet data sources, including Wikipedia, travel websites and social media. While no doubt complicated to build if it is to aggregate multiple private API data sources, the NIC team built a mapping example as a proof of concept and will share a public repository of the work created so far.

Following the event, NIC released a press release noting:

Collectively, the solutions presented by NIC during the federal hackathon demonstrate that by working together, government and technology companies can create a richer repository of information about public lands and waters and provide greater access to those seeking recreational activities. The apps will soon be released into GitHub as open-sourced software and will be available for anyone’s use.

NatLand Facts

Using the RIDB API, Hipcamp built a content platform to facilitate sharing of additional facts and information about parklands in order to encourage families to visit and enjoy national parks.

WanderList

In one of three Booz Allen Hamilton team projects, WanderList draws on the RIDB to allow users to curate their own outdoor adventure lists.

Trailhead Labs’ Monteau sees the RIDB API as a significant example of the way open data can be used in both a citizen engagement and an economic development context. “Open data is transitioning from being an abstract general concept and into giving rise to more vertical applications for open data," he says. "This shows how open data can be more tangible and useful, in this case, all the way through to the park visitor.”

Developers can register for an API key or review the GitHub repository of open source projects already created out of the myAmerica Developer Summit.

Chris Taggart, Founder of OpenCorporates believes that their API will be the principal way that their data is consumed by end users within the next five years.

Speaking at API Strategy and Practice / APIDays in Berlin today— and only three days after the release of Version 4 of the OpenCorporates API — Taggart shared some of the origin and unique challenges facing the open data business that aims to build the world’s most comprehensive underlying dataset of company information.

“Government information on companies is siloed in various registers,” says Taggart. He believes that by de-siloing that data, more useful analysis can be performed, for example, analyzing the health and safety reputations of companies where transgressions are stored in one dataset and the company data in another (as occurs in the UK).

Taggart — a former journalist who ran magazine publishing companies — used the investment he made from the sale of a previous business to start OpenCorporates, and has now reached the stage of having a sustainable business model.

“A company is a legal construct and we have assembled this data from company registers around the world,” Taggart says. Key to the approach taken is that OpenCorporates automatically ingests all its data from primary sources, so there have been no manual imports. Instead, a variety of webscraping, PDF scraping, CSV dumps, and API ingestion is used, with specific code written where needed to pull in the data. While datasets are then normalized so that company names can be matched up globally (thus helping identify, for example, foreign branches of home companies), the underlying data is not changed.

“Identifiers is a pretty critical issue,” says Taggart, a challenge also being faced by other company information data businesses like Owler, which have an internal mantra of ‘death to DUNS’ (the proprietary company identifier used by Dun and Bradstreet). Taggart sees proprietary identifiers leading back to the very core problem that OpenCorporates is trying to solve, that is, open access to business information: “The Dun and Bradstreet identifier is a proprietary identifier with lock-in and no visibility,” says Taggart.

OpenCorporates’ mission goal is to aggregate every bit of company-related public data and match it to the relevant company, says Taggart.

“We have already made a good start, but a very small start: including trademarks and corporate structure. Our current focus is on U.S. business licenses (we have a product being released in a month or two), financial licenses, and government gazettes."

OpenCorporates' roadmap is driven in part by user demand, and by the core team identifying things that are structurally useful. “For example,” says Taggart, “getting data on U.S. businesses is difficult. Businesses must register at the state level but that doesn't have any information on industry codes, who the people managing the business are, no trading addresses or trading names, no financials. But if the business starts to employ people, or open up a shopfront, well, you need licenses for that. So these are disparate datasets, but by pulling this stuff together we can build one complex structure that gives you a much better idea about things.”

Data Quality in Proprietary Business Information Services

Compared with proprietary business information data systems, Taggart sees OpenCorporates as better suited to providing high quality, accurate information.

He points to six common data quality issues commonly seen in proprietary, vendor-locked company information data services. In his presentation, Taggart said:

• Data accuracy is problematic:“Most proprietary business data has been re-keyed, so you have errors happening that are not being picked up.”
• Gaps in data
• Lack of granularity: ”Legacy systems of business data are often unable to provide the level of detail needed.”
• Errors go unchecked: Taggart notes that OpenCorporates receives around 20 emails each day pointing to errors in their corporate datasets, and that is when the data is being drawn from official sources. Fraudulent data can be more easily discovered when open sourced and made available in the way OpenCorporates does.
• Black box/no provenance:“We always say where we get the data from, and what year it was. Often proprietary data is the same data that goes round and round between various company purchases. Sharing data on the date and source of company data also means it is better able to provide context, and allows appropriate comparisons to be made between datasets.”
• Isolated: Proprietary identifiers create barriers to sharing data openly and prevent others from helping improve data quality.

Version 4 of OpenCorporates API

Taggart announced that Version 4 of the OpenCorporates API was released earlier this week, noting: “We expect that in five years, all of our data will be accessed solely by API.”

New features in the latest version include:

• companies can be search by their registered address
• searches for companies can begin with a given phrase (e.g., ‘Barclays Bank’)
• advanced filters allow searching by multiple jurisdictions (e.g., 'Ireland and UK'), by country, and by inactive and branch companies; or that filter results so that only nonprofit companies are shown, or can be excluded from private company searches
• filtering of companies by officer’s addresses, dates of birth, or position
• greater date searching
• a revamp of how industry codes are allocated to companies that allows for more granular search filtering.

While some API calls do not require an API registration, the new version also allows those with an API key to make calls to get company officers’ addresses and dates of birth programmatically.

The new searches help surface company data for a variety of use cases. For example, users can discover what businesses operate in a particular building, or can assess the likelihood of fraudulent record-keeping, for example, by filtering for only those companies with an active company officer over 105 years of age (Taggart says there are about 400 in the UK alone).

A  recent blog post on the OpenCorporates website by Tony Hirst shows some of the powerful company data information functions that can be performed with the API, including mapping what other companies are owned by one company’s largest shareholders, what companies (and their subsidiaries) receive payments from city governments, and which corporate entities might share the same location (or even trade under multiple names at the one location).

The OpenCorporates Business Model

Taggart says that being “Open is critical here: everything we do is available for free, without registration, on the web.”

While having many links with non-profit organizations like the Open Knowledge Foundation, Taggart doesn’t want people to be confused about the business model: “We are a for-profit company with a strong social mission. This is also critical to quality. By being open in this way we can ensure the quality the world needs.”

He says OpenCorporates drew on successful open source approaches as a way to define its innovative business model: a share-alike or paid service.

“If you use the API, anything you use it in has also got to be open sourced,” explains Taggart, defining the share-alike part of the equation. “Or you can build proprietary tools and you pay for API access,” he adds, explaining the profit-generation-side. “This allows us to be a public-good oriented company and, at the same time, to be sustainable. Now, the next challenge is how do we scale this up: how do we handle the sales cycle?”

Taggart says with their approach to date, they have been able to prove themselves to their paying clients and with Taggart’s own initial investment, this gave the room for the business to grow to a point where it is now sustainable. “It has been difficult but it is now working,” he said.

Developers can read the API documentation online, begin making API calls from the website, or sign up for an API key.

A new tool is available to perform gender analysis on any large data set. Gender Gap Grader uses a databank of world names to analyze any data set by likely gender, taking into account regional differences that might mean that a name used in one part of the world is typically male, while in another region it's more likely to be female.

“The conversation around gender is pretty much center stage,” co-founder Elena Rossini said in a presentation at API Strategy and Practice/APIDays in Berlin last week, “especially now with the Clinton Foundation, the United Nations and the Bill & Melinda Gates Foundation working together to spotlight gender equality with the #GenderDataRevolution initiative.” Rossini said she is inspired by one of Hillary Clinton’s remarks that "gender data does not just measure progress, it inspires it."

Gender Gap Grader provides an API toolkit based on the NamSor name recognition software. Believing that “APIs can be a powerful tool for social change,” co-founders Rossini and Elian Carsenat have created the API tools specifically to help analyze gender in large data sets, with a goal of publishing gender gap estimates in all industries at a granular level. Studies so far include one of airline pilots, which found that only 7% of the world’s pilot workforce is female.

“Names can be very complex,” said Carsenat. “Our mission is to support the gender data revolution by allowing any individual to run an analysis of names to see the ratio of men and women.”

Gender and racial inequality in the fast-growing, economic power engine of the tech sector has come under increasing scrutiny in the last six months, as many begin to ask why an emergent industry that encompasses new disciplines in mobile, data science and the cloud should be repeating the same power and inequality differences seen in more traditional labor markets.

Recent IT studies include a data science survey that found that women data scientists earn $13,000 less than their equivalent male counterparts, while interactive data published on The Wall Street Journal using the U.S. Department of Labor Statistics API shows that female Web developers earn 79.4% of what male Web developers earn, with female computer support specialists hardly faring any better, earning 79.8% what their male counterparts earn.

Rossini showed an angel investing study by Gender Gap Grader that analyzed data using the AngelList API and revealed that among the 650,000 names in the database, only 9% were women and that of the 19,000 angel investors, only 7.4% were women.
 

At the Berlin conference, Rossini and Carsenat also discussed preliminary findings of a study to be released in June that reveal that women account for between 13% and 33% of people who have their research published in the sciences.

Rossini pointed out that there is a “powerful economic case that when a company is more diverse, they are more profitable.”

Joelle Emerson, CEO and founder of Paradigm, is familiar with the gender gap tool and told ProgrammableWeb:

Analyzing the gender divide in big data sets enables us to see concretely how gender gaps play out in different contexts and can inspire people in positions of power to act. Marc Benioff's recent announcement that he is evaluating pay across Salesforce and giving raises to close existing pay gaps is a great example. (Outside the workplace context, FiveThirtyEight has done some interesting research.)

In the workplace alone, there are many areas beyond pay in which we can and should be analyzing gender-based disparities. On the internal side, we can look at promotion rates (are women being promoted at a slower rate than men?) and performance evaluation data, and on the hiring side, we can look at whether there are any gaps in the rate at which men and women pass through each stage of the recruiting process.

It's exciting to see a growing number of tech companies thinking about how they can use data to better identify and respond to gender gaps in their workforce.

4 Examples of How the Gender Gap Grader API Could Be Used

While the Gender Gap Grader website highlights a few large-scale studies of gender analysis, there is also a wealth of opportunities for using the Gender Gap Grader toolkit across business and government.

Here are four examples of how the tool could be used:

Reducing the Risk of Women Leaving the Tech Industry

Analyzing payroll differences and career trajectories can help tech companies identify inequalities that may prevent them from leveraging the benefits of having a diverse workforce with women in leadership positions.

"We support and encourage companies to analyze their payroll and bonuses for systemic gender inequities," Alaina Percival, CEO of Women Who Code, told ProgrammableWeb. "We also recommend analyzing career progression rates to see that on average women are being promoted at the same rate as male counterparts."

Percival added:

Data is showing that women are leaving their tech careers midcareer at a rate of 56%, and they are reporting feeling stalled in their careers at the same time their male counterparts are reporting their careers taking off. Analyzing for systemic inequalities can bring to light hidden biases and quickly reduce the exit rate of women from tech.

Analyzing Company Ownership by Gender in Specific Industries

The Gender Gap Grader could be used in conjunction with the OpenCorporates API to identify which industries have the most inequitable leadership and where women leaders are most recognized.

We're looking forward to what @gendergapgrader come up with! #apisberlin#gendergaphttps://t.co/8Ko3C8x5cC

— opencorporates (@opencorporates) April 24, 2015

Reviewing Gender Differences Within a Large Corporation or Government Body

Last year, the city of Palo Alto, California, used Gender Gap Grader’s API along with other NamSor APIs in a RapidMiner integration tool to analyze the differences between male and female employees within the city authority. City authorities, corporations, government bodies and other enterprises could use the Gender Gap Grader toolkit to review their payrolls and management teams to analyze whether women are being paid equally to their male colleagues, and to measure the level of women in management positions within the organization.

Analyzing Economic Development Initiatives by Gender

Governments at all levels, investment funds and philanthropies all have funding programs available to support new economic development from microenterprise and seed funding to series level investments. These initiatives could apply the Gender Gap Grader to analyze whether they are distributing their funding disproportionately.

Developers can access an API toolkit, which includes a RapidMiner Starter Edition and NamSor APIs. An API key is not required to experiment with the tool, but developers can register for more professional plans with higher-volume processing. Professional-level toolkits also include diversity analytics by country, culture and ethnicity to enable additional equity analysis.

APIs are increasingly being released by city authorities around the world as a programmatic way for community organizations and businesses to interact with city open data. Cities are running hackathons or civic hacking events to encourage reuse of city datasets. APIs are discussed in terms of their benefits to civic engagement through greater transparency, for more efficient delivery of government services, and as an enabler of a new wave of local industry innovation. The growing international focus on the “smart city” — in which open data, e-government, and real-time sensor feeds contribute to more automated and sustainable city functioning — will also rely heavily on APIs in order to make much of that agenda possible.

But the truth is, cities around the world are only starting on the API journey. Many have commenced with open data portals that were published with an ad-hoc collection of historical data released, and then left to stagnate as uptake was limited. Others focused on civic hacking events that gave rise to sporadic events that built some app prototypes and not much more. Few cities are focused on creating transactional APIs that would enable citizens and local businesses to engage with services directly via API, with perhaps Open 311 being the only example of civic engagement and service delivery provided via API. While there are many successes across the globe, there is also much more work to do.

Here is a look at four city examples from around the globe. We look at some of the key trends demonstrated by these case studies, and highlight the progress that is emerging and the challenges ahead.

Barcelona

Barcelona continues to win international awards as one of the most innovative smart cities in the world. Some of their recent acclaimed projects include:

• A CityOS project that enables tech makers and devs to prototype sensor and API-enabled projects using an area of the city as an experimental lab
• The business incubator BDigital program for apps development 
• New models of procurement for city services, like Citymart (which posts a city’s urban problems, and then facilitates submissions from businesses who are encouraged to create innovative solutions, rather than the usual city tendering process where the city completely defines the service they want to contract in advance).

ABOVE: Some of the smart city awards won by Barcelona City

APIs are published across various government departments, including:

• Area Metroplitana de Barcelona: a collection of agencies that publish transport, environment, land use, and business data. API documentation is published in Catalan and includes datasets of publications, city news, research study results (on transport, the environment, etc.), current projects and works, urban plans, and datasets that outline city infrastructure and resources (beach locations, bus stops, taxis, business services, etc.). Documentation is a webpage explaining the various query parameters, etc., that can be made, with links to the available JSON datasets. An API key is not necessary to make the calls.
• Sentilo: Sentilo is Barcelona city’s open source, Internet of Things (IoT) smart city infrastructure platform aimed at enabling access to sensor and actuator data (monitoring temperature and air quality, garbage collection, parking, pedestrian flows, etc.). A REST API provides access to the city’s sensors and actuators. Documentation includes example code snippets and an active Google Groups forum is used to respond to developer issues.
• Open Data BCN: An open data catalog publishes 118 datasets in ODATA format, and 41 in XML format.

The city hosts Apps4BCN, a directory site of available apps built using city data and APIs. The directory includes reviews, where anyone can register as an expert user of apps and contribute, which in turn gives them a profile page. (So savvy app developers could use these experts as early adopters for user feedback on their future creations.)

Barcelona also invests in a Code for Europe initiative that currently has a Fellowship based at the city, working with internal staff on implementing Open 311 API standards so that citizens can register and communicate local issues such as broken city infrastructure directly via an application.

Barcelona has a number of partnerships aimed at fostering standardization in the use of APIs so that civic tech solutions created in another city are transferable locally. This includes CitySDK, iCity, and Open-DAI.

The city is also currently hosting a Smart City App Hack event. Unlike the hackathons that the city has hosted in the past, the Smart City App Hack allows dev teams to build apps over a period of months instead of a weekend.

Barcelona’s open data portal averages around 10,000 website visits each month.

Challenges

• Navigation: While there is a lot of activity across the city’s operations, it takes some navigation to try and find APIs available for developers. There is no single point of access to discover the city’s APIs.
• Documentation: Where APIs have been made available, there is no interactive documentation and no client libraries are provided for any of the APIs or datasets.
• Developer marketplace: While Apps4BCN has an app user social network, it is difficult to locate a similar network or marketplace of developers who have expertise or interest in building civic tech solutions.
• Terms of use: There are contradictions in Barcelona’s Open Data terms of use. Under Creative Commons Attribution 3.0, data can be used for commercial use and can be altered. However, the same terms of use also say the data is governed by Article 8 of the Spanish Act 37, which says that data cannot be altered in reuse. The city also reserves the right to charge for the data if it wants to in future.
• Data ecosystem: While there are some links to other websites and partners that also host API data for the city, this is fairly ad hoc, and there is no mention of things like national statistics that might include some Barcelona-relevant datasets, and no mention of the work of the Catalan regional statistical agency, which also offers Barcelona-relevant data and has an API Developer Portal.

New York City

New York City has 8 APIs available via its developer portal, including a geoclient, an Open311 API, an event calendar, city government spending data, and several news and data feeds, including transport.

In addition, New York City has a Socrata-based open data portal, which means that all of the published datasets are available as APIs using Socrata’s Open Data API (SODA) standard. These are intended for community and hackathon use without a registration key, although it is possible to build small scale applications with up to 1,000 request calls per hour (with a registered key).

A roadmap clearly articulates the city’s planned data releases up until 2018. Current datasets are often provided in close-to-realtime. For example, building permits data includes data up until the previous working day.

Data visualizations and stories of how city APIs and open data are being used by business, researchers, and the community are showcased on the open data portal.

An annual NYC Big Apps competition has replaced the city’s hackathon approach, and aims to foster more sustainable civic tech product development over a six-month period. This includes social media profile pages for teams competing in events and progress updates on their product development. Team profile pages show what APIs they are using in their civic tech solutions. These pages could be used as a developer marketplace or directory for those seeking to collaborate with city API developers beyond the life of the competition.

Via the NYC Big Apps website, the city does signpost to a wider data and API ecosystem, including state open data sources, and proprietary API data services relevant to New York City, like Foursquare, Yelp, The New York Times, Mapbox, and SeatGeek. Interestingly, wider government open data APIs such as the Federal U.S. Census Bureau APIs are not referenced.

New York City’s developer portal currently has an average of around 6,000 visits each month, with half of visits coming from searches predominantly for the Geoclient mapping API.

Challenges

• Data ecosystem: Within the NYC dev portal, the wider NYC data ecosystem is not fully signposted. For example, the NYC Metropolitan Transit Authority’s data feeds are not linked or mentioned on the dev portal. While technically, the MTA is a public authority overseen by New York State, that is not something that many developers or businesses would necessarily know. This results in the API for the Department of Transport referencing data feeds for traffic updates and truck routes through the city, without making any reference to MTA’s public transport system, or to other modes of transport such as the city’s bike share program or the city’s car share programs.
• API Registration: Accessing API documentation is a little convoluted. Users must sign up for an API key first and then fill out some forms as if they are building an application. Once that workflow is in process, the interactive sandbox with the API documentation can then be used to test API calls.

• Lack of activity: On the API Developer Portal, all of the requests from users for new APIs and comments on site feedback have gone unanswered for over a year. Where certain datasets are linked from the API to the wider open data portal (for example, looking at bike parking locations across the city) requests for updates are already 6 months old with no sense of engagement from city representatives.
• Terms of use: API terms of use don’t say anything about use of NYC APIs in commercial applications. The terms stipulate that data can only be drawn from the city’s API endpoints if they are actively being used in the developer’s application. The final clause of the terms give the city the right to change the terms of use at any time, with immediate effect. All applications that use NYC APIs must have a privacy policy with their end users that explains how the application will use, share, and transfer data. The policy must also be published on the application’s landing page and in any marketplace where the application is available for download.

Continue reading on page two to learn more. 

Continued from page one. 

Melbourne

Australian city open datasets can be accessed via the Australia Government’s open data CKAN catalog (which enables all datasets to be called via the CKAN API). Alongside this, open data platform OpenDataSoft has local agents keen to sell their product to Australian city authorities. Meanwhile, Melbourne has opted for Socrata as their data portal, which — like New York City — means all datasets are available as Socrata Open Data API endpoints.

To aid discoverability, the city has introduced a simple flowchart for data use, based on the potential role of the API developer-consumer. Potential API users interested in social change, entrepreneurial opportunities, community participation, and urban planning are routed to their own separate data catalog landing pages with a curated set of data chosen for them to initially explore.

In the past six months, Melbourne’s open data portal has seen an average of 5,000 monthly visits.

Challenges

• Timeliness: Melbourne’s data supply is quite patchy in regards to the timing of the data that is being shared. Three datasets that are suited to being regularly updated demonstrate the variety in updating machine readable data:
  • The development activity monitor that lists all of the city’s major commercial and residential property development was last updated in November 2014.
  • The city’s bike share data is provided as a live feed in realtime so that it can be used in third-party apps.
  • The immensely popular dataset that users sensors to count pedestrian activity (with over 20,000 visits to date) is available as hourly counts at 30 locations across the city … but the dataset is only updated online once a month.
• Terms of use: Melbourne’s open data policy is available as a footer menu option, but cannot be read online. Instead, you have to save the Word document to your computer and then open it from there. The policy confirms that data is provided for free, is reusable, and can be used for commercial product development. The city is committed to releasing all data in machine-readable format, but no assurances are given regarding frequency of updates or reliability of data access.
• Lack of developer engagement: While this is a fairly new initiative, at this stage there isn’t really any focus on cultivating a developer ecosystem around using the APIs generated from Melbourne’s open data catalogue. City-led participatory events have focused on crowdsourcing data — for example, a biodiversity initiative last year encouraged citizens to post photos of local animals, insects, birds, and plants to an open data set — but not so much with application development, as of yet. There is also no official showcase of apps made using Melbourne-related open data.

Singapore

Singapore has a longstanding partnership with UP Singapore, a non-government collective that regularly runs hacklab events aimed at fostering API skills development amongst community organizations, independent developers, and entrepreneurs making use of Singapore APIs. From May 11th to May 16th, UP Singapore will host the Smart Health CoLab, aimed at encouraging skills development around civic apps focused on citizen health. A week-long series of events includes workshops on city health data, smart health services, and a group hack day.

The Singapore developer portal lists four key sets of APIs covering environment and weather, maps, traffic, and library services.

A marketplace is available that shows what apps have been created using government data sources and APIs.

Singapore’s data website, which includes the API developer portal, sees on average 30,000 visits per month. The majority of API searches are for machine-readable data to create dengue fever maps.

Challenges

• Terms of use: Use of the Singapore APIs for commercial applications require developers to register their app first. All uses of APIs in applications must include mention of the dataset source. As with all API government terms of use, there are no guarantees on provision of data or on accuracy or uptime availability. In addition to the catch-all Singapore Government open data terms of use, each API has its own terms of use.
• Documentation: The majority of API documentation is in PDF format. Some APIs require registration first before documentation can be viewed. The mapping API documentation is provided as non-interactive HTML pages, with some code samples included.
• Developer marketplace: Despite the multiple initiatives aimed at supporting developers to create apps from Singapore’s APIs, there is no direct developer showcase or easy way to see which local businesses are building civic tech solutions. The app showcase provides links to app marketplaces but not directly to the developers’ websites.

The Road to City API Usability

These four examples demonstrate that even amongst leadership cities around the world, there is still a lot of work to be done to make APIs usable and reliable for community and business use.

Some of the key issues these city API strategies demonstrate include:

Lack of Uniform Approaches to Metadata

It is difficult in many cases to see a standard format for metadata being prepared for each API. Socrata (being used in New York City and Melbourne) provide a simple metadata table for each of their datasets. Singapore also has its own table formatted metadata. 

Limited Recognition of Data Ecosystems

One of the difficulties facing many startups wanting to enter the civic tech space is the fragmented nature of government responsibilities. Data and APIs may be managed or owned by a particular tier of government. For example, in Australia, it is more common for state governments to manage both crime and transport data in most jurisdictions (while in the U.S., these are city datasets). City data catalogs don’t tend to make any mention of this split in responsibilities, which could leave developers searching fruitlessly for relevant APIs within the city catalog without any signposting or reference to other government bodies that could be publishing city-relevant data via API.

Old School Documentation

City API documentation is challenging. Much of the API documentation cited in this research was still in PDF format. Several city websites have tables with links that promise to lead to API documentation, but that instead lead to another landing page with links back to the original page. There is limited use of interactive API documentation where developers can test API calls. Client libraries are pretty much non-existent.

Lack of Developer Marketplaces

While city authorities recognize the economic development potential of releasing APIs, and encourage the use of APIs for civic transparency, few are showcasing the expertise amongst civic tech developers, startups, and businesses that can assist new entrants to make use of city APIs. While several cities include showcases of apps that have been developed for the city, these often link directly to app marketplaces without highlighting the developer teams or providing direct links to the developer websites. In addition, app showcases are now facing the same discoverability issues as open data catalogs themselves: i.e., with an increasing number of apps listed, it is harder to filter search queries to surface those apps of specific interest to the end user.

No Guarantee of Supply

For now, city APIs use terms of use policies that request some acknowledgement of the API source in the end use, and that allow for commercial and community use of data in applications, with few rate limit restrictions on API calls. Some cities require that developers register any commercial application as part of the API terms of use.

Overall, data provided by API is still very much seen as a non-essential service being provided by city authorities. Unlike utility supply such as water and energy, or public transport infrastructure and services, governments are not yet prepared to treat data supply as an essential part of their service delivery and their terms of use make no promise of reliability, accuracy, or regularity of data supply.

No Acknowledgement of Equity Impacts

To date, few APIs are building-in an equity dimension to ensure that new government service design addresses the city-level inequalities that occur between those with access to local resources and those who traditionally have been marginalized. For example, transport data available via API tends not to include parameters on accessibility within the main release. Instead, this is seen as a separate dataset, often released at a later date. In addition, open 311 APIs and building permit API data are rarely discussed in terms of how developers could overlay this data supply with any available census data in order to integrate a socio-economic context into the data feed.

Will City API Supply Grow and Thrive?

Initially, city APIs have focused on providing machine-readable access to static open datasets held by city authorities, but real-time sensor feeds are rapidly being published, with the future hope that transactional APIs will be part of the next wave of API availability at the city level.

So it is exciting to see global cities dip their toes into these API waters. Hopefully, a new era of digital government service delivery, increased civic transparency and accountability, and sustainable innovation will emerge from the increasing provision of city APIs.

To aid the potential rollout of city APIs, new international projects aimed at encouraging open API standards are also blossoming. These projects may help speed up the pace of civic tech innovation, as solutions created in one city can be deployed in more settings. City APIs will require some flexibility to meet local cultural and geographical needs, but standards around nomenclature and data model design can enable civic tech startup agility across city and national borders.

This is just an initial look at city government API strategies and how they encourage discoverability of APIs for use by citizen groups, advocates, entrepreneurs, and businesses. For now, many cities are aligning their API strategies with their open data policies. This undersells the potential that APIs can play in building a city-as-platform model. There are many sparks of promise being shown by the four cities outlined here, and many others around the globe, but much work to do as well to create a solid API infrastructure that can adequately engage developers and end-users in building new tech solutions for the cities we live in.

The Sunlight Foundation has announced that it is rolling out new websites, mobile applications, a unified API and an enhanced data explorer tool in the coming months. The Sunlight Foundation is also in the process of developing two major platforms, one that focuses on the federal government and one that focuses on state governments.

The Sunlight Foundation, a nonprofit that promotes transparency in government, is organizing and consolidating its data under two flagship platforms so that both federal and state government data will be comprehensive and easier to use. Building two platforms also makes it possible to build a single, unified Sunlight API that will allow programmatic access to all of Sunlight's data.

ProgrammableWeb reached out to James Turk, director of Sunlight Labs, who provided some additional information about the flagship platforms in development and the upcoming unified Sunlight API. Turk explained that the Web development team is building on Sunlight's existing technical stack, which is largely Python and Django. He also said that the team is "evaluating technical options for pieces yet to be built."

When it comes to the existing Sunlight APIs, the organization’s ultimate goal is to replace all of them with a single, unified API. "There will be one base URL with many endpoints," said Turk. "Our aim is to eventually replace all of those APIs, but our immediate focus is on Open States, Influence Explorer and Congress." He also said that "the API will use simple key-based authorization. Users will be able to use their existing Sunlight API keys."

Quite a few APIs that provide programmatic access to Sunlight Foundation projects are available:

• Capitol Words API– This provides access to word-frequency data, which is used in the Capitol Words project and Scout.
• Congress API v3– This is a live API that provides information for legislators, bills, votes, real-time notice of hearings, floor activity and more.
• Docket Wrench API– This provides access to metadata for documents, dockets and agencies. It also provides access to clustering and text analysis tools featured in Docket Wrench.
• Influence Explorer API– This provides programmatic access to campaign contributions and lobbying records.
• Open States API– This provides access to legislator, bill and activity information for all 50 states as well as Washington, D.C., and Puerto Rico.
• Political Party Time API– This provides access to the raw data created by the Sunlight Foundation based on fundraising invitations collected in Party Time.
• Real-Time Federal Campaign Finance API– This API provides "up-to-the-minute" campaign finance information. This includes PACs, federal candidates and other groups that file electronic documents with the Federal Election Commission.

In addition to a unified Sunlight API, an enhanced data explorer tool is set to be launched later this year. The new data explorer tool will allow users to perform complex queries across data sets. "The initial concept is simply to provide researchers and interested parties easy access to tabular data," Turk said.

The Sunlight Foundation is working hard on building the upcoming new websites, mobile applications, unified API and enhanced data explorer, all of which are planned to be released later in 2015.

"This new approach stems from our desire as an organization to put what we've learned over the last nine years to work to create tools that will be easier to use and accessible to more people," Christopher Gates, president of the Sunlight Foundation, told ProgrammableWeb.

For more information about the Sunlight Foundation and its projects, visit SunlightFoundation.com.

The U.S. Food and Drug Administration has announced the openFDA Developer Challenge to encourage developers and researchers to use their medication adverse reactions and product labeling APIs. The openFDA APIs reflect an emerging U.S. government API strategy approach, but to date developer adoption has been muted. It is hoped that launching challenges will ignite a community of interest around the APIs.

OpenFDA has released three APIs for adverse events, labeling and enforcement reports, but it is probably better to think about them in terms of the three public health risk sources for which each API is monitoring: medications (drugs), medical devices and foods.

Adverse events include all occasions where a medical impact is recorded for either a drug or a medical device — for example, if a patient reports feeling dizzy after taking a particular type of medication. The product labeling API is specific to drugs; it records data on the FDA-approved labeling of all prescription and over-the-counter drugs. Enforcement reports track all FDA-instigated product recalls for drugs, devices and food.

The new challenge approach is aimed at generating a sense of curiosity in what is possible with the APIs and encouraging researchers and developers down a path of growing expertise in analyzing big data on public health risks from medications.

The two challenges focus on the medications data, with one challenge on adverse reactions and the other on product information. For adverse events, devs are encouraged to identify a spike in adverse events for a particular medication, normalize against other public health data sets such as emergency admissions, and then seek out an algorithm to automatically identify and justify such spikes.

For the product labeling challenge, devs can visualize black box warnings on drugs, develop a language model and then analyze whether specific classes of drugs approach consumer education in differing ways.

Once challenge participants have worked through each challenge level, they are encouraged to post their work on a subreddit, where the community is invited to comment and give feedback.

It is a novel approach to engaging with developers and researchers that falls somewhere between a certified developer program, where expertise with an API is rewarded by encouraging proficient developers to market their services to those who want to build products using an API, and a speed hack, where developers are encouraged to complete specific tasks (such as visualizing complex queries) for an API.

The openFDA developer portal is highly lauded by API strategists as a strong example of how government agencies can provide API documentation and engage with a developer community. Resources include:

• A dedicated Twitter account for communicating to users
• Interactive documentation that allows search queries to be tested on the page
• Clear links to a developer forum on StackExchange where devs are invited to post their questions about using the APIs
• A GitHub repository where much of the dev portal is available as open source
• A status page documenting current uptime status of the APIs

The site clearly stipulates that it is in beta stage and is not intended for production or clinical use.

To date, the site has seen an average monthly engagement of around 20,000 website visitors, according to SimilarWeb’s website statistics.

This is comparable to other medication- and related-health APIs and interactive data websites. For example, the U.S. National Library of Medicine's interactive data sources for medications, RxNav, and the Pillbox website that lets users search for a pill name based on a visual description (for which an API is also available) have similar monthly visit numbers. E-health data aggregator Human API also has similar monthly visits to its website, suggesting that overall, e-health is still very much in its nascent stage, with lots of opportunities to start leveraging newly released APIs.

Mapsense, a startup specializing in tools and services for understanding big location data sets, has announced the launch of Mapsense Developer, a suite of tools that developers can use to build location-based applications as well as analyze and visualize massive data sets of location-based information. Mapsense Developer includes a D3-based mapping library, built-in geographic data sets and Mapsense API. The company has also launched Mapsense Enterprise, a set of sophisticated geovisualization, analysis and segmentation tools for terabytes of data.

The Mapsense team consists primarily of highly skilled engineers, some knowledgeable in large-scale data processing and computational geometry. The team spent two years developing the Mapsense platform and working with a select group of enterprises, government agencies, banks and other organizations, helping them analyze and leverage massive amounts of geotagged data.

The Mapsense platform is cloud-based and designed to be highly scalable, with infrastructure capable of streaming billions of rows of location data. Mapsense utilizes a custom-built spatial data store with dynamic vector map tiles, allowing modern browsers to render raw map data instead of relying on static images. The Mapsense API allows the platform to be integrated with enterprise systems and consumer-facing products. Developers can use the API to build location-based applications and advanced map visualizations that include features such as hovercard, choropleths, interactive legends and animations.

ProgrammableWeb reached out to Erez Cohen, co-founder and CEO of Mapsense, who provided some insight into the Mapsense platform. He explained that smartphones, sensors, satellites and other Internet-connected devices are generating constant streams of location data and that there are billions upon billions of lat/longs. Cohen said that "many businesses don’t know how to use all this data" and that Mapsense helps users "find commonality across data sets using lat/longs."

Cohen also explained that the company designed and built a suite of APIs that powers the Mapsense platform and is now allowing access to some of those APIs. Many organizations such as Mapsense, TigerText and the U.S. Census Bureau use a "dogfooding" approach when it comes to APIs, developing APIs to power their platforms and later opening the platforms to developers via the existing APIs.

One of the key features of Mapsense is a suite of built-in geographic data sets, which includes public data sets and data sets curated by Mapsense. At the time of this writing, built-in geographic data sets included:

• Mapsense Earth— Tiles that are suitable for use as a basemap. Based on OpenStreetMap and Natural Earth.
• Mapsense Demographics— Demographic data from U.S. Census Bureau 2008-2012 American Community Survey five-year estimates and TIGER/Line Shapefiles.
• Mapsense Earth— Tiles that are suitable for use as a basemap. Based on OpenStreetMap and Natural Earth.
• Twitter— Every geocoded tweet from the past week. This data set is streaming data only.
• World Airports— Every airport in the world. Data provided by OurAirports.com.
• Farmers Markets— Every farmers market in the U.S. Data provided by the U.S. Department of Agriculture.
• Earthquakes— All earthquakes reported by the U.S. Geological Survey. This data set is streaming data only.
• Meteorites— All known meteorite landings. Data provided by NASA.
• Hurricanes— Every Atlantic hurricane since 1851; every Pacific hurricane since 1949. Data provided by the NOAA National Hurricane Center.

Cohen explained that Mapsense features queryable vector map layers that contain underlying meta information. For example, zooming in and clicking a feature in a Mapsense Earth map will show the specific metadata for that feature. "For some cities in Europe," Cohen said, "Mapsense has more buildings metadata than Google Maps." Cohen also said that Mapsense basemaps are based on OpenStreetMap and that the platform actually renders its own basemaps; Mapsense does not rely on any external basemaps.

Cohen told ProgrammableWeb that the company plans to add many more built-in geographic data sets in the coming months and that eventually users will be able to push their own data sets to the Mapsense platform. Cohen said he would like to see Mapsense become a GitHub-like platform where users can store, share and use location-based data sets.

The Mapsense platform featuring free, open source developer tools is available now. Features such as the ability to upload custom data, embeddable maps and nontechnical user-friendly UI are coming soon.

"With the advent of cellphones, IoT and connected cars, billions of lat/longs are now being generated by companies across a wide swatch of verticals," said Cohen. "We've developed in-house spatial-data stores optimized for dealing with the analysis and visualization of this sort of data; the implications of these tools are starting to resonate strongly with both the Fortune 500 and the developer community."

For more information about the Mapsense platform and suite of free, open source developer tools, visit Mapsense.co.

Microsoft has released a preview of the Microsoft Health Cloud APIs. The APIs represent an early step in Microsoft’s goal to provide programmatic access to Microsoft Health products and associated data. Apps and services that integrate with Health Cloud APIs lead to extensive health and fitness features (e.g., observations, insights, personalized recommendations, fitness coaching, individual baseline development, and more). During the preview period, the APIs remain read-only.

Microsoft launched Band in October 2014, and promised the developer community that the Microsoft health ecosystem would exist as an open environment accessible across platforms. The Microsoft Band SDK, launched in February, marked Microsoft’s first move toward its promise. The SDK was updated in April and presented at the Microsoft Build Developer Conference in April. The Health Cloud API preview invites the broader developer community to incorporate Microsoft-collected health data into third-party services and apps.

The API preview includes four APIs: profile, devices, activities, and summaries. The profile API provides the user profile for an individual user. The devices API provides the devices (Microsoft Band, phones, etc.) associated with a particular user and the details of such devices. The activities API provides activity data for a determined amount of time or user ID. Activities include sleep, run, workout, bike, guided workout, and golf. The summary API gives hourly or daily summaries for a certain date range. The API can summarize data for steps, calories, distance, and heart rate. As previously mentioned, the APIs are read-only during the preview period.

To supplement the API preview, Microsoft announced the Microsoft Band Web Tiles preview. The Web Tiles preview makes health data easier to view with a “glance.” Once a Tile is authored, the view is supported across platforms (iOS, Android, and Windows). Developers can author via their favorite text editor or utilize the Web Tiles Authoring Tool that provides user-friendly UI-based, step-by-step guide to create Web Tiles. Keep an eye out for more developments to come in the Microsoft health space.

Data is playing an increasingly-prominent role in business as more records are made more easily available through open data portals and public APIs. Writing for DigitalGov back in April, Bill Brantley discussed the importance of government data to the US economy.

Brantley references a TechRepublic interview with Ian J. Kalin, where the CDO for the Department of Commerce discussed his reliance on government data while working in the energy sector.

“If we didn’t have the Census and Bureau of Labor Statistics producer price indices for certain manufactured goods, we would never have been able to build some of our most successful products,” claimed Kalin.

Combining data in innovative ways can provide valuable informational products and services, and this is a solid argument for encouraging the DOC to release more data via APIs. For example, comparing National Oceanic and Atmospheric Administration information with Census information allows the creation of a model for the aquatic commerce system. This could then be complimented with Bureau of Labor Statistics and information from NASA’s Earth observations to complete the picture.

This information could be used to determine commercial opportunities or environmental impacts, or to track any series of metrics to determine the health of the industry. The establishment of Census and the Patent and Trademark Office in the Constitution shows the importance of information in the past. Now, the spirit of open access continues as information is made available through government APIs.

With Indiana enacting the Religious Freedom Restoration Act earlier this year, the Open States team wanted to access the data to provide the text to the public. However, in a subsequent post on the Sunlight Foundation’s blog, software developer Rachel Shorey discussed some trouble the team encountered while trying to scrape the data.

Bill text is usually provided by the state as a PDF, and Open States provides a link to that specific bill on the state’s website. The organization gets the majority of its information from Indiana through its legislative API, MyIGA, which requires an API key even for PDFs. With no way for Open States users to download the PDF this way without a key, the nonprofit resorted to scraping an ungated download link from the bill’s page.

However, it seemed like the link URL was generated on the fly and required a document-specific hash value that the team needed to find. Some custom code was able to locate this ID, but it left the scraper needing to visit a slow site multiple times for a single bill, often with multiple timeouts, leaving the scraper crashed or hung at most attempts.

This method for generating link URLs on the go could be viewed as a preventive measure against website scraping, despite the nature of open data, and may have applications elsewhere. Reverse engineering the document ID returned nothing about how the IDs were constructed, so the team returned to the API.

The state legislative service failed to return Open States’ calls about the matter, but terms of service allow Open States to use data gained via the API key to create an app. So the team used available data to construct a simple proxy URL that retrieves the document for download, circumventing the hash-generated URLs.