For seven years, Benefacts provided Irish nonprofits and their stakeholders with free access to a high quality database of trusted, neutrally presented information about the nonprofit sector, harvested and aggregated from their regulatory disclosures and requiring no additional effort on the part of the nonprofits concerned.
In consultation with its public and nonprofit sector stakeholders, Benefacts devised five distinct processes for re-using public data. First, it defined the entities in scope by reference to international classification norms. It identified public sources of data for these including any limitations on the availability, quality or timeliness of public data. Having established and documented rules for processing it, Benefacts created governance processes to harvest, process and store the data, and in turn to push the data to online and other services.
These processes could be adapted by anyone interested in building a similar database, in Ireland or elsewhere.
Processes for producing and deploying the Database of Irish Nonprofits
- Specify entities in scope
- Identify sources of public data for entities in scope
- Establish and document data processes, rules and standards
- Harvest, process and store the data
- Push the data to online and other services
1. Specify entities in scope
For a functional definition of Irish nonprofits in scope, Benefacts followed the Johns Hopkins Comparative Nonprofit Sector Project (CNSP), whose approach to sector classification internationally was implemented in turn by the UN and Eurostat.
The CNSP recognised civil society organisations around the world using these five qualifications:
a) Organised or institutionalised to some extent
The organisation may have some form of institutional existence such as a legal charter of incorporation, or a constitution and rules which have been agreed by its members. Excluded are purely ad-hoc and temporary gatherings of people with no enduring structure or organisational identity
b) Institutionally separate from government
The organisation may receive significant State funding but to be considered a nonprofit it must be structurally separate from the instrumentalities of government, and not exercise governmental authority. Excluded are State Bodies that may be registered as charities such as the Arts Council or the Health Service Executive.
c) Non-profit-distributing
A nonprofit – for example a social enterprise – may accumulate a surplus in a given year but this must be reinvested in the basic mission of the entity and not distributed to the organisation’s owners, members, founders or governing board.
d) Self-governing
The organisation has its own internal governance procedures, controls its activities, and enjoys a meaningful degree of autonomy. Excluded are organisations that may be incorporated as nonprofit bodies and/or regulated as charities, but are wholly controlled by Government, such as the Institute for Public Administration, or Pobal.
e) Voluntary
The organisation engages volunteers in its operations and/or leadership on its board and/or through the use of volunteers. Excluded are organisations in which membership is required or otherwise stipulated by law such as professional associations that require membership in order to be licensed to practise a trade or profession.
2. Identify sources of public data for entities in scope
In Europe, re-use of public information regulations place an obligation on State bodies to make data open by default and design. This “open data” mandate was the basis on which Benefacts built the Database of Irish Nonprofits, going to nine distinct public sources and one non-public source of regulatory disclosures by nonprofits – many of which make routine disclosures to more than one State body.
2.1 Charities
In Ireland, charities are regulated by the Charities Regulatory Authority which maintains a public register of charities, updated periodically and available – with a subset of charity disclosures – as a downloadable file. About 40 registered charities were excluded because they are State bodies and therefore out of scope.
2.2 Charter bodies
A very small number of non-governmental nonprofits in Ireland (for example, the Red Cross) have their own legislation, and lay their accounts each year before the Houses of the Oireachtas after which they may be accessed from its digital library.
2.3 Companies
All companies including nonprofit companies (limited by guarantee without share capital or CLG) are required under Irish Company Law to make regular compliance returns to the Companies Registration Office. Not all CLGs belong in a database of public benefit nonprofits: for example, many are property management companies, which were excluded from the database by reference to their NACE code.
2.4 Sports bodies and charities availing of tax relief
In Ireland, sports bodies (which are not recognised as charities) and eligible charities including education institutions can avail of relief from certain taxes. A subset of these can also register for tax relief on donations, subject to approval by the Revenue Commissioners which publishes lists of eligible entities on its website.
2.5 Political parties
Political parties in Ireland are regulated under the Electoral Act, 1992. A quarterly-updated Register of these is published by the Clerk of the Oireachtas in the Library of the Houses of the Oireachtas. Benefacts began collecting and digitising this data including published financial statements from the Standards in Public Office Commission in 2017, and updated it periodically thereafter.
2.6 Primary and secondary schools
Primary and secondary schools are regulated in Ireland under the Education Act 1998. Registers of these are maintained live and published as downloadable files by the Department of Education. Most schools are also now registered as charities; some schools are also incorporated as CLGs.
2.7 Friendly, industrial and provident societies, trades unions
Cooperatives and other kinds of mutual societies are registered in Ireland under the Industrial and Provident Societies Acts 1893-2021, Friendly societies are registered under the Friendly Societies Act 1896 – 2021. Trade unions fall under the provisions of the Trade Union Acts 1871 – 1990. The legislation in each case specifies the kinds of public regulatory disclosures required. Benefacts analysed the subset in scope by reviewing their constitutions – some cooperatives for example are commercial in nature.
The Registrar in each case is the Registrar of Friendly Societies who provides access to the register, updated annually, on request. Some friendly and mutual societies are also registered as charities.
2.8 Higher education institutions
In Ireland, higher education institutions are regulated under the Irish Universities Act 1997, and funded by the Higher Education Authority, which maintains and publishes an up-to-date register of universities, colleges and other institutes of higher education.
2.9 Social housing providers
The Approved Housing Bodies Regulatory Authority was established in February 2021. It provides a register of approved housing bodies, replacing the list that was previously maintained by (and accessed by Benefacts from) the Department of Housing, Local Government and Heritage.
2.10 Public participation networks
Under local government reform legislation the Irish State has a system for registering small local groups to facilitate democratic participation and transparency. The majority of these local clubs, associations and societies are not found on any national register. Their names and purposes are published in each of Ireland’s local authority jurisdictions but the data is not available for public re-use. For its own research purposes Benefacts collected the data from websites published county-by-county, but with one exception did not re-publish it, and only used the data in anonymised form in its sector analysis reports. Fingal Public Participation Network cooperated with Benefacts in providing a live dataset of its members for inclusion on Benefacts public website.
Not included in the database
Missing from any of these sources is the untold number of local affinity, cultural, recreational or social clubs or societies in Ireland not listed on any national or local register. Some are the local member organisations of national governing bodies of more than 60 sports organisations, and about 4,000 local religious bodies listed in church records but not included on the public register of charities.
3. Establish and document data processes, rules and standards
Given the intensity of sector regulation and the extent of available regulatory data for Irish nonprofit organisations, Benefacts relied almost exclusively on mandatory filings to public regulatory authorities, and made no provision for additional (online) voluntary data disclosures.
However, data provided for regulatory compliance purposes is not always readily susceptible to re-use for other purposes. This is chiefly because
a) regulators do not always verify the data, nor do they publish all of the disclosures they receive
b) data is often produced in non-digital (e.g. image files) and in non-standard (e.g. financial statements, constitutions) formats
c) nonprofits can be inconsistent in their returns to different regulators
d) certain data is personal and subject to privacy legislation
Benefacts informed each regulatory authority of its plans, and in consultation with its two stakeholding forums, established these core business rules and processes to assure the quality and consistency of its database:
3.1 Validate and classify each entity
Having acquired lists of the entities in scope from all regulatory sources and verified they were in scope, Benefacts collected core registry data (alternate names, date of incorporation, address, institutional form etc) and assigned each one a unique 8-digit Benefacts ID, mapped against all of its other regulatory IDs. In this way, the Database of Irish Nonprofits was a register of registers (or “base register”) of the Irish nonprofit sector.
Following a consultation with its stakeholders, Benefacts produced a classification standard for Irish nonprofits that drew on established international norms. Using information provided in their constitutions or websites, Benefacts classified each nonprofit according to this schema.
The classification structure adopted by Benefacts had 12 categories and 54 sub-categories.
1 Arts, culture, media
1.1 Arts organisations
1.2 Museums & libraries
1.3 Heritage & visitor attractions
1.4 Media, film
2 Recreation, sports
2.1 Recreational clubs & societies
2.2 Agricultural fairs
2.3 Sports organisations
3 Education
3.1 Pre-primary education
3.2 Primary education
3.3 Secondary education
3.4 Vocational & technical education
3.5 Third level education
3.6 Research
3.7 Education support
3.8 Adult and continuing education
4 Health
4.1 Hospitals and hospices
4.2 Residential care centres
4.3 Residential mental health services
4.4 Health services and health promotion
4.5 Mental health services
4.6 Addiction support
5 Social services
5.1 Pre-school childcare
5.2 Family support services
5.3 Youth services
5.4 Services for older people
5.5 Services for people with disabilities
5.6 Services for Travellers and ethnic minorities
5.7 Homeless and emergency relief
6 Development, housing
6.1 Local development
6.2 Job creation
6.3 Social enterprise
6.4 Sheltered housing
6.5 Social housing
7 Environment
7.1 Animal welfare
7.2 Group water schemes
7.3 Environmental enhancement
7.4 Environmental sustainability
8 Politics, advocacy, law
8.1 Political organisations
8.2 Advocacy organisations
8.3 Civil and human rights
8.4 Legal services and citizen’s advice
9 Philanthropy, voluntarism
9.1 Philanthropy
9.2 Fund-raising
9.3 Voluntarism
10 International
10.1 International development organisations
10.2 International affiliation organisations
11 Religion
11.1 Places of worship
11.2 Religious associations
11.3 Diocesan, parish organisations
12 Professional, vocational
12.1 Trades unions & employer organisations
12.2 Chambers of commerce
12.3 Professional or sector representative bodies
3.2 Hash personal data
Directors’ birthdates and addresses, required for verification purposes, were machine-read for verification and analysis purposes but were inaccessible to Benefacts staff and were not stored or published.
3.3 Digitise content from image files
Using optical character recognition wherever possible, Benefacts extracted the narrative content (e,g. organisation purpose statements) from nonprofit constitutions. The contents of financial statements – a key source of regulatory, governance, narrative, financial and employment data – were manually captured using an online tool and a set of data harvesting rules devised by Benefacts for the purpose, validated by external financial reporting experts, approved by its Board and regularly updated.
3.4 Define data hierarchy rules
Regulators operate different compliance regimes in relation to the timeliness, completeness and external validation (e.g. by audit) of regulatory filings. Benefacts established a hierarchy based on these three quality criteria.
4. Harvest, process and store the data
‘Open data’ can be freely used, re-used and redistributed by anyone, subject to the requirement to attribute and share alike. The Open Data and re-use of Public Sector information Directive mandates the release of public sector data in free and open formats across the EU, and it provided the legal basis for Benefacts re-use of public data from these public sources.
In practice, despite the obligation on public bodies to maximise the availability of data in machine-readable form, only a minority of the 600 data fields harvested by Benefacts were readily digitally accessible. Financial statements and constitutions – the most extensive and reliable sources of governance, financial and employment information – are available in Ireland only as image files. Benefacts purchased these from the Companies Registration Office via commercial re-sellers under the terms of a data re-use license .
Benefacts developed its own data entry application (or ‘data harvester’) which ensured that the financial data was entered in a well-defined format, following a consistent set of taxonomies designed specifically to capture data from the financial statements of Irish nonprofit organisations. A team of professional data analysts was responsible for the digitising process. Benefacts uses a workflow management system to ensure that financial data received from third-party sources was prioritised, keyed in the harvester, reviewed and imported to the Database of Irish Nonprofits. The same workflow system was used to ensure that auditors’ opinions and legal object extracts were reviewed prior to storage in the database.
Benefacts financial data capture process recognised that financial reporting was undergoing a number of processes of change driven by regulatory reforms including Financial Reporting Standards, new provisions in the Companies Act 2014 permitting companies limited by guarantee to file accounts in abridged format and to file unaudited accounts (subject to scale threshold limits), the adoption on a voluntary basis of Charities SORP by some Irish charities, the small number of group companies, and the instances of prior year adjustments. As financial standards changed, the harvester was flexible enough to accommodate the changes.
Benefacts set up various processes to access data periodically from each public regulatory source, in every case doing what any other member of the public could do. Often, the data included returns made by some nonprofits to multiple regulators – for example in Ireland, social housing providers make separate regulatory returns to charity, company, charity tax and social housing regulatory authorities. To maintain its database live and up-to-date, Benefacts:
4.1 purchased a daily feed of regulatory compliance data and updated constitution and financial accounts files for c.10,000 incorporated nonprofits annually from the Companies Registration Office.
4.2 periodically downloaded the data file providing extracts from the regulatory returns of c.11,000 registered and de-registered charities from time to time updated by the Charities Regulatory Authority. It was the policy of the Regulator not to publish the financial statements filed by unincorporated charities (mostly church bodies, philanthropies and trusts) pending the introduction of regulations determining their reporting rules.
4.3 accessed the financial statements and updated constitutions of about 1,500 friendly, industrial and provident societies, trade unions and political parties from their respective registrars once a year, when fresh data was published.
4.4 downloaded the published files providing lists of schools, higher education bodies and nonprofits availing of tax relief from their respective registrars, updated periodically.
4.5 annually visited the websites in each of the 28 local authority areas where classified lists per county of public participation network members were published online by their local convenors.
5 Push the data to online and other services
Benefacts used the Microsoft Azure Cloud platform for all its data products and services. This provided the scalability, security and flexibility needed to develop fault-tolerant and reliable products, and to deploy them quickly.
Benefacts software stack consisted of a data harvester, the Database of Irish Nonprofits, the Benefacts free public website, Benefacts customised Analytics service, and a range of public API (including “Who Funds What”) and various Open Data services. Benefacts used Microsoft Azure Search to provide bespoke, customisable search facilities across all its products. The search technology included natural language processing, and geospatial search facilities.
5.1 Data harvester
The in-house ‘harvester’ was the platform used by Benefacts financial analysts to extract information from financial statements. This included OCR technologies as well as manual data entry, and evolved continuously over the life of the project.
The technical architecture was designed to ensure that, although hosted in the Azure infrastructure, the harvester software and the core database sat in an Azure Virtual Network, only accessible from within Benefacts premises or via a VPN. For added security, there was no external connectivity to the core database or its supporting services, not even from Benefacts own webservers.
Because Benefacts’ products and services did not access the database directly, subsets of the data were pushed out to NOSQL databases designed to support the specific requirements of each service.
5.2 Database of Irish Nonprofits
The Database of Irish Nonprofits was a large Microsoft SQL Server-based database hosted in Azure. Various in-house .NET services were used to extract data and publish it to databases supporting the websites and other services. Those were Microsoft CosmosDB databases, which integrated easily with website technologies.
5.3 Benefacts websites
The in-house data harvester, and Benefacts websites were a combination of .NET APIs, and Node.js. They ran as Azure App Services, which provided the required fault-tolerance and auto-scaling features.
5.4 Public API and Open Data Services
Benefacts public API and Open Data services, as well as other APIs built for specific end-users’ needs, were also Azure App Services. The public API was provided via Microsoft’s API Management service, which provided out-of-the box signup and product management facilities that allowed Benefacts to create combinations of different services and also to provide them as a single product.