Data Management for Geoinformatics A short course on good data management for taught postgraduate students in geoinformatics and related data sciences. . John Murtagh, UEL. Data Collection. Data sources. 3.
Data Management forGeoinformaticsA short course on good data management for taught postgraduate students in geoinformatics and related data sciences.
John Murtagh, UEL
1. Finding data – this involves searching and finding data that has already been released
2. Getting hold of more data – asking for ‘new’ data from official sources e.g. through Freedom of Information requests.
3. Collecting data yourself – This means gathering data and entering it into a database or a spreadsheet – whether you work alone or collaboratively
Sometimes data is public on a website but there is not a download link to get hold of it in bulk – but don’t give up! This data can be liberated with what datawranglers call scraping. More later…
“A piece of data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the requirement to attribute and/or share-alike.”
…is an online tool to make the process of extracting "useful bits of data easier so they can be reused in other apps, or rummaged through by journalists and researchers.“
Most of the scrapers and their databases are public and can be re-used.
A group for open geospatial data with an emphasis on use in teaching and research.
It not only provides access to the know-how and tools to discuss and create linked data and data aggregation sites, but also enables access to, and the creation of, large aggregated data sets providing powerful and flexible collections of information.
“a landmark site for academia providing a single point of contact for linked open data development.”
A number of startups are emerging, that aim to build communities around data sharing and re-sale. This includes Buzzdata and Figshare — a place to share and collaborate on private and public datasets — and data shops such as Infochimps and DataMarket.
An interesting Google subsidiary, Freebase, provides "an entity graph of people, places and things, built by a community that loves open data.“
While they may not always be easy to find, many databases on the web are indexed by search engines, whether the publisher intended this or not. Here are a few tips:
When searching for data, make sure that you include both search terms relating to the content of the data you’re trying to find as well as some information on the format or source that you would expect it to be in.
Google and other search engines allow you to search by file type.
You can also search by part of a URL. Googling for ‘inurl:downloadsfiletype:xls’ will try to find all Excel files that have “downloads” in their web address
Another popular trick is not to search for content directly, but for places where bulk data may be available.
(if you find a single download, it’s often worth just checking what other results exist for the same folder on the web server). You can also limit your search to only those results on a single domain name, by searching for, e.g. ‘site:agency.gov’.
For example, ‘site:agency.gov Directory Listing’ may give you some listings generated by the web server with easy access to raw files, while ‘site:agency.gov Database Download’ will look for intentionally created listings.
The information requested must be provided unless an exemption or exception allows the institution not to disclose it.
The request could be addressed to anyone in the University organisation, & there are only 20 working days to respond.
Freedom of Information (FoI) & Environmental Information (EIR) legislation provides the public with a right to access information (also research data) held by a UK public authority, which includes most universities, colleges, or publicly-funded research institutions.
You can make an FOI request using a website whatdotheyknow.com
The Act is mainly concerned with the UK Statistics Authority and applies only to data designated as Official Statistics. It defines how 'personal information' can be disclosed to an 'Approved Researcher' i.e. an individual to whom the Statistics Authority has granted access, for the purposes of statistical research, to personal information held by it.
Although the Act does not apply to individual researchers managing confidential research data not designated as Official Statistics, such researchers might wish to adapt the Approved Researcher model for access to confidential data.
This Act gives the public access rights to environmental information held by a public authority (including universities) in response to requests (similar to the Freedom of Information Act). Freedom of access does not imply free access. There are circumstances under which requests may or must be refused, for example if the data contain personal information.
This means gathering data and entering it into a database or a spreadsheet – whether you work alone or collaboratively.
Finding more Data using Google
You can search for CSV files on Google by typing +filetype:csv in the search bar. Searching for "South Africa +filetype:csv" will result in CSV files mentioning South Africa. You can try different other filetypes as well (such as: "xls" for excel spreadsheets or "pdf“)
As defined by Open Knowledge Foundation
University's Research Ethics Committee (UREC) has specific responsibility for institutional oversight of matters relating to ethics and governance in research undertaken by both staff and postgraduate research students that involves
personal sensitive data
or human material.
Ethics of carrying out research with data
Further information from the Quality Assurance and Enhancement officer firstname.lastname@example.org
The Data Protection Act (1998) covers personal or sensitive personal data, but not to all research data in general, nor to anonymised data or if the participants are no longer living
To gain access to police data or records you may be subject to a Disclosure or a DBS check (previously CRB check) which provides details of any criminal record data held on you.
Data Management for Geoinformatics by John Murtagh as part of the Jisc funded project TraD (University of East London is licensed under a Creative Commons Attribution Share Alike Licence