Annotated-directory-big-data

From Earlham CS Department
Revision as of 12:39, 14 October 2011 by Charliep (talk | contribs)
Jump to navigation Jump to search

This is an annotated directory of public, freely available, "large" data sets. For now they are in no particular order.

Metadata

http://www.drewsullivan.com/database.html

Google ngrams

  • URL - http://books.google.com/ngrams/datasets
  • Description - The ngram databases on which Google's ngram viewer is built. A variety of corpora are available, e.g. by language, the "Google Million", English fiction, etc. Each set contains a list of ngrams, frequency, and date information.
  • Curator - CharlieP

MusicBrainz

  • URL - http://musicbrainz.org/doc/MusicBrainz_Database
  • Description - In a nutshell, the musical equivalent of IMDb, except editable by anyone
  • Complete Size - Without information about editors, 3.47 GB
  • Format - PostgreSQL "COPY TO" format
  • Curator - Jahelton07

freedb

World Cubing Association Database

Large Data Sets on AWS

  • URL - http://aws.amazon.com/publicdatasets/#1
  • Description - A list of large data sets on Amazon's AWS, more data sets within the four links in the list.
  • Download: 2 - 250Gb, Various formats
  • Curator - Twright09

Starcraft 2 Hit Analysis

Starcraft 2 Combat Analysis

Twitter Users by Location

The AOL Search Data

Center for Disease Contol Data

  • URL - http://www.cdc.gov/nchs/
  • Description - The National Center for Health Statistics under the CDC has a lot of nice downloadable datasets on mortality and health in its data warehouse. You can download the complete 1998 ICD-9 and 2000 ICD-9 here as well (the coding manual for cause of death used by many state and federal agencies) along with a guide to the ICD-9. Data is in Lotus 1-2-3 and ASCII formats.
  • Curator - jrhurst08

IRS Statistics

  • URL - http://www.irs.treas.gov/tax_stats/index.html
  • Description - The IRS Statistics of Income program tracks all sorts of data but always in summary form. You can find all sorts of information on non-profits (including the database of tax data for approved non-profits - downloadable in ASCII fixed-length form) and other stats on income earned, migration and foreign taxes paid. All are downloadble often in spreadsheet form. Some databases are VERY big and the site is VERY slow.
  • Curator - jrhurst08

National Oceanographic and Atmospheric Administration - Storm Prediction Center

  • URL - http://www.spc.noaa.gov/climo/
  • Description - This NOAA site includes a nice archive with downloadable files on tornadoes and tornado deaths since 1950 . There's also data on hail and wind damage data.
  • Curator - jrhurst08


Freebase

  • URL: http://wiki.freebase.com/wiki/Data_dumps
  • Description: Full data dumps of every fact and assertion in Freebase,an open database of the world's information, covering millions of topics in hundreds of categories.
      Set - Quad dump is a full dump of Freebase assertions (quad dump) as tab separated utf8 text.
      Set - Simple Topic Dump is a tab-separated file containing basic identifying data about every topic in Freebase. 
      Set - TSV per Freebase type is a tab-separated file for each type in Freebase, suitable for loading into spreadsheets.Each line represents an instance of a Freebase type and columns represent the available properties for the type.
  • Size: Total- 6.0 Gb compressed with bzip2
      Set - Quad dump: The Link Export is approximately 3.5 Gbytes compressed with bzip2 (35 GB uncompressed) 
      Set - Simple Topic Dump: approximately 1.2 Gbyte compressed with bzip2 (5 GB uncompressed). In June 2011, there were over 22 million rows. 
      Set - TSV per Freebase type: The full download is approximately 1.3 Gbytes compressed with bzip2.The browseable set contains approximately 7500 TSV files in 100 domains.
  • Format: This is a complete "low level" dump of data which is suitable for post processing into RDF or XML datasets.
  • Schema:
      Set - Quad dump:The format of the link export is a series of lines, one assertion per line.The lines are tab separated quadruples, <source> (mid - a machine-generated id), <property> (a particular kind of quality of the entity mentioned in the "source" column), <destination> (holds the name of a namespace), <value> (a key within that namespace). 
      Set - Simple Topic Dump: mid,English display name, Freebase /en keys, numeric English Wikipedia keys, Freebase types, short text description
      Set - TSV per Freebase type: type, type's description
  • Curator: eosergi10

"DBpedia"

  • URL:http://blog.dbpedia.org/2011/09/11/dbpedia-37-released-including-15-localized-editions/
  • Description: The dataset release is based on Wikipedia dumps dating from late July 2011.DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data.
  • Size:The dataset consists of 1 billion pieces of information out of which 385 million were extracted from the English edition of Wikipedia and roughly 665 million were extracted from other language editions and links to external datasets. Totoal is approximatly: 2.5Gb.
  • Format:RDF triples
  • Schema:Every DBpedia resource is described by a label, a short and long English abstract, a link to the corresponding Wikipedia page, and a link to an image depicting the thing (if available).

If a thing exists in multiple language versions of Wikipedia, then short and long abstracts within these languages and links to the different language Wikipedia pages are added to the description.

  • Curator: eosergi10

IMDB

  • URL: http://www.imdb.com/interfaces
  • Description: All the data used to create IMDB, available from any of the 3 ftp sites listed under "Plain Text Data Files"
  • Curator: gaschue08

ies

  • URL: http://nces.ed.gov/ipeds/datacenter/
  • Description: Data for all US colleges since 1980, available in different sizes based on how much data you wish to retrieve.
  • Curator: gaespin07

Enron Email Dataset

Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Annotated-directory-big-data&oldid=12362"