Difference between revisions of "Annotated-directory-big-data"

From Earlham CS Department
Jump to navigation Jump to search
(Research and Innovative Technology Administration)
(Freebase)
Line 45: Line 45:
 
==== Freebase ====
 
==== Freebase ====
 
*URL: http://wiki.freebase.com/wiki/Data_dumps
 
*URL: http://wiki.freebase.com/wiki/Data_dumps
*Description: Full data dumps of every fact and assertion in Freebase,an open database of the world's information, covering millions of topics in hundreds of categories.
+
*Description: Full data dumps of every fact and assertion in Freebase,an open database of the world's information, covering millions of topics in hundreds of categories.  
 +
      Set - Quad dump is a full dump of Freebase assertions (quad dump) as tab separated utf8 text.
 +
      Set - Simple Topic Dump is a tab-separated file containing basic identifying data about every topic in Freebase.
 +
      Set - TSV per Freebase type is a tab-separated file for each type in Freebase, suitable for loading into spreadsheets.Each line represents an instance of a Freebase type and columns represent the available properties for the type.
 
*Curator: eosergi10
 
*Curator: eosergi10
 +
*Size: Total- 6.0 Gb compressed with bzip2
 +
      Set - Quad dump: The Link Export is approximately 3.5 Gbytes compressed with bzip2 (35 GB uncompressed)
 +
      Set - Simple Topic Dump: approximately 1.2 Gbyte compressed with bzip2 (5 GB uncompressed). In June 2011, there were over 22 million rows.
 +
      Set - TSV per Freebase type: The full download is approximately 1.3 Gbytes compressed with bzip2.The browseable set contains approximately 7500 TSV files in 100 domains.
 +
*Format:  This is a complete "low level" dump of data which is suitable for post processing into RDF or XML datasets.
 +
*Schema:
 +
      Set - Quad dump:The format of the link export is a series of lines, one assertion per line.The lines are tab separated quadruples, <source> (mid - a machine-generated id), <property> (a particular kind of quality of the entity mentioned in the "source" column), <destination> (holds the name of a namespace), <value> (a key within that namespace).
 +
      Set - Simple Topic Dump: mid,English display name, Freebase /en keys, numeric English Wikipedia keys, Freebase types, short text description
 +
      Set - TSV per Freebase type: type, type's description
  
 
===="DBpedia" ====
 
===="DBpedia" ====

Revision as of 03:03, 14 October 2011

This is an annotated directory of public, freely available, "large" data sets. For now they are in no particular order.

Google ngrams

  • URL - http://books.google.com/ngrams/datasets
  • Description - The ngram databases on which Google's ngram viewer is built. A variety of corpora are available, e.g. by language, the "Google Million", English fiction, etc. Each set contains a list of ngrams, frequency, and date information.
  • Curator - CharlieP

MusicBrainz

World Cubing Association Database

Large Data Sets on AWS

Starcraft 2 Hit Analysis

Starcraft 2 Combat Analysis

Twitter Users by Location

The AOL Search Data

Freebase

  • URL: http://wiki.freebase.com/wiki/Data_dumps
  • Description: Full data dumps of every fact and assertion in Freebase,an open database of the world's information, covering millions of topics in hundreds of categories.
      Set - Quad dump is a full dump of Freebase assertions (quad dump) as tab separated utf8 text.
      Set - Simple Topic Dump is a tab-separated file containing basic identifying data about every topic in Freebase. 
      Set - TSV per Freebase type is a tab-separated file for each type in Freebase, suitable for loading into spreadsheets.Each line represents an instance of a Freebase type and columns represent the available properties for the type. 
  • Curator: eosergi10
  • Size: Total- 6.0 Gb compressed with bzip2
      Set - Quad dump: The Link Export is approximately 3.5 Gbytes compressed with bzip2 (35 GB uncompressed) 
      Set - Simple Topic Dump: approximately 1.2 Gbyte compressed with bzip2 (5 GB uncompressed). In June 2011, there were over 22 million rows. 
      Set - TSV per Freebase type: The full download is approximately 1.3 Gbytes compressed with bzip2.The browseable set contains approximately 7500 TSV files in 100 domains. 
  • Format: This is a complete "low level" dump of data which is suitable for post processing into RDF or XML datasets.
  • Schema:
      Set - Quad dump:The format of the link export is a series of lines, one assertion per line.The lines are tab separated quadruples, <source> (mid - a machine-generated id), <property> (a particular kind of quality of the entity mentioned in the "source" column), <destination> (holds the name of a namespace), <value> (a key within that namespace). 
      Set - Simple Topic Dump: mid,English display name, Freebase /en keys, numeric English Wikipedia keys, Freebase types, short text description
      Set - TSV per Freebase type: type, type's description

"DBpedia"

  • URL:http://blog.dbpedia.org/2011/09/11/dbpedia-37-released-including-15-localized-editions/
  • Description: The dataset release is based on Wikipedia dumps dating from late July 2011.DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data.
  • Curator: eosergi10

IMDB

  • URL: http://www.imdb.com/interfaces
  • Description: All the data used to create IMDB, available from any of the 3 ftp sites listed under "Plain Text Data Files"
  • Curator: gaschue08