Leif-big-data: Difference between revisions

From Earlham CS Department
Jump to navigation Jump to search
Charliep (talk | contribs)
Bureaucrats, nsAppliedGroupsD, nsAppliedGroupsR, nsAppliedGroups_, nsCS382DEL, nsCS382RW, nsCS382_TalkDEL, nsCS382_TalkRW, nsDepartmentDEL, nsDepartmentRW, nsDepartment_Tal, nsFundingDEL, nsFundingRW, nsFunding_TalkDE, nsFunding_TalkRW, nsGradingDEL, nsGradingRW, nsGrading_TalkDE, nsGrading_TalkRW, Administrators
1,991 edits
Created page with "* Project title * Project data set ===== Project Tasks ===== #Identifying and downloading the target data set #Data cleaning and pre-processing #Load the data into your Postg..."
 
Lnulric09 (talk | contribs)
179 edits
No edit summary
 
(11 intermediate revisions by 2 users not shown)
Line 1: Line 1:
* Project title  
* Project title: Stories in Words
* Project data set  
* Project data set: Google Ngrams - 1gram (English)


===== Project Tasks =====
===== Project Tasks =====
#Identifying and downloading the target data set
#Identifying and downloading the target data set
#Data cleaning and pre-processing  
#*This project uses Google Ngrams - 1gram (English) which can be downloaded from Google Books at [http://books.google.com/ngrams/datasets] 0-10 CSV files.
#Load the data into your Postgres instance  
#Data cleaning and pre-processing
#Develop queries to explore your ideas in the data  
#*The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv
#Load the data into your Postgres instance
#*I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.
#Develop queries to explore your ideas in the data
#* I wrote a script to fish my database for the data I specify and that is included in my shared directory
#Develop and document the model function you are exploring in the data
#Develop and document the model function you are exploring in the data
#* Exploring what stories I can say about graphing key words
#Develop a visualization to show the model/patterns in the data
#Develop a visualization to show the model/patterns in the data
#* I have included a keynote presentation in my public directory
===== Tech Details =====
* Node: as6
* Path to storage space: local machine
* Path to project files: ~lnulric09/public/big_data/
===== Results =====
* The visualization(s)
* The story

Latest revision as of 16:43, 12 December 2011

  • Project title: Stories in Words
  • Project data set: Google Ngrams - 1gram (English)
Project Tasks
  1. Identifying and downloading the target data set
    • This project uses Google Ngrams - 1gram (English) which can be downloaded from Google Books at [1] 0-10 CSV files.
  2. Data cleaning and pre-processing
    • The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv
  3. Load the data into your Postgres instance
    • I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.
  4. Develop queries to explore your ideas in the data
    • I wrote a script to fish my database for the data I specify and that is included in my shared directory
  5. Develop and document the model function you are exploring in the data
    • Exploring what stories I can say about graphing key words
  6. Develop a visualization to show the model/patterns in the data
    • I have included a keynote presentation in my public directory
Tech Details
  • Node: as6
  • Path to storage space: local machine
  • Path to project files: ~lnulric09/public/big_data/
Results
  • The visualization(s)
  • The story
Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Leif-big-data&oldid=12585"