Leif-big-data - Revision history

Lnulric09 at 16:43, 12 December 2011

2011-12-12T16:43:26Z

← Older revision		Revision as of 16:43, 12 December 2011
Line 10:		Line 10:
	#*I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.		#*I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.
	#Develop queries to explore your ideas in the data		#Develop queries to explore your ideas in the data
	* I wrote a script to fish my database for the data I specify and that is included in my shared directory		#* I wrote a script to fish my database for the data I specify and that is included in my shared directory
	#Develop and document the model function you are exploring in the data		#Develop and document the model function you are exploring in the data
	* Exploring what stories I can say about graphing key words		#* Exploring what stories I can say about graphing key words
	#Develop a visualization to show the model/patterns in the data		#Develop a visualization to show the model/patterns in the data
	* I have included a keynote presentation in my public directory		#* I have included a keynote presentation in my public directory

	===== Tech Details =====		===== Tech Details =====

Lnulric09 at 16:43, 12 December 2011

2011-12-12T16:43:06Z

← Older revision		Revision as of 16:43, 12 December 2011
Line 10:		Line 10:
	#*I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.		#*I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.
	#Develop queries to explore your ideas in the data		#Develop queries to explore your ideas in the data
			* I wrote a script to fish my database for the data I specify and that is included in my shared directory
	#Develop and document the model function you are exploring in the data		#Develop and document the model function you are exploring in the data
			* Exploring what stories I can say about graphing key words
	#Develop a visualization to show the model/patterns in the data		#Develop a visualization to show the model/patterns in the data
			* I have included a keynote presentation in my public directory

	===== Tech Details =====		===== Tech Details =====
	* Node: as6		* Node: as6
	* Path to storage space: local machine		* Path to storage space: local machine
			* Path to project files: ~lnulric09/public/big_data/

	===== Results =====		===== Results =====
	* The visualization(s)		* The visualization(s)
	* The story		* The story

Lnulric09 at 17:51, 6 December 2011

2011-12-06T17:51:38Z

← Older revision		Revision as of 17:51, 6 December 2011
Line 1:		Line 1:
	* Project title: Stories in ~~a Word~~		* Project title: Stories in Words
	* Project data set: Google Ngrams - 1gram (English)		* Project data set: Google Ngrams - 1gram (English)

	===== Project Tasks =====		===== Project Tasks =====
	#Identifying and downloading the target data set		#Identifying and downloading the target data set
	#*This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.		#*This project uses Google Ngrams - 1gram (English) which can be downloaded from Google Books at [http://books.google.com/ngrams/datasets] 0-10 CSV files.
	#Data cleaning and pre-processing		#Data cleaning and pre-processing
	#*The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv		#*The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv
Line 16:		Line 16:
	* Node: as6		* Node: as6
	* Path to storage space: local machine		* Path to storage space: local machine
	* Path to Data Setup Script
	* Path to Data Extraction Script
	* Path to GNUPLOT Script
	* Path to Raw Extracted Data
	* Path to Presentation

	===== Results =====		===== Results =====
	* The visualization(s)		* The visualization(s)
	* The story		* The story

Lnulric09 at 18:25, 3 December 2011

2011-12-03T18:25:21Z

← Older revision		Revision as of 18:25, 3 December 2011
Line 1:		Line 1:
	* Project title: ~~Influence of the Hippie Movement Bringing Indian Themes into Western Literature~~		* Project title: Stories in a Word
	* Project data set: Google Ngrams - 1gram (English)		* Project data set: Google Ngrams - 1gram (English)

Lnulric09 at 19:01, 2 December 2011

2011-12-02T19:01:09Z

← Older revision		Revision as of 19:01, 2 December 2011
Line 15:		Line 15:
	===== Tech Details =====		===== Tech Details =====
	* Node: as6		* Node: as6
	* Path to storage space: ~~/scratch/big-data/leif~~		* Path to storage space: local machine
			* Path to Data Setup Script
			* Path to Data Extraction Script
			* Path to GNUPLOT Script
			* Path to Raw Extracted Data
			* Path to Presentation

	===== Results =====		===== Results =====
	* The visualization(s)		* The visualization(s)
	* The story		* The story

Lnulric09 at 18:56, 2 December 2011

2011-12-02T18:56:24Z

← Older revision		Revision as of 18:56, 2 December 2011
Line 3:		Line 3:

	===== Project Tasks =====		===== Project Tasks =====
	*Identifying and downloading the target data set		#Identifying and downloading the target data set
	**This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.		#*This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.
	*Data cleaning and pre-processing		#Data cleaning and pre-processing
	**The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv		#*The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv
	*Load the data into your Postgres instance		#Load the data into your Postgres instance
	**I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.		#*I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.
	*Develop queries to explore your ideas in the data		#Develop queries to explore your ideas in the data
	*Develop and document the model function you are exploring in the data		#Develop and document the model function you are exploring in the data
	*Develop a visualization to show the model/patterns in the data		#Develop a visualization to show the model/patterns in the data

	===== Tech Details =====		===== Tech Details =====

Lnulric09 at 18:49, 2 December 2011

2011-12-02T18:49:27Z

← Older revision		Revision as of 18:49, 2 December 2011
Line 5:		Line 5:
	*Identifying and downloading the target data set		*Identifying and downloading the target data set
	**This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.		**This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.
	#Data cleaning and pre-processing		*Data cleaning and pre-processing
	The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv		**The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv
	#Load the data into your Postgres instance		*Load the data into your Postgres instance
	I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.		**I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.
	#Develop queries to explore your ideas in the data		*Develop queries to explore your ideas in the data
	#Develop and document the model function you are exploring in the data		*Develop and document the model function you are exploring in the data
	#Develop a visualization to show the model/patterns in the data		*Develop a visualization to show the model/patterns in the data

	===== Tech Details =====		===== Tech Details =====

Lnulric09 at 18:49, 2 December 2011

2011-12-02T18:49:02Z

← Older revision		Revision as of 18:49, 2 December 2011
Line 3:		Line 3:

	===== Project Tasks =====		===== Project Tasks =====
	#Identifying and downloading the target data set		*Identifying and downloading the target data set
	This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.		**This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.
	#Data cleaning and pre-processing		#Data cleaning and pre-processing
	The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv		The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv

Lnulric09 at 18:06, 2 December 2011

2011-12-02T18:06:47Z

← Older revision		Revision as of 18:06, 2 December 2011
Line 4:		Line 4:
	===== Project Tasks =====		===== Project Tasks =====
	#Identifying and downloading the target data set		#Identifying and downloading the target data set
	#Data cleaning and pre-processing		This project uses Google Ngrams - 1gram (English) which can be downloaded from [http://books.google.com/ngrams/datasets] 0-10 CSV files.
	#Load the data into your Postgres instance		#Data cleaning and pre-processing
	#Develop queries to explore your ideas in the data		The raw CSV file values are separated by TABS so I had to use a script to replace TABS with COMMAS as follows: tr '\t' ',' <input_file.csv>output_file.csv
			#Load the data into your Postgres instance
			I used a script which when piped into postgres drops existing tables, creates the tables, copies the data in, and then indexes the tables.
			#Develop queries to explore your ideas in the data
	#Develop and document the model function you are exploring in the data		#Develop and document the model function you are exploring in the data
	#Develop a visualization to show the model/patterns in the data		#Develop a visualization to show the model/patterns in the data

Lnulric09 at 17:32, 2 December 2011

2011-12-02T17:32:25Z

← Older revision		Revision as of 17:32, 2 December 2011
Line 1:		Line 1:
	* Project title		* Project title: Influence of the Hippie Movement Bringing Indian Themes into Western Literature
	* Project data set		* Project data set: Google Ngrams - 1gram (English)

	===== Project Tasks =====		===== Project Tasks =====