Mobeen-big-data - Revision history

Mmludin08: /* I. Develop and document the model function you are exploring in the data */

2011-12-18T06:32:22Z

I. Develop and document the model function you are exploring in the data

← Older revision		Revision as of 06:32, 18 December 2011
Line 61:		Line 61:
	===== I. Develop and document the model function you are exploring in the data =====		===== I. Develop and document the model function you are exploring in the data =====

	*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern		*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern the users are rating a movie. For example i tried to find if the same users rated a movie high, and if they watch the same type of movies and in the same period of time. the queries develop to do this could be found on /cluster/home/mmludin08/Bi-Data-M/Clean-Data/bigdata.sql.

	===== II. Develop a visualization to show the model/patterns in the data =====		===== II. Develop a visualization to show the model/patterns in the data =====

Mmludin08: /* 2. Data cleaning and per-processing */

2011-12-18T06:22:52Z

2. Data cleaning and per-processing

← Older revision		Revision as of 06:22, 18 December 2011
Line 38:		Line 38:
	-- test.txt -- was created for conversion and cleaning test on a small scale.		-- test.txt -- was created for conversion and cleaning test on a small scale.
	-- python_script.py -- works Fine for conversion to date. And this was used for the actual data conversion.		-- python_script.py -- works Fine for conversion to date. And this was used for the actual data conversion.
			-- insert_script.pl -- Works Fine for converting the .csv files to .psql. Insertion statements so i could upload to bigfe.

	* '''Files: '''		* '''Files: '''
Line 49:		Line 50:
	-- ''' ratings.csv'''		-- ''' ratings.csv'''

	-- ''' tags.csv'''		-- ''' tags.csv'''

	==== 3. Load the data into your Postgres instance ====		==== 3. Load the data into your Postgres instance ====

Mmludin08 at 13:54, 14 December 2011

2011-12-14T13:54:59Z

← Older revision		Revision as of 13:54, 14 December 2011
Line 42:		Line 42:

	-- ''' bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.		-- ''' bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.

			These file contain the actual data for the project, and were uploaded to database and used for queries.

	-- ''' movies.csv'''		-- ''' movies.csv'''
Line 48:		Line 50:

	-- ''' tags.csv'''		-- ''' tags.csv'''

	~~These file contain the actual data for the project, and were uploaded to database and used for queries.~~

	==== 3. Load the data into your Postgres instance ====		==== 3. Load the data into your Postgres instance ====

Mmludin08 at 13:54, 14 December 2011

2011-12-14T13:54:18Z

← Older revision		Revision as of 13:54, 14 December 2011
Line 43:		Line 43:
	-- ''' bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.		-- ''' bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.

	-- ''' movies.csv'''		-- ''' movies.csv'''

	-- ''' ratings.csv'''		-- ''' ratings.csv'''
	-- ''' tags.csv''' These file contain the actual data for the project, and were uploaded to database and used for queries.
			-- ''' tags.csv'''

			These file contain the actual data for the project, and were uploaded to database and used for queries.

	==== 3. Load the data into your Postgres instance ====		==== 3. Load the data into your Postgres instance ====

Mmludin08 at 13:53, 14 December 2011

2011-12-14T13:53:28Z

← Older revision		Revision as of 13:53, 14 December 2011
Line 24:		Line 24:
	* '''Directories:'''		* '''Directories:'''

	'''-- Backupfiles:''' The Backupfiles directory contains the data set that was downloaded from Movielens.		-- ''' Backupfiles:''' The Backupfiles directory contains the data set that was downloaded from Movielens.

	'''-- Clean_Data:''' The Clean_Data directory has all the data files that were formated by using the perl/python scripts.		-- ''' Clean_Data:''' The Clean_Data directory has all the data files that were formated by using the perl/python scripts.

	'''-- Q_results:''' The Q_results directory has has the out files for each queries that were passed to the postgres from bigdata.sql file.		-- ''' Q_results:''' The Q_results directory has has the out files for each queries that were passed to the postgres from bigdata.sql file.

	'''-- Scripts:''' This directory contain all the scripts written for this project. some of the script were written for test purposes.		-- ''' Scripts:''' This directory contain all the scripts written for this project. some of the script were written for test purposes.

	-- bashscript.sh -- Works for cleaning the data (this script was used mostly)		-- bashscript.sh -- Works for cleaning the data (this script was used mostly)
Line 41:		Line 41:
	* '''Files: '''		* '''Files: '''

	'''-- bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.		-- ''' bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.

	'''-- movies.csv'''		-- ''' movies.csv'''
	'''-- ratings.csv'''		-- ''' ratings.csv'''
	'''-- tags.csv''' These file contain the actual data for the project, and were uploaded to database and used for queries.		-- ''' tags.csv''' These file contain the actual data for the project, and were uploaded to database and used for queries.

	==== 3. Load the data into your Postgres instance ====		==== 3. Load the data into your Postgres instance ====

Mmludin08 at 13:51, 14 December 2011

2011-12-14T13:51:46Z

← Older revision		Revision as of 13:51, 14 December 2011
Line 24:		Line 24:
	* '''Directories:'''		* '''Directories:'''

	'''--. Backupfiles:''' The Backupfiles directory contains the data set that was downloaded from Movielens.		'''-- Backupfiles:''' The Backupfiles directory contains the data set that was downloaded from Movielens.

	'''--. Clean_Data:''' The Clean_Data directory has all the data files that were formated by using the perl/python scripts.		'''-- Clean_Data:''' The Clean_Data directory has all the data files that were formated by using the perl/python scripts.

	'''--. Q_results:''' The Q_results directory has has the out files for each queries that were passed to the postgres from bigdata.sql file.		'''-- Q_results:''' The Q_results directory has has the out files for each queries that were passed to the postgres from bigdata.sql file.

	'''--. Scripts:''' This directory contain all the scripts written for this project. some of the script were written for test purposes.		'''-- Scripts:''' This directory contain all the scripts written for this project. some of the script were written for test purposes.

	-- bashscript.sh -- Works for cleaning the data (this script was used mostly)		-- bashscript.sh -- Works for cleaning the data (this script was used mostly)
Line 41:		Line 41:
	* '''Files: '''		* '''Files: '''

	'''--. bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.		'''-- bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.

	'''--. movies.csv'''		'''-- movies.csv'''
	'''--. ratings.csv'''		'''-- ratings.csv'''
	'''--. tags.csv''' These file contain the actual data for the project, and were uploaded to database and used for queries.		'''-- tags.csv''' These file contain the actual data for the project, and were uploaded to database and used for queries.

	==== 3. Load the data into your Postgres instance ====		==== 3. Load the data into your Postgres instance ====

Mmludin08 at 13:50, 14 December 2011

2011-12-14T13:50:56Z

Mmludin08: /* 5. Results */

2011-12-14T13:46:42Z

5. Results

← Older revision		Revision as of 13:46, 14 December 2011
Line 54:		Line 54:

	==== 5. Results ====		==== 5. Results ====
	===== I. Develop and document the model function you are exploring in the data =====		===== I. Develop and document the model function you are exploring in the data =====

	*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern		*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern

	===== II. Develop a visualization to show the model/patterns in the data =====		===== II. Develop a visualization to show the model/patterns in the data =====

	* The visualization(s)		* The visualization(s)
	* The story		* The story

Mmludin08: /* 5. Results */

2011-12-14T13:46:03Z

5. Results

← Older revision		Revision as of 13:46, 14 December 2011
Line 54:		Line 54:

	==== 5. Results ====		==== 5. Results ====
	===== I. Develop and document the model function you are exploring in the data =====		===== I. Develop and document the model function you are exploring in the data =====

	*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern		*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern

	===== II. Develop a visualization to show the model/patterns in the data =====		===== II. Develop a visualization to show the model/patterns in the data =====

	* The visualization(s)		* The visualization(s)
	* The story		* The story

Mmludin08 at 13:45, 14 December 2011

2011-12-14T13:45:22Z

← Older revision		Revision as of 13:45, 14 December 2011
Line 53:		Line 53:
	* SQL statements with results are on cluster: /cluster/home/mmludin08/Big-Data-M		* SQL statements with results are on cluster: /cluster/home/mmludin08/Big-Data-M

	==== 5. Develop and document the model function you are exploring in the data ====		==== 5. Results ====
			===== I. Develop and document the model function you are exploring in the data =====

	*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern		*For this project my aim was to discover the movie genres time line. In more words, I wanted to find out at what period of time people watch what type of movies. I also tried to look for the pattern

	==== 6. Develop a visualization to show the model/patterns in the data ====		===== II. Develop a visualization to show the model/patterns in the data =====

	~~===== Results =====~~
	* The visualization(s)		* The visualization(s)
	* The story		* The story

← Older revision		Revision as of 13:50, 14 December 2011
Line 24:		Line 24:
	* '''Directories:'''		* '''Directories:'''

	'''1. Backupfiles:''' The Backupfiles directory contains the data set that was downloaded from Movielens.		'''--. Backupfiles:''' The Backupfiles directory contains the data set that was downloaded from Movielens.

	'''2. Clean_Data:''' The Clean_Data directory has all the data files that were formated by using the perl/python scripts.		'''--. Clean_Data:''' The Clean_Data directory has all the data files that were formated by using the perl/python scripts.

	'''3. Q_results:''' The Q_results directory has has the out files for each queries that were passed to the postgres from bigdata.sql file.		'''--. Q_results:''' The Q_results directory has has the out files for each queries that were passed to the postgres from bigdata.sql file.

	'''4. Scripts:''' This directory contain all the scripts written for this project. some of the script were written for test purposes.		'''--. Scripts:''' This directory contain all the scripts written for this project. some of the script were written for test purposes.

	bashscript.sh -- Works for cleaning the data (this script was used mostly)		-- bashscript.sh -- Works for cleaning the data (this script was used mostly)
	perl_script.pl -- Didnt quite work the way i wanted to. Conversion and cleaning together.		-- perl_script.pl -- Didnt quite work the way i wanted to. Conversion and cleaning together.
	conversion_script.pl -- Works fine for epoch conversion to only date. (test version )		-- conversion_script.pl -- Works fine for epoch conversion to only date. (test version )
	py_conv_script.py -- Works fine for epoch conversion to data and time.		-- py_conv_script.py -- Works fine for epoch conversion to data and time.
	test.txt -- was created for conversion and cleaning test on a small scale.		-- test.txt -- was created for conversion and cleaning test on a small scale.
	python_script.py -- works Fine for conversion to date. And this was used for the actual data conversion.		-- python_script.py -- works Fine for conversion to date. And this was used for the actual data conversion.

	* '''Files: '''		* '''Files: '''

	'''1. bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.		'''--. bigdata.sql''': This file has the queries that were written to generate results for my project. most of them works fine. some didnt or took longer even after using indexes.

	'''2. movies.csv'''		'''--. movies.csv'''
	'''3. ratings.csv'''		'''--. ratings.csv'''
	'''3. tags.csv''' These file contain the actual data for the project, and were uploaded to database and used for queries.		'''--. tags.csv''' These file contain the actual data for the project, and were uploaded to database and used for queries.

	==== 3. Load the data into your Postgres instance ====		==== 3. Load the data into your Postgres instance ====