Tristan-big-data: Difference between revisions
Jump to navigation
Jump to search
| Line 3: | Line 3: | ||
===== Project Tasks ===== | ===== Project Tasks ===== | ||
-Identifying and downloading the target data set | |||
The WCA Dataset was easily downloaded as a set of SQL inserts. The file can be downloaded from [http://worldcubeassociation.org/results/misc/export.html here]. | ::The WCA Dataset was easily downloaded as a set of SQL inserts. The file can be downloaded from [http://worldcubeassociation.org/results/misc/export.html here]. | ||
-Data cleaning and pre-processing | |||
The issue was that the .sql file was in MS-SQL or OracleSQL, so some mass modifications to the file had to be made. Primarily it was with changing smallint(n) to int, and `tablename` without the `. | ::The issue was that the .sql file was in MS-SQL or OracleSQL, so some mass modifications to the file had to be made. Primarily it was with changing smallint(n) to int, and `tablename` without the `. | ||
-Load the data into your Postgres instance | |||
It took a few times to get everything from the script all working, but the script was successfully run on my directory on BigFe. | ::It took a few times to get everything from the script all working, but the script was successfully run on my directory on BigFe. | ||
#Develop queries to explore your ideas in the data | #Develop queries to explore your ideas in the data | ||
Revision as of 15:26, 3 December 2011
- Examining Trends in a Performance Sport
- Data set: WCA Database
Project Tasks
-Identifying and downloading the target data set
- The WCA Dataset was easily downloaded as a set of SQL inserts. The file can be downloaded from here.
-Data cleaning and pre-processing
- The issue was that the .sql file was in MS-SQL or OracleSQL, so some mass modifications to the file had to be made. Primarily it was with changing smallint(n) to int, and `tablename` without the `.
-Load the data into your Postgres instance
- It took a few times to get everything from the script all working, but the script was successfully run on my directory on BigFe.
- Develop queries to explore your ideas in the data
- Develop and document the model function you are exploring in the data
- Develop a visualization to show the model/patterns in the data
Tech Details
- Node: as3
- Path to storage space: /scratch/big-data/tristan
Results
- The visualization(s)
- The story