SE2006:group bar:minutes

Feb 15

Possible meeting time: Working M/W/F in the evening for 1 or 2 hours starting the 27th. Toby, Kevin, and I are going to be working this weekend on clustcomp projects and then going to a conference on Feb 22-26.

• Languages:
  • php - most of us are familiar. big libraries. web stuff.
  • java with tomcat - web stuff. good development environment.
  • python - learn a new language.
  • perl - most of us are familiar. less good, but good web stuff.
  • ruby - learn a new language. there may be other reasons but we don't know what they are because we don't know ruby.

• Caching Location data:
  • with memcache (http://www.danga.com/memcached/)
  • postgres
  • mysql

• Communication
  • using a mailing list for now.
  • developing new methods as they seem fit (wiki,cvs,bulletin board).

• Group name:
  • kartgoolers.
  • cartooglers.
  • pwners
  • colin's team.
  • group-b
  • group bar

• Assignment:
  • send us your group name choice to list. we accept write ins. if anyone feels super strong about a name then tell us why.
  • submit your leader (self?)nominations.

Feb 28

• We have decided on some of the roles:
  • Aybars: Libarian
  • Alex & ColinC: Developers
  • Tobias: Leader
  • Kevin: Architect

• We have roughly agreed on meeting Monday, Tuesday and Wednesdays.

• We have discussed which language to use, and decided that we'll decide on that later after we break the project up a little bit more (possibly tomorrow).

• Tomorrow we will discuss the documentation.

• We will look at the lab 2 solutions soon.

• Error checking?

Mar 1

• We worked on breaking this project into smaller pieces.

• For database we are going to use SQL and for the rest of the project, we will probably use Java.

• We will use Javadoc for documentation of the API

• Next meeting we are going to talk about:
  • Lab 2 solutions
  • Time budgeting
  • Eclipse review
  • Java prep
  • Kevin will bug Skylar about getting Java API installed on Quark.
  • We have two solutions so far on the table
  • We will talk more about documentation

Mar 6

• we went over our lab 2 solutions:

We compared three of our Kerasotes lab scripts. We apparently all wrote them in PHP. The best innovations we found were Toby's data point definition abstraction and test suite. In essence, the data point definition abstraction allows one to describe a data source and data points that one would like to harvest in a general manner. The test suite is a basis for more general testing of the software that we will write. Also of note, were some of our findings as far as regular expressions. For instance, as a group we learned how to make a regular expression not greedy and how to not save a group (?).

Attached you will find an image of our general model thus far. We have already incorporated some of the ideas discussed in this session, including a more general data point abstraction and streamlined data flow between the programmer API, database, and WUI. As part of our plan, we intend to use Java as the base language, XML for temporary data storage and transfer, PostgreSQL or MySQL for data storage, and XHTML Transitional and JavaScript for the WUI.

• eclipse overview and java prep held till next mtg.
• java API + doc has been installed on quark

Mar 7

• Toby went over Eclipse basics with us
• We set up a CVS repository on Quark in /clients/groups/bar/cvs/
  • So far, we're planning on using the repository through Eclipse's interface
• We went over Java basics, and described the Javadoc idea

• On the boiler plate:
  • Have a discussion and get in writing exactly what our individual jobs entail.
  • Show how Javadoc works
    • Develop our group's commenting style
  • Think about our coding style, especially as regards code formatting
  • CVS plumbing (CVS spam, bugzilla, etc.)

Mar 8

• We decided on the roles. SE2006:group_bar:roles

• We went over using Javadoc feature of Eclipse.

• Next time we are going to:
  • Flesh out the model.
  • Write the stubs of the code

Mar 13

• Today we worked on finding some sources. Next time we will be discussing about either to use one of these or look for another one.

• Sources
  • [fact finder]
  • [Surface Condition Weather Forecasting for Air Sports Aviators]
  • [Energy Information Administration]
  • [Geostat Center: Collections]
  • [US Census Bureau: International Database (IDB)]
  • [CIA World Factbook GDP per capita ranking of countries]

• Perhaps useful sources
  • [Demographic Data Viewer]

Mar 14

• Today we worked on the stubs for the scraping API (scrAPI) and had some preliminary discussions about how to create the SQL schemas and connect them to this API.

• Tomorrow we are planning to work more on this API and possibly improve its relations with our SQL schemas.

Mar 15

• We worked more on scrAPI and added more regular expressions into it. There are some new ideas emerging about some new data types. It seems that we are making good progress on scrAPI. :-)

• We have found some more standardized websites that allow us to automatically scrap them. Moreover, we also tried to find ideas on how to scrap some of these websites that we have found. We have communicated them to each other either by e-mail and/or by talking.

• Next meeting we are going to try to incorporate the structure of these new websites we have found into our scrAPI design.

Mar 27

• We examined some code done and made decisions about upcoming parts.

• Next days will be work-on-code days that will happen in XP style pairs.

• Created Todo List

Mar 28

• The Zip Code Database Project
• Google Maps + US Census Information = Analysis of your Area
• Maps.Huge.Info Mapping Programs
• gCensus
  • FAQ talks about how the data is organized.
  • I like the "Totals for viewable area"
• Mapki
• bookmarks
• Created Sources List
• http://raws.wrh.noaa.gov/roman/

Mar 28

• We worked on creating regular expressions for scraping data.

• We did code clean-up.

• Until friday, we will be working on the regular expressions and try finding ways to place our scraped data onto the database.

Mar 31

• We are working on practically everything, tomorrow is the first due date. Examples of what we are doing includes but not limited to:
  • Creating the classes and regular expressions needed for the classes.
  • Debugging existing classes
  • Error handling
  • Documentation
  • Code clean-up
  • Going crazy

April 3

(Toby standing in for Aybars)

• reassessed our roles in the group: currently we all do a little of everything and we like that. will remain in current "roles."
• discussed code licenses: we will adopt the LGPL for the API (and probably the GPL for the UI)
• we divided upcoming work into 3 projects:
  • geocoding API - Colin & Toby will incorporate support for group fu's data store
  • user interface - Kevin will start researching possible alternatives to Google maps (both web- and java-based)
  • API cleanup and data source additions - Alex & Aybars
    • Alex interested in veterans vs. house value overlay
• Toby will write tests for his code (Scheduler, SourceDefinition, etc.)

Questions for class tomorrow:

• what metrics and what dimensions do our customers want in the final product?
• what should the interface look like?
  • what kind of data displayed in what way?
  • what kind of controls?
• do we need to test each data source individually? (probably)

Apr 4

• Worked out intra-group communication problems.

• Found some new sources and found some sources about nuclear power plants. The last one we found was the most comprehensive, so we will probably end up using it.

• Examined Dave's Perl codes and analyzed the output data files (the *.DAT files etc).

• Talked about what connections we have between our sources and what connection there should be in the upcoming ones. We are developing a theme for our sources.

• Talked about what to do next. It seems that more or less all we have is one or two more scrAPI parts to create on one or two more new sources we find around a theme/topic, write a front-end for Dave's geocode, some debugging, a few more documentation and we are mostly done!

Apr 5

• Have found more or less all the data sources we need.

• Started scrAPIng the new data sourcees we have found.
  • Nuclear power plant source
  • Wind energy source

• Tried to get Geo-coder working
  • Waiting on the other group for the geocoder database

Apr 10

• Geocoder ideas (Colin & Toby):
  • use http://geocoder.us/service/csv/geocode for city & address geocoding (waiting on e-mail from services@geocoder.us)
  • use Dave's database for geocoding states, zip codes, and counties
    • see postgresql get_shape_group function
  • create a local cache of at least the geocoder.us data, possibly including Dave's data (to simply access at runtime)
  • standardize column names for location data for source tables (e.g., (lat, long), state, county, zip code, city, address)
  • write functions that can generically group entities in a data table at different summary levels based on the above column names
  • for tomorrow: create geocoding API stubs for Alex

• Data formatter (Alex):
  • for tomorrow: create XML schema for markers & polylines (to be fed to UI)

• Estimation of an average time left per person to be submitted on Tuesday. According to our statistical extrapolations we have a gigantic amount of more work to do.

Apr 11

• Created a brain-storm-like-map where we created the user interface on board. We discussed a range of problems and solutions. Topics included:
  • The user interface design
  • XML
  • Tomcat
  • ScrAPI to Tomcat connection
  • Tomcat to user interface connection

Apr 12

• We split into groups and:
  • Developed the user interface design. We are trying create:
    • Website intra-communication where different parts of the website can change the other parts on event calls
    • Create the basis for connection with Tomcat server
    • Nice design
  • Streaming database insertion
  • Worked on the Tomcat server

Apr 17

• Tomorrow is the due date for the beta version, everyone is pretty much stressed out. We worked on everything, which was mostly the user interface and debugging.

• We worked on database and database-user interface relation.

• Debugging on scrAPI and some efforts for better standardization.

Apr 19

• We have created a list of tasks, and decided on their priorities and how much more we need to learn to accomplish those tasks.

• We created a bunch of design ideas for the user interface.
  • How will the data be displayed?
  • Where will it be displayed?
  • How will a bunch of data be displayed?
  • What will be the displaying options?

Apr 24

• At this time, most of our work focuses on debugging. Most of our ideas of the big picture are implemented, but a lot of things work incorrectly or insufficiently. Today we concentrated on:
  • Testing
  • JavaScript related problems (these problems are bugging the group deeply) and understanding JavaScript abnormalities
  • General debugging

Apr 25

• We worked again mostly on debugging (we are trying to get this mostly done until our next deadline, thursday morning).

• Debugging list includes:
  • Bug #2: "Save source metadata in database"
  • Some test files
  • Objects within JavaScript

Apr 26

• Tomorrow is the last deadline before the actual submission date and the presentation.

• We are overstressed and working on getting things done. Our list includes:
  • User Interface, especially the "Regional Stats." Being the middle box in the right, this gives summarized average information about the points in the screen.
  • Some JavaScript debugging
  • Some general debugging to get it generally ready
  • Metadata/database stuff

Apr 30

• Tomorrow is the actual submission and presentation day. Today is the last group meeting for software engineering.

• We are doing some last minute polishing stuff and getting ready for presentation. Our software seems mostly ready for presenting and is open to further development.