SE2006:group bar:minutes

Feb 15

Possible meeting time: Working M/W/F in the evening for 1 or 2 hours starting the 27th. Toby, Kevin, and I are going to be working this weekend on clustcomp projects and then going to a conference on Feb 22-26.

• Languages:
  • php - most of us are familiar. big libraries. web stuff.
  • java with tomcat - web stuff. good development environment.
  • python - learn a new language.
  • perl - most of us are familiar. less good, but good web stuff.
  • ruby - learn a new language. there may be other reasons but we don't know what they are because we don't know ruby.

• Caching Location data:
  • with memcache (http://www.danga.com/memcached/)
  • postgres
  • mysql

• Communication
  • using a mailing list for now.
  • developing new methods as they seem fit (wiki,cvs,bulletin board).

• Group name:
  • kartgoolers.
  • cartooglers.
  • pwners
  • colin's team.
  • group-b
  • group bar

• Assignment:
  • send us your group name choice to list. we accept write ins. if anyone feels super strong about a name then tell us why.
  • submit your leader (self?)nominations.

Feb 28

• We have decided on some of the roles:
  • Aybars: Libarian
  • Alex & ColinC: Developers
  • Tobias: Leader
  • Kevin: Architect

• We have roughly agreed on meeting Monday, Tuesday and Wednesdays.

• We have discussed which language to use, and decided that we'll decide on that later after we break the project up a little bit more (possibly tomorrow).

• Tomorrow we will discuss the documentation.

• We will look at the lab 2 solutions soon.

• Error checking?

Mar 1

• We worked on breaking this project into smaller pieces.

• For database we are going to use SQL and for the rest of the project, we will probably use Java.

• We will use Javadoc for documentation of the API

• Next meeting we are going to talk about:
  • Lab 2 solutions
  • Time budgeting
  • Eclipse review
  • Java prep
  • Kevin will bug Skylar about getting Java API installed on Quark.
  • We have two solutions so far on the table
  • We will talk more about documentation

Mar 6

• we went over our lab 2 solutions:

We compared three of our Kerasotes lab scripts. We apparently all wrote them in PHP. The best innovations we found were Toby's data point definition abstraction and test suite. In essence, the data point definition abstraction allows one to describe a data source and data points that one would like to harvest in a general manner. The test suite is a basis for more general testing of the software that we will write. Also of note, were some of our findings as far as regular expressions. For instance, as a group we learned how to make a regular expression not greedy and how to not save a group (?).

Attached you will find an image of our general model thus far. We have already incorporated some of the ideas discussed in this session, including a more general data point abstraction and streamlined data flow between the programmer API, database, and WUI. As part of our plan, we intend to use Java as the base language, XML for temporary data storage and transfer, PostgreSQL or MySQL for data storage, and XHTML Transitional and JavaScript for the WUI.

• eclipse overview and java prep held till next mtg.
• java API + doc has been installed on quark

Mar 7

• Toby went over Eclipse basics with us
• We set up a CVS repository on Quark in /clients/groups/bar/cvs/
  • So far, we're planning on using the repository through Eclipse's interface
• We went over Java basics, and described the Javadoc idea

• On the boiler plate:
  • Have a discussion and get in writing exactly what our individual jobs entail.
  • Show how Javadoc works
    • Develop our group's commenting style
  • Think about our coding style, especially as regards code formatting
  • CVS plumbing (CVS spam, bugzilla, etc.)

Mar 8

• We decided on the roles. SE2006:group_bar:roles

• We went over using Javadoc feature of Eclipse.

• Next time we are going to:
  • Flesh out the model.
  • Write the stubs of the code

Mar 13

• Today we worked on finding some sources. Next time we will be discussing about either to use one of these or look for another one.

• Sources
  • [fact finder]
  • [Surface Condition Weather Forecasting for Air Sports Aviators]
  • [Energy Information Administration]
  • [Geostat Center: Collections]
  • [US Census Bureau: International Database (IDB)]
  • [CIA World Factbook GDP per capita ranking of countries]

• Perhaps useful sources
  • [Demographic Data Viewer]

Mar 14

• Today we worked on the stubs for the scraping API (scrAPI) and had some preliminary discussions about how to create the SQL schemas and connect them to this API.

• Tomorrow we are planning to work more on this API and possibly improve its relations with our SQL schemas.

Mar 15

• We worked more on scrAPI and added more regular expressions into it. There are some new ideas emerging about some new data types. It seems that we are making good progress on scrAPI. :-)

• We have found some more standardized websites that allow us to automatically scrap them. Moreover, we also tried to find ideas on how to scrap some of these websites that we have found. We have communicated them to each other either by e-mail and/or by talking.

• Next meeting we are going to try to incorporate the structure of these new websites we have found into our scrAPI design.

Mar 27

• We examined some code done and made decisions about upcoming parts.

• Next days will be work-on-code days that will happen in XP style pairs.

• Created Todo List

Mar 28

• The Zip Code Database Project
• Google Maps + US Census Information = Analysis of your Area
• Maps.Huge.Info Mapping Programs
• gCensus
  • FAQ talks about how the data is organized.
  • I like the "Totals for viewable area"
• Mapki
• bookmarks
• Created Sources List
• http://raws.wrh.noaa.gov/roman/

Mar 28

• We worked on creating regular expressions for scraping data.

• We did code clean-up.

• Until friday, we will be working on the regular expressions and try finding ways to place our scraped data onto the database.

Mar 31

• We are working on practically everything, tomorrow is the first due date. Examples of what we are doing includes but not limited to:
  • Creating the classes and regular expressions needed for the classes.
  • Debugging existing classes
  • Error handling
  • Documentation
  • Code clean-up
  • Going crazy

April 3

(Toby standing in for Aybars)

• reassessed our roles in the group: currently we all do a little of everything and we like that. will remain in current "roles."
• discussed code licenses: we will adopt the LGPL for the API (and probably the GPL for the UI)
• we divided upcoming work into 3 projects:
  • geocoding API - Colin & Toby will incorporate support for group fu's data store
  • user interface - Kevin will start researching possible alternatives to Google maps (both web- and java-based)
  • API cleanup and data source additions - Alex & Aybars
    • Alex interested in veterans vs. house value overlay
• Toby will write tests for his code (Scheduler, SourceDefinition, etc.)

Questions for class tomorrow:

• what metrics and what dimensions do our customers want in the final product?
• what should the interface look like?
  • what kind of data displayed in what way?
  • what kind of controls?
• do we need to test each data source individually? (probably)

Apr 4

• Worked out intra-group communication problems.

• Found some new sources and found some sources about nuclear power plants. The last one we found was the most comprehensive, so we will probably end up using it.

• Examined Dave's Perl codes and analyzed the output data files (the *.DAT files etc).

• Talked about what connections we have between our sources and what connection there should be in the upcoming ones. We are developing a theme for our sources.

• Talked about what to do next. It seems that more or less all we have is one or two more scrAPI parts to create on one or two more new sources we find around a theme/topic, write a front-end for Dave's geocode, some debugging, a few more documentation and we are mostly done!

Apr 5

• Have found more or less all the data sources we need.

• Started scrAPIng the new data sourcees we have found.
  • Nuclear power plant source
  • Wind energy source

• Tried to get Geo-coder working
  • Waiting on the other group for the geocoder database

Apr 10

Aybars, where are our notes?

• Geocoder ideas (Colin & Toby):
  • use http://geocoder.us/service/csv/geocode for city & address geocoding (waiting on e-mail from services@geocoder.us)
  • use Dave's database for geocoding states, zip codes, and counties
  • create a local cache of at least the geocoder.us data, possibly including Dave's data (to simply access at runtime)
  • standardize column names for location data for source tables (e.g., (lat, long), state, county, zip code, city, address)
  • write functions that can generically group entities in a data table at different summary levels based on the above column names