spring 2010 plans

The following is being submitted as our URP application. I hope it will get you up to speed with where we are right now and where we hope to go this semester.

For the past two URP sessions, I have worked on goffe, an offline file/folder editor designed to make offline filesystem changes to a networked volume. Peter Healy has joined me for the Fall 09 session. Progress was substantial during the summer when time was a little less restricted, and nearly the entire back-end was completed. Work began on the GUI during the fall, as well as furthering a utility to generate XML from the filesystem data.

After attempting a couple stop-gap solutions, often using the utility “tree”, we agreed the best way to get the flexibility we needed was to contribute a solution to the tree project directly. We did hear from the maintainers of the tree project during the semester and plan to work with them and develop a contribution immediately. As for the GUI, progress has been slow due to some poor planning by myself in the early development stages of goffe as well as an overbooked fall semester.

At present, we are deciding whether to rewrite the majority of the back end based on existing code, patch what is currently there, or contribute to a different project after completing our contribution to tree. I do not want to abandon this code as I believe the utility is very unique and most of the legwork on the back end has been completed. I believe the processes used and the pseudocode behind them are solid, however, since I have no formal training in programming with classes, Pete has found it difficult to adapt what is currently there for use in a GUI. I am hoping we can resolve these differences by simplifying some methods and combining others to make them more accessible to additional code.

In summary, our outlook for spring is very optimistic. We have a project ready to contribute to in tree, and we are looking forward to implementing our changes and finalizing a deliverable. Additionally while goffe's future is uncertain, should we decide that the necessary changes are far beyond what can be completed this semester, we look forward to joining a different project. Alternately if we continue work on goffe as I am hoping, we will be working together to determine more exactly what must be incorporated in the back end so I can implement the changes and Pete can begin progress on the GUI.

progress: 11/17/09

so a lot's been going on but not a lot's been happening.

to start with; the XML.
from the beginning, i had planned to implement something to generate the XML by very cleverly manipulating strings with bizzare regular expressions. i didn't need it to be perfect or something i could really build on later, i just needed it to work. after the main aspects of goffe made more progress i could incorporate building the XML into java, so just getting reliable XML that would validate against a DTD was all i was concerned with. and i figured doing it through strings would be simple.

the best tool for the job was awk. unfortunately i learned quickly that awk wouldn't separate fields based on whitespace; whenever it found a match, it would collapse all whitespace around it. that collapsed whitespace was the way awk would determine its nesting level, so i was pretty screwed. i fiddled around with it, pretty extensively, such as examining trailing slashes on folders, etc, but no luck.

so i read more on java. well, java sucks for dealing with *NIX conventions, something they are apparently working on for java 7 (i.e. retreiving user, group, permissions, etc). i really don't want to port my code over to java7 any time soon, so now my back burner tool takes on more importance.

enter perl, which while slightly more verbose than awk, could keep the whitespace i needed. after a week of coding, tweaking, and tuning, i got it working through up to 10 levels of nesting, fairly impressive for manipulating 35000 lines of complicated strings, but not good enough. i am considering continuing the endeavor in perl but without using the strings.

then after my presentation i got an unsolicited email from an anonymous code donor, written in python and walking the filesystem directly instead of using strings. first off, whoever sent it to me, thanks. unfortunately, it doesn't work. i dont have time to play with it right now, but it looks like the way it prints its tags prevents it from nesting, period. so there is exactly one file in every folder, which dont actually correspond to real folders. sorry to be the bearer of bad news, but thanks again for the effort.

and last but not least i considered donating to the tree project, and actually sent them an email. well i heard back not long ago, and apparently they still work on it from time to time. i would really like to contribute, as the versitility of tree is pretty remarkable. if i implemented it in tree, it could be very user defineable, i.e. print attributes or elements, trailing slashes for folders, character sets, the list goes on. so, yeah, i'm going to do that. and hopefully it works.

so languages that XML generation will have gone through: sh/awk, perl, java(ish), python, and hopefully it will find its home in C where it rightfully belongs.

in other news, pete has been working on researching the GUI pretty intensely. i will try to get him to come post on here. we are trying to asses how viable it is to use more than one existing GUI to work on the files (one parses and generates JFrame data, one performs operations) versus start almost completely from scratch with a typical browser. hopefully we will make progress on that soon.
-bryan

progress 10/1/09

and we are back for fall 2009.
unfortunately, updates kind of went on hiatus for a while there.

so i'll take a moment to get you up to speed
toward the end of the summer, progress went very well.
nearly all of the pre-GUI blocks are in place; albeit incompletely in some instances.
some current issues:
there is still no way to generate XML reliably (either in java or awk)
cut/paste element issues
and completing the script generator

for this semester, Pete Healy has come on board to assist with more of the code tidiness, and making sure the methods and classes flow well.
Aside from making the corrections above, we are still discussing a few options, including making the code more multiuser friendly, making it *NIX only vs. cross platform, etc.

hopefully these will get resolved and we will be on our way soon.
again, thank you to prof. moorthy and sean o sullivan for the opportunity to further our projects.
-bryan

status update 6/25/09

after paving over the initial hurdles last week things have gone incredibly smoothly.

functionality has been added bit by bit and code is almost where i hoped it would be by the end of the month.

works:
takes input and opens corresponding xml file
parses XML, builds nodelist and arraylists of contents of root directory
selection of file or folder by index, opening/rebuilding contents of subdirectories
delete and rename functions for in-memory representation
creates and modifies current working path
input validation, exception handling (every bug i've caught but one)
etc.

to be continued:
fix one nagging exception
start writing changes to file
save new XML upon leaving
work on copy, paste functions, etc.

then onto the GUI.

status update 6/16/09

for something that was supposed to be so easy and standardized, learning how to make the DOM do what i want it to do has been a huge pain in the ass.

a lot of it is my fault too, to be sure, like not updating the DTD for XML validation when i changed the layout of the fields in the XML.
so, the DOM was inserting null characters in my nodelists, which is just bad for everyone.

in any case, those simple mistakes have all been cleared up, after hours of debugging, and hopefully a text-based tree walker type majigger will be ready by the end of the night.

with how quickly these tiny errors multiply, i'm beginning to doubt it.

what works:
xml builds (or close to it)
xml validates and loads
given a folder element, the code builds arraylists of file and folder objects. then it lists them.

what doesn't work:
everything else

status update 6/11/09

i suppose using 'tree' was a bad idea, but ive been stuck in this quicksand so long trying to float i never bothered to grab the rope and switch to ls.

major issues have come up as a result, and trying to iron them out has been a major goal before proceeding into the java section of the code. i wanted to use tree to build the XML due to its functionality, it did everything i wanted it to, nothing more, nothing less. line by line data that was practically XML already.

everything worked fine until i went to validate the XML and found there were missing tags. enough sleuthing around and i found the issue: subfolders, while further indented than their parents, didnt have any kind of delimiting character i could use when awk recomputed its position in the subfolder nesting.

long story short, when attempting to close folder tags, if it had to do more than one at a time, it couldn't. due to the way the code has been designed thus far, there is no way to correct the issue (not that i see, anyways). somehow i missed this in the slew of vertical lines tree spits out.
if i wanted to correct it, i could switch the script to use ls, but since i'm hoping to put the XML building functionality into the java later on, this seems like a waste of time.

anyways, when i manually put the tags in, the code validated with no errors against the DTD i had written. what a shame to waste such "almost perfect" code.

since the XML did validate, i can move onto the java tomorrow, and hopefully build something like a DOM tree crawler to lay the bricks for the GUI down the road.

status update from start-5/27/09

as of right now, here is what has happened and is happening with goffe

- doing a lot of reading into taking full advantage of java, mainly the API's and how i'm going to get the main app started up.
- almost finished with a "good enough" bourne shell/awk script for generating XML data.

the script is far from perfect, but it provides a good starting point until i'm ready to integrate the functionality into the more final java app. it does not properly close tags yet, but pseudocode is already in place to make the necessary corrections.
plenty of other reasons why this wouldnt be in any final code. for example, it currently relies on the package "tree" found in the repos to generate data. also there is a terrible hack-job for determining file versus folder, but i won't get into that. look in the code (once it's posted) if you're really interested.

next steps include fixing the tag issues and trying to poke around with a command line app and DOM or SAX to make sure it's all well-formed.