Lisa de Larios-Heiman
lisadlh@sims.berkeley.edu
Carolyn Cracraft
cdclph@sims.berkeley.edu
Overview
The SylViA Data Model
The SylViA Viewer Application Infrastructure
The SylViA Editor Application Infrastructure
Usability and the SylViA Viewer
User Interface Design and the SylViA Editor
Work on SylViA began with Lisa's final project for Alex Milowski's XML and Related Technologies class in the Spring of 2004. This initial work consisted of a data model and XML schema for a syllabus, a handful of XML-encoded syllabus instances, and a series of XSLT transforms which provided various views across the syllabus data. These components would form the basis for the SylViA Viewer. When Carolyn joined Lisa and we decided to use the work as a basis for their masters project, we retained the conceptual framework, but the model, schema, and transforms were significantly overhauled.
During the summer of 2004, we used Bob Glushko's Document Engineering techniques to develop, from scratch, a rigorous model of syllabus data. We then took the basic structure of Lisa's original transforms and recoded them to work with our new data model. Finally, we selected about a dozen of the most popular SIMS courses, including all the core courses, and hand-coded XML instances of their syllabi. By the first weeks of the Fall 2004 semester we had rolled out a beta version of the syllabus viewer and began efforts to promote the project among students and teachers and gather feedback. During the Fall semester, we made a series of improvements to the viewer based on user input, and we also struggled to stabilize the platorm on which the viewer was running.
As the Spring 2005 semester rolled around, we again encoded about a dozen SIMS class syllabi into XML instances. Early in the semester, we held a model and schema review with members of the Center for Document Engineering. Based on the results of this review, we made significant changes to the schema and transformed all our existing instances into this new model.
We had two main points of focus for work during the Spring semester — Lisa worked on improvements to the viewer in correlation with Nancy Van House's Needs and Usability Assessment class, while Lisa, Carolyn, and SIMS first-year Sarai Mitnick worked on developing the SylViA editor in Marti Heart's User Interface Design and Development class. Work on the editor was particularly important, since we knew that convincing professors to use SylViA to create and manage their syllabi would require a significantly inviting and intuitive interface.
By the end of the Spring semester and the time of this report, a final version of the SylViA viewer has been rolled out as the culmination of Lisa's usability testing. The final prototype of the editor has been established, and much of the backend code needed to connect this prototype to our database has been completed. During the summer, we will be polishing up the editor code and performing a final round of user testing to make sure that editor functionality is up to our standards. We will also work with Kevin Heard and the SIMS computing staff to come up with a plan to deploy SylViA at SIMS so it can live on beyond our graduation.
In developing a data model for SylViA, the Syllabus Viewing Application, our first decision was whether to base our model on the work that Lisa had done for her final project for INFOSYS 290-8: XML and Related Technologies during Spring 2004. Ultimately, we decided to start from scratch, since Lisa's original model was not developed in a thoroughly rigorous manner and, more importantly, some aspects of her model were developed with the goal of exploring interesting aspects of XSD and XSLT, rather than the goal of long-term usability and interoperability.
Once we had determined to develop a model from a clean slate, we endeavored to apply the Document Engineering approach. We began by establishing our scope and context. The scope was and is an important driver for model development. Our overriding desire on this project is to develop a fully-functional application that would live on and be used at SIMS after our graduation, and this top priority informs all our decisions. For instance, we decided early on to focus only on syllabi at SIMS with a target audience of SIMS professors and TA's. While we would certainly love to see SylViA deployed in other departments or even other campuses, we felt that such ambitions were beyond the capabilities of two people in less than a year of time. Furthermore, we felt that having a broader, campus-wide scope would ultimately dilute our focus on developing the best possible application for use at SIMS. We decided to keep the campus perspective in mind but resolve any issues in favor of building a better application for SIMS.
With our priorities established, issues of context fell into place. Indeed, working on a masters project at SIMS is one of the more favorable environments in which to develop a model-based application. First of all, every SIMS syllabus is published online in one form or another, so amassing an inventory of sample documents was very straightforward. Also, it is the general disposition of SIMS professors and especially SIMS students to embrace new technologies and new applications of technologies, so we felt our chances of establishing support for our efforts was better than average. And while there are a few systems in place on campus that manage syllabi in one way or another, such as Blackboard, WebCT, and Study.Net, none of these were in common use at SIMS, so there was no institutional resistance to our project. Finally, in terms of our larger context of managing information and metadata in an educational institution, our work in areas such as the SIMS core class INFOSYS 202: Information Organization and Retrieval, in efforts like the System Map and Event Calendar projects, and in libraries and archiving gave us a strong theoretical grounding for the work we were doing.
With the scope and context in mind, we began to collect an inventory of sample documents to analyze. We decided to use one representative syllabus from each SIMS professor who had taught in the last year. We also included all the core courses. Since we found that most professors' syllabi were fairly consistent across different courses, we felt that one from each would give us a fair overview of how our target audience thought about this problem space. Our final list of syllabi analyzed in this inventory/harvesting phase is as follows:
We then divided up this list of documents and each harvested components from our half of the samples. We collected a list of components in a separate spreadsheet for each document. We captured the basic parameters of name, data type, and description for each component, plus a list of possible values where applicable. The spreadsheets for each document can be found in the appendices.
Even within the very limited scope of recent SIMS syllabi, we found a great variety of document types. We were not really surprised by this discovery, since the lack of any standard approach to class scheduling was a constant frustration for students and a major motivation for the SylViA project. Our syllabi ranged from fairly transactional forms, with tables of class titles, readings, and assignments, to more narrative documents with long descriptions of topics and discussion questions and a notable lack of specific dates and assignments. Still, this range of document types yielded reasonably consistent lists of candidate components.
Proceeding with the Document Engineering approach to modeling, we reconvened with our various lists of components and consolidated them into a master list of components that we wanted to include in our final model. The major factor in deciding on the consolidated list of components was a reevaluation of our scope. We knew we wanted to focus on SIMS syllabi, but during our harvesting phase, we tended to look at all parts of the website for a class whenever a site was present. As such, we ended up with unique elements like "instructor portrait," "rules of the road," and "FAQs." We quickly realized that developing a full-featured course website was beyond the scope of what we hoped to accomplish. We decided to make it a new priority to focus exclusively on elements related to the schedule of classes, assignments, and exams within a course. We knew that we would have to collect some basic course information to make the schedule meaningful, but mostly we tried to excise components that were not extremely common or did not relate directly to the course schedule.
On the other hand, we were faced with something of a conflict during the consolidation phase. Much as we had tried to analyze a reasonable subset of existing syllabi, we now needed to choose a reasonable subset of our components. Our main goal was to develop a clean, robust, and elegant model of syllabus data. But we were also determined to build Sylvia into a fully deployed application that could be adopted by professors at SIMS and used well after our graduation. We knew that some professors would be attached to the idiosyncrasies of their own syllabi, and that we might encounter resistance to adoption of SylViA if we did not embrace as many of these unique features as was feasible. In general, if we felt we could accommodate a component from a single professor without compromising the integrity of the model or causing too many problems for transformation, we would try to include it. But if a component was too far out of our scope we would remove it from the list. Our list of consolidated components (with a column indicating from which syllabi a component was derived) can be found in the appendices.
Armed with our list of consolidated components, we began to work on aggregating these components into logical groupings based on functional dependency. We then wrote these aggregations on index cards and laid them out on a table and began to form the network model of our component groups. This was an iterative process, as some aggregations emerged rather naturally (i.e. office hours, with its components of day, start time, end time, and location) and other components did not have an obvious home (i.e. units). We grouped and arranged and grouped and arranged until the outlines of a reasonable model began to take shape.
As this document component model became clearer, we began to think about how we might construct our document assembly model, that is, how we might create a hierarchical structure out of the network of associations. This task was difficult at first, because our gut feeling was that syllabus ought to be the root of any document we were constructing, but our basic course information felt out of place within this structure. Finally we realized that, just as a syllabus is one part of a greater course website, it is really just one part of a course, and course ought to be our root element, with syllabus as one of its children. Furthermore, we found that handling for courses with labs and discussions became very easy within this hierarchy. We simply allowed for multiple syllabus elements within a course, since each type of meeting (lecture, lab, etc.) had its own schedule and location and assignments.
Perhaps our biggest stumbling block was in modeling readings. We knew that we would have to have at least a few different types of readings in order to capture the disparate bibliographic components of, say, legal cases and websites. But all types of readings shared some common components, like requirement and page span. And we did not want to make the model too confusing to the user by having a very complicated model of readings. We settled on having a generic reading component with some of the shared elements, and then contain within that one of four broad categories of readings - articles, websites, cases, and readings from textbooks.
Once we had worked out this assembly model, we captured it in a master spreadsheet modeled on those used by UBL. In order to capture something of the hierarchy in the extremely flat format of Excel, color coded rows are used. Each aggregate component is indicated by a pink background, with any leaf components contained therein in white, and any children which are themselves aggregates are highlighted in green. Our final draft of the spreadsheet can be found in the appendices.
Armed with the spreadsheet, we moved on to encoding the model in XML Schema. We chose to use all elements and no attributes, which we feel enhances usability for a less technical audience, and we used Venetian blind style, with the root element as our only global element and otherwise all global types. As we encoded, we decided to take a look at the BABL schemas developed by previeous members of the CDE as part of their effort to model the course approval system. We wanted to reuse parts of their schemas in order to create continuity among projects to model university domains and to take advantage of their codelists for things like grading options and academic departments. We identified a handful of BABL elements that also appeared in our model, imported the BABL schema, and reused these elements. We then updated the spreadsheet to reflect the source of these elements. The rest of schema development went smoothly, and we were ready to test our model by encoding syllabi for the Fall 2004 semester.
Since Lisa was TA for INFOSYS 255: Foundations of Software Design, and Carolyn was TA for both INFOSYS 290-16: XML Foundations, and INFOSYS 290-17: Model-Based User Interfaces, we first attempted to encode those syllabi. We also recruited other TAs who were generously volunteered to use our model in their classes, including Alison Billings, Tu Tran and Vivien Petras in the fall, and Margaret Spring and Joshua Solomin in the spring. We even had one volunteer, Judd Antin who was not a TA but liked SylViA so much that he volunteered to maintain a couple of instances.
As we began to roll the model out to different users, we found two major adjustments had to be made. First, we found that we would need to allow some basic HTML tags in the more narrative elements in the model, such as course description and comments. We considered using strict XHTML, but we felt that would be too restrictive for a casual audience that wanted to mark up their content in any way they were used to. Also, we really wanted to protect the TA's who nice enough to help with the project from the added burden of namespaces. We developed our own lightweight implementation of the most common HTML tags, using substitution groups to allow arbitrarily nested mixed content in our few narrative elements.
Second, we discovered that we would need to allow more flexibility in class scheduling. Part of the conceptual model for the syllabus app is that we keep the syllabus separate from any particular semester calendar, which allows for reuse from year to year. To achieve this, the class meetings in the syllabus are numbered, not dated. We have a separate calendar instance for each semester, and based on the day pattern (i.e. Tuesday/Thursday, Monday/Wednesday/Friday) of the course, we match up class numbers to dates for the current semester. One of our users, however, had to miss a class and wanted to reschedule it on a day that fell outside of the normal day pattern. We decided to add an option to assign an actual date to a class and not just a number in the class sequence.
Aside from those changes, the model remained fairly stable during Fall semester, and we concentrated on incorporating user feedback into our transforms and the workflow of our application. But when Spring semester began, we were faced with another set of modeling challenges.
When choosing our document inventory, we only looked at professors who had taught a class in the past year. During that time, there had been very few course offerings in the legal aspects of information, though that subject is a SIMS core competency. But in Spring 2005, there were two legal courses being taught, and we found that our model was woefully unequipped to deal with the subtleties of those syllabi. On the most basic level, there were many more kinds of legal readings than just cases, so we had to shoehorn things like briefs and bills into our model. But more importantly, we found that some of the legal professors at SIMS needed to use full Harvard Blue Book citations in the readings, since their syllabi were being used as references for others in the field. Other professors were assigning readings such as articles from anthologies, which did not fit well with our existing article model. These hurdles, combined with a forthcoming design review with the Center for Document Engineering, motivated us to consider overhauling our reading model altogether.
In February 2005, we sat down with members of the CDE to review our schema and get suggestions for improvements. It was a very fruitful meeting, and afterwards we identified a handful of important changes that would require a fairly serious overhaul of our existing instances, but which we felt were necessary to make a better model. First, on the advice of Alex Milowski, we moved required elements to the top and optional elements to the bottom of almost all our aggregates. This structure will make the schema easier for others to reuse and repurpose. Also, we chose to put in wrappers for elements which could occur zero to many times. For instance, a course can have no textbooks or multiple textbooks, and in the original model textbooks were just allowed to occur as siblings under the parent course element. In the new model, we added a textList wrapper, with would contain the list of textbooks and could optionally be empty. Having these empty wrapper elements rather than lists that might not be there makes XSLT processing smoother.
A significant decision was made to excise the BABL schema and make our syllabus schema entirely self-contained. Other than the codelists, the only BABL element used in our schema was personal name, and we didn't feel it was a particularly strong name model. Also, BABL had been created two years before, and the practices for modeling and schema development in the CDE had changed significantly since then. We copied the necessary codelists into our schema and namespace, created our own name model, and removed the BABL imports.
In response to user requests, we also decided to add handling for take home exams. We had felt that a take home exam was really an assignment and could be modeled as such, but our users really wanted to keep it in the exams section of the model. We created an exam substitution group to handle both take home exams with assigned and due dates, and in class exams with a time span and location.
Most significantly, we decided to throw out our previous models of readings and redo them from scratch. After much debate, we decided that readings should indeed be modeled as substitution groups. We had resisted this construct in our original model, feeling it was too complicated for the casual user. But since we were now building an interface for entering syllabus data that would protect the user from raw XML, we felt we could implement a more sophisticated model. Also, we realized that if we had used substitution groups in the first place, we could have added on elements for each new type of readings we were encountering, and then we wouldn't have had to throw out our original model. Our new reading types included brief, bill, case, journal article, anthology article, book chapter, and website. Each type included all necessary fields for full bibliographic citation. To offset the tedium of having so much content to markup, we only made the most essential elements required for each type of reading.
Those changes represented the penultimate stage in our modeling process. We now felt very confident in our model's ability to gracefully accommodate any SIMS syllabus, and we moved on to other aspects of the project, including, most importantly, building the input interface. The last change that we might consider after this interface is up and running is replacing our lightweight HTML implementation with actual XHTML, though in the transitional rather than strict form. When users have a clean interface to use and we can handle the namespace issue behind the scenes, we feel that it might be proper to use the existing standard for XML-style HTML. After all, the most important thing a modeler must keep in mind is to reuse standard models when applicable and not reinvent the wheel with every modeling effort.
See the final syllabus schema and model diagram in the appendices
.The decision to develop SylViA on Cocoon was informed by its roots as a final project for the Spring 2004 course INFOSYS 290-8: XML and Related Technologies, taught by Alex Milowski. The project requirements stipulated that at least one technology taught during the course should be implemented. Lisa chose Cocoon because it supports the creation of a dynamic website where content and presentation are completely separated. When Carolyn joined Lisa in developing the syllabus application as a masters project, there seemed to be no need to change the application's infrastructure, and development on Cocoon continued.
Cocoon is a Java-based open source XML publishing framework developed under the auspices of the Apache Software Foundation. Cocoon can be installed on several different servlet engines, though the most common is Apache's Tomcat. Cocoon utilizes pipelines to handle requests to a web application. Based on the requested URL, the correct XML files are generated, then transformed using XSL files, and then serialized to the browser. A single document, the sitemap, is used to control the functionality of the web application. Also in the sitemap are the components that define the functionality of the Cocoon installation. The most common components, used extensively in SylViA, are the pipelines, generators, transformers, serializers, readers and matchers. Others include selectors and actions, resources and action sets. For a more detailed explanation of the components, please see Langham and Ziegeler's Cocoon: Building XML Applications. The overall structure of the sitemap is very complex because of the number of components that are defined and the number of parameters that can be set.
The most important component for a web application creator is the pipeline. Within the pipelines, a developer creates the URL match patterns, and, within the matchers, combines generators, transformers and serializers to generate the content. A simple example of a matcher within a pipeline for the URL "webapp/view/243.info" would display the course information for the SIMS course INFOSYS 243: Document Engineering. The matcher would look like:
The first asterisk in the pattern attribute maps to "243" and Cocoon generates the file 243.xml from the directory named "xml". The second asterisk maps to "info", and Cocoon then transforms 243.xml with info.XSL from the directory "xsl". Cocoon serializes the results as HTML, and the user is able to see the results of the transformation in her browser. An alternative way of constructing a matcher is to use readers instead of generators, transformers and serializers. Readers are used when the document should be outputted unchanged. An example of a reader is:
In this matcher, a CSS document is sent to the browser directly. Using readers simplifies the sitemap, however, for every document mime-type, a separate matcher must be written.
Cocoon supports further functionality beyond simply outputting dynamic content, including form handling, though that functionality has been under rapid development and is not sufficiently robust or well-documented for our needs. For further information on Cocoon's functionality, please see the Cocoon website: cocoon.apache.org. However, documentation on the official Apache site is slight and vague, and the Cocoon wiki and mailing list should be referenced for more explicit instructions.
Sylvia uses the most basic of Cocoon's functionality - its ability to dynamically generate web pages from an application's XML and XSL files. Much of our development on Cocoon has been focused on writing matchers that support a variety of different views of our data. As we have developed Sylvia's output interface, the most important changes we have made have been to the navigation, and much of our effort has been focused on making Cocoon support better navigation.
The original version of Sylvia had minimal navigation, and what little there was had been hard coded into the individual view transformations. It was possible to select different views of a single syllabus, but it was impossible to go from one syllabus to another. There was not even any way to go to the application's home page. For reasons of usability, it was decided that a navigation bar should be added to the site that would broaden the navigation. However, the top navigation bar was still hard coded into the individual view transformations. Also, to create a single view of multiple courses, users had to type in a complex URL themselves. It became critical that a more sophisticated navigation be made available and that the navigation code be stripped from the individual views.
We discovered aggregation, a feature of the Cocoon sitemap that could be utilized to combine individual syllabus views with a single global navigation bar. The aggregate function allows a matcher to call any number of other matchers and combine all their outputs together, and run transformations on the aggregation's results. At the same time, we discovered that Cocoon can scrape a directory, outputting the contents as an XML list of directories and files. Putting these two functionalities together meant that the navigation bar could be modularized and separated from the content display, and that the content of the navigation bar could be dynamicly created depending on the syllabi in the XML directories. The initial version of the new site navigation meant that 5 matchers were utilized for every syllabus used. This resulted in a serious performance issue, discussed below. Since the new navigation made SylViA much more usable we did not want to roll it back, and we began to explore how to overcome the performance issues. The number of matchers called was reduced to three, which reduced the performance problems slightly (see diagram below). However, the change meant that the XSL that combines all of the elements together, aggregate.xsl, contains header and footer content. While this does not match the ideal of separating content and presentation, the performance benefits were sufficient to make the compromise worth while.
The initial sitemap was largely unchanged for the first 6 months, as Sylvia development focused on the schema and transformations. However, several technical issues that would become critical first became apparent in those first six months. The first was Cocoon's handling of errors. There is no default error page, and when an error occurred, the user was shown the stack trace for that error. As the URLs required to view content were complicated, incorrect URLs were frequently entered. Within the pipelines of the sitemap, at the same level as the matchers, a handle-error statement was added. Users were now shown an error page that gives tips on how to access the information.
A more critical problem that manifested during the first six months was a caching issue. SylViA was being used as the only syllabus for several courses during the Fall 2004 semester, most notably INFOSYS 255: Foundations of Software Design. It became clear over the semester that Cocoon was displaying inconsistent data depending upon which syllabus view was chosen. Eventually, it was determined that Cocoon was caching incorrectly. If a URL was visited too frequently, it would not always be refreshed with the most up-to-date information. A less frequently visited URL would be more current. Cocoon's sitemap allows developers to set caching variables for every pipeline. They can be set to "caching" or "noncaching", and the length of caching can be set as well. Initially, all of the pipelines were set as caching, which appears to be the default. Changing them to noncaching ensured that stale data was no longer served.
Efforts to solve the caching issue, however, resulted in the most severe performance issues to date. Cocoon began to serve completely blank pages. Any modifications that were made to resolve the stale data issue increased the likelihood that blank pages would be served. Careful determination was made as to which matchers could be cached because the information is relatively static, and which should not be because the information is dynamic. The matchers were then grouped in to the appropriate pipelines. Also, the significant improvement to the site navigation, described above, increased the frequency of the crashes. Restarting the Tomcat instance was the only way to get data outputted rather than the blank pages. Eventually, Tomcat was set up to restart three times a day, though on days with heavy traffic, Cocoon would still begin serving blank pages before a restart. In an effort to minimize the problem, SylViA's Tomcat was upgraded from 4 to 5.0, but the issue persisted. At the same time, caching settings in the sitemap continued to be refined. Cocoon mailing list entries were combed, and though some suggestions helped they did not entirely resolve the issue. Eventually, it was determined that the cause of the blank pages was a memory issue, primarily because the JVM was set up with too little memory allocated to it. Mika Fonsén, in his "solution: performance and use-persistent-cache" posting to the XML-cocoon-users mailing list, recommended several changes that could be made to resolve this problem. Once all of Fonsén's recommendations had been followed, the blank page issue disappeared.
The instability of Cocoon as a platform made us look into alternative pipelining technology. We eventually decided to use smallx, a much more lightweight solution created by Alex Milowski. As we do not need much of Cocoon's functionality, the smaller function set offered by smallx met all of our needs. Also, given the problems we are having with memory usage, a less resource-intensive solution was desirable.
Smallx does not cache in the same way that Cocoon does, resulting in lower memory requirements. Rather than caching the results of pipelines, smallx caches the pipelines themselves. The XML file that is run through the pipeline is never cached, but the transformations that are used by the pipelines are cached. This means that each time a file is called, it is pulled into the pipeline, but because of smallx efficiency, there has not been any noticeable change in response time. However, the first URL visited after a server is started always takes a long time to load because it triggers the caching of pipelines.
Smallx does not use a sitemap to match and parse URLs like Cocoon. Instead, URL Rewrite is included as part of the smallx installation. The application is configured to first try to match a URL against the URL patterns defined in URL Rewrite's urlrewrite.xml document. If no matches are found, the application then tries to return the requested content based only on the URL. Not having a sitemap in many ways makes serving content easier. Cocoon's sitemap had to have matchers for every kind of file that might be requested. To serve a CSS file, a CSS matcher had to be added to the sitemap. On smallx, no configuration has to happen for the CSS file to be served.
Rather than being grouped into a single document, the pipelines are each a discrete document with a .pd file extension. The pipeline files are implemented by smallx when a URL is matched in urlrewrite.xml. If the URL for viewing an individual syllabus is entered, URL Rewrite rewrites the URL to a form that a pipeline can consume and interpret, and then passes it to the pipeline that generates the individual views. If a URL for viewing multiple syllabi is entered, then URL Rewrite passes the information to the pipeline for multiple views. A smallx pipeline is very different from a Cocoon pipeline in that it constructs a document rather than acts upon a document. Cocoon holds an XML document in memory, and then runs XSL transformations on it. A smallx pipeline instead pulls in the XML document, wrapping it within itself. Transformations are then run on the pipeline document which holds the XML document. However, it is possible to choose what parts of the pipeline document a transformation should be run on, allowing for a degree of control and focus that is hard to obtain in Cocoon. However, it also makes it impossible to pass variables from the URL to the transformations. Those variables have to be embedded into the pipeline document and the transformations have to be able to pull the variable from the document. This can result in transformations that are tightly bound to the smallx framework and not easily transferred to other environments.
Once the Cocoon-related memory issues had been resolved, development of SylViA on Cocoon was frozen. However, during the Spring 2005 semester, Lisa took INFOSYS 214: Needs and Usability Assessment. For that class, she performed a three part usability review of the SylViA viewer because it had been developed with little or no feedback from users. After each part of the review, she revised SylViA for the next part. She decided to use this project as an opportunity to test implementation of smallx as an alternative to Cocoon. She made her usability related changes to a second installation of SylViA based on smallx. For the most part the development already done on Cocoon could be transferred to smallx. Alex Milowski provided invaluable assistance by setting up the initial pipelines and by changing them to meet SylViA's requirements. However, because of the significant differences between the pipelines in smallx and Cocoon as described above, many changes had to be made.
Several features in Cocoon do not exist in smallx. Cocoon's aggregation functionality, which had been so critical to improving the site navigation, has no comparable feature in smallx. Instead, each pipeline is an aggregator, pulling content into itself. A separate transformation for generating the information used for site navigation content was no longer necessary. A Cocoon feature that was critical to implement, and first seemed impossible in smallx, was choosing a transform dynamically based on the requesting URL. This feature allows for the various views of coursework that are central to SylViA's appeal. Cocoon's pipelines were able to choose which view transformation to apply to an XML document. However, because smallx caches the entire pipeline at startup, the calls to each XSL were hard-coded into the pipelines. Adding a new XSL or changing an XSL's name required updating the pipeline and restarting the server. It was critical that future developers of SylViA be able to add new transformations without having to restart the server. Alex Milowski rewrote the pipelines he originally provided to accommodate this requirement. If a new view transformation is added to SylViA, it can utilized without restarting the server simply by adding it to the "xsl" directory and adding its name to the appropriate matchers in the urlrewrite.xml document.
One nice piece of functionality that smallx provided but Cocoon did not was the ability to view an infinite number of courses at once. By the time development on Cocoon ended, it was still not possible to view multiple courses without specifying how many courses were being viewed in the URL. This limited the number of courses viewable at once in Cocoon to whatever was in the matchers. Smallx makes it possible to view as many syllabi as are available in a semester. URL Rewrite passes the list of courses to the multiple view pipeline, each course separated by a period. An XSL transformation within the pipeline splits the list at the periods, and then pulls in each XML file. The transformation is able to handle a list with a single course, or 5 or 10 or any number of courses. While there is currently no link on SylViA that makes use of that functionality, it makes the application more flexible for future development.
The smallx pipelines make it possible to tightly control the output. Smallx supports checking to see if a directory or XML document exists. Within the pipeline it is then possible to choose an error page based on whether a directory or file does not exist. If a semester that has no syllabi is requested, users see an error message telling them that their semester is not in SylViA. Likewise, if they enter a course number for syllabus that is not in the SylViA, they receive an error message telling them that. Another example of changing the output based on the context is on the printable versions of the syllabi views. On the default page views with the site navigation, there is a link to see a printable view. When looking at the printable view, however, it makes no sense to include that link. The printable view pipeline includes a call to a transformation that is run only on the block-level element that contains the link, replacing it with a link back to view with the site navigation. Overall, these refinements improve the usability beyond what would have been possible in Cocoon.
Overall, smallx has proven to be a better alternative to Cocoon for SylViA. Its memory usage is smaller, which has increased SylViA's stability. It is also a smaller servlet, making the SylViA viewer more portable. The URL Rewrite matching patterns can be defined using regular expressions, giving better matching control. Finally, as it focuses exclusively on pipelining XSL, it does not overload SylViA with unused functionality. And it must be noted that having a close relationship with Alex Milowski, smallx's developer, allows us to get code revisions and customizations to meet our particular needs.
There are four kinds of pages that SylViA provides. They are: static HTML pages; the dynamic index pages; views of syllabi; and RSS-related pages. The syllabi can be viewed individually and in different combinations, and there are several different views of the syllabi providing different levels of comprehensiveness.
Though the HTML pages can be viewed without being served by smallx, urlrewrite.xml includes a matcher for them. This is done because the pages need to be wrapped in the site navigation. The navigation is dynamic, and dependent upon the context of the URL. When an HTML page is requested, the pipeline for static content is called, which generates the site navigation content. The HTML page is then wrapped into the pipeline document, and the final transformation, navigation.xsl, is run on the document. This pipeline process is used for the top level index page, the help page, and the error pages.
The dynamic index pages are exclusively for the semesters. Each semester index page displays a list of the syllabi that have been included in the application. Originally, these pages were manually created and served as HTML pages. However, it was important that they be generated automatically so that SylViA can live on at SIMS with minimal administrative intervention. URL Rewrite is able to differentiate between the top level index page and the semester index pages. It calls the dynamic index pipeline, which is generates the list of courses for the requested semester. A transformation is run on the list, which outputs the content of the index page. The navigation transformation is then run on the pipeline document.
The syllabus views are much more complicated and require several pipelines depending upon what category of view is requested. The four categories are individual views, printable version of individual views, multiple syllabi views, and printable version of multiple syllabi views. Each category has its own pipeline, and the printable versions do not include the site navigation. URLs are structured so that URL Rewrite is able to distinguish between the categories, and URL parameters are passed to the appropriate pipelines. In all of the pipelines, the date assignment transformation is run on the syllabus XML content. This transformation interprets the class number and translates it into a date. This allows a syllabus to be moved from semester to semester without having to change the dates for classes, assignments, and exams. Once the date transformation has been run, a view transformation is then run on the content. Each view determines what parts of the XML content and to what detail are displayed on the final page. The views range from displaying all of the content to just this week's course work. If the user wants to see the printable version, the document is then output. However, if they have chosen one of the other categories, the navigation transformation is then applied.
The final set of pages support SylViA's RSS functionality. Users are able to subscribe to an individual course or a combination courses, and in their RSS aggregators or readers they are able to see their coursework without visiting the SylViA viewer. To make this functionality possible, there are two pipelines. The first is for generating the RSS file that is read by the RSS reader. The RSS file is just an XML document that is parsed by the reader. The second pipeline generates the subscription page. This page displays the URL for the RSS file, as well as tips on how to use the subscription service.
As we debated the relative merits of Cocoon and smallx, we were also deciding on a technology and interface for managing syllabus input. While several members of the SIMS community have been willing to work with raw XML instances, many more are uncomfortable even with HTML. For SylViA to be successful and to live past our graduation, it was crucial that an interface and workflow be developed that hides the creation of XML instances from syllabi creators.
We considered a variety of options and a variety of tools for the transparent creation of marked up syllabus data. A logical approach might have been to use the forms and database functionality available in Cocoon, but Carolyn knew from prior experience that would not be possible. As part of the System Map project team last year, Carolyn had investigated accessing a database from within Cocoon and found the handling to be almost completely undocumented. She found one code snippet that demonstrated one narrow database interaction, but nothing like the robust handling necessary to develop a full-featured database-driven web application. Furthermore, Kate Ahern and Dave Schlossberg had tried working with Cocoon's forms handling and found that they could not adequately validate entered data. So we knew Cocoon would not be a reasonable platform for the input interface. The functionality of smallx is focused entirely on XSL pipelining and excludes the bloat of Cocoon's forms and database handling, so using smallx would clearly not be an option.
Since Cocoon and smallx were not a viable option, we next looked at Smart Draw's Visual Script. The Visual Script software allows a programmer to take shapes, icons, and pictures from standard libraries and define visible fields and invisible XML to go with each image. These images with the customized code are packaged as a template, and the end-user just drags and drops images, fills in fields, and generates XML instances without having to see the code. For SylViA, we could imagine that a professor would open up a syllabus template and see a large shaded area which represents the syllabus itself. On the drawing clipboard they might see an image of a book with fields attached for title, author, publisher, etc. This image would represent course text, and the professor could drag it onto the syllabus and fill in the information for a textbook. Code would be associated with each object that allows Visual Script to compile the proper XML behind the scenes when the user is finished with a drawing.
Visual Script is an interesting and compelling tool, but we ultimately decided against pursuing it as our input interface. When experimenting with the program, we found it more difficult to create complex hierarchies than we had anticipated, and so we worried whether we would be able to manage our fairly complex model. We were also concerned that, given the complexity of creating a syllabus, merely presenting users with a wealth of icons to negotiate and no real guidance would not be the optimal solution. On the one hand, Visual Script gives the user quite a bit of flexibility; on the other hand, we felt that designing a thoughtful workflow in a more traditional forms-based interface might be even more beneficial to our users. And finally, we were worried about the notion of requiring our users to install a new program and learn how to use it. We needed to make adoption of our application as painless as possible for users, and without a really compelling value proposition, we did not want to require the use of new proprietary software.
Another option emerged during the fall 2004 semester as part of our work in INFOSYS 290-3: Web Services: Concepts, Design and Implementation, taught by Adam Blum. While web services per se do not address the problem of user interface design, we had a guest lecturer, Roger Sippl, who gave us a demonstration of a very promising piece of software called Above All Studio. As a final project for INFOSYS 290-17: Model-Based User Interfaces, Lisa analyzed whether it would be appropriate for us to use Above All Studio to create the interface for our application.
Above All Studio allows users to create applications that are a composite of many web services. The applications are form based, and those forms are automatically generated from web services' definition documents, or WSDLs. A Studio user adds any number of WSDLs into her Studio Dictionaries, and Studio analyzes the WSDLs and can automatically generate input forms based on the web services' expected elements. Various forms can then be grouped together in an Application, and Applications can be deployed either in a proprietary format that can be run on any computer that has Above All Runner installed, or as ActiveX in an HTML solution that allows users to access the form via a web browser.
In order to use Above All as our interface generator, we would need to parse our data model into smaller chunks which would be published as web services. Our users would use the Above All forms to submit the necessary data, and our server would capture the information to build up syllabus instances. Lisa discovered a variety of problems with this approach, however. First of all, the backend of our service was developed in Apache Axis, and we found this particular piece of open-source software to be exceedingly difficult to configure and severely lacking in documentation. And, given our experiences with Cocoon, we were concerned about whether an Axis-based solution would be scalable.
Even more problematic, Above All Software's Studio is based on C#, creating compatibility issues with web services that are run on Java, such as Axis. The date/time data types in C# are not conformant with the data type in Java or XML. We would have to change all of our date/time fields in our model into strings, and this would increase the chance of users submitting mal-formed data. And finally, Studio has been optimized for creating forms that display information from relational databases, which can have serious consequences for displaying data that is captured in an XML instance. If we tried to create a web service definition that utilized XSD's choice option rather than select, Above All threw an error stating that such nesting is not currently supported. We were not able to accurately define our web service based on SylViA's schema, but had to make compromises based on C# and Studio's implementations.
Given these limitations, we felt that Above All Studio would not be the best solution for developing our syllabus input interface. Also of concern were platform demands - in order for Above All to serve ActiveX solutions through a web browser, it needs to be hosted in Windows, and SylViA as well as SIMS writ large is pretty committed to an open-source Unix/Linux environment. As much as we appreciated Above All's ability to maintain an explicit connection between our model and its forms, we decided to continue looking for input solutions.
Our next serious consideration was the use of XForms. In a sense, XForms would be a natural fit for us, since the architecture would mimic what we were already using to serve the syllabus output. We could imagine writing a transformation that would take our schema and output XForms code that would be displayed and consumed by an XForms-capable browser. The catch is that browser vendors have barely begun to implement even beta versions of XForms handling. So far, the only one we know of is a plug-in for Mozilla released in February 2005. While this was an encouraging sign, our struggles with Cocoon and Axis made us extremely wary of committing to yet another nascent, unproven XML technology. Furthermore, having performed a round of user interviews as a first step towards developing our input interface, we felt more strongly than ever that the input for SylViA must run in any professor's preferred browser and not require the adoption of a particular browser or the installation of plug-ins.
In the end, we are building the input using Java Server Pages, a choice determined by practical as well as educational considerations. For the spring 2005 semester, Carolyn enrolled in INFOSYS 290-17: Java Web Applications, where she learned to work with JSP and servlets. While JSP is not an explicitly XML-driven technology, the Java language does provide fairly robust XML handling, so we will be able to deal more directly with our instances than would be possibly in web scripting languages like PHP or ColdFusion. Furthermore, JSP can be deployed under the same Tomcat instance that runs smallx and Cocoon, so there are no architecture conflicts. And we feel the established nature of JSP and the wealth of related books and documentation will give us a much greater chance of producing a fully-functional application, given our limitations of time and human resources. Finally, a JSP solution is the only one which will meet our requirements of running reliably in any common browser, so our users will face no external obstacles to adoption of SylViA.
As we moved forward with JSP development, we had to decide on a database to comprise the backend data storage for the editor. Based on the database options offered at SIMS, we narrowed the choice down to Oracle 9i with XML extensions or Postgres. Since Oracle offered XML handling, it was a natural choice for our application, but in her previous experience with Oracle, Carolyn had found it quite unweildy and difficult to administer. We had heard good reports on Postgres from fellow SIMS students, but in the absence of XML functionality we would not be able to explore the intersection of our hierarchical viewer data model and a relational editor data model. After soliciting advice from database experts including Ronald Bourret and Matt Thorn, we decided that we would not compromise our vision for the application based on theoretical concerns about administrative overhead, and we decided to go with Oracle.
Having made the decision to use an XML-enabled database, we proceeded to design a data model that incorporated both relational and hierarchical elements. In general, we chose to store snippets of XML rather than shredded relational data in places where we did not anticipate searching across the data or where we were trying to avoid overly complicated joins. For instance, we have nine different types of readings in our model each with its own set of fields. In order to properly shred these readings into a relational model, we would have had to create nine different tables and face a serious challenge in querying across all the table to retrieve readings for a particular class. Instead, we chose to create a single readings table that stores a snippet of XML in the reading column and allows us to consolidate all reading types.
We created database tables and keys and populated some sample syllabus data, and we then proceeded to explore various options for coding JSP. In Carolyn's Java Web Applications class she learned about servlets and classic model-view-controller application architecture. In discussions with other students doing JSP development for their final projects, however, we learned that JavaBeans had emerged as the favored approach. Ultimately, we felt that the benefits of developing in parallel with other teams with whom we could share code and discuss challenges outweighed the possible philosophical superiority of the servlet approach. Also, developing with JavaBeans was simply faster than using servlets, and time was of the essence as project crunch time approached. So, with the help of Matt Meiske and Jesse Mendelsohn of the RDIR team, Carolyn got ramped up on JavaBeans and developed backend code for the editor.
In its original instantiation, the SylViA viewer provided several different views of individual syllabi and provided different levels of detail, both in content and temporally. It also allowed students to combine up to four syllabi into a single view. The SylViA viewer had been used as the primary syllabus for several courses in Fall 2004 and Spring 2005 semesters. However in that time, no study of its use had been done as all efforts have been on stabilizing the technology supporting the component.
The goal of the usability project was to increase the usability of the SylViA viewer component. To improve SylViA's viewer component, Lisa chose three methods taught in INFOSYS 214: Needs and Usability Assessment: surveying, usability testing, and heuristic evaluation. The first step was to perform a survey to study the current use of SylViA. Using the results from the survey, Lisa made changes to SylViA to better match current use and feature requests. Next, Lisa performed a usability test to ensure that novice users and expert users of the previous version would be able to navigate easily through the site. After making changes based on the results of the usability test, she then coordinated three heuristic evaluations of SylViA. After making the final set of changes, the final version was released to the public.
The purpose of the survey was to find out how users use SylViA. Specifically, we wanted to know what views users are using and which they find useful. We also wanted to know how many syllabi they combine using the multi-course view, and how many they want to combine. We also wanted to know what information users are going to SylViA for. And we wanted to find out how many users viewed the syllabi that were available in comparison to how many actually took the courses. The information from this survey was used to determine what features need to be changed or added. Our secret agenda for the survey was to use the survey invitation emails to raise awareness of SylViA.
The target group was students who are currently using SylViA, or have used it in the past. This group is mostly made up of first year MIMS students, since we made a special effort to include their MIMS core courses in the application. Two core courses, INFOSYS 255: Foundations of Software Design and INFOSYS 207: Analysis of Information Systems, even used SylViA exclusively to present their syllabi. However, many non-core courses are in SylViA, and the target group also included second year MIMS and PhD students. We also sent a plea to faculty to take part in the survey.
The survey takers were primarily first and second year MIMS students at SIMS, though a very few PhD candidates and Faculty members completed the survey, as well as one member of the SIMS staff. As we expected, first year MIMS students accounted for only 42% of the respondents but represented 70% of the SylViA users. Only 30% of the SylViA users were second year students. Of the PhDs, Faculty, and Other respondents, none had used SylViA. Common reasons for not using SylViA were: not knowing about its existence; classes enrolled in do not use SylViA; not being a student; and not using syllabi in general.
Users of the multiple syllabi function were exclusively first years. This is not surprising, as many second years did not have sufficient classes in SylViA to make use of the feature. However, of the 16 first-year SylViA users, only 11 used the multi course view. The reasons given for not using the feature were finding it confusing the first time it was used; and not realizing that the feature existed. Both reasons have hopefully been addressed by improvements to site navigation that occurred after the initial release of the multiple syllabi feature.
There did not seem to be a strong preference amongst survey takers for one view over another. They were asked about five different views: table (a tabular view of all course information), work (a non-tabular view of the same), to do (only required work), assignment (only assignments and exams), and administrivia (general course information). We asked survey takers to rate the five individual views on two scales. The first scale was based on frequency of use, and the second was about utility.
There was no view that was clearly used more frequently or seen as more useful than the others. The more comprehensive table, work, and to do views were rated higher than the limited assignment and administrivia views, but there did not seem to be a strong preference for any one view over another, as we had hoped there would be. This may indicate that people really do find the views equally useful, that the views were indistinguishable to users, or that the views are not sufficiently distinct.
We also found that users wanted to combine more than the previously available four classes. Most wanted to combine 4 to 7 courses in SylViA, except for those who wanted to combine as many as possible. We believe this last response is from people who want to use SylViA to see what is going on in all courses at once, not just to see what is happening in their own courses.
We wanted find out if people are using the internal links provided by SylViA. These links that do not appear in the left navigation bar but are just above the display of syllabus information. While the newly added link for a printable version of a syllabus was rarely used, the link to this week's course work was used often. This suggests that people are able to find the internal navigation links, and that other functionality can be linked to from there.
It became clear from comments that the reason people do not use SylViA is a lack of awareness. Some effort has to be made to educate students and faculty about what SylViA can do, what its URL is, and what courses are using it. An important part of this is making SylViA appealing to teaching teams so that they use it instead of their current syllabus publishing methods. A part of this process is developing a user interface for inputting the syllabus data, discussed below. As part of both these projects, we have involved many students and professors in the testing process, which is helping to raise awareness. Also, we will publicize the final release of the viewer and hope that our masters project presentation will make SylViA even more popular.
Though most of the changes that we made were prompted by the survey, some changes were also informed by informal feedback we have received in conversations with other SIMS students. We made significant changes to the look and feel of the site, such as changing the color scheme from grays and blues to greens and adding a leaf logo to the sidebar. Also, the font was changed from serif to sans-serif in an effort to increase legibility. We also eliminated the assignment descriptions from the bottom of the table and work views.
We decided to eliminate the to do view which was the least popular. We then added a week view that displays the readings, assignments and exams for the next week and a half for a course or combination of courses. Several people have informally indicated that they found it frustrating having to scroll through the entire semester, and that the "Go to this Week" link was insufficient.
We improved the side navigation bar by making the drop down menus default to whatever course and view a user was looking at. In response to user demand, we added a fifth course selector to the multi-course dropdown menu. This may be somewhat optimistic, because the addition supposes that a student will be taking five courses that have all their syllabi in SylViA.
We've also had several requests for SylViA to keep track of users' work that they have to do, and even have done. This was a more difficult problem to solve, as the actual technology of SylViA as it had been developed does not lend itself easily to creating a personalized work management system. However, Lisa realized that there is an XML-based technology in use that we could utilize to provide students with their course work without their having to go to SylViA every day, and that would allow them to "check things off their list." We created a new "view" that supplied SylViA's information in RSS, or Real Simple Syndication. If a user adds the new RSS URL for SylViA to their RSS reader, their reader will download their list of readings, assignments and exams. The RSS allows people to subscribe to their combination of classes, or to just a single class. This means that people will not have to visit SylViA to find out about their course load.
The next step was to perform a usability test to ensure that users are able to quickly navigate to the information they are seeking. As SylViA is a very shallow site, it is vital that the views are intuitively labeled so that users understand what they display. The behavior of the view selecting forms in the left navigation bar must also be studied so that users are not surprised by the results of their selection. The testing will also ensure that the help page provides information in a clear and concise manner, and that users do not discover functionality by reading it.
As we were looking at a system that was already in use, and one that would continually have new users coming to it, we decided to talk to users who had never used SylViA as well as users who were familiar with the original version. The original usability test consisted of three parts. The first was a questionnaire about the terminology used on SylViA, the second was a performance test of SylViA, and the third was a review of the help page. While exposing the user to terminology before they see it risks affecting their behavior on the performance test, we felt that the reverse order would have a worse effect on the questionnaire.
The testing was performed with eight people: two first-year MIMS students and one professor who had used SylViA, and three second-year MIMS students and two professors who had not used SylViA. The one of the latter professors was a family member who had not used SylViA, but is a professor of Computer and Information Science at Santa Rosa Junior College and a web design consultant. Instead of having all the testers use a single computer, they used their own computers. This became an advantage because we were able to see the interface and identify bugs on a wide variety of platforms. As it quickly became apparent that the initial terminology questionnaire was stressful to users and detrimental to the testing, it was abandoned after two users and the procedure was whittled down to the performance test and help screen review.
Overall, the testers expressed that they liked SylViA. They felt that it was a good-looking site that functioned well and easily. For the most part, they were able to perform the task scenarios quickly, much more quickly than expected. Some users chose to explore the different views to discover the differences, or would explain in detail why they liked or did not like something, which would lengthen the time spent on at task, even if they were able to complete it quickly.
The most important result that emerged from the debriefing sessions were that the views have to be more distinct from each other. The debriefing sessions were also useful for soliciting feedback on the help page, which became progressively less wordy. Also, after every two to three tests, changes were made to SylViA to fix the most severe usability problems. By the end, therefore, testers were able to discover the navigation functionality on their own, select the appropriate view, use the multiple course views, and approved of the help page. In general, the usability testing was enormously worthwhile, as we discovered many points of confusion that we would never have known had existed.
The changes made between tests ranged from very minimal to fairly major. Minimal changes included adding links and changing category names. More major changes centered on information that was available in a view. Other changes are very subtle, but required some significant work on the backend of the application. Also, some changes were performed in between usability tests, while others were performed after the tests had been completed.
An important intra-test change involved navigating from a semester index page to the SylViA home page, which proved impossible for the first two testers, in spite of the leaf logo on every page linking to SylViA's home page. We realized this web convention was not obvious to our users and added a home page link to the left navigation bar, so all subsequent testers were able to perform the task quickly.
Another change made during testing involved the quick view; users thought it was not different enough from the detailed view, so the quick view was reduced to just required readings, assignments, and exams. Recommended and optional readings were removed from the quick view, along with comments about readings and classes. This significantly streamed-lined the view and truly made it a "quick" view. Also, the default view was changed from the quick view to the full view, and users who triggered the default view responded positively.
One issue during the tests was confusing SylViA, the syllabus application, with a full course website, which was both understandable and ironic. During development of the syllabus model, we had made every effort to exclude the larger question of the course website and focus simply on the schedule. While it is possible to use SylViA as a course website, we wanted to make sure that users understood that they were looking at the syllabus and not the course, so we changed many instances of the word "course" to "syllabus", especially in the left navigation bar.
Our fourth tester pointed out that the discrepancy between the views available for single courses and multiple courses was serious. Single courses had six different view options; this week, quick, detailed, assignments, administrivia, and the default full. She found it confusing that there were not similar options for multiple course views. Multiple courses had only this week, and semester. So we reworked the semester view to make it the same as the detailed view and renamed it to match. We also added a quick view of multiple courses and an assignment view.
Only one tester commented on the RSS subscription, beyond saying that it seemed like a good idea. When she clicked on the "Subscribe to this Course" link, she expressed that she was completely confused by what she saw on the screen. It made us realize that the "Subscribe to..." link needed to link to a page that explained how to subscribe to a course, rather than to the RSS feed itself. We were able to add an introductory page to SylViA, and subsequent testers seemed less confused by what they saw.
The other changes to SylViA were made once testing was completed, either because they too complex to fit between tests, or because they were refinements that did not severely affect usability.
One professor indicated that she did not like having to move from the list of courses on a semester's index page to the left navigation bar to see a course. She wanted to click on a link in the main part of the page to get to a syllabus. We was concerned that the semester index pages were at that point being manually created. In the future there might not be anyone to create the page, and we wanted the list of syllabi on the semester index page to be dynamically generated. We were able to add this functionality to the application, and in future semesters, as syllabi are added to the application, the list will change dynamically. At the same time, the transformation that generates the list was written so that each syllabus links to its full view.
During every debriefing session, the testers reviewed the help page. Once they had all read and revised it, we posted the page to SylViA and added links to it in various places, using a bright orange that shows up against the green colors of the site.
Though the full view was now the the default view, it was not an option in the dropdown list. We decided that it had to be added, because we wanted people to return to the view easily without having to remember how they got to it the first time. However, we did not like the name full view, and decided instead on complete view. We felt that "full" was too vague, and "complete" was more evocative of the level of information available on the view. Another small change was made to the dropdown menu. For courses in the Fall 2004 semester, which employed an early version of the syllabus model, the this week view was not available, but the option still appeared in the menu. Testers were confused by being offered an option which led to a blank page, so we changed the dropdown menu and made the this week view available only for the current semester.
At the beginning of this project, we had switched the technologies that SylViA was served on. The most significant change as a result of this switch was the loss of any error handling in the site. If a malformed URL was entered, the user was no longer taken to a page explaining that an error was made and giving them the opportunity to fix it. The users did sometimes make errors, and we realized that we could not get away with not having any error handling. We decided that we needed to re-add error handling to SylViA before the heuristic evaluation. Instead of being presented with a server error page, or a blank SylViA page, users now see a modified help page.
We made two more small changes to the look and feel of the site. The first was to choose a slightly lighter green for the main color. This was because the orange question marks and the links in the left navigation bar did not visually "pop" as much as we wanted them to. The second was to change "INFOSYS" to "IS". We did this because it made lists look less cluttered, and it put the emphasis on the course numbers instead of the department code.
One tester made a suggestion we were not able to implement. She wanted the courses she had selected in a multiple view to stay selected in the dropdowns even when she went into a single course view. However, the technology that SylViA is served on, and our own technical limitations, made that impossible. Users will have to continue using the back button, even if this breaks them out of their normal pattern of use.
For the next step, the heuristic evaluation, one professor and two students served as evaluators. Each evaluator used Jakob Nielsen's list of ten heuristics, since the students had all used the heuristics in their course work, and the professor teaches it in her courses.
The student evaluators provided severity ratings for their violations, but the professor did not. However, she noted that there were no severe violations. Her comments were often in the form of suggestions rather than violations. The students rated the severity on a scale of 4 (usability catastrophe) to 0 (not a usability problem). The first student's severity ratings tended to be much higher than the second's. She also found more violations, a total of 13 violations. However, many of her violations came from the same page, the top level index page of SylViA. Her single level 4 violation and a level 2 violation were both known issues that had not yet been resolved. The other student found only 7 violations, and he told Lisa this is an unusually low number for him, and that he typically find many more violations when evaluating sites.
Overall, the violations and comments were about very nuanced issues, or were suggestions instead of violations. The evaluators' lists of violations had very little overlap. There was no single problem or area that all evaluators had severe problems with. Therefore, the list of violations tended to be informed by the evaluators' aesthetic preferences. The evaluators told me that they found the SylViA viewer to be robust and well done, and that many of their violations were comments on what they would like to see, as opposed to changes that need to be made. This made it very difficult to correlate the violations into a single condensed violation list.
Several of the violations centered on the side navigation bar. The professor felt that the information that was specific to the course being viewed should be highlighted in some way, to differentiate it from the other items in the side navigation bar. She also suggested that the dropdown menus for individual syllabi and multiple syllabi should be combined, so that there is only one set of dropdowns. Her final recommendation for the side navigation was to remove the link for the SylViA project's email address and to change "Contact SylViA" to "SylViA Contact". One student felt that the "Course Email and Links" section should be split into to separate sections. The other student felt that the side navigation changed too abruptly when navigating from the top level index into the semesters.
The title heading and the internal links at the top of the views also received several violation comments. Both students admitted that they had never read the headings. One student thought the this week view should include the week span, and the other thought there should be a "breadcrumb" that would indicate where the user is. She said that even though the SylViA viewer is very shallow, she still had to look in several places to figure out what course, semester or view she was looking at. Our professor wanted the "Subscribe" link to be moved to the side navigation bar and out of the view header. The first student felt that the "Subscribe" link should be available on all of the schedule views, rather than just the this week view.
Within the views, the professor and a student both commented on the overwhelming consistency of the display of information. The professor found it difficult to find this week's work in the quick or detailed views. The student was concerned that new users would find it difficult to skim the schedule. The professor also commented that the assignments blended into the schedule too much, and that they needed to be visually pulled out better. She also felt that the view should be renamed from assignments to something else because the view included exams, and she suggested deliverables instead.
The lack of consistency across all of the violations meant that we had to make many small changes throughout the SylViA viewer, rather than focusing on resolving a single issue. We had to decide whether suggestions should be implemented or not, and for those instances where the evaluators disagreed, we had to decide whose advice to follow.
In the case the top level index page, we decided that we would change it on the recommendations of our professor. This was partly because we felt that the professor understood our intentions with the page better than the objecting student evaluator. Also, the student was not able to make any suggestions on what she would rather have the top level index page display. We did not add the list of available semesters to the index page as the professor suggested, but we did rearrange the information, and we changed one of the headings to "Getting Started". No comments were made on the semester-level index pages, and they were left unchanged.
The side navigation underwent another round of changes. We thought the first student's idea of splitting the course email and link list into two separate lists was very good and implemented it just as he suggested. Though we liked the idea of combining the Individual and Combination dropdowns, the implementation proved to be difficult. Also, we did not feel that the change should be made without further usability testing. One of our concerns was that the dropdown list of views would have to change depending upon whether a single syllabus was selected or several syllabi. Realizing that we had some dropdown inconsistencies, we fixed them. In the individual view dropdowns for a course with multiple syllabi, such as Infosys 255 which has both lectures and labs, the Lecture and Lab view options appear after the user has selected that course. We decided instead to move the links to the Course Link section, so that the dropdowns never changed, except for this week only being available during the current semester.
We also liked the professor's idea of visually separating the Course Email and Course Links from the rest of the side navigation content. Once that had been done, however, the rest of the side navigation looked odd, and in the end we highlighted every section individually. We hope that this will in some way diminish the confusion of having the Individual and Combination dropdowns separated. The consistency of the highlight will also hopefully diminish the incongruency when moving from the top level index to the semester level.
The view header had changed little during the development of the SylViA viewer. However, after receiving our heuristic evaluation, we realized that several changes needed to be made. We were surprised to learn that the student evaluators had not even looked at the headings. We decided to decrease the font size of the course title so that more of the views' information is visible. For combination views, however, we decided to have a line of text for each course so that the course titles could be added.
Following our first student's suggestion, we added a "breadcrumb" to the view header, underneath the course title. The breadcrumb allows the user to navigate up to the semester index and to the syllabi's complete view. The view title is then listed after the link to the complete view. Following the second student's advice, we added the this week date span to the breadcrumb. We did not follow the professor's recommendation that the "Subscribe" link be moved to the Course Links section of the side navigation bar. This was because there was no comparable section in the side navigation for multiple views.
To address the issues of the difficulty of skimming the view information or finding assignments, we made several changes to the display of the information. We added an icon to the date heading that appears only in the current week's dates. It closely mimics the appearance of the orange bullet in front of the "Go to this Week" link, hopefully making its purpose clear. To make the assignments and exams more visible, we changed the layout of their information. Instead of being condensed on a single line, the information is broken up across several lines, making the assignments and exams more visible.We also added an icon, a white exclamation point in a circular color field, in front of those assignments that are due in the future. Finally, we changed the assignment view name to deliverables, at the suggestion of the professor.
Several changes were not made because we felt they should not be implemented without further usability testing. One such suggestion, from our professor, was that each class within the combination views should have its own color. She did mention that the site could look like a "jagged rainbow", but she was not the first to comment that the courses were hard to distinguish within the combination views. Another suggestion was sorting the courses in the combination view by their start time. This would cause the earliest class to always be listed first. The first student suggested this, thinking that the courses are displayed in random order. However, the courses are in fact displayed in their order in the dropdown menus. Before making a change, we would want to know if users would want their courses sorted for them automatically, or if they would want to control the order.
Though many of the changes made after the heuristic evaluations were quite minor, they sometimes required a lot of work on the backend. The date span in the this week view header is a particular example of this. However, the changes altogether significantly improve the navigation and look of the site.
Performing the usability assessment has proven to be invaluable. Every group of people who saw the newest version was more complimentary than the previous group. The site navigation has improved dramatically, and we are more confident about the site's quality. We wish that we had been able to perform the usability assessment earlier in the development process, as many issues and suggested that arose could be addressed and solved in a more complete manner if we had had more time to study and implement them. Regardless, we were glad to perform the usability assessment and implement the changes that were possible.
In order for SylViA to succeed and be widely adopted at SIMS, we need as many professors as possible contributing syllabi each semester. As our final project in Infosys 213: User Interface Design and Development taught by Marti Hearst, we designed an interface for inputting syllabus data. We were joined by SIMS first-year Sarai Mitnick. We designed the interface so that it is useful and intuitive enough that professors will actually want to maintain syllabi in SylViA. In addition to professors, we considered that TAs will be helping maintain syllabi, and multiple people might be working on the same syllabus. Also, the syllabus model is comprehensive and fairly complex, so we needed to wrangle a lot of information into a clear and concise interface.
Two methodologies are taught in User Interface Design and Development, User-Centered Design and Rapid Prototyping. User-Centered Design allows the design team to focus on the needs of the user to determine functionality and feature sets, rather than the requirements of the application architecture. Rapid Prototyping allows design teams to quickly create prototypes of an interface so that new ideas and solutions can be tested immediately rather than when it is too late to make changes. These two methodologies are used together, so that after each user test, a new prototype was created for the next user test. The results of our work in this class can be seen on our project website.
We began by conducting surveys with professors to determine what they would want in the editor interface. We then created a paper prototype of the editor, and conducted four usability tests on the paper prototype. We then created our first interactive prototype, and submitted it to another class project team for a heuristic evaluation. A second interactive prototype was designed based on their feedback, and a second round of usabiltiyt testing. The third interactive prototype was designed from that feedback, and submitted as our final prototype for that class. All of the interactive prototypes were built on Tomcat using a combination of HTML, JavaScript and JSP. The third prototype forms the basis of the final SylViA editor
An important part of the User-Centered Design methodology is to create personas. Personas are representative of possible users, and allow developers to talk about a particular, though imaginary, user in concrete terms. Personas also help in limiting the feature set to a deployable level. The basis for our personas and goals was the series of 7 interviews we conducted, as well as Carolyn and Lisa's own experience as TA's. In choosing people to interview, we tried to get a representative sample of potential SIMS users, and this breakdown is ultimately reflected somewhat in the personas. We interviewed a professor who has been at SIMS for a while but is not overly wedded to a particular format for an online schedule; another professor with a long history at SIMS who is very attached to his particular HTML syllabus; a fairly new professor without much in the way of technical skills; and our advisor, Bob Glushko, was an example of a professor who is particularly excited by the goals and technical architecture of the project and is therefore an early and enthusiastic adopter.
Among TAs, we chose a TA who has been maintaining a Sylvia-compliant raw-XML version of his course's syllabus as a favor to our team; and a TA who has been very involved in syllabus development for his course, and had designed and maintains his own HTML version of the schedule. Worries about getting interviews scheduled in time also led us to call on a third student who is officially TA for our second professor's class and also somewhat assists our third professor with scheduling in her capacity as GSR. She has also been maintaining an XML syllabus as a contribution to Sylvia. Furthermore, Carolyn was a TA for Bob's document engineering course. We found that having this overlap, where both professors and their TAs were interviewed about the same courses, gave us valuable perspective.
After conducting interviews, we gathered as a team and began to brainstorm about what types of personas we would need. Since our set of potential users is fairly small and familiar to us, we found the hardest part of developing personas was to abstract away from the actual people we know here at SIMS and make up fictional composites. Though we initially created five personas, we decided to focus on only two for development. The first was George Wilson, a relatively new professor at SIMS. The other persona we designed for was Parvita Spencer, a visiting lecturer. The three personas that were dropped were Barbara Levine, a longtime SIMS professor, and the TAs Calvin Masaharu and Anna Mendes.
We also developed scenarios that described our persona's interaction with the SylViA editor. The scenario that we used most was George's. As a new-ish professor, George had already used the My SylViA editor, but was creating a syllabus for a class he designed. His scenario allowed us to focus on the initial set up of a syllabus, and the entry of basic schedule information. Our usability testing task lists were based on his scenario.
The My SylViA interface is comprised of five parts:
Development of our prototype focused on the workflow for creating and managing syllabi. The most important workflow was the creation of the syllabus and the entry of basic course data. We also developed workflows for setting up course topics, editing the syllabus information, logging on to the editor, and managing the syllabus editors. Diagrams of the workflow can be viewed in the appendix.
The login page will be linked to from the SylViA viewer.
From the mySylViA home, the user is taken through the Set Up Syllabus wizard. The four steps of the wizard are:
The final page of the wizard is a confirmation page. The user can choose between:
Once the syllabus has been created, the user can edit the syllabus by selecting that option in the mySylViA home.
The user can then choose to edit the Schedule, Administrivia, Exams or Assignments pages.
Each piece of information, such as classes, readings, topics or office hours, has its own edit page. Once changes are submitted, the user is taken back to the originating page.
The user can preview what the syllabus would look like in the SylViA viewer
Once the user is finished, they are taken to a page that gives them the option of publishing their changes, or saving them as a draft for later review.
The user is taken to the Set Up Topic wizard under three different conditions
The three steps of the wizard are:
The user is then taken to the Schedule page.
The user can access editor management from the mySylViA home page. All management is done on a single page. Users can delete or add users, and transfer management. Once management is done, the user can return to the mySylViA home page.
Though in many ways our interface did not undergo any dramatic changes between the prototypes, each version implemented several small changes that altered the user experience significantly. Sometimes it was as simple as reordering or renaming fields. However, there were several pages or page sets that we focused much of our attention on.
A major change between our paper prototype and all subsequent prototypes was the overall layout of the site. We went from a classic "L" shape, with a top banner and a side nav bar, to just having a side nav bar. We placed a logo at the top of the side nav, and we added the calendar to the side nav bar. Removing the header eliminated a lot of wasted space, and adding the calendar made use of unused space.
|
Paper prototype side nav:
|
Final prototype side nav:
|
We worked through several drafts of the home page, though the structure of it changed little. Changes were mostly focused on how to clearly group information, whether than what information is grouped. The design of the home page was settled on by the second interactive prototype.
First interactive prototype home page:
Final interactive prototype home page:
The syllabus set up pages went through the most drastic transformations. We orginally had four single pages for entering all of the information, however there was little to connect the four pages in a workflow. For the first interactive prototype the pages became part of a wizard. This metaphor was reinforced by using a page counter across the tops of the pages. We continued to refine the fields within the forms. Between the first and second interactive prototypes we changed the second and third pages, moving some fields to the other page and changing the title of the third page from "More Course Information" to "Class Information". We also added page labels below the page counter. We eventually eliminated the questions related to setting up topics between the second and the third prototypes.
Paper prototype prototype Add Teaching Team page:
Final interactive prototype Add Teaching Team page:
The transition from the set up wizard to the syllabus editing pages proved to be problematic in each prototype. For the paper prototype, we took the user directly to the Schedule page. This proved to be an unsuccessful solution. We then tried incorporating a confirmation message into the tabbed pages (see below), so that users would see the confirmation and would be told to use one of the tabs to start editing. This also proved to be unsuccessful. We finally incorporated a fifth page in the set up wizard that displays a confirmation message, and then allows the user to choose where they want to continue to.
Second prototype confirmation page:
Final prototype confirmation page:
On the paper prototype, users were supposed to use links in the side nav to navigate between three different sections of the syllabus, the Schedule, Administrivia and Assignments & Exams pages. However, this failed miserably, and we moved to using tabs across the top of the pages to allow users to move between the pages for the first interactive prototype. There was also a tab that allowed the user to view a preview of the syllabus as it would look in SylViA. This was a more successful solution, and we kept it for all subsequent prototypes. After the pilot study with the second interactive prototype we realized that the Preview tab did not match the other tabs. We instead made it a button in the upper right hand of the screen and split the assignments and exams tab into two.
Paper prototype side bar navigation:
Final prototype tabbed navigation:
Setting up topics proved to be another very problematic feature of our interface. Topics allow users to group classes into subject areas within a course. However, this concept was very difficult to explain within the context of the course set up wizard, where the topic questions were originally placed. We tried several times to reword the questions about topics, but users continued to not understand what topics were or why they should care. After the second prototype, we decided that the topic questions should not be a part of the course set up wizard, since topics deal specificly with the schedule and the wizard dealt with the course more generally. We have created second wizard that the user must go through the first time they try to access the Schedule page.
Second interactive prototype topic question:
Final prototype topic question:
The preview option was always a part of our prototype, but it was not implemented until the third prototype. However, several changes were made to how it could be accessed. Originally, it was linked to in the side nav bar. Then it was part of the tabbed pages. Finally we decided to place the link to the preview in the upper right hand corner. The actually preview itself is based on the SylViA viewer, though the colors are match the scheme used for the My SylViA editor.
First interactive prototype preview tab:
Final prototype preview button:
Final prototype preview view:
The evolution of the process for finishing editing a syllabus proved that our first design was the best. We originally designed the process so that the user would click a link that would take the user to a page where she could choose between saving her changes in draft form, or publishing them to the viewer. In an effort to reduce the number of steps, we instead designed buttons that let the user make the choice from the tabbed pages. This turned out to be confusing the pilot usability testers, and we returned to our original design.
First interactive prototype save draft and save/publish buttons:
Final prototype finish button:
Matthew Langham and Carsten Ziegeler, Cocoon: Building XML Applications. (Indianapolis, Indiana: New Riders) 2002.
Mika Fonsén, solution: performance and use-persistent-cache, http://marc.theaimsgroup.com/?l=xml-cocoon-users&m=107666537511906&w=2.
Alex Milowski, smallx, https://smallx.dev.java.net/.
