Coming up on a 1.0 release

Well, it’s been over 5 weeks since my last post (eep). The good news is that development in this time has been going just as strong as ever. Development actually sped up towards the end of the coding period of the Summer of Code, and the code is pretty much ready for a 1.0 release. The only thing that it is waiting on is the Logic Service project to be integrated into the alpha branch from the logic branch. So, before I stop updating this blog again, I will go through a short description about how the project has come along.

First, the Logic Web Service module. This part of the project went through a good number of changes from its start to finish. First, we planned to use the REST module, a module already implemented with simple functionality to access some data about patients. However, after some discussion and work on that, we discovered that the REST Module would need some considerable workarounds to allow the kind of functionality we needed. It lacked the ability to work with parameters in the URL, and required two separate handlers for the old functionality and the new functionality. In light of this, we set up a new module, the Logic Web Service module, which you’ll find the code for at the OpenMRS svn repository. The new module has a servlet named “api” that responds to api functions: getFilters, getTokenTags, getTokens?token=X, and getData?filter=X&filtertype={static | dynamic}&token=Y&token=Z&token=A&token=…. The Logic Web Service very roughly gives access to some basic Logic Service functionality through an XML interface using REST.

The ODA driver has come along nicely too. The UI finished milestone 3 fairly quickly, then went on to add in milestone 4, which dropped in another box, a “selected tokens” box. The user moved tokens between the available and selected tokens boxes. From that, which was supposed to be the final GUI implementation for the version 1.0 release, there were numerous changes that came with the addition of a way to group tokens–token tags. Token tags work in the same way as the much-hyped tags of the web 2.0 movement. To find tokens easier from the list of 1000s, you may just select a particular token tag from a list, and only the few tokens that share that tag would be selectable. The UI and Runtime drivers were updated to allow for this, and soon after the UI was updated slightly to more easily follow the many steps required to create a data set.

The goals for the version 1.0 release were all met, as far as I can tell. I’m really happy with the results, and I can’t wait until this gets into the hands of some actual users of OpenMRS. I can’t wait to get started on the next version of the BIRT ODA Plugin and the Logic Web Service, and while school makes it so I no longer have that much time, I’ll still continue working on it. There’s still many places to take the ODA driver (for instance, choosing datatypes, dynamic logic criteria creation and multiple-column tokens), and I can’t wait to take it there.

I’m not sure on whether or not this blog will continue, but you can track further progress on the Reporting Framework Integration project at http://openmrs.org/wiki/Reporting_Framework_Integration_Project. Thanks for reading!

BIRT ODA Driver GUI Milestone 2

Well, that was quick. :)  I’ve reached milestone 2 of the GUI for the OpenMRS ODA driver.  There were few changes that needed to be added.  However, the code is starting to look less like a hack, as functions start to be called from the right place and such.

Now one selects an entity on the left side, and this triggers the right box, for filters, to fill with entries.  It also triggers the filling of the token selection.  Of course, this only downloads the selection currently, as there is no box to update that holds the tokens as of yet.

Then the user selects the filter, which is fetched depending on the entity.  Coming next is a third step, choosing the tokens.

BIRT ODA Driver GUI Milestone 1

I have reached the previously mentioned GUI Milestone 1 today. Milestone 1 was letting the user select a filter/cohort, but make it only work with Patients, and for it to select all tokens. This is close to what is now committed. The slight difference is that it selects from all the tokens three hard coded token names that I have in my testing file. As the other part is not quite working yet, I could either do this or remove some tokens from the token test file. In any case, this is a non-issue, as the REST Module does not yet work.

The User Interface consists of a left aligned label, and a “combo” drop-down box that lets the user select from a list of filters retrieved from the server. It displays the names of the filters to the user, and enters the id number as a String into the query string. In other words, all intended behaviors for the filter selection are implemented. Coming next is the intended behavior for the entity selection, which will also involve filling the filter selection box when a new entity is selected.

Midterm Update

So I find myself just past the midterm of the program, at the end of week 7, looking back at the progress that I’ve made so far. I’m quite happy with what I’ve achieved so far. Not because I’ve got a monstrous load of code written, or because I’ve finished ahead of schedule. Rather, I’m doing fine on the code front, but the thing I’m most happy with is what I’ve learned about doing real work in the software development world. But I won’t go into that now… I need some material for an end-of-summer blog post, don’t I?

I’ve all but finished the runtime driver (a task that feels like it should be given more than half a sentence), and work on the user interface has started. There probably should have been more time for work on the interface, but I’m going to have much fewer false starts on this, now that I know where I can find information. Documentation has been created and updated, both at the Reporting Framework Integration Project page and at the BIRT ODA Plugin User Guide page. The user guide is incomplete, as the plugin has yet to be finished, and there is nothing to return data as of yet. The Project page now outlines the project better, and includes an outline for the project in what I’ve done so far and what is planned.

The project page also includes some of the latest news in the XML schema, which includes dropping of the <table> tag. We couldn’t really imagine people needing multiple tables, so it was dropped. I’m starting to think that some functions may change before long, so expect that page to be updated soon.

So what’s left in the project, you ask? Well, there are a few that need to be addressed big time.

• User interface needs to be implemented
• Each of the api functions of the Logic Web Service needs to be implemented in the REST Module

In addition to that, there are few things that would be nice to change before the project is done.  For one, it would be nice to update the plugin to play the best it can with the new version of BIRT, BIRT2.2.  It’s currently designed for 2.1, and I’ve made a few changes to make it compatible, so it does work, but I’d like to make sure everything is working just right.  Also, the user interface that gets implemented at first will likely be a stub that is not so polished, and could use work. So, the interface will need work. In addition to that, we may want to include searchable boxes of tokens, instead of forcing the user to search through the tokens themselves. We may also want the same kind of fancy setup that exists in the Flat File driver, allowing users to select tokens from a box on the left and move them over to a box on the left, choosing those tokens easily without having to click multiple tokens all while holding a key to select them all.

Milestones for the User Interface:

1. User can select a filter/cohort. Patient is the only allowed entity. All tokens are returned, and can be selectively placed in the report.
2. User can select an entity, which will affect the choices for filters. All tokens are returned based on which entity is chosen.
3. User selects entity, which affects the choices for filters and tokens. A filter is chosen, and tokens are selected from a list in a box.

This is the end of the user interface stub, which works fine. At this point, the REST Module will have to be implemented. There’s been a lot of planning, but I think lots of work is going to have to go into this, including many many questions to my mentor Justin.

All in all, though, the project is going well and I look forward to these final weeks.

Recapping Lots of Stuff

It’s been a fairly long time since my last entry, so I figured it’s time to write another one. A good amount of work has been getting done.

Firstly, actual work on the plug-in has started. I most recently committed code letting the user ping the server with their connection settings. I’ve also been working on the rest of the base code for the ODA driver, and am nearing completion on the easy parts. The hard part is figuring out how to use JAXP (the Java API for XML Processing) to process the XML. I imagine it can’t be too hard, though it will take a few days I’m sure. However, the rest of the runtime driver is going well.

Design for the UI portion of the driver is coming along, and has deviated a good bit from the original design. It’s still not done, but it’s coming along. Originally, I planned to allow the user to choose from the api functions “patient,” “findPatient,” and “cohort,” each of which was specified differently, and filled out the query to the api (patient identifier, patient search string, and cohort respectively). They then selected the tokens they wished to include for the columns. However, this ignored the eventual inclusions of other entities (e.g., encounter, observation) and only cared about the patient entity. So now we allow the user to select from a list of entities (we’re actually only caring about patient now, but the option is there). The user then selects tokens based on the entity (how this will be updated is still undecided). Also, the user selects a constraint/filter, which in the case of the Patient entity is the cohort. As of now, if I’m correct there are no filters/constraints for the other entities yet, but they are possible and will probably arise in a short time.

Work has also been happening on the REST Module that fulfills the need for a Logic Web Service. I made a small bugfix the other day, updating it for the current OpenMRS API, we’ve been planning the new functions for the module, and we’ve been planning the XML formats that will be used, taking into account how other there will be applications of the REST module other than the BIRT plug-in. I have a feeling that work will continue on the REST module well after the summer ends, as more and more applications that can make use of it appear.

As for the XML formats that are going to be used, we’re actively developing them. Currently, we’re working with something that defines a dataset with tables (only one table will be used for our purposes), which in turn defines columns by their metadata, and rows by their data. An example looks something like this:

<?xml version="1.0" encoding="utf-8"?>
<dataset>
 <table entity=”PATIENT”>
  <columns>
   <column token=”PATIENT.PATIENT_ID” />
   <column token=”LAST WEIGHT” />
   <column token=”LAST CD4 COUNT” />
  </columns>
  <rows>
   <row>
    <value>1</value>
    <value>180</value>
    <value>209</value>
   </row>
   <row>
    <value>2</value>
    <value>200</value>
    <value>198</value>
   </row>
  </rows>
 </table>
</dataset>

This isn’t really finalized, but it’s making progress. This is just a small snapshot into what’s been going on. I want to emphasize again that this is definitely the most planning I’ve ever done for a coding project, but it’s actually nice. I feel like this is what software development is really all about.

Of course, adding to the things on my list of things to check out is the release of BIRT 2.2. Congrats to the developers.

The Ins and Outs of an ODA Plug-in

The last few days have been spent learning about and understanding the workings of the ODA plug-in. It’s fairly complicated, and the actual documentation on how it interacts with BIRT is sparse. So, among other things recently, I’ve put together a small document that basically does a walkthrough of the user’s interaction with BIRT and the ODA plug-in. It includes some of the main functions called, to give an idea of what kind of data is being created and requested at each step, using the FlatFile ODA plug-in as a basis. I’ve uploaded a copy of that at this link, though the original is part of a larger, growing document in Google Docs.

I believe that eventually, the (larger, currently growing) document will become quite helpful to those who are trying to build an ODA plug-in of their own. The document actually includes a lot of OpenMRS specifics, but I’d like to create a sort of partially genericized version eventually.

Other news of late includes me finally understanding how the OpenMRS database is accessed through the code. I’ll talk about that more some other time, as currently there’s still more I have to learn, such as how I can work through the Logic Service, and I feel it’d be an incomplete entry as of now. I don’t usually do this much planning before jumping into a coding project, but this is such a complex system that any coding I had written a week ago I’d likely want to throw away today. The project is progressing nevertheless.

A Shell of a Plug-in

Well, I’ve finally got a basic, no-functionality plug-in. I tried copying various aspects of other ODA adapters, such as the Flat File and JDBC ODA Plug-ins. However, nothing worked better than just going with a lot of pre-set-up options. So, here is a walkthrough on how to get a shell of a plug-in. I should note that these instructions are for BIRT release 2.1, the official release as of this posting. Soon BIRT 2.2 will be out–it’s currently in RC0 (release candidate 0), so it should not be more than a few weeks until BIRT 2.2 is unleashed. I haven’t worked with BIRT 2.2 yet, but it should definitely be something to look forward to. I haven’t worked with any release other than BIRT 2.1, so if you’re using another version (such as BIRT 2.2 RC0), then this may not work. I’d be surprised if it did not work on an official release though, so go ahead and try anyway if you’re on a full release.

First, we’ll probably want to switch to the Plug-in Development Environment (PDE) in Eclipse. This is accessible by accessing the menus Window => Open Perspective => Other… and selecting Plug-in Development from the pop-up box.

Now onto the creating. Create a new project, select Plug-in Project and click Next.

Enter a Project name. For this, I used “openmrsdb”. The rest should probably stay the same, unless you don’t want this project in your normal workspace. Click Next.

Our Plug-in ID will stay the same, as will the version number and name. I did enter a Plug-in provider, “OpenMRS,” but you don’t need to enter anything. Classpath I left blank for now, even though some tutorials included some things. I didn’t have anything I felt needed to be included, so it stayed blank. If this actually changes in the future, then I’ll update this post then.

Leave the default Activator settings (checked, default name). As for the next checkbox, uncheck it. We are currently building the Runtime Plug-in, which is only a backend. We will make a separate plug-in for the UI soon. And leave the setting for rich client application (No). Click Next.

We do want to use a template, so leave that checkbox marked and click on “ODA Data Source Runtime Driver.” Click Next.

Most of this can stay as it is right now. However, we do want to change “Number of Data Source properties” to 3; you’ll see why soon. Also, I changed “Data Source Display Name” and “Data Set Display Name.” This is purely cosmetic, but who wants a bad user experience? By default, these will be “Openmrsdb Data Source” and “Openmrsdb Data Set,” which are right, but a little odd to say or think about. I change the Data Source name to “OpenMRS Database,” and the Data Set name to “OpenMRS Data Set.” The reason is that we are clearly drawing data from the OpenMRS database, not drawing a source from the OpenMRS database. As for the Data Set name, I thought it sounded redundant to have “Database” in between “OpenMRS” and “Data Set,” but still wanted it to say “Data Set.” Those name changes are completely up to you, and may vary depending on what your ODA driver does. Finally, click Finish.

When you finish, an editor should come up showing some properties about your new plug-in. Click the “Extensions” tab on the bottom, as this is where we have to do some work. Under “All Extensions,” click the arrow to explore the entry. Click the arrow for the first, “openmrsdb (dataSource).” Again, click the arrow for (properties) underneath that. You should see a list of three properties (property1, property2, property3). Click on property1, and you should see a few boxes come up on the right. We are going to fill in the information for these. Enter this data, leaving the rest of the boxes alone:

property1: name = SERVER; defaultDisplayName = %data.source.server; defaultValue = http://
property2: name = USERNAME; defaultDisplayName = %data.source.username
property3: name = PASSWORD; defaultDisplayName = %data.source.password; isEncryptable = true

These are generally self explanatory. These represent the properties that are saved and will be passed when creating a connection. We will want to specify a server, a username, and a password. However, the default display names I gave them don’t currently mean anything. We’re going to have to set those now.

From your project browser, open the file “plugin.properties” at the root of the project. Add these lines to the end of the file:

data.source.server=OpenMRS Server Path
data.source.username=Username
data.source.password=Password

These will be read by the UI to be displayed when you are setting the connection properties. The reason they are in this file is for internationalization support. You may make other language’s versions of these strings, so that people of other languages may more readily use the driver. You may notice that the other values that were optionally changed earlier (data source and data set names) are in this file. You can change them here if you want to change those names ever again. Go ahead and close that file and the editor for the project information, saving if you haven’t already. If you want to get to the editor again, you can try to open one of the files from the project, such as plugin.xml or MANIFEST.MF, and these will get you to that editor, though possibly under a different tab.

Now, we want to create the UI Plug-in. This is probably even simpler. Create another Plug-in Project, and name this one openmrsdb.ui (or equivalent). It doesn’t really matter what the name is, though having the original name plus “.ui” seems natural to me. Leave the rest of the options the same, unless you want to move this out of your normal workspace. Again we don’t really change anything, though you can change the plug-in name or provider here again. Unlike last time, however, leave checked the box “This plug-in will make contributions to the UI.”

On the next screen, we want to use the template “ODA Data Source Designer.” Note that there are a lot more templates this time, due to the fact that the UI contributions checkbox was checked. I guess there aren’t many plug-ins that don’t have to do with the UI. Click next.

This screen is important. Many of the values here must match up with the values from the previous plug-in. If you didn’t change anything other than what I told you you could, you should be fine. Again at the bottom, you can change the names. I changed them to be the same as the ones I set with the runtime plug-in, just to be sure. Couldn’t hurt to change them, right? Finally, click finish.

Now, you’re done. You’ve got a shell of a plug-in. To see it in “action,” you’re going to want to run the project as an Eclipse application. Right click on one of the projects and select Run As => Eclipse Application. When the new instance of Eclipse starts up, you’re going to want to add a new reporting project and add a new report to the project. Switch to the BIRT reporting perspective, and open the “Data Explorer” view. Right click on Data Sources, and click “New Data Source.” Select “OpenMRS Database” from the list (or whatever it is called. As default, it would be “Openmrsdb Data Source”) and click Next.

Aha! It is the connection properties for our plug-in! Success! You should see something that looks like this:

Except yours won’t have data filled in until you fill it in. It doesn’t matter what you put, as it doesn’t do anything yet (the Test Connection button will always work too :-). There’s not too much more to see, but you can try creating a data set. It will ask for a query in a box. This is not what will eventually be there, as we want our users to see something nicer than a plain old query box. No, the purpose of this driver is to smooth the connection.

That will be a good adapter. But for now, all we have is a shell. It’s the beginnings of a wonderful connection between report designer and database. From here, it’s..well, still uphill. But it’s a good start.

Many thanks to Scott Rosenbaum for writing a nice Primer on ODA Extensions and BIRT, available at EclipseMag.net (registration required). It was the first thing that got me a nice shell, and it’s going to help tremendously in adding functionality.

Working with Eclipse and Plug-ins

Over the past few days, I’ve been experimenting with the plug-ins and the ODA API. I finally got a nice Javadoc set up of the ODA API, generated from getting the source to the ODA interfaces and getting it into Eclipse and generating the Javadocs from within Eclipse. I’ve put them on the Internet for anyone to see, where they can hopefully stay for a while, accessibly on my Cornell site. I’m quite glad to have made this, as now my complaints from last time no longer really apply. Though I question why this is not made readily available already. But now, on to the Eclipse work…

First, I must say that I first tried following a small tutorial on creating “your first Eclipse plug-in.” As there are not any tutorials that I can find on ODA drivers specifically, I settled for a generic plug-in. I followed a tutorial I found somewhere on the Eclipse website, that was hosted over at EclipseZone. It wasn’t terribly exciting, but it did give me a little button in my Eclipse toolbar that, when clicked said “Hello, World!” Yes, everyone’s favorite phrase makes a comeback. I must admit that I love writing “Hello, World!” programs. One of these days I’ll find out who started this trend. Anyway, on to the Flat File ODA plug-in.

Apparently, the version of the Flat File plug-in that was in CVS on the dev.eclipse.org website is a newer version than the release version. That’s perfectly fine, normally. Unfortunately, this means that it’s completely incompatible with old versions of the ODA framework, such as the one that is currently available to the general public from the Eclipse datatools project website and through the Eclipse download manager.

Luckily for me, I figure out that in plug-in development mode, I can access the plug-ins that are currently usable in Eclipse, and import the source as a project to my workspace. It’s just one tab over, next to the Package Explorer. How convenient!

I’ve been playing with those a good bit since I found that. I made some changes, just to figure out how it all works. First in property and XML files, so that I could change the name from “Flat File Data Source” to “Flat File Duck Source” (first change I could think of):

I’ve made a few other changes as well, such as editing the group on the next page of the New Data Source selector to have 5 instead of 4 rows, and filled the 5th row with another radio button. The previous four were CSV (comma-separated values), SSV (semicolon-separated values), TSV (tab-separated values), and PSV (pipe-separated values). The new one was QSV (quack-separated values). I guess ducks are very important to my code. I didn’t implement the feature, mostly because it would involve a few different source files, but it wouldn’t have been too hard.

On that front, I am currently trying to set up new projects, openmrsdb and openmrsdb.ui, which at first will just be a copy of flatfile and flatfile.ui. Eventually these will be going into the OpenMRS source code in openmrs-contrib, as opposed to a module or the main code, as they plug into Eclipse and BIRT, rather than OpenMRS.

On the front of the Web Service module for OpenMRS, I’ve gone through the instructions on the OpenMRS wiki about Creating your first OpenMRS module. Of course, I’ll be working instead on Burke’s older REST module, so I checked that out as well, and added it as a project that I can directly work on. For that part, I am completely set up, as far as my developing environment goes. I will get some general practice with web services soon, so that work on the REST module can get started.

The next order of business is getting at least a shell of the ODA modules set up. But then again, that may wait a little bit, as today I got Integrating and Extending BIRT by Weathersby et al. in the mail. I would like to note here that the USPS is an odd service. I saw the mail truck drive by today, and went to check the status of my package, which wasn’t supposed to arrive until June 15th (10 days early!). As I noted that the tracking said it was still in Jersey City, NJ, USA, there was a knock on the front door, which was the mailman with my package. I can understand the fastness, but I would really like some better tracking. I live in New York, which is at least a day away from Jersey City by mail, if it was going fast. In any case, I’m glad to have the book, so that I can have a reference other than the API and the Flat File implementation.

Background Reading

Over the past few days, I’ve been getting set up.  I’ve mostly got up an Eclipse development setup, and I’ve been reading up on various API documentation.  Not the most glamourous part of this summer’s project, but I’ve got to gain background.

As for the Web service side of things, I’ve been reading on the Javadocs for the packages javax.servlet and javax.servlet.http.  Quite useful information, paired with a reference.  As I read the documentation, I was also viewing some servlet examples that came paired with Tomcat, as well as part of Burke Mamlin’s implementation of a REST module for OpenMRS.

Good news on the Web service front: we may be able to use Burke’s module as the base for the web service module.  Burke’s may have implemented what we need, and I would simply add on handlers for the kinds of URLs that need to be handled.  We’re not sure yet if we’ll have to change some things (e.g., to handle multiple tokens), but for now this seems like a great help.

As for the BIRT side of things, I’ve been trying to read through APIs, but it’s harder to find them.  To some degree, it is described on the DTP Open Data Access Overview, but it says for “detail ODA API documentation, see its Javadoc API Reference Documentation included in the DTP download.”  The DTP download, as far as I can tell, is available from the Data Tools Platform Project download page, but the Javadocs are not well organized, at least if you ask me.

I have the source to the Flat File ODA Plugin, which as far as I can tell is just a CSV File Plugin.  I believe this will be the most indispensable reference I will have in building my ODA adapter.  The CSV driver is “an exemplary implementation of the DTP ODA run-time API,” so I believe that following in its example will be my best option.  It won’t be the easiest to program without a highly organized and accessible API documentation, but hey, that’s half the fun.

About the Reporting Framework Integration Project

While dev.openmrs.org is down for a Python update, I’m going to take time to go over my view of the project as of now.

The basic premise of my project is that OpenMRS lacks sufficient reporting capabilities, and needs to be able to better integrate with reporting tools, such as BIRT. BIRT, Business Intelligence and Reporting Tools, is an open source Eclipse-based reporting system. My project goal is to build a direct link between OpenMRS and BIRT, via two components.

The first component is a module to OpenMRS. This functions as a web service that runs and responds to certain queries. Our service will be a REST service, so that it takes standard HTTP queries and returns some kind of XML document. This is pretty cool in and of itself. The REST module responds to queries like

http://demo.openmrs.org/openmrs/getData?cohort=All_Patients&entity=Patient&token=CD4COUNT&token=AGE

Using an analogy to SQL, this looks up the table (entity) of patients, filters out some with a qualifier (WHERE in SQL, here it is the cohort, and All_Patients actually doesn’t filter out any), and returns the data from the columns (tokens) named CD4COUNT and AGE. I don’t have much experience with SQL, but I’d say that this is a good analogy, and the way I understand the web service. This module is a way to interface with the OpenMRS database without having to actually directly access the MySQL database.

The second component is an adapter for BIRT. In this part of the project, I build an Open Data Access (ODA) adapter that allows BIRT to interface with a new source, the OpenMRS web service. Whereas now BIRT is forced to communicate directly with the OpenMRS database through JDBC, the ODA adapter I build will allow BIRT to communicate transparently with OpenMRS. It is as of yet unsure how complicated the adapter will be, possibly allowing custom building of cohorts, or perhaps just cohorts defined in the web interface.

At the end of the project, generating compelling reports in BIRT should be only as hard as designing any report would be; the interface to OpenMRS will be as transparent as possible. The user will not have to worry about the connection to the database, but will rather only have to worry about making the report pretty.