Akoma Ntoso, HTML5, LegisPro Sunrise, Standards, technology, Track Changes, Uncategorized, W3C

LegisPro Sunrise!!!

LegisPro Sunrise is almost done!. It has taken longer than we had hoped it would, but we are finally getting ready to begin limited distribution of LegisPro Sunrise, our productised implementation of our LegisPro drafting and amending tools for legislation and regulations. If you are interested in participating in our early release program, please contact us at info@xcential.com. If you already signed up, we will be contacting you shortly.

LegisPro Sunrise is a desktop implementation of our web-based drafting and amending products. It uses Electron from GitHub, built from Google’s Chromium project, to bundle all the features we offer, both the client and server sides into a single easy-to-manage desktop application with an installer having auto-update facilities. Right now, the Windows platform is supported, but MacOS and Linux support will be added if the demand is there. You may have already used other Electron applications – Slack, Microsoft’s Visual Studio Code, WordPress, some editions of Skype, and hundreds of other applications now use this innovative new application framework.

Other versions

In addition to the Sunrise edition, we offer LegisPro in customised FastTrack implementations of the LegisPro product or as fully bespoke Enterprise implementations where the individual components can be mixed and matched in many different ways.

Akoma Ntoso model

LegisPro Sunrise comes with a default Akoma Ntoso-based document model that implements the basic constructs seen in many parliaments and legislatures around the world that derive from the Westminster parliamentary traditions.

Document models implementing other parliamentary or regulatory traditions such as those found in many of the U.S. states, in Europe, and in other parts of the world can also be developed using Akoma Ntoso, USLM, or any other well-designed XML legislative schema.

Drafting & Amending

Our focus is on the drafting and amending aspects of the parliamentary process. By taking a digital-first approach to the process, we are able to offer many innovative features that improve and automate the process. Included out of the box is what we call amendments in context where amendment documents are extracted from changes recorded in a target document. Other features can be added through an extensive plugin mechanism.

Basic features

Ease-of-use

While offering sophisticated drafting capabilities for legislative and regulatory documents, LegisPro Sunrise is designed to provide the familiarity and ease-of-use of a word processor. Where it differs is in what happens under the covers. Rather than drafting using a general purpose document model and using styles and formatting to try to capture the semantics, we directly capture the semantic structure of the document in the XML. But don’t worry, as a drafter, you don’t need to know about the underlying XML – that is something for the software developers to worry about.

Templates

Templates allow the boilerplate structure of a document to be instantiated when creating a new document. Out-of-the-box, we are providing generic templates for bills, acts, amendments, amendment lists, and a few other document types.

In addition to document templates, component templates can be specified or are synthesized when necessary to be used as parts when constructing a document.

For both document and component templates, placeholders are used to highlight area where text needs to be provided.

Upload/Download

As a result to our digital-first focus, we manage legislation as information rather than as paper. This distinction is important – the information is held in XML repositories (a form of a database) where we can query, extract, and update provisions at any level of granularity, not just at the document level. However, to allow for the migration from a paper-oriented to a digital first world, we do provide upload and download facilities.

Undo/Redo

As with any good document editor, unlimited undo and redo is supported – going back to the start of the editing session.

Auto-Recovery

Should something go wrong during an editing session and the editor closes, an auto-recovery feature is provided to restore your document to the state the document was at, or close to it, when the editor closed.

Contextual Insert Lists

We provide a directed or “correct-by-construction” approach to drafting. What this means is that the edit commands are driven by an underlying document model that is defined to enforce the drafting conventions. Wherever the cursor is in the document, or whatever is selected, the editor knows what can be done and offers lists of available documents components that can either be inserted at the cursor or around the current selection.

Hierarchy

Document hierarchies form an important part of any legislative or regulatory document. Sometimes the hierarchy is rigid and sometimes it can be quite flexible, but either way, we can support it. The Sunrise edition supports the hierarchy Title > Part > Chapter > Article > Section > Subsection > Paragraph > Subparagraph out of the box, where any level is optional. In addition, we provide support for cross headings which act as dividers rather than hierarchy in the document. Customised versions of LegisPro can support whatever hierarchy you need — to any degree of enforcement. A configurable promote/demote mechanism allows any level to be morphed into other levels up and down the hierarchy.

Large document support

Rule-making documents can be very large, particular when we are talking about codes. LegisPro supports large documents in a number of ways. First, the architecture is designed to take advantage of the inherent scalability of modern web browsing technology. Second, we support the portion mechanism of Akoma Ntoso to allow portions of documents, at any provision level, to be edited alone. A hierarchical locking mechanism allows different portions to be edited by different people simultaneously.

Spelling Checker

Checking spelling is an important part of any document editor and we have a solution – working with a third-party service we have tightly integrated with in order to give a rich and comprehensive solution. Familiar red underline markers show potential misspellings. A context menu provides alternative spellings or you can add the word to a custom dictionary.

Tagging support

Beyond basic drafting, tagging of people, places, or things referred to in the document is something for which we have found a surprising amount of interest. Akoma Ntoso provides rich support for ontologies and we build upon this to allow numerous items to be tagged. In our FastTrack and Enterprise solutions we also offer auto-tagging technologies to go with the manual tagging capabilities of LegisPro sunrise.

Document Bar

The document bar at the top of the application provides access to a number of facilities of the editor including undo/redo, selectable breadcrumbs showing your location in the document, and various mode indicators which reflect the current editing state of the editor.

Command Ribbons and Context Menus

Command ribbons and context menus are how you access the various commands available in the editor. Some of the ribbons and menus are dynamic and change to reflect the location the cursor or selection is at in the editor. These dynamic elements show the insert lists and any editable attributes. Of course, there is also an extensive set of keyboard shortcuts for many commonly used commands. It has been our goal to ensure that the majority of the commonly used documenting tasks can be accomplished from the keyboard alone.

Sidebar

A sidebar along the left side of the application provides access to the major components which make up LegisPro Sunrise. It is here that you can switch among documents, access on-board services such as the resolver and amendment generator, outboard services such as the document repository, and where the primary settings are managed.

Side Panels

Also on the left side are additional configurable side panels which provide additional views needed for drafting. The Resources view is where you look up documents, work with the hierarchy of the document being edited, and view provisions of other documents. The Change Control view allows the change sets defined by the advanced change control capabilities (described below) to be configured. Other panels can be added as needed.

Advanced features

In addition to the rich capabilities offered for basic document editing, we provide a number of advanced features as well.

Document Management

Document management allows documents to be stored in an XML document repository. The advantage of storing documents in an XML repository rather than in a simple file share or traditional content management system is that it allows us to granularise the provisions within the documents and use them as true reference-able information – this is a key part of moving away from paper document-centric thinking to a modern digital first mindset. An import/export mechanism is provided to add external documents to the repository or get copies out. For LegisPro Sunrise we use the eXist-db XML database, but we can also provide customised implementation using other repositories.

Resolver

Our document management solution is built on the FRBR-based metadata defined by Akoma Ntoso and uses a configurable URI-based resolver technology to make human readable and permanent URIs into actual URLs pointing to locations within the XML repository or even to other data sources available on the Web.

Page & Line numbering

There are two ways to record where amendments are to be applied – either logically by identifying the provision or physically by page and line numbers. Most jurisdictions use one or the other, and sometimes even both. The tricky part has always been the page and line numbers. While modern word processors usually offer page and line numbers, they are dynamic and change as the document is edited. This makes this feature of limited use in an amending system. What is preferred is static page and line numbers that reflect the document at the last point it was published for use in a committee or chamber. We accomplish this approach using a back-annotation technology within the publishing service. LegisPro Sunrise also offers a page and line numbering feature that can be run without the publishing service. Page and line numbers can be display in the left or right margin in inline, depending on preference.

Amendment Generation

One of the real benefits of a digital first solution is the many tasks that can be automated – not by simply computerising the way things have always been done, but by rethinking the approach altogether. Amending is one such area. LegisPro Sunrise incorporates an onboard service to automatically generate amendment documents from changes recorded in the target document. Using tracked changes, the document hierarchy, and annotated page and line numbers, we are able to very precisely record proposed changes as amendments. Of course, the amendment generator works with the change sets to allow different amendment sets to be generated by specifying the named set of changes.

Plugin Support

LegisPro Sunrise is not the first incarnation of our LegisPro offering. We’ve been using the underlying technologies and precursors to those technologies for years with many different customers. One thing we have learned is that there is a vast variation in needs from one customer to another. In fact, even individual customers sometime require very different variations of the same basic system to automate different tasks within their organisation. To that end, we’ve developed a powerful plug-in approach which allows capabilities to be added as necessary without burdening the core editor with a huge range of features with limited applicability. The plugin architecture allows onboard and outboard services to be added, individual commands, menus, menu items, side panels, mode indicators, JavaScript libraries, and text string libraries to be added. In the long term, we’re planning to foster a plugin development community.

Proprietary or Open Source?

There are two questions that always come up relating to our position on standards and open source software:

Is it based on standards? Yes, absolutely – almost to a fault. We adhere to standards whenever and however we can. The model built into LegisPro Sunrise is based on the Akoma Ntoso standard that has been developed over the past few years by the OASIS LegalDocML technical committee. I have been a continual part of that effort since the very beginning. But beyond that, we always choose standards-based technologies for inclusion in our technology stack. This includes XML, XSLT, XQuery, CSS, HTML5, ECMAScript 2015, among others.
Is it open source?
- If you mean, is it free, then the answer is only yes for evaluation, educational, and non-production uses. That’s what the Sunrise edition is all about. However, we must fund the operation of our company somehow and as we don’t sell advertising or customer profiles to anyone, we do charge for production use of our software. Please contact us at info@xcential.com or visit our website at xcential.com for further information on the products and services we offer.
- If you mean, is the source code available, then the answer is also yes – but only to paying customers under a maintenance contract. We provide unfettered access to our GitHub repository to all our customers.
- Finally, if you’re asking about the software we built upon, the answer is again yes, with a few exceptions where we chose a best-of-breed commercial alternative over any open source option we had. The core LegisPro Sunrise application is entirely built upon open source technologies – it is only in external services where we sometimes rely up commercial third-party applications.

What does it cost?

As I already alluded to, we are making LegisPro Sunrise available to potential customers and partners, academic institutions, and other select individuals or organisations for free – so long as it’s not used for production use — including drafting, amending, or compiling legislation, regulations, or other forms of rule-making. If you would like a production system, either a FastTrack or Enterprise edition, please contact us at info@xcential.com.

Coming Soon

I will soon also be providing a pocket handbook on Akoma Ntoso. As a member of the OASIS LegalDocML Technical Committee (TC) that has standardised Akoma Ntoso, it has been important to get the handbook reviewed for accuracy by the other TC members. We are almost done with that process. Once the final edits are made, I will provide information on how you can obtain your own copy.

Akoma Ntoso, HTML5, LegisPro Sunrise, LEX Summer School, Standards, technology, Track Changes

The Sun is Rising on Akoma Ntoso — and LegisPro too!

Two great news piece of news this week! First, the documentation for Akoma Ntoso has now been officially released by OASIS. Second, we’re announcing the latest version of our LegisPro drafting platform for Akoma Ntoso, codenamed “Sunrise”.

After several years of hard work, we’ve made a giant step towards our goal of setting an international XML standard for legal documents. You can find the documents at the OASIS LegalDocML website. A special thanks to Monica Palmirani and Fabio Vitali at the University of Bologna for their leadership in this endeavour.

legispro250 Later this week, Xcential will be announcing and showing the latest version of “Sunrise” version of LegisPro, at both NALIT in Annapolis, Maryland and at the LEX Summer School in Ravenna, Italy. This new version represents a long-planned change to Xcential’s business model. While we have a thriving enterprise business, we’re now focusing on also providing more affordable solutions for smaller governments.

Part of our plan is to foster an open community of providers around the Akoma Ntoso standard for legislative XML. With Akoma Ntoso now in place as a standard, we’re looking for ways to provide open interfaces such that cooperative tools and technologies can be developed. One of my goals at this years summer school in Ravenna is to begin outlining the open APIs that will enable this vision.

LegisPro

The new edition of LegisPro will be all about providing the very best options:

It will provide a word processing like drafting capability your drafters demand — along with the real capabilities you need:
- We’re not talking about merely providing a way to style a word processing document to look like legislation.
- We’re talking about providing easy ways to define the constructs you need for your legislative traditions, such as–
  - a configurable hierarchy,
  - configurable tagging of important information,
  - configurable numbering rules,
  - configurable metadata,
  - oh, and configurable styles too.
- We’re talking about truly understanding your amending traditions and providing the mechanisms to support them, such as–
  - configurable track changes, because we understand that a word processor’s track changes are not enough,
  - as-published page and line markers, because we understand your real need for page and line numbers and that a word processor’s page and line numbering is not that,
  - robust typography, because we know there’s a quite a difference between the casual correspondence a word processor is geared for and the precision demanded in documents that represent laws and regulations.
It will be as capable as we can make it — for real-world use rather than just a good demo:
- We’re not talking about trying to sell you a cobbled together suite of tools we built for other customers.
- We’re talking about working with specialists in all the sub-fields of legal informatics to provide best-of-breed options that work with our tools.
- We’re talking about making as many options available to you as we know there is no one-size-fits-all answer in this field.
- We’re talking about an extensible architecture that will support on-board plug-ins as well as server-side web-services.
- We’re talking about providing a platform of choices rather than a box of pieces.
It will be as affordable as we can possibly make it:
- We’re talking about developing technologies that have been designed to be easily configured to meet a wide variety of needs.
- We’re talking about using a carefully chosen set of technologies to minimize both your upfront cost and downstream support challenges.
- We’re talking about providing a range of purchasing options to meet your budgetary constraints as best we can.
- We’re talking about finding a business model that allows us to remain profitable — and spreads the costs of developing the complex technologies required by this field as widely and fairly as possible.
It is as future-proof as we can possibly make it:
- We’re not talking about trying to sell you on a proprietary office suite.
- We’re talking about using a carefully curated set of technologies that have been selected as they represent the future of application development — not the past — including:
  - GitHub’s Electron which allows us to provide both a desktop and a web-based option, (This is the same technology used by Slack, WordPress, Microsoft’s Visual Studio Code, and hundreds of other modern applications.)
  - Node.js which allows us to unify client-side and server-side application development with “isomorphic JavaScript,”
  - JavaScript 6 (ECMAScript 2015) which allows us to provide a truly modern, unified, and object-oriented programming environment,
  - Angular and other application frameworks that allow us to focus on the pieces and not how they will work together,
  - CSS3 and LESS that allows us to provide state-of-the-art styling technologies for the presentation of XML documents,
  - the entire XML technology stack that is critical for enabling an information-centric rather than document-centric system as is appropriate for the 21st century,
  - and, of course, using the Akoma Ntoso schema for legislative XML to provide the best model for sharing data, information, tools, and other technologies. It’s truly a platform to build an industry on.
It is as open as we can possibly make it:
- We’re not talking about merely using an API published by a vendor attempting to create a perception of openness by publishing an API with “open” in the name.
- We’re talking about building on a full suite of open source tools and technologies coming from vendors such as Google, GitHub, and even Microsoft.
- We’re talking about using non-proprietary protocols such as HTTP and WebDAV.
- We’re talking about providing an open API to our tools that will also work with tools of other vendors that support Akoma Ntoso.
- And, while we must continue to be a profitable product vendor, we will still provide the option of open access to our GitHub repositories to our customers and partners. (We’ll even accept pull requests)

Our goal is to be the very best vendor in the legislative and regulatory space, providing modern software that helps make government more efficient, more transparent and more responsive. We want to provide you with options that are affordable, capable, and planned for the future. We want to do whatever we can to allay your fears of vendor lock-in by supporting open standards, open APIs, and open technologies. We want to foster an Akoma Ntoso-based industry of cooperative tools and technologies as we know that doing so will be in the best interests of everyone — customers, product vendors, service providers, and the people who support them. As someone once told me many years ago, if you focus on making the pie as large as you can, the crumbs left on the knife will be plenty enough for you.

Either come by our table at NALIT in Annapolis or join us for the Akoma Ntoso Developer’s Conference in Ravenna at the conclusion of the LEX Summer School to learn more. If neither of these options will work for you, you can always learn more at Xcential.com or by sending email to info@xcential.com.

Akoma Ntoso, HTML5, LegisPro Web, LEX Summer School, Standards, Transparency

Data Transparency Breakfast, LEX US Summer School 2015, First International Akoma Ntoso Conference, and LegisPro Edit reveal.

Last week was a very good week for my company, Xcential.

We started the week hosting a breakfast put on by the Data Transparency Coalition at the Booz Allen Hamilton facility in Washington D.C.. The topic was Transforming Law and Regulation. Unfortunately, an issue at home kept me away but I was able to make a brief pre-recorded presentation and my moderating role was played by Mark Stodder, our company President. Thank you, Mark!

Next up was the first U.S. edition of the LEX Summer School from Italy. I have attended this summer school every year since 2010 in Italy and it’s great to see the same opportunity for an open dialog amongst the legal informatics community finally come to the U.S. Monica Palmirani (@MonicaPalmirani), Fabio Vitali, and Luca Cervone (@lucacervone) put on the event from the University of Bologna. The teachers also included Jim Mangiafico (@mangiafico) (the LoC data challenge winner), Veronique Parisse (@VeroParisse) from the European Union, Andrew Weber (@atweber) from the Library of Congress, Kirsten Gullickson (@GullicksonK) from the Office of the Clerk at the U.S. House of Representatives, and myself from Xcential. I flew in for an abbreviated visit covering the last two days of the Summer School where I covered how the U.S. Code is modeled in Akoma Ntoso and gave the students an opportunity to try out our new bill drafting editor — LegisPro^edit.

After the Summer School concluded, it was followed by the first International Akoma Ntoso Conference on Saturday, where I spoke about the architecture of our new editor as well as how the USLM schema is a derivative of the Akoma Ntoso schema. We had good turnout, from around the world, and a number of interesting speakers.

This week is NCSL in Seattle where we will be discussing our new editor with potential customers and partners. Mark Stodder from Xcential will be in attendance.

In a month, I’ll be in Ravenna once more for the European LEX Summer School — where I’ll be able to show even more progress towards the goal of a full product line of Akoma Ntoso tools. It’s interesting times for me.

The editor is coming along nicely and we’re beginning to firm up our QuickStarter beta plans. I’ve already received a number of requests and will be getting in touch with everyone as soon as we’re ready to roll out the program. If you would like to participate as a beta tester — or if you would just like more information, please contact us at info@xcential.com.

I’m really excited about how far we’ve come. Akoma Ntoso is on the verge of being certified as an official OASIS standard, our Akoma Ntoso products are coming into place, and interest around the world is growing. I can’t wait to see where we will be this time next year.

Akoma Ntoso, HTML5, LegisPro Web, LEX Summer School, Standards, Track Changes, W3C

Coming soon!!! A new web-based editor for Akoma Ntoso

I’ve been working hard for a long time — building an all new web-based editor for Akoma Ntoso. We will be showing it for the first time at the upcoming Akoma Ntoso LEX Summer School in Washington D.C.

Unlike our earlier AKN/Editor, this editor is a pure XML editor designed from the ground up using the XML capabilities that modern browsers possess. This editor is much more robust, more precise, and is very scalable.

Basic Features

Configurable XML models — including Akoma Ntoso and USLM
Edit full documents or portions of large documents
Flexible selection and editing regardless of XML structure
Built-in redlining (change tracking) supporting textual AND structural changes
Browse document sources with drag-and-drop.
Full undo & redo
Customizable attribute editor
Search and replace
Modular architecture to allow for extensive customization

Underlying Technology

XML-based editing component
- DOM 4 support
- XPath Support
- CSS Styling
- Sophisticated event model
HTTP-based resolver architecture for retrieving documents
- Interpret citations
- Deference URLs
- WebDAV adaptors to document repositories
- Query repositories with XQuery or databases with SQL
AngularJS-based User Interface using HTML5
- Component modules for easy customization
XML repository for storing documents
- Integrate any XML repository
- Built-in support for eXist-db
Validation & Publishing
- XML Schema validator
- XSL-FO publishing

We’ll reveal a lot more at the LEX Summer School later this month! If you’re interested in our QuickStart beta program, drop me a note at grant.vergottini@xcential.com.

HTML5, LegisPro Web, Standards, Track Changes, W3C

Building a browser-based XML Editor

Don’t forget the 2014 U.S. House Legislative Data and Transparency Conference this week.

I’m now hard at work on our second generation web-based XML editor. In my blog last week, I talked about the need for and complexities of change tracking in a legislative editor. In this blog, I want to describe more of the overall motivation.

A couple years ago, we built an HTML5-based legislative editor for Akoma Ntoso. We learned a lot from the effort and had some success with a couple customers whose needs matched the capabilities of the editor. The editor was built to use and exploit, to the fullest extent, many of the new APIs added to modern browsers to support HTML5. We found that, by focusing on HTML5, a lot of the complexities of dealing with browser quirks and incompatibilities were a thing of the past – allowing us to focus on building the editing functions.

The editor worked by transforming the XML document into a close representation of the XML, expressed as HTML5 tags. Using HTML5 features such as the @contenteditable attribute along with modern CSS, the browser DOM, selection ranges, drag and drop, and a WebDAV repository API, we were able to implement a fairly sophisticated web-based legislative editor.

But, not everything went smoothly. The first problem involved the complexity of mapping all the intricacies of XML into an HTML5 representation, and then maintaining that representation in the browser. Part of the difficulty stems from the fact the HTML5 is not specifically an XML dialect – and browsers tend to do HTML5 things that aren’t always XML friendly. The HTML5 DOM is deliberately rather loose and forgiving (it’s a big part of why HTML was successful in the first place) while XML demands a very precise and rigid DOM.

The second problem we faced was scalability. While the HTML5 representation wasn’t all that heavyweight, the bigger problem was the transformation cost going back and forth between HTML5 and XML. We sometimes deal with very large legislation and laws. In our bigger cases, the cost of transformation was simply unreasonable.

So what is the solution? Well, early last year we started experimenting with using a browser to render XML documents with a CSS directly – without any transform into HTML. Most modern browsers now do this very well. For the most part, we were able to achieve an acceptable rendition in the browser without any transformation.

There were a few drawbacks to this approach. For one, links were dead – they didn’t inherently do anything. Likewise, implementing something like the HTML @style attribute didn’t just naturally work. Before we could entertain the notion of a pure XML-based editor built within the XML infrastructure in the browser, we had to find a solution that would allow us to enrich the XML sufficiently to allow it to behave like an HTML page.

Another problem arose in that our prior web-based editor relied upon the @contenteditable feature of HTML. That is an HTML feature rather than a browser feature. Using XML as our base environment, we no longer had access to this facility. This wasn’t a total loss as our need for a rich change tracking environment required us to find a better approach that @contenteditable offered anyway.

With solutions to the major problems behind us, we started to take a look at the other goals for the editor:

Track Changes – This was the subject of my blog last week. For us, track changes is crucial in any editor targeted at legislation – and it must work at both the structural and textual level equally well. We use the feature for two things – redlining changes as is common in the U.S. and the automatic generation of amendment documents (amendments in context). Differencing can get you part way there – but it excludes the ability to adequately craft the changes in a way that deal with political sensitivities. Track Changes is a very complex feature which must be built into the very core of the editor – tacking it on later will be very difficult, if not impossible.
Scalability – Scalability is very important to our applications. We need to support very large documents. Even when we deal with document fragments, we need to allow those fragments to be very large. Our approach is to create editing islands within a large document loaded into the browser. This amounts to only building the editing superstructure around the parts of the document being edited rather than the whole document. It’s like building the scaffolding around only the floors being worked on in a skyscraper rather than trying to envelope the entire building in scaffolding.
Modularity – We’re building a number of very different applications currently – all of which require XML editing. To allow this variability, our new XML editor is written as a web-based component rather than a full-fledged application. Despite its complexity, on the surface it’s deceivingly simple. It has no user interface at all aside from the editing canvas. It’s completely driven by a well thought out JavaScript API. Adding the editor to a document is very simple. A single link, added to the bottom of the XML document, adds the editor to the document. With this component, we’re able to include it within all of the applications we are building.
Configurability – We need to support a number of different models – not just Akoma Ntoso. To achieve this, an XML-based configuration file is used to define the behaviors for any XML model. Elements can be defined as read-only, templates can be defined (or derived), and even the track changes behavior can be configured for individual elements. The sophistication being defined within the configuration files is to allow us to model all the variants of legislative models we have encountered without the need for extensive programming-level customization.
Browser Support – We’re pushing the envelope when it comes to browser support. Our current focus is on Google’s Chrome browser. Support for all the browsers aside from Internet Explorer should be relatively easy. Our experience has shown that the browsers are now quite similar. Internet Explorer is the one exception – in this particular area. Years ago, IE was the best browser when it came to XML support. While IE had many other compatibility issues, particularly with CSS, it led the way in supporting XML. However, while Microsoft has made tremendous strides moving forward to match the other browsers and modern standards, they’ve neglected XML. Their circa 1999 legacy capabilities for XML do no match modern standards and are quite deficient. Hopefully, this is something that will soon be rectified.

It’s not all smooth sailing. I have been finding a number of surprising issues with Google Chrome. For instance, whitespace management is a bit fudged at times. Chrome thinks nothing of adding the occasional non-breaking space to maintain whitespace when editing the DOM. What’s worse – it will inexplicably convert this into a text node that reads ” ” after a while. This is a character entity that is not defined in XML. I have to work hard to constantly reverse this odd behavior.

All in all, I’m excited by this new approach to building a web-based XML editor. It’s a substantial increase in sophistication over our prior web-based XML editor. This editor will be far more robust, scalable, and configurable in comparison to our prior editor and other editors we have worked on. While we still have a way to go in our development, we’ve found solutions to all the risky issues. It’s a future-looking approach – support can only get better. It doesn’t rely on compatibility modes or any other remnants of prior eras in web technology. This approach is really working out quite nicely for us.

Akoma Ntoso, HTML5, LegisPro Web, Standards, Track Changes, Transparency, Uncategorized, W3C

Legal Citations and XML Editing for Legislation

It’s been quite some time since my last blog post – almost six months. The reason is that I’ve been very busy. We are doing a lot of exciting development within Xcential. We are developing a number of quite challenging projects around the globe.

If you’ve been following my blog, you may remember that I was working on an HTML5-based XML editor. That development was two years ago now. We’ve come a long way since then. The basic editor has been stripped down, componentized, and has being rebuilt to be a far more robust, scalable, and adaptable solution. There are more details below, which I will expand upon as the editor rolls out over the next year.

Legal Citations

It was almost a year ago since the last Legislative Data and Transparency Conference in Washington D.C. (The next one is coming up) At that time, I spoke about the need for improved citation management in published XML documents. Well, we’ve come a long way since then. Earlier this year a Technical Committee was formed within OASIS to begin developing some standards. The Legal Citation Markup Technical Committee is now hard at work defining markup models for legal citations. I am a member of that TC.

The reference management part of our HTML5-based editor has been separated out as a separate project – as a citation interpreter and reference resolver. In our development tests, it’s integrated with eXist as a local repository. We also source documents from external sources such as LII.

We now have a few citation management projects underway, using our resolver technology. These are exciting projects which will be a huge step forward in improving how citations are managed. It’s premature to talk about this in any detail, so I’ll just leave this as a teaser of stuff to come.

XML Editing for Legislation

The OASIS Legal Document ML Technical Committee is getting ready to make a large announcement. While this progress is being made, at Xcential we’ve been hard at work refining the state-of-the-art in XML editing.

If you recall the HTML5-based editor for Akoma Ntoso from a couple of years back, you may remember that is was based around all the new HTML5 technologies that have recently been incorporated into web browsers. We learned a lot from that effort – both good and bad. While we were able to get a reasonable tagging editor, using facilities that made editing far easier, we still faced difficulties when it came to basic XML editing and scalability.

So, we’ve taken a more ambitious approach to produce a very generalized XML editing platform. Using what we learned as the basis, our new editor is far more capable. Rather than relying on the mapping of XML into an equivalent HTML5 structure, we now directly use the XML facilities that are built into the browser. This approach is both far more robust and far more scalable. But the most exciting aspect is change tracking. We’re building change tracking directly into the basic editing engine – from the outset. This means that we can track all changes – whether the changes are in the text or in the structure. With all browsers now correctly implementing the standardized DOM Range model, our change tracking model has to be very sophisticated. While it’s hellishly complex, my experience in implementing change tracking technologies over many years is really coming in handy.

If you’ve used change tracking in XMetaL, you know the limitations of their technology. XMetaL’s range selection constrains how you can select which limits the flexibility of deletion. This simplifies the problem for the XMetaL customizer, but at a serious usability price. It’s one of the biggest limiting factors of XMetaL. We’re dealing with this problem once and for all with our new approach – providing a great way to implement legislative redlining.

Redlining Take a look at the totally contrived example on the left. It’s admittedly not a real example, it comes from my stress testing of the change tracking facilities. But look at what it does. The red text is a complex deletion that spans elements with little regard to the structure. In our editor, this was done with a single “delete” operation. Try and do this with XMetaL – it takes many operations and is a real struggle – even with change tracking turned off. In fact, even Microsoft Word’s handling of this is less than satisfactory, especially in more recent versions. Behind the scenes, the editor is using the model, derived from the schema, to control this deletion process to ensure that a valid document is the result.

If you’re particularly familiar with XMetal, you will notice something else too. That deletion cuts through the structure of a table!!!! XMetaL can only track changes within the text of table cells, not the structure. We’re making great strides towards proper legislative redlining technologies, and we are excited to work with our partners and clients to put them into practice.

Akoma Ntoso, Hackathon, HTML5, LegisPro Web, Standards, Transparency, W3C

Web-Based XML Legislative Editor Update

It’s been quite a while since I gave an update on our web-based XML legislative editor – LegisPro^web. But that doesn’t mean that nothing has been going on. Quite the contrary, this has been a busy year for the editor project.

Let me first recap what the editor is. It’s an XML editor, written entirely around HTML5 technologies. It was first developed last year as the centerpiece to a Hackathon that Ari Hershowitz and I staged in San Francisco and around the world. While it is designed as a general purpose XML editor and can be configured to model any XML schema, it’s primarily configured to support Akoma Ntoso.

Since then, there has been a lot of continuing interest in the editor. If you attended the 2013 Legislative Data and Transparency Conference this past May in Washington DC, you may have noticed Jim Harper of the Cato Institute demonstrating their “Deepbills” project. The editor you saw is a heavily customized early version of LegisPro^web, reconfigured to handle the XML format that the US Congress publishes legislation in.

And that’s not the only place where LegisPro^web has been adopted. We’re in the finishing stages of a somewhat larger implementation we did for Chile. This is an Akoma Ntoso implementation – focused on debates and debate reports rather than on legislation. One interesting point worth noting – this implementation is done in Spanish. LegisPro^web is quite easily localized.

The common thread between these two implementations in the use case – they’re both implementations focused on tagging metadata within pre-existing documents rather than on creating new documents from scratch. This was the focus of the Hackathon we staged back in 2012 – little did we know how much of a market would exist for an editor focused on annotation rather than document creation. And there’s more to still come – we’ve been quite surprised in the level of interest in this particular use-case.

Of course, we’re not satisfied with an editor that can only annotate existing documents. We’ve been hard at work turning the editor into a full-featured legislative editor that works equally well at creating new documents as it does at annotating existing documents. In addition, we’ve made the editor very customizalble as well as adding capabilities to manage the comments and discussions that might revolve around a document as it is being created and annotated.

Most recently, the editor has been upgraded to the latest version of Akoma Ntoso coming out of the OASIS legal document ML technical committee where I am an active member. Along with that effort, the validator has been separated to run as a standalone Akoma Ntoso validator. I talked about that in my blog last week. I’m busy using the validator as I work frantically to complete an Akoma Ntoso project I am working on this week. I’ll talk some more about this project next week.

So where do we go from here? Well, the first big effort is to modularize the technologies found within the editor. We now have a diverse set of customers right now and they can all benefit from the various bits and pieces that make up LegisPro^web. By modularizing the pieces, we’ll be able to pick and choose which parts we use when and how. Separating out the validator was the first step. We’ll also be pulling out the reference resolver, attaching it to a native XML database, and partitioning out the client-side to allow the editing component to be used without the full editing environment offered by LegisPro^web.

One challenge that remains is handling redlining – managing insertions and deletions. This is a very difficult subject – and one I tackled in the work I did implementing the XML editor used by the California legislature. I took a very different approach in trying to solve the problem with LegisPro^web, but I’m not happy with the result. So, I’ll be returning to the proven approach we used way back when we built the original LegisPro editor on XMetaL.

As you can tell, we’ve got our work for the next year cut out for us.

Akoma Ntoso, HTML5, LegisPro Web, Standards, Transparency

2013 Legislative Data and Transparency Conference

Last week I participated in the 2013 Legislative and Transparency Conference put on by the U.S. House of Representatives in Washington D.C.

It was a one day event that featured numerous speakers both within the U.S. government and in the surrounding transparency community around D.C. My role, at the end of the day, was to speak as a panelist along with Josh Tauberer of GovTrack.us and Anne Washington of The George Washington University on Under-Digitized Legislative Data. It was a fun experience for me and allowed me to have a friendly debate with Josh on API’s versus bulk downloads of XML data. In the end, while we both fundamentally agree, he favors bulk downloads while I favor APIs. It’s a simple matter of how we use the data.

The morning sessions were all about the government reporting the progress they have made over the past year relating to their transparency initiatives. There has been substantial progress this year and this was evident in the various talks. Particularly exciting was the progress that the Library of Congress is making in developing the new congress.gov website. Eventually this website will expand to replace THOMAS entirely.

The afternoon sessions were kicked off by Gherardo Casini of the UN-DESA Global Centre for ICT in Parliament in Rome, Italy. He gave an overview of the progress, or lack thereof, of XML in various parliaments and legislatures around the world. He also gave a brief mention of the progress in the LegalDocumentML Technical Committee at OASIS which is working towards the standardization of Akoma Ntoso. I am a member of that technical committee.

The next panel was a good discussion on extending XML. The panelists were Eric Mill at the Sunlight Foundation who, among other things, talked about the HTML transformation work he has been exploring in recent weeks. I mentioned his efforts in my blog last week. Following him was Jim Harper at the Cato Institute. He talked about the Cato Institute’s Deepbills project. Finally, Daniel Bennett gave a talk on HTML and microdata. His interest in this subject was also mentioned in my blog last week.

One particularly fun aspect of the conference was walking into the entrance and noticing the Cato Institute’s Deepbills editor running on the table at the entrance. The reason it was fun for me is that their editor is actually a customization of an early version of the HTML5-based LegisPro Web editor which I have spent much of the past year developing. We have developed this editor to be an open and customizable platform for legislative editing. The Cato Project is one of four different implementations which now exist – two are Akoma Ntoso based and two are not. More news will come on this development in the not-too-distant future. I had not expected the Cato Institute to be demonstrating anything and it was quite a nice surprise to see software I had written up on the display.

If there was any recurring theme throughout the day, it was the call for better linked data. While there has been significant progress over the past year towards getting the data out there, now it is time to start linking it all together. Luckily for me, this was the topic I had chosen to focus on in my talk at the end of the day. It will be interesting to see the progress that is made towards this objective this time next year.

All in all, it was a very successful and productive day. I didn’t have a single moment to myself all day. There were so many interesting people to meet that I didn’t get a chance to chat with nearly as many as I would have liked to.

For an amusing yet still informative take on the conference, check out Ari Hershowitz’s Tabulaw blog. He reveals a little bit more about some of the many projects we have been up to over the past year.

https://cha.house.gov/2013-legislative-data-and-transparency-conference

Akoma Ntoso, HTML5, Standards, Transparency, W3C

XML, HTML, JSON – Choosing the Right Format for Legislative Text

I find I’m often talking about an information model and XML as if they’re the same thing. However, there is no reason to tie these two things together as one. Instead, we should look at the information model in terms of the information it represents and let the manner in which we express that information be a separate concern. In the last few weeks I have found myself discussing alternative forms of representing legislative information with three people – chatting with Eric Mill at the Sunlight Foundation about HTML microformats (look for a blog from him on this topic soon), Daniel Bennett regarding microdata, and Ari Hershowitz regarding JSON.

I thought I would try and open up a discussion on this topic by shedding some light on it. If we can strip away the discussion of the information model and instead focus on the representation, perhaps we can agree on which formats are better for which applications. Is a format a good storage format, a good transport format, a good analysis/programming format, or a good all-around format?

1) XML:

I’ll start with a simple example of a bill section using Akoma Ntoso:

<section xmlns="http://docs.oasis-open.org/legaldocml/ns/akn/3.0/CSD03" 
       id="{GUID}" evolvingId="s1">
    <num>§1.</num>
    <heading>Commencement </heading>
    <content> <p>This act will go into effect on 
       <date name=”effectiveDate” date="2013-01-01">January 1, 2013</date&gt;. 
    </p> </content>
</section>

Of course, I am partial to XML. It’s a good all-around format. It’s clear, concise, and well supported. It works well as a good storage format, a good transport format, as well as being a good format of analysis and other uses. But it does bring with it a lot of complexity that is quite unnecessary for many uses.

2) HTML as Plain Text

For developers looking to parse out legislative text, plain text embedded in HTML using a <pre> element has long been the most useful format.

   <pre>
   §1. Commencement
   This act will go into effect on January 1, 2013.
   </pre>

It is a simple and flexible represenation. Even when an HTML represenation is provided that is more highly decorated, I have always invariably removed the decorations to leave behind this format.

However, in recent years, as governments open up their internal XML formats as part of their transparency intiatives, it’s becoming less necessary to write your own parsers. Still, raw text is a very useful base format.

3) HTML/HTML5 using microformats:

<div class="section" id="{GUID}" data-evolvingId="s1">
   <div>
      <span class="num">§1.</span> 
      <span class=”heading”>Commencement </span>
   </div>
   <div class="content"><p>This act will go into effect on 
   <time name="effectiveDate" datetime="2013-01-01">January 1, 2013 <time>. 
   </p></div>
</div>

As you can see, using HTML with microformats is a simple way of mapping XML into HTML. Currently, many legislative data sources that offer HTML content either offer bill text as plain text as I showed in the previous example or they decorate it in a way that masks much of the semantic meaning. This is largely because web developers are building the output to an appearance specification rather than to an information specification. The result is class names that better describe the appearance of the text than the underlying semantics. Using microformats preserves much of the semantic meaning through the use of the class attribute and other key attributes.

I personally think that using HTML with microformats is a good way to transport legislative data to consumers that don’t need the full capabilities of the XML representation and are more interested in presenting the data rather than analyzing or processing it. A simple transform could be used to take the stored XML and to then translate it into this form for delivery to a requestor seeking an easy-to-consume solution.

[Note: HTML5 now offers a <section> element as well as an <article> element. However, they’re not a perfect match to the legislative semantics of a section and an article so I prefer not to use them.]

4) HTML5 Microdata:

<div itemscope 
      itemtype="http://docs.oasis-open.org/legaldocml/ns/akn/3.0/CSD03#section" 
      itemid="urn:xcential:guid:{GUID}">
   <data itemprop="evolvingId" value="s1"/>
   <div>
      <span itemprop="num">§1.</span>
      <span itemprop="heading">Commencement </span>
   </div>
   <div itemprop="content"> <p>This act will go into effect on 
      <time itemprop="effectiveDate" time="2013-01-01">January 1, 2013 </time>.
   </p> </div>
</div>

Using microdata, we see more formalization of the annotation convention than microformats offers – which brings along additional complexity and requires some sort of naming authority which I can’t say I either really understand or see how it will happen. But it’s a more formalized approach and is part of the HTML5 umbrella. I doubt that microdata is a good way to represetn a full document. Rather, I see microdata better fitting in to the role of annotating specific parts of a document with metadata. Much like microformats, microdata is a good solution as a transport format to a consumer not interested in dealing with the full XML representation. The result is a format that is rich in semantic information and is also easily rendered to the user. However, it strikes me that the effort to more robustly handle namespaces only reinvents one of XMLs more confusing aspects, namely namespaces, in just a different way.

5) JSON

{
   "type": "http://docs.oasis-open.org/legaldocml/ns/akn/3.0/CSD03#section",
   "id": "{GUID}",
   "evolvindId": "s1",
    "num" : {
      "type": "http://docs.oasis-open.org/legaldocml/ns/akn/3.0/CSD03#num",
      "text": "§1."
   },
   "heading":  {
      "type": "http://docs.oasis-open.org/legaldocml/ns/akn/3.0/CSD03#heading",
      "text": "Commencement"
   },
   "content": {
      "type": "http://docs.oasis-open.org/legaldocml/ns/akn/3.0/CSD03#content",
      "text1": "This act will go into effect on "
      "date": {
         "type": "http://docs.oasis-open.org/legaldocml/ns/akn/3.0/CSD03#date",
         "date": "2013-01-01",
         "text": "January 1, 2013"
      }
      "text2": "."
   }
}

Quite obviously, JSON is great if you’re looking to easily load the information into your programmatic data structures and aren’t looking to present the information as-is to the user. This is a programmatic format primarily. Representing the full document in JSON might be overkill. Perhaps the role of JSON is for key parts of extracted metadata than the full document.

There are still other formats I could have brought up like RDFa, but I think my point has been made. There are many different ways of representing the same legislative model – each with its own strength and weaknesses. Different consumers have different needs. While XML is a good all-around format, it also brings with it some degree of sophistication and complexity that many information consumers simply don’t need to tackle. It should be possible, as a consumer, to specify the form of the information that most closely fits my need and have the legislative data source deliver it to me in that format.

[Note: In Akoma Ntoso, the format is called the “manifestation.” and is specified as part of the referencing specification.]

What do you think?

Akoma Ntoso, Hiring, HTML5, LegisPro Web

My Week in Ravenna at the Akoma Ntoso Developer’s Workshop

Last week I attended the LEX Summer School 2012 and the Akoma Ntoso Developer’s Workshop in Ravenna, Italy. These events were hosted by the University of Bologna. For me, this was my third year that I visited Ravenna. I had originally attended the LEX Summer School 2010 as a student, returned last year as an invited speaker, and this year was again a speaker and a teacher at the Developer’s Workshop.

As always, the visit provided me with a great opportunity to compare notes with my colleagues around the world, learn a thing or two, and share my own experiences in the field of legal informatics. Of course, there has been a lot of development in Akoma Ntoso since 2010. Back then, Akoma Ntoso was a hard to pronounce secret that nobody had heard about. Today, people are slowly learning how to pronounce Akoma Ntoso, the schema is well on its way to becoming a standard, and it is far better known in this field.

The biggest development undoubtedly didn’t happen in Ravenna but rather at the World e-Parliaments Conference in Rome. There a number of parliaments and legislative bodies from around the world committed to providing some support for Akoma Ntoso in the future. Moving forward to a common standard will be the biggest advancement for the still fledgling legal informatics industry.

Back in Ravenna at the LEX Summer School, I once again presented legix.info which demonstrates the application of Akoma Ntoso to California legislation as well as other select legislative documents from around the work.

Once the LEX Summer School concluded, we switched to the Akoma Ntoso Developer’s Workshop. While the group was small, the people involved were all hard-core legisltive information systems developers and the discussions were lively and fruitful. A number of tools were described:

Bungeni – Ashok Hariharan described the Bungeni system based around OpenOffice that has been developed forthe Global Centre for ICT in Parliaments at UNDESA. Bungeni is a full fledged legislative information system incorporating an editor, workflow, calendaring, citizen participation, and other modules.
Italian Senate Editor – Carlo Marchetti and Roberto Battistoni of the Italian Senate describe their editor built on an earlier version of MS Word using the custom XML tagging technology that Microsoft has since been forced to remove. As this technology has been abandoned by Microsoft, the Italian Senate is having to implement an alternative approach using Word content controls.
Unibo Web-Based Editor – Monica Palmirani surprised us all by introducing a web-based editor that the University of Bologna is developing for Akoma Ntoso. It’s quite similar to the LegisPro Web editor having been inspired by my work. As can be expected, they are learning many of the same lessons that I have learned on my journey. Where their approach differs from mine is that they’re relying less on new HTML5 capabilities and more on pre-existing open source frameworks and modules.
AT4AM – Philip Luppens described the AT4AM amendment tool developed for the European Union Parliament. This is a web-based editing environment for creating European style amendments. It is built using the Google Web Toolkit (GWT). The EU have announced plans to produce an updated open source version of AT4AM.
LegisPro Web – I concluded the workshop by describing our new HTML5-based LegisPro Web editor for Akoma Ntoso. I spent much of my time describing the HTML5 technologies we use and the issues I have uncovered in adapting Akoma Ntoso to an editing environment for US style legislation. It was an interesting an engaging conversation. It’s always a bit nerve-wracking to put your own work up on display for other people to tear apart, but the results are always a better product in the end.

In the meantime while I was in Italy, there have been a lot of developments back stateside. There are now two active projects to implement the LegisPro Web editor. One project is based on Akoma Ntoso while the other uses the US House DTD. In addition, we are in the process of concluding a contract to add a third project, using a custom schema developed by one of the states in the USA. And if that wasn’t enough, we have just received another large contract to update a legislative system.

We need a few good people!!! All this progress means that we’re going to be stretched thin going into the future. And what that means is that we’re looking for a few good people. If you or anyone you know is looking for a new position in this fledgling legal informatics industry, please contact us at jobs@xcential.com. We’re looking for people with a passion for this subject first and foremost. You will ideally have good XML experience in the area of document authoring and will be quite comfortable with schemas and DTDs. Our development platforms are quite diverse. Of course, as we do a lot of cutting edge browser-side development, we need good quality JavaScript developers. This is not just any old JavaScript development though. It’s complex AJAX-style development using the latest HTML5 capabilities, lots of sophisticated DOM manipulation, and complex XSL Transforms. Also, our other client platform is XMetaL. Most of the development in XMetaL is also JavaScript based, with some need for Perl experience. On the backend, we do development in both Java and .Net depending on the customer and we use SQL Server, mySQL, and Oracle XDB for our database work.

Legis Info

Applying Legislative Technologies

Category Archives: HTML5