Recently there have been a few articles on how AI slop is creating problems for legislative drafting, including this one.
I’ve been giving this problem a lot of thought. Although I am not a drafter, I’ve been in the field for 24 years now – starting in the State of California and expanding around the world.
The basic problem, as I see it, is that AI is contributing to a breakdown in the rigor of legislative drafting. Rather than having highly skill legislative attorneys do the task, AI agents are being used, often without disclosure, to craft legislation. The result is legislation riddled with errors that have to be cleaned up.
Given my drafting experience in California, this doesn’t make a lot of sense. In California, the swim lanes are clear. Lobbyists and other special interests influence policy, but don’t define it. Politicians define policy, but don’t draft legislation to implement it. Only the team of highly skill drafters in Legislative Counsel get to draft legislation.
However, I have learned that these strict swim lanes don’t always apply and solutions must be devised to work in this environment.
To me, there are three solutions to the problem of AI drafted legislation:
- Like California, implement stricter swim lanes. This might be difficult to achieve. In California, it took a constitutional amendment in 1966 to overcome all the resistance.
- Another obvious answer is to provide more robust legislative drafting capabilities to those drafting legislation outside of the Legislative Counsel, building in some level of tolerance for AI concocted language being part of the mix. Knowing what I know of how complex legislation can be, I remain skeptical.
- So, perhaps there is a third option, one that takes a very different path to finding a solution – having legislation be specified as policy intent rather than as concrete legislative language.
I’m sure this sounds like the idealistic ramblings of a software developer but hear me out.
How do you specify policy intent? How do you describe exactly what you want if you don’t want to take the laborious route of getting enough bandwidth from a team of lawyers in Leg. Counsel that simply aren’t funded to the extent necessary to be at your beck and call?
The answer today is simple; you tell an AI chatbot what you want, and it drafts the bill for you – throwing all caution to the wind. The result may even look pretty good – it’s a plausible bill. It is only when someone actually analyzes the bill closely that the cracks begin to reveal themselves, the original legal text isn’t right, the references aren’t correct, consequential amendments are missing, and so on. There is a myriad of things that could be wrong – and likely are.
This all means that the result of having an AI chatbot generate a bill is throwaway rubbish – a suitable prototype for a bill but certainly not a draft that can stand up to scrutiny.
But what if it was the process that created the flawed bill that was valuable rather than the end result? Chances are, someone using AI to draft a bill took considerable time and effort to explain to the AI chatbot what they wanted. The conversation probably looked very similar to the conversation they would have with a drafting attorney if one were to be assigned to them. Why not capture that conversation rather than just the faulty end result.
It turns out that there is a way to capture a conversation like this and bottle it up as a reusable or replayable task – and one that is easily deliverable. It is a new technology called a “skill” — a reusable package of instructions, workflows, and supporting resources that teaches an artificial intelligence agent how to perform a specific, multi-step task.
A skill is quite easy to create in a platform, like Claude or Skill.yo. You simply explain to the chatbot what you are trying to get, get a response, and then iterate providing more and more information and instructions until you get the result you think is right. Then you save that conversation as a skill.
Skills can be quite sophisticated. They can have a UI. And they are reusable.
Now, imagine that the skill is the description of policy intent. They’ve refined it as far as they can, even though it’s not perfect. But rather than delivering an imperfect result, they instead deliver the skill to Leg. Counsel as their expression of policy intent.
Leg. Counsel, with a skill enabled drafting editor, can then take the policy intent skill they receive, run it against the official texts they have, and under the stewardship of a qualified drafter, refine the result taking into account all the considerations that are invisible to someone trying to draft on the outside.
What is more, a skill enabled drafting editor would have other abilities as well – allowing the drafters in Leg. Counsel to create their own library of skills to perform all the dreary tasks that hamper their productivity without surrendering the overall responsibility for drafting over to an AI chatbot.
There are some very serious concerns that needs consideration and guardrails that need to be implemented. Whenever someone interacts with an AI chatbot, there is a risk of piercing any confidentiality bubbles that might be in place. While the conversation won’t immediately be used by the LLM, the AI vendors don’t guarantee that they won’t use the chats for future training. This is called Vendor Data Retention. The problem can be circumvented by paying for Zero Data Retention (ZDR), but it’s very costly – reflecting the value of using chat conversations for future training. This has to be given a lot of consideration.
All of this is just food for thought…
Legis Info 