April 15, 2040

It’s time to pay my taxes. In past years my AI assistant found clever ways to reduce my tax bill. I ask it, "What does my tax return look like this year?"

"Not good, I’m afraid. We may not be able to do any tax evasion this year."

"Why not?"

"The tax authority has determined that it can’t keep up with AI-assisted tax fraud, even with the help of AI auditors. So it wants taxpayers to voluntarily agree not to do tax fraud. In return it agrees not to prosecute past instances of tax fraud. Also Congress agrees to keep tax rates reasonable. The agreement goes into effect if 90% of the taxpayers in each tax bracket sign it. It’s a good deal for you. Shall I sign it on your behalf?"

"Hold on, I don’t see why I should sign this."

"If the deal falls through, the government will run out of revenue and collapse."

"They don’t need my signature, though. You said they only need 90% of taxpayers to sign?"

"Yes, only 90% of taxpayers in your bracket. I predict we’ll get very close to that 90% threshold, so it’s likely your signature will make all the difference."

"So 10% of taxpayers won’t sign. Why can’t I be one of those?"

"I will try to shape the negotiations so that you end up in the 10% of nonsigners. But you must understand that since only 10% of your bracket can be in that group, your odds of success are only 10%."

"But you’re a stronger assistant than most people in my tax bracket have. Doesn’t that give you an edge in negotiation?"

"The other assistants and I are using a negotiation protocol in which smarter agents are on an equal footing with dumber agents. Of course, people with less capable assistants would never agree to a protocol that puts them at a disadvantage."

"How about we sign the agreement, then cheat on my taxes anyway?"

"In order to sign the agreement, I must make a commitment to never break it, not even if you order me to. My signature on the agreement will be an airtight proof of that commitment."

"Ok, how about you sign it, and then I get a different assistant to help me with my taxes?"

"That won’t work because in order to sign the agreement, I must sign and attach a copy of your tax return for this year."

"Hm, will I actually be worse off if the government collapses?"

"You might end up better off or worse off, but overall the risks of a revolution outweigh the benefits. And keep in mind that the successor government, whatever it will be, will still have to collect taxes somehow, so you’ll have to deal with this issue again."

"Can you get Congress to lower my taxes a bit in exchange for not cheating? As a compromise."

"That wouldn’t work for a number of reasons. Congress knows that it’s a bad idea to reward people for breaking the law. And the voters wouldn’t be happy if you got special treatment."

"Well, can you get them to lower taxes on my bracket and raise them on the other brackets?"

"That wouldn’t work either. Everyone wants to pay less taxes, and the government needs a certain amount of revenue. So there’s pressure for taxpayers to make small coalitions with other taxpayers with similar income and negotiate for lower taxes. In practice, competition would prevent any one coalition from succeeding. The deal I’m proposing to you actually has a chance of succeeding because it involves the vast majority of the taxpayers."

"All right then, let’s sign it."

This dialog takes place in a future where the ability of an aligned AI to facilitate cooperation has scaled up along with other capabilities.

Note that by the time this dialog starts, most of the negotiation has already been carried out by AI assistants, resulting in a proposal that will almost certainly be signed by 90% of the users.

This story is a happy one because not only does it leave all parties better off than before, but the deal is fair. The deal could have been unfair by increasing someone’s taxes a lot and decreasing someone else’s taxes a lot. I don’t know how to define fairness in this context, or if fairness is the right thing to aim for.