Counsel’s Corner: Where AI Ends and Legal Practice Begins

The Consultation Printout

Elder law practitioners know the scenario well. A client’s son arrives at the consultation with a printout. He has already asked a chatbot whether his mother’s home counts as an exempt resource. The answers are fluent and confident. And predictably—for the attorney at least—the answers are incomplete, misleading, and often wrong.

Practitioners have dealt with versions of this for years, starting with internet searches followed by message boards. The difference now is that the output reads like an authoritative and carefully crafted memo. However, the “author” is prone to presenting information in a manner the prompter will find agreeable, rather than information that is accurate and useful.

AI’s collision with the legal profession is driving a variety of responses. Each response attempts to define a meaningful line between AI that assists lawyers and AI that, in effect, “advises” non-lawyers. AI “advises” non-lawyers directly through non-lawyer use of the technology, and indirectly when the technology is used by lawyers with inadequate oversight of outputs reflecting legal judgment.

So how do attorneys compete with an AI subscription that is cheaper, faster, and untethered to rules of professional conduct? Analyzing these issues through attempts at regulating AI use for legal judgment tasks is a useful exercise. New York Senate Bill S7263-A is among the most direct attempts to date.

What the New York Bill Would Do

The bill would add Section 390-f to New York law:

• Who is covered: The bill regulates “operators,” meaning anyone who owns, operates, or deploys a chatbot that interacts with users, such as a company with a branded AI assistant on its website. Developers that merely license their technology to an operator are excluded, though a developer deploying its own consumer-facing chatbot would qualify.
• What is prohibited: An operator may not knowingly permit chatbots to impersonate actual or fictitious practitioners of a covered profession (including attorneys) in a manner that would be unauthorized practice or use of a professional title under New York law if done by a natural person.
• Impact: An operator is deemed to know chatbots cannot lawfully practice a covered profession. Still, a violation requires the operator knowingly permit impersonation. User-facing disclaimers do not avoid liability. Safe harbor applies only where users intentionally circumvent the terms of service or operator safeguards.
• Enforcement: The Attorney General may seek an injunction and civil penalties.

This Is Not Just a New York Story

New York is perhaps the most developed example at regulating AI and its use in legal contexts, but it is not the only approach:

Federal Legislation
CHATBOT Act (H.R. 7985) takes New York’s concept nationwide, making it impermissible for anyone deploying a chatbot to generate output or marketing that implies a license to practice a covered profession or that a licensed human verified the information. This includes “authoritative and conclusive recommendations, advice, or guidance” typically provided by a practitioner, excluding general information. Enforcement would run through the FTC, state attorneys general, private rights of action, and state UPL enforcement.

California’s Approach
California’s approach focuses on lawyers rather than vendors. SB 574 was recently signed it into law and states attorneys “shall not delegate the practice of law to generative artificial intelligence,” may not enter confidential, personal identifying, or other nonpublic information into an AI tool unless access is restricted to authorized persons. Attorneys must verify AI output, personally verify citations, and disclose AI use on court submissions. There are no independent penalties, but violations may draw sanctions and State Bar discipline.

Litigation
In Nippon Life Insurance Co. of America v. OpenAI Foundation, an insurer alleges that ChatGPT encouraged a former claimant to question her attorney, helped her reopen a case, and drafted several filings. Insurer asserts UPL under Illinois statute. OpenAI has moved to dismiss, arguing that ChatGPT is a tool, not a person.

Two Ideas at Once

Each of these responses is more about accountability than technology. Skeptics and critics say AI UPL rules act as guild protection and that the consumer-protection case is overstated. But practitioners can hold two ideas at once: (1) that AI should be curtailed where it poses as counsel or attempts to exercise legal judgment; and (2) that AI should be allowed to roam where it makes lawyers better at what only lawyers can do while still providing non-lawyers with access to information.

Using AI is not acquiescence to technological competition. ABA Formal Opinion 512 recognizes that lawyers may use generative AI subject to familiar ethical duties of competence, confidentiality, communication, etc. Used appropriately, AI can help the lawyer exercise legal judgment more quickly and, hopefully, more cost effectively.

What Practitioners Can Do Now

• Know your tool’s terms. Understand how any AI platform handles client data before confidential information goes in. Confidentiality is the lawyer’s duty most easily breached by convenience.
• Keep the lawyer’s judgment visible. Review every output and verify every citation. Courts have not been kind to lawyers who skipped that step.
• Ask about clients’ own AI use at intake. Clients are already consulting chatbots. Knowing what they have read helps you correct misinformation early, demonstrate competency, and develop trust.
• Audit your public-facing tools. If your website runs a chatbot, consider its disclosures, its labeling, and whether anything  implies a lawyer verified the output.
• Join the conversation. Bar committees, legislative comment periods, and CLE panels are where these conversations happen. The people best positioned to explain what legal judgment consists of are the lawyers exercising it every day.

Closing Thought

Where does legal information end and legal advice begin? Who is accountable when software crosses that line? What duties do lawyers owe to the public when the public is asking a machine instead of consulting with counsel? These questions need answering, and the answers will benefit from more attorney participants, not fewer. AI reticence is reasonable. Abstaining from the discussion is not.

“Decisions are made by those who show up.” -Martin Sheen as Jed Bartlet, The West Wing