Picture a counter with a person behind it. The circulation desk at the Free Library’s Cobbs Creek Branch, photographed in 2007 for the Historic American Buildings Survey, is exactly that: a curved wooden desk, a pigeonhole shelf, a staff member working at the far end. I keep coming back to that image because of a question public agencies are now asking about chatbots. When a resident or a business owner asks something the tool can’t safely answer, where does the question go? Two governments built public chatbots with opposite answers to that question, and the results have been just as opposite. I’m writing this at Gemini Strategic Consultants, where the technology underneath hospitality and economic development is part of our work, so I read both cases closely.

The bot that wrote its own answers

New York City’s MyCity chatbot was announced in October 2023 with a promise that business owners could “access trusted information from more than 2,000 NYC Business web pages,” as The Markup reported in March 2024. The bot didn’t look those answers up. It generated them, and The Markup’s testers found it generating wrong ones.

Asked about workers’ tips, the bot said it was fine to take them. The Markup noted that was wrong, although employers can sometimes count tips toward minimum wage requirements. Asked whether a restaurant could stop taking cash, it said, “Yes, you can make your restaurant cash-free,” and added that there were no regulations requiring businesses to accept cash. A 2020 city law requires exactly that. Asked whether landlords had to accept Section 8 housing vouchers, the bot said no, although source-of-income discrimination is illegal in the city.

Here is the part I find most useful for anyone planning something similar. The City’s response was to add disclaimers and narrow the tool. In The Markup’s account, republished by StateScoop in January 2026, administrators added notices telling users not to treat its responses as legal or professional advice, improved some answers, and appeared to limit the kinds of questions it would take. They did not change how it produced an answer. In January 2026 the mayor called the earlier bot “functionally unusable” and put its cost at around half a million dollars. The Markup reported that building its foundations had reportedly cost nearly $600,000. Those are the mayor’s figure and a reported figure, and I’m attributing them rather than vouching for them.

I checked the bot’s address on 8 November 2026. It now opens an official City page that begins with the sentence “The Chatbot beta test has ended.” The City ended it.

The chatbots that only match

Pennsylvania’s Department of Labor & Industry made a different decision. According to the Office of Administration’s 15 April 2026 update, the department rebuilt its unemployment compensation chatbots in June 2025, one for benefits and one for employer taxes, “with AI that can match a wider range of user phrasing to preapproved, curated responses.” The same update says the chatbots “do not generate novel answers; instead, they escalate to a live representative whenever a question falls outside approved content.”

That’s the whole design, and it’s easy to miss how much work it assigns to people. Someone at the department has to write each answer. Someone has to approve it before it’s live. The AI does one narrow job, which is working out which approved answer a resident’s messy phrasing is closest to. When nothing fits, the question goes to a person.

The Commonwealth reports a result, and I want to describe it carefully. Since the relaunch, it says, the number of customers whose interactions were handled entirely by the benefits chatbot rose by about 65 percent. That counts conversations the bot finished without handing off. It isn’t a measure of accuracy, and I couldn’t find a published accuracy audit, so I wouldn’t read it as proof the answers are always right. What it does show is that a bot restricted to approved answers can take on a growing share of conversations on its own.

I also note what I did not find. The sources I read give no handoff rate, the share of chats that end with a person, and that is the number I would most like to see next to the 65 percent.

What Pennsylvania did with its own staff first

The chatbot design didn’t appear alone. The Commonwealth’s March 2025 announcement of a year-long pilot put ChatGPT Enterprise in the hands of 175 employees across 14 agencies. Results came from biweekly feedback sessions, user interviews and live demonstrations. Nearly half the participants, 48 percent, had never used ChatGPT before, and over 85 percent reported a somewhat or very positive experience.

The headline number was time. Employees reported saving an average of 95 minutes a day on administrative work such as drafting emails and summarizing long documents. Treat that as a self-reported estimate. Route Fifty’s coverage says the figure came from exit surveys, with participants using the tool about 35 minutes a day and estimating that they saved about 95 minutes by doing so. Nobody timed it.

Two details in the pilot matter more to me than the headline. The first is that the pilot’s own findings say human oversight remains essential: employees emphasized that AI augments human expertise rather than replacing it, which reinforces the importance of human review and judgment. The second is that the tool failed in ordinary ways. Route Fifty, citing the pilot report, lists difficulty generating images, errors extracting text from a PDF, and trouble producing accurate citations. An organization willing to publish its tool’s failures is one I trust more when it publishes its successes.

Access then got tied to training. The Office of Administration said in November 2025 that more than 1,500 Commonwealth employees had completed responsible AI training as a prerequisite to use Microsoft Copilot Chat and ChatGPT Enterprise, with another 4,100 enrolled. By April 2026 the figure was more than 3,000 employees across 35 agencies using the tools, and 6,500 more enrolled in training as a requirement for use.

Why I think the design decided it

It would be easy to say Pennsylvania chose a better model, or that New York chose a worse one. Neither case supports that. The difference the sources show is who writes the answer and who handles the exception.

In New York’s design, the tool produced an answer from web pages, and the only check was a reader who might or might not know the law. In Pennsylvania’s, the department that owns the rules wrote each answer ahead of time, and anything outside that set went to a representative. One design lets the tool improvise. The other doesn’t.

There’s a fair objection, and I’d rather state it than avoid it. A bot that only matches is less impressive than one that writes fluent answers to anything, and some residents will find the limits frustrating. That’s true. But a limit that sends a question to a person is a visible limit, and a wrong answer delivered fluently isn’t.

What this means for Philadelphia

Philadelphia already has the internal half. The Office of Innovation and Technology’s leadership page describes a City AI Policy and an AI Council built around “accountability, human oversight, and responsible governance.” The City’s employee newsletter of 6 April 2026 says Microsoft Copilot Chat is the approved enterprise AI tool and tells staff not to enter City data into commercial AI platforms. I can only point to those two public documents, because the policy itself sits on an internal site.

The open decision is the public-facing half. If the City, a business association or a corridor group puts a bot in front of business owners, the Labor & Industry design is the one I’d copy, and it comes down to three commitments. The department that owns each rule writes and signs off each answer. A named person takes every question outside the approved set. And two numbers get published from the first day: the share of chats resolved without a person, and the share handed off.

At the table I’d want the Office of Innovation and Technology, the Law Department, the departments whose rules the bot would explain, and the business groups who would hear about it first when it gets something wrong. None of this asks anyone to use a tool differently. It asks the institutions deploying one to decide, before launch, where the human sits.

I’d start with the desk in that photograph. A tool can sort, match and route, but somebody has to be at the counter when the question doesn’t fit.

Two Government Chatbots, Two Designs: Pennsylvania and New York City

The bot that wrote its own answers

The chatbots that only match

What Pennsylvania did with its own staff first

Why I think the design decided it

What this means for Philadelphia

Sources and references