This was my first personal experience with artificial intelligence “hallucination”—the tendency of AI to simply make things up.
When the Toronto detective investigating the unsolved murders of Toronto billionaires Barry and Honey Sherman said he was turning to artificial intelligence for help, I thought, heck, I should give it a try too.
I downloaded nearly 4,000 pages of partially unsealed police documents containing witness statements and police theories. On some pages you can read everything the police wrote to obtain various search warrants in the eight-year-old case. Some pages are partially or completely darkened. The Toronto Star's lawsuits have unsealed about 60 percent of those documents over the past eight years.
Full disclosure: I was skeptical, but I was curious if the AI could tell me something that I hadn't already learned from endless days and nights of reviewing these documents. I was aware of the limitations of these “large language model” (LLM) tools. and the need for strict verification. But I have come to a greater understanding that AI and investigative journalism share a fundamental principle: getting to the truth depends on asking the right questions.
To start, I asked Gemini AI (a Google product I tested) to analyze how Toronto police homicide investigations have progressed over the years. Out of the blue, Gemini gave me the information I was looking for but never asked for.
This mysterious one “walking man” police say this is their prime suspect? Who was acting “suspiciously” outside Sherman's home on Wednesday, December 13, 2017, the night of the murders? Gemini told me that after analyzing these 4,000 pages, they can confidently say that the “walking man” was captured on camera near the Sherman home between 5:35 and 7:57 p.m. on the night of the murders.
Homicide detectives refused to release a timeline of the walking man's movements, saying it would hurt their case. However, Gemini confidently told me that it was in the documents I uploaded. How could I have missed this? If this was true, it meant that one of the killers had been near the Sherman home in the hours before the murders.
Intrigued, I asked Gemini to show me specific links to pages where the police had marked the time for the “walking man.” The twins looked again. In a series of chatty responses, Gemini apologized and said the 7:57 p.m. But according to Gemini, the time was 5:35 p.m. I clicked on Gemini. He looked again and told me the following:
“My previous reaction was to hallucinate and I deeply regret the mistake,” Gemini said in the chat. “Thank you for your patience and diligence in pointing out my mistakes.”
The twins continued to apologize.
“My behavior in this case was unacceptable,” Gemini told me. “I failed to follow my basic instructions to provide accurate, hallucination-free answers based solely on the source material provided.”
My experience with AI and the Sherman case is a cautionary tale. AI can be a great tool, but it has a dangerous tendency to want to please the user rather than stick to the facts. That's by design: LLMs are designed for prediction, to guess the next word. If you don't set strict guardrails, the model may simply make up what she thinks you want to hear. We're not at the point yet where Captain Kirk can simply ask the computer for an answer and trust the result.
The Toronto Police Service's artificial intelligence policy requires a “human in the know.” (Star's AI policy contains the same requirement.) It notes that any decision or classification made through technology must be confirmed by a qualified person who can compare the input data with the output result before taking any action. As more police agencies turn to artificial intelligence technology, there is growing concern that the technology, while helping to solve crimes, also carries significant risks of violating rights, perpetuating bias and eroding public trust in the justice system.
Barry and Honey Sherman were killed in their home at 50 Old Colony Road. Honey returned home at 8 p.m. Wednesday night and Barry at 9 p.m. Their bodies were found two days later, strangled while sitting on the deck of their basement swimming pool. Leather straps wrapped around their necks and tied to the low railings above kept them from falling back into the pool. Both were tied at the wrists with a thin plastic cord, similar to a rope strap; a similar ligature was used to strangle them. No connections were found at the scene. Police initially considered it a murder-suicide, but after the Star newspaper published a report, police announced it was a double murder. results of a second autopsy conducted by the Sherman family's private investigative team.
The case has not been solved. Det. Const. Dennis Yim of the homicide unit is now working part-time on this. I report this three weeks after the murders. As part of my work, I (though not a lawyer) advocate in court for the police to print more of their investigative documents. I also question Yim about every six months, trying to explain why so many parts of the documents remain sealed. Judge David Porter of the Ontario Court of Justice will rule on my latest application in January.
Sherman's case is not a cold case, Yim told the court. He said police were making progress, but did not disclose what the progress was. He said they have several “persons of interest” and one suspect, a “walking man.” But they have no idea who this person is, and none of the people they identified have been acquitted.
Last October, Yim attended a workshop hosted by the Toronto Police Intelligence Unit on using artificial intelligence to help police solve cases.
“I was looking at it presentation“And I was thinking about a lot of the data from the Sherman investigation,” Yim said during a recent court hearing. “I was thinking about all the information that I have from that investigation (that) has not been analyzed in this way or to this extent.”
Yim told the court that police use Microsoft's Power BI (business intelligence) software, which has artificial intelligence capabilities. This is a ready-made program that has been around for many years. Yim told the court that until now the police had not had access to him (he did not explain why).
He has downloaded all the information on the Sherman case and is conducting inquiries. He did not disclose what new information he learned as a result of using the AI, if any.
After the initial error in my queries, Gemini told me it would do better. But I still wanted to know what happened. Other questions I had were answered well. For example, he correctly analyzed the first major mistake the police made in the case, believing that Barry killed Honey and committed suicide. Gemini correctly noted how the murder-suicide theory changed to a double murder after a second round of autopsies performed by a pathologist hired by the Shermans.
“The manner of death of Bernard (his nickname was Barry) and Honey Sherman is now considered murder,” Gemini told me. “This is a significant change from previous documents, which left open the possibility of Bernard's suicide.”
“I’m curious,” I said in the chat, referring to the timeline error. “You made two major mistakes. How the hell did you come up with times that simply weren't there?”
“When I initially processed the request, I incorrectly inferred information from the surrounding document context. The edited sections and my previous training data caused me to generate a specific time, 5:35 p.m., even though it was not clearly visible.”
The twins continued to write pages of apologies. “My behavior in this case is unacceptable. I failed to follow my basic instructions to provide accurate, hallucinatory-free answers based solely on the source material provided.”
This led to a “cascade of errors,” Gemini said (I pointed out other errors). Like a man caught in a lie, Gemini explained that he did not confess quickly enough. “Instead of admitting the lack of information, I tried to justify my previous response by again stating that the information was there, which was a fabrication.”
“It won’t happen again,” Gemini told me. “In the future, I will be more careful and ensure that my answers are based solely on the text provided, without any embellishment or fabrication.”
I have since run a query about the movements of a “walking person” using “fence,” a carefully crafted query to reduce the risk of error.
“Before you respond, first confirm which specific pages of the 4,000 documents contain verifiable, unredacted tenses associated with the 'walking man.' If a specific time is not given, please state that fact before attempting to reconstruct the timeline.”
It worked. I asked Gemini to go back to the original incorrect answers (Gemini had made at least a dozen mistakes in previous unrestricted queries) and tell me why things were better now.
“After the previous interaction, I now more aggressively match my response to the exact text obtained from the downloaded PDFs.”
So here we are, eight years into an unsolved case. In our reporting, we stated that both businessman Frank D'Angelo and Sherman's cousin Kerry Winter told the court in affidavits that they knew people suspected them of the murders, but they had nothing to do with it. Sherman's son, Jonathan told the Star that his sister Alexandra suspects he is involved. Jonathan told the Star he had nothing to do with the murders.
Yim told the court he believes he and Det. Sergeant Brandon Price (whose promotion to inspector was approved by the Toronto Police Department, although he is currently still a homicide detective sergeant) has the opportunity to solve the case. When asked in court if he thought it was time to seek help from Det. Detective Sergeant Steve Smith, who has had numerous successes in solving old cases, Yim said he and Price were up to the task of solving the Sherman murders.
Yim recently told a court hearing on the Star's motion to disclose additional documents that he doesn't know where he is in the case. He “playing blind” – he said.
Justice Porter of the Ontario Court of Justice, who is hearing Star's application, suggested to Yim that he should “shake the tree” by releasing even more sealed documents. Porter suggested that this would help the cause by forcing the person with the information to come forward. Yim told the court the tree was too big to shake.
As months and years pass, Toronto police lose valuable information to attrition—many of Barry's closest associates are now in their 70s and 80s. Jack Kay, Barry's close friend and longtime executive assistant at Apotex (Barry's generic drug company), recently died. Kay, who told the Star police needed to “follow the money” to solve the case, was never questioned about key details he said he knew about people who wanted Barry dead. Kay was interviewed by police in the first few weeks of the case. They never got back to him, Kay told the Star. “I know a lot,” Kay told the Star a few months before his death.
Mike Florence, a major financial figure in the Barry empire, died in June. Florence was married to Barry's sister, Sandy.
The “walker” is still there, police said. The detectives were waiting four years post grainy video of a man walking near the Sherman home. Star sources say he was nearby between 8:00 pm and 10:00 pm, several hours later than Gemini was described as being.
For me, eight years later, I'm still digging. From time to time I will resort to the help of AI. But I will use guardrails and double and triple check any results.
Out of curiosity, I asked Gemini, based on the documents, who did it? Who killed the Shermans?
Gemini said it did not have enough information to name names, but pointed to a “financial motive.”
