Brazil courts confront AI prompt injection attempts by lawyers

rtificial intelligence is reshaping how courts operate worldwide, speeding up case processing and helping organize legal documents. But that shift is bringing new vulnerabilities with it.

In Brazil, court officials uncovered the first known cases of lawyers attempting to manipulate AI systems used by the judiciary, embedding hidden commands within legal filings to influence how cases were automatically processed.

The scheme surfaced at a Brazilian labor court, where staff discovered a filing containing invisible text, written in white letters on a white background so it couldn’t be read by human eyes.

The court’s AI system, however, could detect and process the text. The hidden instruction told the software to review the case only superficially and avoid challenging the evidence submitted.

The attempt was caught before it could affect the proceedings, but it raised serious concern among officials. Experts are calling it a new form of digital fraud known as “prompt injection,” in which malicious instructions are embedded in documents to manipulate how an AI system behaves.

Courts in many countries increasingly rely on AI tools to summarize lengthy case files, search relevant legal precedent, and support the daily work of judges and staff. While final rulings remain solely in human hands, these systems now play a role in preparing and organizing cases, making them a potential target for bad actors.

Following the incidents in Brazil, judicial authorities are considering new safeguards, including automatic detection of invisible text, formatting checks on documents before processing, and stronger human oversight at every stage.

Discussions are also underway about stricter ethics rules for legal professionals who use AI in their work.

The case is a reminder that technological progress brings new kinds of threats, ones that now target not just people but the digital tools built to support them. How safely AI is used in the justice system will depend on developing effective safeguards and preserving human control over critical decisions.

Source: tovima.com