According to city officials and national housing advocates, San Jose’s use of AI surveillance to find homeless encampments is the first of its kind in the country.
Residents have complained to the city’s 311 phone line about homeless encampments 914 times so far in 2024. They reported illegal dumping 6,247 times, graffiti 5,666 times, and potholes 769 times last year. The goal of the surveillance pilot is to address these complaints more efficiently.
Last July, San Jose issued an open invitation to technology companies to mount cameras on a municipal vehicle that began periodically driving through the city’s District 10, collecting footage of the streets and public spaces. The images were fed into computer vision software and used to train the companies’ algorithms to detect the unwanted objects, according to interviews and documents the Guardian obtained through public records requests.
City employees are driving a single camera-equipped vehicle through sections of District 10 “every couple weeks”, said Khaled Tawfik, director of the San Jose information technology department. The city sends the training footage to participating companies, which include Ash Sensors, Sensen.AI, Xloop Digital, Blue Dome Technologies, and CityRover.
Ken Salsman, chief technology officer for Ash Sensors, said his company, which specializes in sensors that monitor the structural health of buildings, had not explored homelessness detection before learning of San Jose’s pilot. The experiment provided an opportunity to create potentially marketable technologies by solving challenging computer vision problems, such as distinguishing an empty RV parked outside a home from an RV that is a home.
“Are the windows covered inside the vehicle? Are there towels to provide privacy? Is there trash outside the vehicle, suggesting they’re using food and having trouble getting rid of the waste?” Salsman said. He added that successful detection of lived-in vehicles would probably require frequent scanning of city streets to establish whether the vehicles have moved.
A report from the company Sensen.AI shows that its system detected 10 lived-in vehicles in footage collected from two streets on 8 February.
Tawfik said the goal of the pilot was to encourage companies to build algorithmic models that could detect a variety of different objects from car-mounted cameras with at least 70% accuracy. The participating companies are currently detecting lived-in RVs with between 70 and 75% accuracy, he said, but the accuracy for lived-in cars is still far lower: between 10 and 15%.
City documents state that, in addition to accuracy, one of the main metrics the AI systems will be graded on is their ability to preserve the privacy of people captured on camera – for example, by blurring faces and license plates. Tawfik said that the city did not “capture or retain images of individuals” through the pilot and that “the data is intended for [the city’s housing and parks departments] to provide services.”
