Episode 58: Ubiquitous License Plate Recognition

With a Raspberry Pi, a camera and some open source software, anyone can record, recognise and store number plates on mass. What does that mean for the privacy of car owners and passengers now and in the future?

This week, The Private Citizen is back with another thought-provoking episode. This time, I talk about what it means for all of us if anyone can just capture and recognise number plates easily and cheaply.

No More Privacy for Car Owners #

Producer jonathan posted on our Discord server about this guy building his own license plate scanner. That gave me the idea to make this issue a topic on the show today.

I’ve known about OpenCV for years. In fact, I used it once to build a system (based on a Raspberry Pi) that would turn on a camera aimed at our hamster terrarium when it detected motion. It turns out that, bypassing OpenCV and using machine learning, you can stick the Pi in your car, record license plates and send them to a cloud server to be analysed.

First, the YOLOv3 model detects the bounding boxes of each license plate in each frame received from the camera. The predicted bounding boxes are not recommended to be very precise – encompassing more than what the detected object takes is a good idea. If it’s too cramped, then the subsequent processes’ performance might be hindered. And this goes hand in hand with the following model.

CRAFT text detector receives the cropped license plates from YOLOv3. Now, if the cropped frames were to be too cramped, then there would be a very high chance that part of the license plate’s text would be left out and then the prediction would fail miserably. But when the bounding boxes are larger, we can just let the CRAFT model detect where the letters are found. This gives us very precise locations of each letter.

Finally, we can pass on the bounding boxes of each word from CRAFT to our CRNN model to predict the actual words.

Basically, anyone with a bit of technical knowledge can do this and it is relatively cheap.

With the current configuration, running this costs about ~USD 4.00/hr. Only possible by using spot instances.

And, as it turns out, more people are doing it. And there are existing open source solutions, like OpenALPR and the more grassroots diy-alpr, so you don’t have to build the whole system yourself. And you can also recognise data offline.

This program is built for Raspberry Pi. It continuously captures photographs of license plates along with current GPS coordinates. The captured photos are analyzed and any license plate numbers that are recognized are output to CSV or/and JSON (along with the time and location of the image). The application is multi-threaded and designed to capture at high speed (3-5 images/sec). The Rapsberry Pi can’t recognize as fast as it can capture, so image files are queued up in a capture directory and worked through as fast as the recognizer is able.

Now, this of course raises a large number of ethical and legal issues. I like how naive this guy just assumes that positive effects can be harnessed, without automatically descending into an Orwellian nightmare.

While it’s easy to get caught up in the Orwellian nature of an “always on” network of license plate snitchers, there are many positive applications of this technology. Imagine a passive system scanning fellow motorists for an abductors car that automatically alerts authorities and family members to their current location and direction.

Teslas vehicles are already brimming with cameras and sensors with the ability to receive OTA updates – imagine turning these into a fleet of virtual good Samaritans. Ubers and Lyft drivers could also be outfitted with these devices to dramatically increase the coverage area.

Of course, people will be eager to point out cases that were solved due to license plate recognition, like the German Autobahn shooter case. This case led to an examination of the laws governing such technology in Germany and its legal use is considered difficult. For example, police was specifically not allowed to use the Toll Collect truck toll data gathered all over the country on motorways.

Wikipedia says the following about the current legal situation in German:

On 11 March 2008, the Federal Constitutional Court of Germany ruled that some areas of the laws permitting the use of automated number plate recognition systems in Germany violated the right to privacy. More specifically, the court found that the retention of any sort of information (i.e., number plate data) which was not for any pre-destined use (e.g., for use tracking suspected terrorists or for enforcement of speeding laws) was in violation of German law. These systems were provided by Jenoptik Robot GmbH, and called TraffiCapture.

It’s a different picture in the UK entirely:

The Home Office states the purpose of automatic number-plate recognition in the United Kingdom is to help detect, deter and disrupt criminality including tackling organised crime groups and terrorists. Vehicle movements are recorded by a network of nearly 8000 cameras capturing between 25 and 30 million ANPR “read” records daily. These records are stored for up to two years in the National ANPR Data Centre, which can be accessed, analysed and used as evidence as part of investigations by UK law enforcement agencies.

In 2012, the UK Parliament enacted the Protection of Freedoms Act which includes several provisions related to controlling and restricting the collection, storage, retention, and use of information about individuals. Under this Act, the Home Office published a code of practice in 2013 for the use of surveillance cameras, including ANPR, by government and law enforcement agencies. The aim of the code is to help ensure their use is “characterised as surveillance by consent, and such consent on the part of the community must be informed consent and not assumed by a system operator. Surveillance by consent should be regarded as analogous to policing by consent.”

ANPR systems are also in use in the US, of course. Sometimes with amusing, and frightening, consequences:

The Department of Homeland Security has proposed a federal database to combine all monitoring systems, which was cancelled after privacy complaints. In 1998, a Washington, D.C. police lieutenant pleaded guilty to extortion after blackmailing the owners of vehicles parked near a gay bar. In 2015, the Los Angeles Police Department proposed sending letters to the home addresses of all vehicles that enter areas of high prostitution.

Early private sector mobile ANPR applications have been for vehicle repossession and recovery, although the application of ANPR by private companies to collect information from privately owned vehicles or collected from private property (for example, driveways) has become an issue of sensitivity and public debate. Other ANPR uses include parking enforcement, and revenue collection from individuals who are delinquent on city or state taxes or fines. The technology is often featured in the reality TV show Parking Wars featured on A&E Network. In the show, tow truck drivers and booting teams use the ANPR to find delinquent vehicles with high amounts of unpaid parking fines.

So what does all of this mean for car owners and passengers? Is privacy still possible? And do people even care?

Producer Feedback #

ezequiel17 says:

Hey Fab, listener from the UK here trying out Matrix/Element for the first time. I was a big fan of Linux Outlaws back in the day, and an occasional listener to GNR afterwards, though as I’m not a gamer I didn’t listen to many and none for a long time. I Looked you up the other day after talking to someone about my favourite podcasts and was happy to see I was a year late to The Private Citizen! Anyway, I’m subscribed again and trying to work my way through the back catalogue – glad you are well and please keep up the great work podcasting – it’s very much appreciated.

Evgeny Kuznetsov chimes in with comments on the previous episode about medical data:

Some feedback re medical data ownership here. From the perspective of a psychiatrist: I see the point in the general sentiment that the medical data should be owned by the patient, but… As it is now, the healthcare institution owns the data and is responsible for its completeness, attributability, legibility, traceability, etc. There’s a trail of what data has been collected when and by who. The data is reliable. Even more important: the lack of data is reliable. If the data is tampered with, there are state watchdogs that will punish the institution.

If the patient owns the data, how do we make sure I, as a psychiatrist, have complete and reliable data about this patient? How am I supposed to sign off on a gun permit if I can’t be sure there wasn’t an occasion last month when the voices in the patient’s head told her to go kill her neighbour? Currently, I can be sure of that: I have hospital records, I have outpatient records, I have records of the outpatient state psychiatric facility that serves the district where the patient lives, and I can make sure there’s no record about her in the facility’s database; because the data belongs to the state (in Russia the psychiatric care is almost exclusively state-provided). If the data belonged to the patient, I could no longer rely on the data available to me being explicit, could I?

Another aspect: say some medical record turns out to be incorrect. Now that it is owned by the medical institution, measures can be taken to a) correct it and b) identify the reason the incorrect data appeared in the system and make sure it doesn’t happen again. That’s because we (the institution) own the data and are responsible for its correctness. If the data is owned by the patient, whose responsibility its correctness is? Whom does the state watchdog prosecute in case there are discrepancies, mis-collection and even forgery of medical data? Individuals?

Owning medical data is a great responsibility. I seriously doubt it’s feasible to put that responsibility on the individuals.

As Dr. House aptly put it, everybody lies. The patients do have incentives to manipulate their medical data. Paid sick leaves and disability pensions, medical limitations for particular professions, insurance premiums for diseases that are results of occupational hazards, etc, etc. We simply can not trust the patient to own this data and be in charge of it, sorry.

An anonymous producer raised another point about this topic:

As some recent examples in the American justice system showed, that even if you are not directly impacted, your relatives could be adversely affected by releasing health records. Spinning the threat of your thoughts regarding a “for profit health system” a little bit further, it isn’t to far off imagining insurance companies inferring from you (great-)parents DNA possible health problems of you and subsequently rejecting you.

If you have any thoughts on the things discussed in this or previous episodes, please feel free to contact me. In addition to the information listed there, we also have an experimental Matrix room for feedback. Try it out if you have an account on a Matrix server.

Toss a Coin to Your Podcaster #

I am a freelance journalist and writer, volunteering my free time because I love digging into stories and because I love podcasting. If you want to help keep The Private Citizen on the air, consider becoming one of my Patreon supporters.

You can also support the show by sending money to via PayPal, if you prefer.

This is entirely optional. This show operates under the value-for-value model, meaning I want you to give back only what you feel this show is worth to you. If that comes down to nothing, that’s OK with me. But if you help out, it’s more likely that I’ll be able to keep doing this indefinitely.

Thanks and Credits #

I like to credit everyone who’s helped with any aspect of this production and thus became a part of the show.

Aside from the people who have provided feedback and research and are credited as such above, I’m thankful to Raúl Cabezalí, who composed and recorded the show’s theme, a song called Acoustic Routes. I am also thankful to Bytemark, who are providing the hosting for this episode’s audio file.

But above all, I’d like to thank the following people, who have supported this episode through Patreon or PayPal and thus keep this show on the air: Georges, Butterbeans, Niall Donegan, Michael Mullan-Jensen, Jonathan M. Hethey, Dave, Steve Hoos, Mark Holland, Shelby Cruver, Vlad, Jackie Plage, 1i11g, Philip Klostermann, Jaroslav Lichtblau, Kai Siers, ikn, Fadi Mansour, Dirk Dede, Michael Small, Joe Poser, Matt Jelliman, Bennett Piater, David Potter, Mika, Martin, Larry Glock, Dave Umrysh, RikyM, drivezero, MrAmish, Jonathan Edwards, Barry Williams, tobias, avis, Neil, Matt, Captain Egghead, Sandman616 and D.

Many thanks to my Twitch subscribers as well: Mike_TheDane, Galteran, indiegameiacs, m0dese7en_is_unavailable and Sandman616.

End of show song: I’m Exactly Where I Need to Be by Vincent Vega.

End of show music and some sound effects licensed via Epidemic Sound.