The License Plate Recognition System
My senior deisgn project for ECE 4006 was a group project consisting of a
Signal Processing problem of our choosing.
My group members were Devin Porter, Chike Lindsay-Ajudua, Leo Dachevski, Muhammad Raza.
I would also like to take the time the praise Leo and thank him for his hard work and
"above and beyond" attitude in getting the project done while we were working together for hours in the CoC.
Back to the topic at hand though, we began by deciding on our project by choosing from several options
which we had compiled into a list with pros and cons for each idea. we decided on a system for
recognizing license plates for several reasons:
It had not been done before by any previous ECE 4006 project group
It was challenging enough, but still possible to complete in the time provided
It just sounded like something very cool
After consulting a few professors who specialized in image processing, we decided on a method where
the license plate would be first converted to binary (black and white) according the some threshold.
Second, the picture would be processed into two signals, one horizontal and one vertical. Third,
The signals would be stretched, clipped according to smart algorithms, and analyzed in order to find
the location of the license plate in the image.
Lately, the signals would be compared using cross correlation to find matches between license plates.
using cross correlation. Cross correlation will be described later, since I would like to now
go through the whole process in an example. Please also note that code is available to test our system
as well, although it is very limited it only exists to prove the system is feasible, but could
easily be modifiable and extended to more robust solutions.
Also, you can download the full source code for the system including testing images at the bottom
of this page.
Getting the Pictures of the cars
Several compromised had to be made on retrieving the pictures of the cars. First, it was assumed
that the license plates would be standard Georgia state license plates, or at least a plate that
was dark text on a light colored background. Second, it was assumed that either a light was being
shown on the back of the car or that a flash was used when taking the pictures. The light used
illuminated the license plate's reflective surface making the job of finding the plate within the
picture much easier, especially on cars with a white paint job.
These compromises were designed to make the project possible in the given
amount of time due to the complexity of the problem and the lack of any previous project
similar enough to simply improve upon.
Examples of a plate without a light or flash, and a plate with light or flash are given below. Note the
drastic difference in the pictures. Also, please note that both pictures were taken during the
same time of day, while it was daylight outside. Although our system would work with less drastic
differences in lightness between the plate and car, for the purpose of our project this advantage
was utilized mainly to ensure that the a working example could be demonstrated above and beyond
any other goals. Also included below is a picture of a binarized image.
Please note that this image is never actually generated
in order to do further analysis, it is just a visual. In reality, the image is turned into two
vectors directly, using a threshold calculated by the original image. More of this will be discussed in the
below though.
No Flash used
Flash Used
Binarized Image
Convert the Images to Binary
In order to better analyze the image, the images were binarized so that all pixels were cast to either
0 or 1 (black or white respectively). In order to binarize the image though, a calculation had to be
done first to know the threshold where pixels should be cast to either black or white. From the
sample pictures we had taken, a percentage of 95% black on the image seemed to fit well when
trying to show the plate as all white. Below is the code which took that 95%, and calculated
what level of color in the picture represented that 95% cutoff.
// get total number of pixels in the area we are analyzing
int total = (h_offset2 - h_offset) * (v_offset2 - v_offset);
// create vector, level it to all zero.
unsigned int tmp[256];
for( i=0; i <= 255; i++ )
tmp[i] = 0;
// make sure offsets make sense
if( h_offset2 < h_offset )
return 0;
if( v_offset2 < v_offset )
return 0;
// count shades into tmp
for( i=v_offset; i < v_offset2 ; i++ ){
for( j=h_offset; j < h_offset2 ; j++ ){
t = (unsigned int)input[i][j];
if(t>=0 && t<=255){
tmp[ (unsigned int)input[i][j] ] += 1;
}
}
}
// calculate the number of pixels to stop making black.
total = total * percent / 100;
// go through tmp, find best shade to make threshold at.
for( i=0; i <= 255; i++ ){
m = total;
total -= tmp[i];
if( total <= 0 ){
if(tmp[i] > m*1.5 && i>0)
return i-1;
else return i;
}
}
Vectorizing the image
Once the threshold is known, we can know which pixels should be white and which should be black.
Using this, we can then convert the image into two vectors. The first vector
simply counts the number of white pixels in each column (horizontal vector) and the second
vector counts the number of white pixels in each row (vertical vector).
Below is the code used to vectorize the image. Note that this looks at the original image, and decides
if each pixel should be black or white on the fly. It does not actually save a binarized version of
the image first. The binarized picture above is generated just for visual and testing purposes.
Sample vectors (taken from the same car pictures shown above) are displayed below. These were generated
for testing purposes, and show how for each white pixel in a given row or column, the vector height
raises by one pixel as well. Essentially, it just shows how the white pixels in the rows and columns
of the binarized image are added up to show a function.
This is the Horizontal Vector (sum of columns)
This is the Vertical Vector (sum of rows)
There are problems with these vectors though. Notice how some other spikes come up nearly as far as
the license plate spikes? Also, notice how some dips in the license plate spikes come far enough
to cause confusion, even farther below the top of spikes that are not within the license plate.
To account for this, the vectors are then filtered several ways. First, the signal is put through
an averager to smooth the signal and remove very dramatic changes in the vector. Second, the
vectors are put through a multiplier to help bring out the taller spikes for better analysis. Finally,
the vectors are put through a custom "despiker" filter which removes all spikes in the signal that are not
as wide as the smallest expected width of the license plate spike.
void averager(unsigned short* sig, int len, int num){
int j,i,start,stop,avg,tmp = 0;
int m[9] = {0,0,0,0,0,0,0,0,0};
// checking
if(num%2 == 0)
num += 1;
if(num > 9)
num = 9;
if(num < 0)
return;
// get the avg of the signal
for(i=0; i<len; i++){
avg += (int)sig[i];
}
avg /= len;
m[0] = avg;
m[1] = avg;
m[2] = avg;
m[3] = avg;
// start binarizing and vectorizing.
for( j=0; j < len ; j++ ){
m[0] = m[1];
m[1] = m[2];
m[2] = m[3];
m[3] = m[4];
m[4] = (int)sig[j];
if(j+1 < len) m[5] = (int)sig[j+1];
else m[5] = avg;
if(j+2 < len) m[6] = (int)sig[j+2];
else m[6] = avg;
if(j+3 < len) m[7] = (int)sig[j+3];
else m[6] = avg;
if(j+4 < len) m[8] = (int)sig[j+4];
else m[6] = avg;
// test threshold, add to signatues appropriatly.
start = 4 - (num - 1)/2;
stop = 4 + (num - 1)/2;
for(i=start; i<=stop; i++){
tmp += (int)m[i];
}
sig[j] = (unsigned short)(tmp/num);
tmp = 0;;
}
}
void despiker(unsigned short* sig, int len, int thresh, int diff){
int j = 0;
int i = 0;
int k = 0;
// first run, remove taller spikes
for( j=0; j < len ; j++ ){
if(sig[j] > thresh && j < len-1)
i++;
else if(i < diff){
for(k=j-i; k<j; k++){
sig[k] = 0;
}
i=0;
}
else i=0;
}
// second run, remove sides of spikes left by first run
thresh /= 4;
i=0;
for( j=0; j < len ; j++ ){
if(sig[j] > thresh && j < len-1)
i++;
else if(i < diff){
for(k=j-i; k<j; k++){
sig[k] = 0;
}
i=0;
}
else i=0;
}
}
void multiplier(unsigned short* sig, int len, int m){
int j = 0;
for( j=0; j < len ; j++ ){
sig[j] *= m;
}
}
The results of these filters can be seen in the images below. The images clearly show how some of the
original large spikes were rounded off, how the image was made taller, and how spikes to the sides of
the plate spikes were cut off. This leaves us the ability to look for the sides of the license
plate spike to know the boundaries of the license plate in the image.
This is the filtered Horizontal Vector
This is the filtered Vertical Vector
Turning the Plate into a Signature
Once the filtered vectors are generated, the sides of the license plate spike is found. These two
sides from both images represent the borders of the license plate within the original image.
Using these borders, the license plate by itself is turned into vertical and horizontal vectors
just like the original image was before. This time however, the vectors will become the actual
signature of the plate itself, and a different threshold is found since within the plate there is a
different expected ratio of black and white pixels. Please note that we did not use the already
calculated vector of the plate because it originally took into account all the background information
as well when generating the signal. To get the cleanest signature possible, we did a new vectorization
of only the plate to ensure that no background noise around the car or plate would compromise the
signature.
Once the license plate is vectorized, it is put through the same averager as the vector of the
orignal image. However, the plate vectors are not put through the despiker function or the
multiplier as these would only degrade the signature in unpredictable ways. Instead, the plate
vectors are put through a normalizer function which sets the average amplitude to a set position.
This must be done in order to compare the signatures later on, since it is the shape of the
vector signal and not the height of the signal that we are concerned about.
Shown below are the edges of the plate found by the functions, and the vectors of the plate itself after
being averaged and normalized.
The plate showing the calculated edges
Horizontal signature of the plate
Vertical signature of the plate
Comparing License Plate Signatures
Lets begin with a real world example; a parking garage deck. As you drive up to the gate, a
camera behind your care snaps a picture of the back of your car. The system them uses the methods
above to calculate the signature of your license plate, and then checks to see if that signature
exists in a known database of other license plate signatures. The database of signatures would
be of cars that are allowed to park in that parking deck, so if your plate's signature is found
in the database the parking deck's mechanical arm would lift up and allow you to drive into the deck.
How does the system check to see if your signature exists in the database? The problem is more
complicated that it sounds, since slight variations in light, position of the car, and a variety of
things I can't even think of could potentially effect the signal so that two signatures
from the same plate taken on two different days would probably not match exactly.
To overcome this problem, we used a cross correlation function to calculate if the signals matched
as a percentage. Not only that, but the cross correlation function has the added bonus of checking
the signal across different offsets, so that if a license plate signature was slightly off to one side
when compared to the same plate's signature in the database, the function would be able to account
for this. The greatest percentage of matching between the different offsets is used as the
true matching percentage of the signatures. The cross correlation function is shown below,
and returns the matching percentage.
double correlation(unsigned short *x, unsigned short *y, int n){
int i,j, delay;
double sx,sy,sx2,sy2,sxy,dx2,dy2,dxy,r,max=0;
unsigned short val;
for (delay=-_MAX_DELAY; delay<_MAX_DELAY; delay++){
// reset values
sx,sy,sx2,sy2,sxy = 0;
// Calculate the mean of the two series x[], y[]
for(i=0; i<n; i++){
j = i - delay;
if(j<0 || j>n)
val = 0;
else
val = y[j];
sx += (double)(x[i]);
sy += (double)(val);
sx2 += (double)(x[i] * x[i]);
sy2 += (double)(val * val);
sxy += (double)(x[i] * val);
}
// calculate the matching percentage between the two signals
dx2 = sx2 - ( (sx*sx) / ((double)n) );
dy2 = sy2 - ( (sy*sy) / ((double)n) );
dxy = sxy - ( (sx*sy) / ((double)n) );
r = dxy / sqrt( dx2 * dy2 );
// if this round matches more, use this match percentage
if(fabs(max) < fabs(r))
max = fabs(r);
}
// return the best matching percentage
return max;
}
Conclusions and Final Thoughts
For two different pictures of the same car, the matching
percentage between the two functions was typically above 95% during our testing, while matching
percentages between two different cars typically varied between 20% and 80%. It should also be
noted that the vertical signature would typically match much more between different plates that
the horizontal signature, and therefore the horizontal signature's match percentage was more
trustworthy and was made more influential when calculating if the two plates actually matched or not.
One of the questions we regularly got when describing our project to others was "why didn't you read
the characters on the plate and save those instead of the signature?" The answer is that character
recognition is an extremely difficult problem, and though we wanted the system to work that way
in the beginning, it was decided that we might not be able to get a working system up and running
in the time allotted for the project. Therefore, the simpler but nearly as effective
method described above was used with very promising results.
What could be done better? Right now, the code has several percentages and thresholds hard coded into
the program which were manually changed during testing for different photographs in order to accurately
find the edges of the license plate in those images. For instance, for a white car or when glare is
coming off the back window of the car there is a much greater amount of white in the picture which can
result in much greater differences in the binarization of the image. We did not get a chance to
write code to automatically calculate the thresholds and averages for our system, but doing so would
make the system much more robust.
Secondly, the plate signatures were normalized vertically (set with
the same average amplitude), but they were not normalized horizontally. This means that if pictures
are taken at different distances, the signals would be different lengths and would probably not match
anymore. For this reason, the pictures need to be very close to the same distance away, or
a stretching/shrinking function needs to be put into the system to normalize each signature to a set
length to account for such an issue. Other than that, I cannot think of anything off the top
of my head but I'm sure other drawbacks probably exist which I have not thought of.
Test it out yourself!
Compile the code on a Linux system by simply running "make", but you will need to have the
stdio.h, stdlib.h, memory.h, setjmp.h, and jpeglib.h libraries available on the system.
For Ubuntu Linux, this means having the libjpeg62-dev, glibc-headers, and build-essential
packages installed (in addition to the gcc package obviously). I cannot say what packages are
needed for other distros though, but a google search for your distro and the library files
mentioned above would probably come up with them.
Also note that there is no "make install" or "configure",
this is not an installation, just a demonstration. Not familiar with "make"? Just extract the files
into a directory on Linux, open a command line in that directory, and type the command "make", and
the program will compile (simple eh?). Once compiled, you run the file created named "test" by
typing "./test", and then the program will give you a menu system with options to choose from.
The "database" is actually just an array in memory and can only hold 10 signatures, but it gives a
good demonstration at least. There are also a lot of debug messages outputted so you can see
exactly what is going on.
There are some images included in the zip file (including the image shown above)
for use in testing. Also note that it will save output images in the same directory, which
show the different stages of the image and vectors. These are used for debug purposes so you can
see where things might be going wrong if it's not working for you. Lastly, please note that the
system expects a picture that is EXACTLY 600x800 pixels, so if you are testing with other images,
be sure to resize them to that size. That's it!
