CCTV and imaging technology
System Evaluation
Since 2007 HOSDB has been running regular i-LIDS based VBDS trials, evaluating systems for Government use. Manufacturers meeting the highest level of performance classification will be entitled to use the trademarked i-LIDS logo in their trade literature.
VBDS may be submitted for evaluation in any scenario, and for either an 'Operational Alert' or 'Event Recording' role. In the 'Operational Alert' role systems provide real-time detection of of suspicious events to be dealt with by a human controller. In the latter systems act as a trigger for recording suspicious events and the recordings are analysed later.
We assess the VBDS's performance using a combination of the detection and false alarm rate. This is known as the F1 measure and its value depends on a parameter known as the recall bias. The recall bias determines the relative influence of detection rates and false alarm rates on the outcome.
Systems for the 'Event Recording' role are assessed using a higher value of recall bias as the detection rate is more important. For the 'Operational Alert' role false alarms are a more significant problem and so a lower value of recall bias is used.
Systems which have been evaluated will be awarded one of five possible classifications.
Evaluation application dates
The evaluation schedule until 2010 is:
-
Abandoned Baggage - September 2009
-
Parked Vehicle - November 2009
-
Doorway Surveillance - February 2010
Reminder notices will be issued to owners of i-LIDS datasets, indicating application submission deadlines for forthcoming evaluations. The application submission deadlines for the 2009-10 financial year are as follows:
-
Abandoned Baggage - 1 September 2009
-
Parked Vehicle - 22 October 2009
-
Doorway Surveillance - 7 January 2010
Manufacturers of systems that have been selected for evaluation will be asked to sign an evaluation agreement containing the terms and conditions for the evaluation. For a copy of the agreement email i-LIDS.
Submission process
Manufacturers wishing to submit a VBDS for a forthcoming scenario trial should complete the evaluation application with:
- the details of their system
- its performance on the test dataset in the scenario
- role for which they wish to be evaluated.
The form should then be mailed or faxed to HOSDB.
After the deadline for application submissions, HOSDB will ask the manufacturers of selected systems to agree to a full set of terms governing the trial. These terms require the manufacturer to provide their system on free loan for the duration of the trial. Systems should be provided optimised ready for the scenario and role in question, with one channel set up to monitor each camera view (stage.) Provision must be made for HOSDB to irretrievably delete any video recorded by the systems during the trials.
Systems demonstrating a sufficient overall F1 value across HOSDB's private evaluation dataset, with no significant weakness in any particular subgroup of imagery (certain stages, weather conditions etc.) will be recommended for Government use in that scenario and role, subject to the terms of the trial.
After the trials, HOSDB will provide the manufacturers with feedback on the performance of their system(s), any weaknesses found and whether they are to be recommended for Government use.
If you have any queries about the evaluation process, please contact:
The i-LIDS team
Home Office Scientific Development Branch
Langhurst House
Langhurstwood Road
Horsham
West Sussex. RH12 4WX.
Email: i-LIDS@homeoffice.gsi.gov.uk
Voicemail: (+44) (0)1403 213823
Fax: (+44) (0)1403 213827 - please mark ‘FAO: i-LIDS team’
