CCTV and imaging technology
Back to i-LIDSSystem Evaluation
Since 2007 HOSDB has been running regular i-LIDS based VBDS trials, evaluating systems for Government use. Manufacturers meeting the highest level of performance classification will be entitled to use the trademarked i-LIDS logo in their trade literature.
Video-Based Detection System evaluation trials
The evaluation schedule for the 2008-2009 financial year is:
-
Sterile Zone – June 2008
-
Abandoned Baggage –August 2008
-
Parked Vehicle – October 2008
-
Doorway Surveillance – January 2009
Reminder notices will be issued to owners of i-LIDS datasets, indicating application submission deadlines for forthcoming evaluations. The application submission deadlines for the 2008-9 financial year are as follows:
-
Sterile Zone - Tues 10 June 2008
-
Abandoned Baggage – Tues 19 August 2008
-
Parked Vehicle – Weds 22 October 2008
-
Doorway Surveillance – Weds 7 January 2009
Manufacturers of systems that have been selected for evaluation will be asked to sign an evaluation agreement that contains the terms and conditions for the evaluation. Copies of the agreement can be requested using the i-LIDS email address.
System role and performance criteria
VBDS may be submitted to HOSDB for evaluation in any scenario, and for either an ‘Operational Alert’ or ‘Event Recording’ role. In the former role, systems provide real-time detection of suspicious events which must be dealt with by a human controller. In the latter, systems act as a trigger for recording of suspicious events, where all the recordings obtained are to be analysed at a later time.
HOSDB assess the performance of VBDS using a combination of the detection and false alarm rates known as the F1 measure, an established criterion defined in the user documentation with each dataset. The F1 value is dependent on a parameter called the recall bias which determines the relative influence of detection rate and false alarm rate on the outcome.
A higher value of recall bias is used to assess systems for the ‘Event Recording’ role since in this role detection rate is comparatively more important. A lower value of recall bias is used to assess systems for the 'Operational Alert' role since in this role false alarms are a more significant problem.
System Classification
The System will achieve one of the following classifications:
- Operational Alert (Primary) - Recommended as primary detection system in the operational alert role for parked vehicle detection applications.
- Operational Alert (Secondary) - Recommended as secondary detection system in the operational alert role for parked vehicle detection applications.
- Approaching Practical Recommendation – System demonstrates performance within a modest range of that required for classification as a secondary detection system for parked vehicle detection applications.
- Event Based Recording – System recommended for use in the event based recording role for parked vehicle detection applications.
- No Classification.
Systems which achieve the Operational Alert (Primary) classification will also be given permission to use the i-LIDS logo The logo is scenario, company, system type and version specific. The evaluation date refers to the financial year in which the system passed.
Primary detection systems are recommended for sole operation as a method of detection for operational alert applications.
Secondary detection systems are recommended for use in combination with a supplementary method of detection for operational alert applications.
Event based recording systems are recommended as a basis for triggered event recording where captured footage is to be reviewed later.
Of the systems evaluated, only those gaining the Primary and Secondary Operational Alert and Event Based Recording classifications will be listed in the Government Catalogue of Security Equipment.
Submission and evaluation process
The scenario recall bias values for both ‘Operational Alert’ and ‘Event Recording’ roles are published with each dataset. This enables manufacturers to set up their systems to maximise the F1 performance measure in whichever role they intend to apply for HOSDB evaluation.
Manufacturers wishing to submit a VBDS for a forthcoming scenario trial should complete the evaluation application with:
- the details of their system
- its performance on the test dataset in the scenario
- role for which they wish to be evaluated.
The form should then be mailed or faxed to HOSDB.
After the deadline for application submissions, HOSDB will ask the manufacturers of selected systems to agree to a full set of terms governing the trial. These terms require the manufacturer to provide their system on free loan for the duration of the trial. Systems should be provided optimised ready for the scenario and role in question, with one channel set up to monitor each camera view (stage.) Provision must be made for HOSDB to irretrievably delete any video recorded by the systems during the trials.
Systems demonstrating a sufficient overall F1 value across HOSDB's private evaluation dataset, with no significant weakness in any particular subgroup of imagery (certain stages, weather conditions etc.) will be recommended for Government use in that scenario and role, subject to the terms of the trial.
After the trials, HOSDB will provide the manufacturers with feedback on the performance of their system(s), any weaknesses found and whether they are to be recommended for Government use.
Multiple Camera Tracking Evaluation
Evaluations for Multiple Camera Tracking will begin in 2009. Specific dates will be announced this year and will be available on the website. More information on the evaluation procedure can be found in the User Guide.
Further details of the evaluation process are given in the pdf user documentation supplied with every i-LIDS dataset.
If you have any queries about the evaluation process, please contact:
The i-LIDS team
Home Office Scientific Development Branch
Langhurst House
Langhurstwood Road
Horsham
West Sussex. RH12 4WX.
Email: i-LIDS@homeoffice.gsi.gov.uk
Voicemail: (+44) (0)1403 213823
Fax: (+44) (0)1403 213827 - please mark ‘FAO: i-LIDS team’
