Presentation

Brian Bailey
Email: [email protected]
Tel: 503 632 7448
Cell: 503 753 6040
Web: brianbailey.us
Copyright © 2008 Brian Bailey Consulting
Verification is all about answering the question:
Have we implemented something correctly ?
Copyright © 2008 Brian Bailey Consulting
!"
#
$
%&
'
(
'
)
Copyright © 2008 Brian Bailey Consulting
Propagation success
Bug
Checker
Stimulus
1. Activate
2. Propagate
3. Detect
Design
Propagation failure
'
*#
#
$
%
$
$
&
+
Copyright © 2008 Brian Bailey Consulting
$
)
%
(
)
&
, &
.
.
/
0
(
(
1
)
2
3
(
&
Copyright © 2008 Brian Bailey Consulting
(
4
5
6
7
(
1
(
Copyright © 2008 Brian Bailey Consulting
(
)
(
)
9
9
9
(
)
(
Copyright © 2008 Brian Bailey Consulting
8
$
Propagation success
Bug
Checker
Stimulus
1. Activate
Design
Propagation failure
$
#
! "
"
"!
!
! "
!
%
&
!
#
#
!
!
!
#
Copyright © 2008 Brian Bailey Consulting
7
(
#
4
:
.
#
'+1
7
Copyright © 2008 Brian Bailey Consulting
)
'
!
Propagation success
Bug
Checker
Stimulus
1. Activate
2. Propagate
Design
Propagation failure
$! "
## "
! !
"
! ! !#
!
Copyright © 2008 Brian Bailey Consulting
!
(
.
)
"
;
:
)
)
9
<
>
>
)
)
(
)
(
)
?
5
=
)
)
Copyright © 2008 Brian Bailey Consulting
#
9
9
9
#
)
Copyright © 2008 Brian Bailey Consulting
#
:
&
:
@
> )
#
9
)
)
9
%&
Copyright © 2008 Brian Bailey Consulting
9
7
)
9
/
5
2
3
(
:
(%
9 )
,
%
Copyright © 2008 Brian Bailey Consulting
0
1
)
*
-)
9
:
8
-:4
.
.
.
-)
)
:
?
Copyright © 2008 Brian Bailey Consulting
$
!
#
+
5
9
4)
4)
#
4)
A
4)
:
7
4)
4)
#
-
A
-
1
(
4)
4)
#
1
-
9
)*
1
-
1
A
1
;B
>
1
;B
>
"
Copyright © 2008 Brian Bailey Consulting
-
1
:
'
7
9
+
%
+1
+
=
+
1
0
!
)
:
C
7
#
&
Copyright © 2008 Brian Bailey Consulting
>
*
)
)
)
(
%
))
)
)
Copyright © 2008 Brian Bailey Consulting
)
'
(
4 %
%=
*)
;
1
)
)
7
DEFD
DEGH
Copyright © 2008 Brian Bailey Consulting
*
Principles
Induce mutations (or faults) in the HDL code and
show that the verification suite detects those mutations
by executing testcases
Original program code:
a = b or c;
Faulty program code:
a = b and c;
Copyright © 2008 Brian Bailey Consulting
+
'
5
'
'
&
'
&
I)
B&=
5
)
J)
JK&KK
LK
)
/
0
Copyright © 2008 Brian Bailey Consulting
)
)
<
7
9
==
#
5
$
%
$
C
#
$
&
$
)
(
#
)
1
% )
)
Copyright © 2008 Brian Bailey Consulting
$
(
-
@
;
&
;
#
;
+
M
Copyright © 2008 Brian Bailey Consulting
,
5
*
M
%
:
)
?
9
)
Copyright © 2008 Brian Bailey Consulting
)
Qualification Steps
Model phase
Activate phase
Report
Static analysis
of the design
Report
Analysis of the
verification
environment
behavior
Iterate if needed
Detect phase
© Certess, Inc. – 2008
Report
Measure the
ability of the
verification
environment to
detect mutations
,
1
7
9
7
)
:
)
7
7
)
Copyright © 2008 Brian Bailey Consulting
,
"
(
)
*
D #
5
N '
!
O #
, )
%
P
F #
)
%
G #
E '
)
E
O
%
Copyright © 2008 Brian Bailey Consulting
F
(
-+.
/
0
Q
0
120
1 0
# 0
,R
K DR
<<
"
/
/
/
S/
4
5/
3#
&& 67#63#
" 8& 67#94
8& 67#94
!
#
QG
QGK DR
%0
R
%/
&
D&
QGK HR
<<
Q
Q K DR
S/
%0
R
%/
&
D&
Q K HR
R
34
"0
*
12 && 67#63#
" 8& 67#64
!
4
4
<299:9=
1 && 67#63#
8& 67#64
K
4
0
R
0
R
;/
4
#
&
/
#
" 3:6;#94
/
>
A#?0
>
%&
/>
0
!4
3#
0? @
0
34
:@
Copyright © 2008 Brian Bailey Consulting
A
# A# #
)0*DTHR
-
Copyright © 2008 Brian Bailey Consulting
!
#
#
!
(
Q
<<
*
,R
K DR
QG
QGK DR
S/
%0
R
%/
&
D&
QGK HR
<<
)
)
<<
<< =J
K DR
S/
%0
R
K HR
<< L=
<<
Q
Q K DR
S/
%0
R
%/
&
D&
Q K HR
R
0
R
*
! -+.
&
#
/
" B
3:67#94
0
R
!
/
>
A#?0
!
%&&
/>
0
3#
0? @
0
34
:@
Copyright © 2008 Brian Bailey Consulting
A
# A# #
Global Metric
Representing the overall quality of the Verification Environment
ST reference : 75%, but usually higher
Activation Score
Measures the ability of the test suite to exercise all the RTL of the
IP
Similar to code coverage
ST reference : 95%, & 100% explained
Missing % should deeply studied & fixed or explained
Propagation Score
Measures the ability of the test suite to propagate mutations to the
outputs of the IP
ST reference : 80%, but should probably be enhanced by adding
more test scenarios to reach 90%
Detection Score
Measures the ability of the environment to catch errors
ST reference : 90%, but usually higher
DAC’2008 - Anaheim
:
#
7
•
Case study 1:
•
•
•
•
Activation Score (A/F)
Application: 3rd party IP
HDL Directed Environment
~300 tests, 30 minutes
Code Coverage ~100%
Detection Score (D/P)
•
IP
Activation Score (A/F)
Propagation Score (P/A)
Global Metric (D/F)
Detection Score (D/P)
95%
90%
85%
80%
75%
70%
65%
60%
ST Ref ST Avg 3rd Party IP
95%
97%
97%
80%
90%
80%
75%
80%
66%
90%
93%
85%
Propagation Score (P/A)
Challenges
•
•
Convince 3rd Party IP provider
High revenue, high visibility chip;
reduce respin risk
ST Ref
ST Avg
3rd Party IP
Global Metric (D/F)
•
Results
•
•
Helped us to push IP provider to improve verification environment
• and monitor progress
Low detection score highlighted manual waveform checks
DAC’2008 - Anaheim
:
#
7
!
(
#
(
%
#
51
%
9
Copyright © 2008 Brian Bailey Consulting
-
C
7
)
#
'+#
)
)
,
.
:
4
)
5
%
Copyright © 2008 Brian Bailey Consulting
+!
D
,
*
Q
S
( * OHNPN F!!G
* OHNFONPH!H
. *