Industry Leaders Panel: Did We Create the Verification Gap? - Part 6
Industry Leaders Panel: Did We Create the Verification Gap? - Part 6
(Jeff): My name is (Jeff) from Verilab. So my question is regarding something that Harry had brought up earlier which was, other than tools, one of the biggest areas of problems is that you get a design spec that gets perhaps misinterpreted or misunderstood, and people are doing – or I think it might have been even Jim that said it, were doing tons of great work on the wrong thing. Right?
Male #2: I've seen it before.
(Jeff): I've seen it, too. And so the question comes down to – and this is actually what Harry brought up – is that you end up – sometimes you end up in situations where there are these (sounds like: super stars), and I'm going to call them design spec (sounds like: super stars), which create great design specs that verifiers can work with. But you're not going to find them all the time. Is there any type of perhaps technological solution to this problem that you can perhaps identify?
Bill Grundmann: So the last thing I need is another tool to be honest. What I've seen effective is getting the system guys together like we talked about earlier with the software guys, but get the system guys involved with the hardware guys, with the software guys, together, up front – not only up front, but regularly throughout the development of the product. A lot of times, I mean, you know, it sounds silly, but put them in a room together with a white board and some markers and a cup of coffee. I've seen that be really effective.
Mike Stellfox: That's the same exact thing I've seen. That's the most effective thing where, let's face it, a lot of places there's not really good specs, especially in consumer where, you know, the market's moving really quickly. But a lot of times a good verification plan is one of the best specs that you can see. And a good verification plan is something that's actually captured where you bring in those stakeholders and you actually work out a lot of those misassumptions that people might get just from reading a document or something is being left unspecified in making their own interpretation. Every single verification planning meeting I've ever been in with a customer where we bring in, you know, like a software architect designer verifier, you find always tons of misinterpretations of what it is we're actually trying to build and then verify. So spending the time up front, you know – I've talked to a lot of customers who say, "Oh, I don't have time, you know, for that." Right? "We're already stretched," you know. You know, "We're too busy chopping wood. Don't give me a chainsaw, you know. I don't need that right now." Well, the problem is, it's like anything, you know, it's nothing special to verification. But if you don't have a good plan that is really based on the assumptions of what the design intent is, then you could be spending lots of cycles verifying things that really are irrelevant or wrong.
Jim Caravella: And it's never a static plan. Right? I mean, that's another key, is it's got to be dynamic…
Mike Stellfox: Evolving.
Jim Caravella: …as…
Male #3: I definitely agree, but I want to tag on I think – because you were asking is there any type of tool or something to help, you know. And I think I've said this on every (inaudible) panel I've ever been, there's no substitute for thinking. And this…
Jim Caravella: Do you have a tool for that?
Male #3: But this goes to your point. You know, there's nothing better than processing, getting guys sitting down and discussing this thing, you know. So…
Male #1: (JL)?
Male #8: So I had a question for you. I think you probably would all agree that no verification engineer has ever introduced a bug in a chip…
…right? But verification engineers deal with a lot of pain and anguish to find these bugs so that the designers can be happy and focused on their work. So my question is – it's a good thing it's a panel discussion – so the question is, have we made verification too easy? Are the designers not feeling the pain of the problems that they're causing the verification team enough? Should they feel more pain? Should we wire them with electrodes or something like that?
So I guess as I – maybe start with Jim or Bill, I don't know. Do you think – I mean, should the designers be more impacted by the verification? Do you think that would help them learn what to do, how to design better for verification?
Jim Caravella: That's an interesting one. What I have seen a lot of is – and I've had to deal with in managing teams – is a lot of times the designers get really pissed off because, "You found a bug in my design and, you know, how can that be?" So there is kind of a feedback mechanism. Maybe it's not electrical shocks, but I've seen where, you know, the designers take it personally. And, actually, I think that's a healthy thing. I don't know if we need electrodes, but – yes, I'm still trying to think if I've ever seen where a verification has introduced a bug. I'm going to (inaudible)…
Male #3: I actually know a story…
Jim Caravella: Okay.
Male #3: No, this goes back to the – it was actually partially my fault. This was back before we had assertion languages. I developed this mechanism (inaudible) assertions and design. Unfortunately, it synthesized into some – in the actual design and it created (inaudible)
Mike Stellfox: So, Janick, it was actually Harry who introduced the (inaudible)…
No, actually, what I've seen interestingly is, a lot of times, like ten years ago, the biggest gap I was seeing was the lack of verification culture. And I – in companies, in other words, not really fostering verification as a career, having people that are really experts in that. And I – and unfortunately there's still some of that out there, but it's gotten a lot better. But it interestingly is, in those companies that really were doing really good verification, they had a separate verifiers, you know, kind of doing it by the book and really doing the best approaches and verification. A lot of times – sometimes you would see kind of a wall forming where the designers were just throwing things over the wall. They were doing very little verification of their blocks and then throwing it to these guys who were these, you know, verifiers, expert guys building these environments. But the fact is, nobody knows the design better than the person who captured and implemented it. And there is a set of bugs, you know, especially micro architectural things, that the designer is much better to find those bugs.
And so you know – and I know of some specific companies that that caused an artificial gap and they went back and corrected that by putting some more verification burden on the designers. You know, for example, you know, using assertion-based verification very early doing some formal or some automated, you know, (sounds like: lenting) type of formal approaches and so on.
So I have seen that occur, and that's definitely one place where, yes, trying to, you know, kind of go by the book can kind of artificially steer you off into the wrong direction.
Male #2: And, come on, let's be honest, half the bugs are in our own test bench. So we're no better than the design guys. It was that they – we keep it to ourselves. Right?
Mike Stellfox: You don't log those bugs.
Jim Caravella: Not reportable, yes.
Male #8: Thanks.