Finding a Needle in a Haystack: A Novel Log Analysis Method with Test Clustering in Distributed System

As the complexity and number of gates in SOC increase, the cost for verification also increases [1][2]. In such a trend, verification engineers are struggling with boring and repetitive debugging process, such as debugging similar tests with same errors in very limited resources, i.e., time and licenses. To reduce such efforts, similarity-based studies were conducted in both the software testing industry [3][4][5][6] and design verification industry [7]. In this paper, we propose a novel approach of test clustering which uses components of simulation logs. The proposed methods are based on automated regression system suggested in our previous study [8], and the system can run more efficiently in three ways: sampling representative tests from each cluster generated by our proposed algorithm, finding related tests with the same errors and automated pipelines in distributed system. By sampling tests from clusters, all tests are prioritized and 30~40% of them are performed as representative tests. As this method detects distinguished errors in the earlier stage of the verification process, this results in the efficient use of resources. By finding related tests, verification engineers can debug once without repetitive works on similar tests and run related tests automatically. Furthermore, to speed up handling millions of data in a limited period, we propose a distributed architecture for machine learning in design verification.

Jin Choi, Samsung Electronics Co., Ltd.
Sangwoo Noh, Samsung Electronics Co., Ltd.
Sooncheol Hong, Samsung Electronics Co., Ltd.
Hanna Jang, Samsung Electronics Co., Ltd.
Seonhee Yim, Samsung Electronics Co., Ltd.
Seonil Brian Choi, Samsung Electronics Co., Ltd.