LargeLanguage Model-Based Extraction of Logic Rules from Technical Standards for Automatic Compliance Checking

LargeLanguage Model-Based Extraction of Logic Rules from Technical Standards for Automatic Compliance Checking

automatic compliance checking;logic rules;technical standards;large language model;prompting

n this research, we design logic rules as a representation of technical standards documentsrelated to ship design,which will be used in automatic compliance checking. We present a novel design of logic rulesbased on a general pattern of technical standards’ clausesthat can be produced automatically from text using a large language model (LLM).We also present a method to extract said logic rules fromtext.First, we design data structuresto represent the technical standards and logic rules used to process the data. Second, the representation of technical standardsis produced manually and tested to ensure that it can give the same conclusionas human judgmentregarding compliance. Third, a variation of prompting methods, namely pipeline method and few-shot prompting, isgiven to LLM to instruct it to extract logic rules from text following the design. Evaluation against the logic rules produced shows that the pipelinemethod gives an accuracy score of 0.57, a precision of 0.49,and a recall of 0.62. On the other hand, logic rules extracted using few-shot prompting havean accuracy score of 0.33, precision of 0.43,and recall of 0.5. Theseresults show that LLM is able to extract a logic rulerepresentation of technical standards. Furthermore, the representation resultingfromtheprompting technique that utilizes the pipelinemethodhas a better performance compared to the representation resultingfrom few-shot prompting.

Rizky Nugroho1*, Adila Krisnadhi2, Ari Saptawijaya3

https://jurnal.iaii.or.id/index.php/RESTI/article/view/6285/1045

Faculty of Computer Science, Universitas Indonesia, Depok, Indonesia

17-04-2025

FAJAR BAGUS W

PDF

ENGLISH

TEXT