2024-04-08-minutes-public-works-study-session

Home Browse Search

Appendix B After receiving data sets from each government, we conducted preliminary assessments on their reliability. These tests included checking to ensure that the number of records matched what we were told were sent and comparing the totals to other information sources. However, few governments had previously published this data and so this resulted in inconclusive tests to verify the completeness of the data. Although we could not verify the full completeness of the data, we believe we have sufficient data to perform our analysis. We also checked for appropriate date ranges, missing data fields and illogical results. We found in most data sets there were some records with dates in an illogical order, so we excluded those records from the final analyses. We also followed up with government staff as necessary to clarify any unusual data observations to determine if they were caused by errors in the data. Next, we conducted detailed accuracy testing on a sample of permit records. We specifically tested key dates in the data - such as the start of review, final decision and when applications were sent back for revisions, when available - to determine if the data matched underlying records. We designed our tests to give us a 95 percent confidence that the error rate for the tested dates was within 5 percent. The tests went as follows: 1. Randomly selected 16 land use permit records and 16 building and civil permit records 2. Compared the key dates in each record to underlying documentation, such as stamped applications, notices of final decision and communications between government staff and permit applicants 3. If any of the records had dates that were more than four days different between the data and supporting documents, we randomly sampled approximately 16 additional records to review We determined that the data was sufficiently reliable to determine whether permits were reviewed within 120 days. However, we determined the data was not reliable for our original audit purpose of reporting the average number of days that permit review takes. We made our assessment based on the combined results of all analyses described above. Two key issues were: 1. Inconclusive completeness tests for most data sets because few governments had previously published reports of the data 2. Inconclusive or failed accuracy tests for some data sets, meaning we found differences of more than four days When we determined the data was not reliable for our original purpose, we used an alternative approach. We specifically analyzed permit review times in several ranges as further described below. The difference in the determination for this alternative analysis was that in the previous analysis, we considered records a match if they were within four days. In the new analysis, records are a match if they are within the same range of time (that is, 0-60 days, 61-120 days, etc.). We designed our alternative tests to give us a 95 percent confidence that the error rate for the tested dates was within 5 percent (with one exception, see Limitation: Richland). We determined the data was reliable for this alternative purpose because it relies on a lower accuracy of data (ranges of time versus matches within four days). Despite shortcomings of the data, we expect there is a high level of interest for the data in the final report and we believe we have sufficient data to perform our analysis. In addition, our overall data reliability assessment includes a limitation for one government, and an exception for another government. Growth Management Act Appendix B 1 52