Currently if you have a file open with errors, then shut it, the errors persist. The reason is that we don't have any sophisticated removal of non-interesting files, unlike what we have with DAML.