3.8 KiB
Requirements (Ordered by importance)
1. The program should work with Unity.
The purpose of this program is to allow Unity projects to interpret code within the ShintenScript language. If the program cannot integrate with Unity, it is unable to do anything.
Testing
Unity provides the Unity Test Framework package, which allows for testing programs in the Unity environment. This can be used to ensure the program maintains its intended behaviour while used in Unity.
Unity integration should be tested as early as possible as the code is being developed, as part of the Create phase.
2. The program should not access data outside of what it is given.
The input to the software may not be from a trusted source. Ensuring malicious input cannot cause bad behaviour is important, as a malicious acter could cause damage to systems running the software.
Testing
C# is a memory safe language (at least the subset used by the project is) therefore ensuring memory-based exploits cannot be used is not required. Ensuring that no other exploits exist can be done by avoiding any unsafe APIs. While tests could be used to check for the presence of these, the nature of the program should not require any, and violations should be caught in review.
Code review should eb performed during pushing/merging code, which happens in the Create phase.
3. The program should be parsed and executed correctly with respect to the language specification.
Writing code is hard enough without the interpreter containing bugs. Ensuring the program behaves as expected is required for it to be a useful way to reresent behaviours.
Testing
Programming languages are complicated, and valid programs is a large and complex input space. Unit testing for lexing, parsing, and evaluating every type of operation should be used to ensure they all behave correctly. There is still the posibillity that some cases are missed here, so ensuring full coverage is important. Larger programs can also be tested, to ensoure that the various operations can be composited properly.
Due to the importance of these tests, they should be created and performed alongside the development of the code, as part of the Create phase.
4. Once parsed, code should execute fast enough to be used in a real time application.
Unity is a real time program, code may need to be run each frame, and finish before the next one begins. While arbitrary programs cannot be made to run in finite time, simple programs and algorithms should run within a short timeframe, even in the worst case. If this fails, the application user may experience lag.
Testing
Instrumenting generated code is difficult, and also may vastly alter how the code executes. This makes instrumentation based profiling difficult, as it becomes not representative of how the code would run on actual production systems. As such, timer based benchmarking is likely the best approach. Many executions should be run in order to get a large sample of times. Ideally, there should be tests run in multiple different environments, such as different hardware (Different ranges of PCs, mobile devices, consoles, etc.) and execution backends (.NET, Mono, il2cpp, etc.)
Performing sensible benchmarks requires the codebase to be near done, and require data of what users program, so these are best performed in the Verify phase.
Scaffolding
The program consists of mostly pure functions, which do not require scaffolding for testing. Larger subsystems may require it, as the software runs in layers (passes). For example, providing a scaffold lexer should be used to create parser tests that are lexer independent. Likewise, ensuring ASTs can be geenrated without a parser is required for testing code evaluation without invoking the lexer or parser.