Validation
After the system finds a network and training method that provides a high performance value, I will perform a modified Jackknife validation process. In the standard Jackknife validation process, the testing set is divided into M parts. Tests are then run to determine how many of these M parts the network needs to see during training in order to generalize correctly for the complete set. If we call this number N, then we train with N/M of the testing set and test on the other (M-N)/M of it. The process is repeated M-choose-N times, each time using different pieces of the set for training and for testing.

The process of Jackknife validation
When validating a network, I will use a value of M = 4. How many of these fourths to use for training (the value of N) will be based on how much data was used during the experiment, as determined by the corresponding genotype; The amount of data per sentence group used for a particular network during evolution will be converted to the next highest fourth, and then that number will be used for validation training. Jackknife validation will then occur by testing the network under all possible combinations of these fourths of each group. This will give, as a maximum, 216 test runs (4-choose-N has its maximum value of six when N=2; 63=216). After these test runs are performed, I will perform up to six final validation runs. These six runs will be done by choosing fourths of the testing set as a whole, instead of choosing fourths from the three individual groups. By performing these final runs I will be able to judge how important it was to have the data divided into three difficulty groups for this particular network.

For example, to validate a network that was trained with 2/4 of the first group of sentences, 1/4 of the second group, and 3/4 of the third group, I will run a test that uses the first 2/4 of the first group, the first 1/4 of the second, and the first 3/4 of the third. Then the test is repeated, this time using a different 2/4 of the first group, and leaving the 1/4 of the second group and the 3/4 of the third group the same. When all possible 2/4's of the first group have been used, I will use a different 1/4 of the second group, use the first 2/4 from the first group and the first 3/4 of the third group again. This process continues until all combinations have been tested. The last tests are performed, in this example, by using a training set that consists of 2/4 (= 2/4 + 1/4 + 3/4) of the total data, selecting it in as many ways as possible.
Back to the Table of Content

Back to the previous subject

To the next subject