The following is a chart describing the flow of processing in the So algorithm, starting with the epoch process, since higher levels do not interact with the details of particular algorithms:
EpochProcess: { Init: { environment->InitEvents(); // init events (if dynamic) event_list.Add() 0 to environment->EventCount(); // get list of events if(order == PERMUTE) event_list.Permute(); // permute if necessary GetCurEvent(); // get pointer to current event } Loop (trial): { // loop over trials SoTrial: { // trial process (one event) Init: { // at start of trial cur_event = epoch_proc->cur_event; // get cur event from epoch } Loop (once): { // only process this once per trial network->InitExterns(); // init external inputs to units cur_event->ApplyPatterns(network); // apply patterns to network Compute_Act(): { // compute the activations network->layers: { // loop over layers layer->Compute_Net(); // compute net inputs layer->Compute_Act(); // compute activations from net in } } network->Compute_dWt(); // compute weight changes from acts } } if(wt_update == ON_LINE or wt_update == SMALL_BATCH and trial.val % batch_n) network->UpdateWeights(); // after trial, update weights if necc GetCurEvent(); // get next event } Final: if(wt_update == BATCH) network->UpdateWeights(); // batch weight updates }
The layer->Compute_Act()
function has several sub-stages for
different versions of algorithms, as detailed below:
For non-input layer hard competitive learning units:
ClLayerSpec::Compute_Act() { SoUnit* win_u = FindWinner(lay); float lvcnt = (float)lay->units.leaves; lay->avg_act = // compute avg act assuming one winner and rest losers.. (act_range.max / lvcnt) + (((lvcnt - 1) * act_range.min) / lvcnt); win_u->act = act_range.max; // winning unit gets max value win_u->act_i = lay->avg_act; // and average value goes in _i }
For non-input layer soft competitive learning units:
SoftClLayerSpec::Compute_Act() { float sum = 0; for(units) { // iterate over the units unit->Compute_Act(); // compute based on netin unit->act = exp(softmax_gain * unit->act); // then exponential sum += unit->act; // collect sum } for(units) { // then make a second pass unit->act = // act is now normalized by sum act_range.min + act_range.range * (unit->act / sum); } Compute_AvgAct(); // then compute average act over layer }
The code for the SOM case is more complicated than the description, which is just that it finds the winner and pastes the kernel onto the units surrounding the winner.