What is the difference between an FNN and an SFNN?
An SFNN is a Sparse FNN, which makes it somewhat cheaper and much more scalable than an FNN. Fundamentally, an FNN is a network that ensures all possible pairs of nodes can communicate with single-switch latency. An SFNN only ensures single-switch latency for a specific set of node pairs. The set is generally specified as all pairings that occur in any of a given set of communication patterns. The interesting thing is that, even taking the union of all the communication patterns commonly used in parallel processing, the number of node pairs doesn’t grow as fast as O(N*N) — so the network complexity increases relatively slowly. Node pairs that were not expected to communicate still can communicate with good bandwidth in an SFNN, but they might see higher latency.
