what are the different methods to detect network attacks within encrypted traffic ..?
The only way I am aware of for detecting network attacks in encrypted traffics, as well as IP tunneling for outbound data leaking, is by performing statistical traffic anomaly detection. For the detecction to be succesful, you have to relay on some characteristics of a bidirectional traffic attack/tunnel, mainly: 1. A Higher volume of traffic coming from a single IP address, or a higher number requests from internal clients where little usually take place (for outbound tunnels). 2. Total amount of data transferred over specific ports / protocols higher than usual. This is the first warning to raise attention. 3. Use of atypical encrypted traffic. Many trojans and network attacks use simple encryption, as XOR, this may give you some idea of an attack in progress is being done. For further information, you could google for statystical packet anomaly detection engine (a plugin for the opensource de facto standard IDS, snort) or related papers on tunneling schemes and its countermeasures,
