US Patent 7548911 Diagnosing problems in distributed systems

Diagnosing problems in distributed systems. In one aspect, a model of a distributed system defines trace statements are generated by various nodes of the system and placed in log files. The log files are sent to an automatic validator that attempts to reconcile the trace statements against the model. Results of the attempt are indicated. In another aspect, trace statements are received by a multiplexer that creates an ordering via sequence numbers. The ordering is forwarded to an automatic validator to determine whether the trace statements indicate correct behavior.