BIG SYNTHETIC DATA WITH MUSKETEER

Transcription

1 BIG SYNTHETIC DATA WITH MUSKETEER CHICAGO BIG DATA ANALYTICS MEETUP A. Sasha Gutfraind Lauren A. Meyers and Ilya Safro University of Illinois at Chicago 2014

2 THE WHOLE STORY Claim 1: Big Data is often a NETWORK Claim 2: Big Data is Not big enough Solution: The MUSKETEER Algorithm

3 THE WHOLE STORY Claim 1: Big Data is often a NETWORK Claim 2: Big Data is Not big enough Solution: The MUSKETEER Algorithm

4 THE WHOLE STORY Claim 1: Big Data is often a NETWORK Claim 2: Big Data is Not big enough Solution: The MUSKETEER Algorithm

5 E XAMPLES OF N ETWORK DATA Colorado Springs social network Al-Qaida network Internet A. Sasha Gutfraind Lauren A. Meyers and Ilya Safro Power grid Big SYNTHETIC Data with MUSKETEER

6 MOTIVATION FOR SYNTHETIC DATA 1 Networks are the central part of many systems considered by analysts, e.g. social, infrastructure, neural systems 2 We need to evaluate ideas/methods/algorithms on them 3 Challenges: 1 Difficult or Impossible to get real data 2 Insufficient: want to show robustness on 10 2 to 10 6 networks

7 APPLICATIONS OF SYNTHETIC DATA Synthetic data are needed to Model networked populations Simulate what-if scenarios Compensate for missing/insufficient data Anonymize data

8 METHODS FOR NETWORK GENERATION 1 Network model: Erdős-Rényi, Kronecker Graph, ERGM, Watts-Strogatz, Liu-Chung expected degrees, Barabási-Albert, etc. 2 Mechanistic model 3 Randomize empirical data 4 An application-specific topology generator: BRITE, INET, Tiers, GT-IGM, PLOD, GridG, GeNGe, etc. New (5.): MUSKETEER Ref: Multiscale Network Generation. Free and Open source. arxiv.org/abs/

9 METHODS FOR NETWORK GENERATION 1 Network model: Erdős-Rényi, Kronecker Graph, ERGM, Watts-Strogatz, Liu-Chung expected degrees, Barabási-Albert, etc. 2 Mechanistic model 3 Randomize empirical data 4 An application-specific topology generator: BRITE, INET, Tiers, GT-IGM, PLOD, GridG, GeNGe, etc. New (5.): MUSKETEER Ref: Multiscale Network Generation. Free and Open source. arxiv.org/abs/

10 TRADITIONAL MULTISCALE ALGORITHM What is a multi-scale algorithm? 1 Iteratively coarsen i.e. reduce the number of variables in a problem: L 0 L 1 L k L 1 L 0 e.g. L i+1 = P T L i P 2 Solve in level k and then refine it back to level 0 Strengths: O(m) or O(m log m) performance for Polynomial or NP-hard problems Pitfalls: Enforcing constraints & Precision Very successful in large linear/nonlinear equation solvers Ref: Knepley/UC - PETSc

11 ARCHITECTURE OF MUSKETEER The MUSKETEER algorithm: 1 Generates a hierarchy of coarsened networks 2 Edits at any level of coarsening 3 Synthethic nodes are resampled 4 Synthetic edges preserve locality Version 1.2 (Dec): Fast editing algorithm

12 NETWORKS Examples Statistics Let s make some networks...

13 Examples Statistics PRESERVATION OF HIDDEN PROPERTIES

14 Examples Statistics REPLICATION WITH A RANDOM KRONECKER GRAPH

15 Examples Statistics EXAMPLE: COAUTHORSHIP Collaboration network (Newman): GCC 379 nodes growth rate: nodes [0, 0.3]; edges:[0, 0.1]

16 Examples Statistics EXAMPLE: POWER GRID Western Interconnection - a power grid with 4941 nodes edit rate: nodes [0, 0.1]; edges:[0, 0.1]

17 Examples Statistics EVALUATION OF RANDOM NETWORKS

18 Examples Statistics QUALITY OF RANDOM NETWORKS Experiment with MUSKTEER Level 0 edits: 8% nodes, 8% edges Level 1 edits: 7% nodes, 7% edges Generally, the choice of edit rates is based on the problem Colorado Springs HIV (left) and replica (right) Ref: Potterat et al.

19 Examples Statistics QUALITY OF RANDOM NETWORKS FIGURE: Colorado Springs Network Median of replicas num nodes num edges num comps clustering avg degree total deg*deg assortativity avg eccentricity avg distance harmonic avg distance avg between. centrality modularity powerlaw exp Relative to real network 1.02 Diversity: 30% of nodes and 60% of edges are new or removed

20 SELECT USE STORIES Examples Statistics S Leyffer, I Safro Developed an algorithm for blocking cyber attacks on large networks MUSKETEER helped discover implementation errors MUSKETEER data provide performance evaluation M Bergner, ME Lübbecke, J Witt Investigate the packed cuts problem Developed a new Branch-Price-and-Cut Algorithm MUSKETEER data provide performance evaluation

21 Examples Statistics SUMMARY & EVALUATION MUSKETEER: Synthetic network data Controlable: fine and global editing; size expansion Suitable for many types of networks Runs in O(m) G, Meyers and Safro. Multiscale Network Generation. THANKS DTRA, Argonne & Los Alamos LDRD program, NIH/MIDAS; I Safro & LA Meyers

22 ABSTRACT Examples Statistics "Multiscale network generation and modeling" Networks are widely used in science and engineering to represent relationships between entities, such as social ties, ecological links and computer infrastructure. However, the structures of real-world networks are often not known completely, and they may exhibit considerable variation so that no single network is sufficiently representative of a system. In such situations, researchers may turn to synthetic network data for modeling the system and performing simulations. In the talk I will introduce a flexible strategy for synthesizing realistic network data using ideas inspired by multigrid methods. The strategy, termed MUSKETEER, is to start from a known network dataset, perform a series of mappings that repeatedly coarsen and later repeatedly uncoarsen the network, while applying perturbations to create diversity. Using examples from several domains, I will show that MUSKETEER can generate diverse ensembles of networks including their edge and node labels. Statistical analysis shows that MUSKETEER can also achieving greater fidelity across a suite of network properties than do other commonly used network generation algorithms. Bio: A. "Sasha" Gutfraind - University of Illinois at Chicago Sasha Gutfraind received a Bachelor s and a Master s from the University of Waterloo in Applied Mathematics and a Ph.D. from Cornell University. He develops mathematical models to illuminate problems in complex networks, public health and security using methods from the theories of complex systems, mathematical optimization and dynamical systems. Prior to coming to UIC, he worked at Los Alamos National Laboratory and at the University of Texas at Austin