BIG SYNTHETIC DATA WITH MUSKETEER

BIG SYNTHETIC DATA WITH MUSKETEER CHICAGO BIG DATA ANALYTICS MEETUP A. Sasha Gutfraind Lauren A. Meyers and Ilya Safro University of Illinois at Chicago 2014

THE WHOLE STORY Claim 1: Big Data is often a NETWORK Claim 2: Big Data is Not big enough Solution: The MUSKETEER Algorithm

E XAMPLES OF N ETWORK DATA Colorado Springs social network Al-Qaida network 3430 3416 3417 3374 3 3 9 13 694 3431 3377 3373 3481 3695 3375 3480 3398 3610 3444 3735 3734 3494 3397 3451 3499 3495 3496 3615 3376 3442 3699 3500 2 3396 3625 3501 3414 3493 3452 3588 3502 3665 3570 3399 3656 3438 3551 3432 3527 3531 3583 3565 3524 3550 3384 3497 3503 3614 3626 3315 3569 3306 3571 4818 3675 3528 3525 3639 3744 3490 3437 3601 3591 3422 3418 3636 3714 3383 3662 3406 3 5 7 33 3526 3491 3479 3530 3518 3514 4697 3677 3666 3319 3592 3535 3659 3467 3676 3522 4635 3553 3303 3520 574 4698 3304 3354 3412 3638 3703 3410 3715 3517 3584 3415 3712 3 7 0 03 3355 3717 3739 3343 4555 4802 3519 3637 3613 3562 4544 4699 3595 3409 3407 3468 3585 4579 4592 701 3305 3727 3482 4612 3711 3411 3536 4776 3594 3620 3542 4521 4762 648 3688 4800 3681 1 4488 3726 3408 4770 4598 4524 4769 3483 3 6 4 73 3425 4529 3578 3736 4514 3521 3713 3350 4679 3679 3469 3424 3366 3716 4589 3443 3358 3731 4623 4586 3576 3740 4562 3293 3394 4568 4504 3547 3419 3344 3316 3487 3488 3586 3426 3577 3733 4767 3515 3607 4583 3546 4659 3600 3516 3510 3507 3674 3673 3608 3670 3560 738 3328 3433 3486 3325 3371 3346 3353 3567 3359 3561 3702 4622 4584 3423 3498 3472 3540 3320 4553 4515 3508 3511 4472 3471 3598 3730 3326 3436 4671 4496 4580 4669 3459 3505 3698 3689 4560 3392 3548 2423 3465 3460 3321 3564 3455 3558 4548 3612 4468 3685 3723 3624 3309 3439 3632 4576 3732 4684 4475 3590 3311 4499 3338 3634 3478 3484 3635 3644 3616 3297 3380 4613 3464 3523 3477 3379 4781 4597 4654 3506 4559 3317 4582 4601 4778 4470 4672 3683 3645 3356 4696 3441 4530 4518 4549 3378 4691 3721 3310 3312 3335 3357 3684 4471 3370 3440 4687 4480 4516 3336 4478 4827 4630 3633 3352 4782 4639 3579 4498 4657 3434 3504 3722 4492 4512 4806 4689 4730 4764 3299 4760 4777 3461 3462 3463 4647 4674 3351 4485 4563 3296 3621 3671 3421 3582 4491 4766 4653 4664 3509 3485 4638 3652 3360 4523 4676 3559 4663 4509 4745 3563 3655 3623 3593 3329 3393 4467 4629 3363 4705 4752 4747 3513 3 5 1 23 3427 3466 3724 3365 4581 4545 3657 3327 3725 3568 3307 3709 4513 3742 3532 3627 3420 4495 4551 4519 4528 4554 3475 4714 4746 4708 3572 3680 3382 3587 4585 4701 4666 4710 4704 4728 4731 3589 3557 4587 4520 4620 4522 4527 4724 4711 4729 4717 4738 3575 3718 3664 3470 3395 4801 4768 4735 4709 4748 4734 4720 3324 3332 3361 4670 4677 4648 4556 3333 4590 4805 4807 4633 3413 4765 3529 3367 4614 4744 4740 4741 4736 4761 4757 3696 4490 4500 4481 4615 4489 3368 4772 4718 4479 4538 4712 4811 3298 4607 3364 3340 3746 4774 4690 4517 4742 4751 4715 4733 4737 4482 4501 3631 3369 4825 4621 3342 4570 3538 4561 4610 4727 4543 4702 3630 3341 3537 4707 4775 3697 4557 4655 4681 4716 4721 4810 4668 4758 4678 4577 4511 4494 4739 4661 3390 4483 4643 4713 3362 4665 4608 4502 4812 4755 3450 3597 3313 4817 4645 4706 4875 3690 3693 4771 3741 4750 4571 4823 3539 3691 3314 4596 3596 4503 4732 3339 3654 3474 4591 4821 3302 4636 3389 4662 4642 3308 3629 4816 4725 3692 3643 3580 4813 3435 4815 4578 4680 4819 4625 4628 4532 4673 4640 4754 4783 4876 3581 4641 4627 4649 4822 4637 4603 4617 3552 4644 4723 4719 4756 4826 4593 3334 4619 3388 4798 3330 4685 4784 4814 4564 3331 3606 4650 4722 4565 4569 4618 4541 4753 3294 4605 4759 3672 3705 4611 4749 3658 3295 4602 4542 4531 4476 3605 3704 3646 4773 4675 4651 4804 3719 3603 4824 4646 4599 4526 3660 3405 4656 4508 4799 4575 4820 4803 4726 4497 4896 4473 4534 3686 4926 3453 3566 4547 2410 4785 4525 4609 4780 4604 4796 4831 3619 4845 3599 3555 3534 3743 3454 4552 4833 4787 4830 3428 4898 4606 3602 3609 4786 4779 3400 3401 3429 4686 4469 2409 4794 3628 3650 3322 3492 4533 3728 4899 3554 3402 4652 4626 3729 3668 4667 4474 3687 4793 3661 4632 4703 4895 4594 4595 4535 3651 3337 4861 3349 3323 3678 3663 3347 3745 3682 3641 4809 4925 4507 4486 4574 4550 4763 4795 4743 4624 4792 3348 4567 3640 4808 4539 4566 4510 4572 4869 2250 3667 4868 3404 3456 4558 4600 4883 4790 4573 4487 4631 3748 4536 4797 4537 4867 2248 2209 3533 3653 4700 3458 3457 3447 4791 3747 4788 4546 4505 3300 2241 3445 3720 3706 4865 3448 4493 2412 4789 3386 3541 3403 3446 4658 4506 2252 2233 4616 3549 3707 3345 3301 3604 4682 4484 3737 4477 2251 2305 4540 3622 2249 4688 3489 3476 2310 4588 3710 4660 4683 3708 4842 2242 3617 3611 3449 4866 4923 3545 4843 4840 4379 4912 4388 4634 4350 4380 3556 3669 3618 4416 4351 2361 4692 2253 2387 4358 4397 4888 2355 4916 3649 4829 4694 4371 3544 4915 4834 4900 4844 4846 4359 4848 4880 2287 3642 2216 4403 4841 4382 4886 4885 4870 4889 4364 4874 3473 4913 4828 2334 4355 2370 4383 4394 2311 4877 4832 4693 4372 4393 4847 2279 4887 4918 4412 4390 4396 4399 4333 4695 4863 2272 4884 2411 4389 4405 4353 3318 4910 4924 4391 4356 4373 3543 4339 4395 4914 4862 4348 4357 4936 4343 4904 4377 4417 4839 4406 2266 4340 4920 2345 4838 4881 4418 4409 4938 4329 2346 4361 4407 4903 4864 4897 4365 4360 4878 2217 4927 4331 3372 4890 4905 2414 4419 4400 4853 4345 4328 3387 4835 2288 4346 4929 4872 4336 4854 4411 4836 4342 2316 4338 4368 4873 4347 2215 4334 4332 4335 4917 4928 4381 2214 2265 4931 4375 2317 4398 2218 4922 4849 2413 4385 4879 3 4404 4402 2306 3381 4930 4413 4387 4349 4327 4850 753 4414 2422 4908 4330 4354 4837 4937 4408 4337 4919 3385 4410 4344 2421 2396 2263 4401 4906 4367 776 4366 4909 797 4935 4384 4386 2245 2357 752 785 4907 2347 786 4352 4392 2315 770 4921 4932 4376 4911 4901 2230 773 4374 767 4939 2337 4933 2291 2256 4415 4940 2236 2303 771 4934 2237 4378 750 751 4369 2247 4902 2348 4370 4341 4891 2238 749 775 2227 2408 805 819 2404 321 263 2293 2231 748 818 4363 2386 768 769 806 765 782 820 2367 817 2359 772 2225 807 2378 2243 2358 766 320 2400 2399 2277 755 2338 774 821 739 740 2269 2325 2373 2270 764 2379 757 999 2377 2406 2271 2267 2219 2389 2336 787 781 2390 2240 824 319 2276 2397 2335 2371 756 803 798 4362 2349 2368 2290 990 2208 2314 2244 2333 2273 823 994 2332 2374 799 2416 2278 2312 2381 747 2254 784 2415 2294 976 2318 1042 2255 1048 2420 2341 936 2419 2382 2385 2323 1012 515 950 942 964 986 4851 1049 2326 2380 970 951 2418 996 2329 2321 969 2369 519 516 1056 2226 978 1001 958 4855 1054 1020 2285 938 1013 2328 2356 2280 1025 1043 971 2286 2282 975 517 2222 1002 1010 2228 1037 397 2220 400 2284 2309 1059 4894 2383 2393 399 2295 2296 1026 1018 1023 1024 973 1011 1045 1007 1022 1046 1009 943 963 2261 396 1008 959 1028 955 952 2262 2403 407 1027 1000 1005 1021 980 4871 437 985 2234 2260 2229 2212 972 991 2258 2235 2376 937 520 2402 398 2392 2211 995 1004 2394 2331 389 2210 2205 944 957 954 2360 974 463 467 2259 2221 2405 2298 448 744 984 1019 1017 2364 1036 2223 408 802 1774 1003 983 2297 2232 795 1155 1038 979 449 1051 940 1040 965 2362 967 1016 465 1006 956 2207 453 1260 968 945 2213 64 2283 982 939 1029 65 953 2401 1015 1031 1039 1214 804 808 1156 456 2340 2330 792 1944 1050 1030 63 395 451 801 966 1057 981 1161 4434 949 1014 935 2388 2384 2324 2391 2304 741 746 813 1259 1996 4432 288 339 333 948 934 2339 2353 2239 2264 2302 790 743 812 2049 1932 4427 238 332 794 2363 4443 4856 961 4420 960 68 793 419 2116 518 800 328 379 338 4435 290 1221 1801 2307 988 997 2342 1270 2365 418 811 4436 1783 1402 1911 987 1052 393 385 421 387 105 4440 277 1058 1055 1230 1228 1501 1933 4442 336 337 327 1032 809 810 239 326 342 2343 0 1227 1935 1437 4421 243 291 372 822 318 270 274 1035 1041 2289 2224 2257 2372 2417 777 4892 4893 2327 2352 2322 2301 2354 760 745 796 4429 4423 354 355 272 325 271 275 269 993 2246 1044 2375 815 759 742 779 1971 1114 4422 353 324 346 268 1034 933 941 2308 2319 2313 814 778 758 762 783 1991 1837 4424 4426 4428 341 962 946 1033 2320 2281 2407 2274 761 789 252 4430 4425 340 331 380 998 992 2395 2206 2350 2299 763 816 780 788 289 376 253 309 374 323 308 297 1047 1053 2366 2300 2275 791 754 251 322 4431 265 300 977 947 2292 2398 2268 370 367 4433 358 294 4857 356 1292 1271 273 1863 4438 386 1639 1288 359 4441 292 1900 240 4035 1934 4859 1250 2351 314 2344 361 4439 1124 343 989 373 1634 2092 362 182 1287 315 1231 1127 378 183 420 244 375 335 1257 446 1901 1120 184 3982 1956 371 1576 293 175 1289 317 307 278 4437 445 152 245 1825 200 66 246 119 56 298 89 2021 1160 303 109 1992 1331 78 247 149 377 79 429 69 1593 148 52 4858 1126 316 87 1066 330 357 1118 299 54 1628 256 198 80 470 204 4860 1809 257 172 259 96 1258 1663 4 1859 90 368 384 1336 329 2069 1063 147 383 1852 310 281 381 160 111 447 195 454 88 1378 258 1354 197 345 144 311 1119 264 1625 360 29 16 840 1128 17 55 334 455 262 344 249 261 280 267 201 192 1353 1399 1337 306 304 312 1400 14 46 107 4067 2004 1064 313 248 347 1428 305 250 352 382 130 2104 254 348 266 255 19 462 12 1388 15 35 1488 193 279 260 287 301 282 302 18 20 178 2109 190 5 428 1401 1997 351 120 166 1379 4852 295 2009 1065 2075 350 276 38 459 1176 369 349 296 37 33 452 13 161 196 176 145 415 36 142 283 1973 27 103 158 363 365 199 39 1367 284 241 286 159 1429 157 1285 366 21 51 153 413 242 34 364 194 1096 1144 2005 285 173 180 154 1729 1624 1420 1504 1222 1092 177 468 1186 2010 146 2141 124 464 450 1097 1086 155 443 45 97 1876 221 60 156 839 1477 1351 83 44 426 2050 220 141 1088 1085 1251 1220 1254 2018 1630 136 225 47 414 4068 417 2017 469 62 1087 1389 59 4882 1496 1240 216 416 1478 424 189 32 1508 121 388 1632 232 213 218 212 179 102 1091 1223 217 25 444 1229 1949 1108 228 31 440 1204 1202 430 58 122 30 40 2086 1090 211 210 1860 1959 425 140 67 441 1203 230 174 167 123 1342 101 466 1788 1524 187 186 70 104 106 226 41 1343 442 227 1364 3987 223 169 423 48 185 1650 1205 1290 229 92 1427 137 219 224 222 2045 1409 168 170 394 202 150 138 1470 110 191 139 1350 2044 171 3994 203 75 1719 4077 391 1449 1849 886 71 181 233 888 2164 164 26 1777 457 1525 1359 2037 165 162 131 887 406 1426 915 404 143 3986 10 913 848 409 405 43 50 914 461 1297 1735 7 8 1165 1206 134 234 889 4078 916 1296 163 6 42 392 460 458 1252 1853 1705 879 3974 1392 1181 1989 1866 94 93 9 1629 1867 436 1094 434 1253 3985 206 918 236 114 1348 1093 2096 76 77 390 2112 2015 1681 1177 1386 2135 2159 86 838 1875 1178 1265 1387 2169 2170 2168 911 237 209 917 61 432 435 1358 11 2190 115 2122 1095 2091 208 188 847 2070 401 1587 912 3984 433 403 2166 205 431 135 1503 2071 125 2101 1765 2079 1295 1842 1349 402 1207 2165 3947 870 853 3975 880 1083 2077 1447 900 2073 2074 151 3973 1446 28 98 884 1588 74 925 852 2042 2134 1347 1312 2090 1311 927 1168 2108 1199 865 2123 1475 2124 1460 1474 859 898 4063 1198 875 1940 1926 1284 1303 4071 4065 57 866 825 909 908 907 872 438 874 871 1660 849 863 899 1965 81 833 885 4066 1636 892 1707 1439 1148 1505 207 85 901 3949 439 1966 1440 1412 1245 854 108 878 1216 1413 1925 1884 876 930 1304 1829 1684 1146 928 3951 1215 1891 1738 2034 1521 1321 1844 100 891 4060 1892 2193 2027 1519 1874 1411 2156 1462 1584 851 1737 1527 1320 1526 2136 1117 1753 897 99 861 1516 855 837 1811 1855 881 929 1637 1339 1717 2121 1951 1522 1463 895 4043 2012 1415 2000 2066 4042 1438 1436 2187 1589 1194 1905 869 858 1827 1366 2149 896 862 2019 1158 1678 1599 2186 82 850 4059 1835 1142 1189 1104 2061 1856 1953 2143 860 1520 1341 2195 2105 1556 2137 2140 2056 923 926 844 890 2084 2085 95 921 864 893 873 3950 3966 1200 1939 2083 2082 922 883 4061 1147 1141 2064 1448 2081 2055 2107 2094 882 902 3995 411 1340 2125 894 857 472 1461 1234 1188 4064 4062 471 1166 2063 2072 2144 2171 931 3954 410 1233 1089 2192 924 846 427 2095 2191 2036 2172 1824 1512 3998 835 4041 1646 1075 1195 1390 843 868 133 831 827 3980 2562 1279 2150 1540 1817 1310 473 1217 845 4445 128 117 919 4447 116 1914 1514 22 49 903 877 1490 1425 118 904 4040 2046 1391 1074 1416 1414 1920 920 1452 1316 836 1076 1159 1322 1770 1209 1435 2198 828 3952 2020 1679 1164 2184 834 826 1511 1865 4448 910 126 127 4459 867 129 235 856 1814 932 1309 2189 1278 1784 1506 2202 1212 2185 2022 1818 2204 2016 4449 4460 4323 4129 4461 24 4457 132 231 599 84 4463 73 2440 4452 601 730 215 214 575 4013 597 593 4325 3948 629 630 661 4453 3957 4113 696 496 690 3992 495 3972 4051 4012 522 3969 3968 608 493 4114 4005 3970 482 4320 691 494 4000 657 497 611 4020 1510 609 489 486 723 603 1157 1762 485 4019 3971 1263 1967 1614 492 4028 484 1750 4120 1968 1577 1486 1983 2173 1980 1649 4119 4039 1261 1728 1408 1822 1430 1509 1990 1954 697 662 1248 1424 1432 1487 1468 592 499 113 4021 4031 1749 1077 1082 1752 620 91 4148 4001 3967 1224 1262 1799 1264 1528 2126 1695 1887 2145 2132 606 604 498 480 4052 721 711 477 4112 1225 1798 3965 3953 4072 1974 1255 1210 3964 4029 1571 722 607 656 476 478 4149 569 3983 1369 4326 712 1211 2188 1654 1306 1903 4073 4075 1913 1368 1883 1235 1236 1237 1305 1361 4076 1902 610 658 483 642 3770 4069 1431 1325 1109 2068 4285 4032 1881 1570 1748 605 479 4074 4014 1846 578 3981 577 628 1495 1747 1618 1362 4070 1723 4093 1471 1370 665 4002 1975 627 841 1919 2008 1244 1363 595 737 4038 1371 596 738 474 842 1326 594 3769 4173 1433 1538 1273 666 4105 540 2183 481 705 4080 1445 591 491 1328 1539 4036 4178 4306 1360 487 541 1730 590 4177 4037 1579 1450 4293 570 4277 4102 2177 1569 4179 1657 1140 1775 4079 1552 4291 4292 1586 1950 2003 589 4095 1444 1274 1272 1626 1928 2178 600 72 112 475 4456 4466 3955 4055 4455 4464 3989 4054 1924 1746 1647 598 4446 4451 514 4027 3978 3956 1736 1916 1507 1395 1529 1173 2026 2076 659 660 23 4454 4465 4008 1125 1727 1396 1631 1943 2157 1536 2131 1557 1904 1712 588 602 4458 4053 3976 3979 1885 1694 1080 1559 1533 2103 53 4324 830 4444 3977 1778 2194 1502 1999 4462 521 3959 3963 4030 4056 1084 1334 1193 2030 1917 1403 1868 4450 3999 412 1683 1713 1731 1541 1893 1761 1864 1485 1484 829 4011 4007 1294 1099 1346 1169 1171 1531 1530 4017 4006 1843 1098 1515 1380 1333 1170 1152 1357 1151 1473 906 832 3997 3958 1594 1677 1941 1175 2203 1277 1100 1356 2176 905 3988 1293 1213 1610 1101 2179 1335 1558 2007 4050 1513 1981 1906 2200 1134 2201 2139 1167 1394 1676 2199 1201 488 1183 1489 2120 2062 3996 422 2047 1467 2138 1276 1464 1908 4049 4048 4016 1352 1466 1465 1706 1675 4163 1327 1112 1564 4096 4018 4010 1620 4305 4111 4276 692 4268 4216 4106 4279 4213 4294 4278 4104 4103 1332 1702 1897 3782 1821 1062 1929 1218 1061 4215 4147 4009 1256 1581 4109 1458 1700 1617 4170 4107 4094 4307 4152 4212 4171 4225 663 1810 1078 1687 539 1763 4047 1764 1619 1318 1836 561 510 694 4098 545 4209 4185 734 621 501 4004 560 701 727 566 507 500 4283 622 612 506 4175 4281 4309 1249 1113 719 728 625 571 542 533 651 580 509 4310 675 720 508 4308 4045 1111 641 576 702 4280 2035 640 736 4201 4311 1459 1365 1923 1116 2180 1542 1616 729 565 4282 2002 1834 698 695 4288 4287 1344 1580 1385 579 735 716 544 4046 1301 664 715 4289 4139 4144 4184 1070 1060 1938 543 4165 4286 1476 1079 1880 1682 1656 1247 1384 699 4151 4108 1153 1952 1673 1655 733 4204 1582 1345 1319 2167 1688 653 652 4312 700 677 4146 1267 1172 1895 1454 1500 1882 4044 676 682 3869 572 4176 1110 584 582 4220 546 1266 613 683 680 3870 4164 670 626 684 550 573 1269 1137 4245 4097 667 4200 709 710 619 672 581 639 645 704 673 638 585 646 4246 615 674 668 634 1382 1139 4127 687 708 511 526 1123 633 636 632 678 559 512 635 618 552 1406 644 4186 1313 648 558 568 527 1381 1105 1410 686 679 4252 538 1987 1994 1543 707 4126 555 547 1317 1138 1115 1686 706 614 649 4247 4248 534 4174 548 3962 1180 1498 1603 693 4125 551 505 1136 1373 1067 1597 1197 1567 616 4128 4207 553 1106 1068 1598 1479 1196 1566 1568 1674 650 4145 1877 1324 1800 1807 681 724 4136 554 1383 1499 1419 1418 1828 1480 1757 713 685 4222 3961 1456 1455 1535 1069 1909 1937 714 671 725 726 549 1405 1453 2006 1286 1751 1936 637 583 4221 1107 1457 1300 2181 1644 1894 718 532 631 587 617 513 1969 1847 1995 1963 1819 643 1179 1268 1970 1611 1397 562 1666 1372 717 535 4317 3781 1621 1608 703 1491 1613 647 623 567 1145 1993 1497 1648 624 4300 1377 1948 1607 1806 1243 1393 557 586 689 1551 2163 2182 2113 1561 1760 1955 1998 1773 1612 1291 1898 1323 1604 1879 1962 1779 2197 1633 4206 4101 523 1242 1848 3939 537 1422 655 3941 1143 1451 669 3780 1443 1555 2088 1133 2110 1404 1635 1718 1374 1472 1823 1964 574 536 1423 1154 563 556 1667 1984 1664 1627 1283 2111 1781 688 3798 3790 504 1421 1376 3779 4244 4100 1759 1549 1813 2098 1609 1103 2065 2097 2118 1776 1680 4241 1308 3893 732 3817 3871 3855 4160 1102 731 4260 4217 564 1861 530 3785 528 1150 1947 1185 1434 1600 3912 3801 3851 4198 3960 1701 3890 2621 4269 4199 3823 1622 1922 2054 1482 1281 2151 3909 3850 531 1481 3797 3799 3834 1690 2080 529 1734 1307 1831 4197 4265 3796 3888 1592 1850 2114 4250 1977 1550 2051 4202 1851 1441 1537 4099 3863 3758 3792 3898 3786 3759 3945 3812 3789 3824 3886 4218 4243 4132 4194 3806 3930 3862 4237 4161 4115 4205 4193 3756 3931 3840 3896 3767 3908 3911 3807 4138 4024 3884 3802 4158 1232 3793 1483 1239 1246 1191 3757 3755 3859 3878 4023 1469 1651 1072 3864 3897 3813 3865 4156 1071 1121 1329 1988 3791 3771 3894 4162 4026 1615 1280 1796 3805 3809 3857 3838 3808 4140 1187 1797 1708 3883 3921 3866 3906 4142 1494 2100 1870 1744 1595 3910 3933 3868 3760 3875 3904 4110 4133 1547 2127 2146 2147 3917 3882 3842 3761 4143 4172 1122 1659 1544 2133 3934 3922 3826 3753 3938 4141 4085 4211 4033 2059 2102 2087 2154 3881 3918 3946 3892 3993 4057 1794 1208 2014 4034 2028 1709 1638 1722 2106 1745 3913 3775 3787 3803 4121 4086 4081 2057 1826 1832 1583 1585 3854 3877 4150 4304 4219 4058 1192 1129 1721 2067 2043 2058 3887 3916 3822 3907 3876 4083 4082 4015 4003 1330 2155 1918 3835 3836 3841 3763 4025 2060 2196 2115 1662 1756 1605 1643 1565 3776 3766 4084 3991 1696 1930 1931 1972 1820 1871 1493 1492 3800 3777 3783 3990 1131 1226 1548 2053 2099 1802 1982 1710 3774 3879 1275 1574 1623 1691 1661 3754 3765 2048 1838 1755 1132 4090 1812 1601 1869 2001 1699 1873 1830 3784 3905 1149 1073 1692 1927 1872 1698 1573 1591 1534 1130 1986 3915 3937 2556 1442 1780 1726 2153 1758 4227 503 1282 2148 2119 1135 1602 1886 3889 524 502 1375 1338 1945 2089 2129 2128 2130 1782 2011 3816 525 1725 2117 1942 654 3940 1732 1715 1961 1841 1703 1803 1652 2029 1081 1560 4190 3891 3843 3795 3928 1795 2594 2557 3768 4137 4258 2939 3788 3820 3923 3847 3831 3936 3804 4231 4242 4302 3751 3794 4134 4230 4249 3169 3818 3935 4259 3880 3902 4232 4267 1163 3858 4187 1190 3925 3901 4191 4087 4303 4022 1546 1299 1238 3811 3814 3825 3223 1523 3772 3885 4251 4234 1724 3844 3839 3750 4192 4155 4301 1553 1733 4154 3872 3762 3773 1976 3810 4238 1888 1704 1182 3778 4157 1805 3932 3752 4233 2716 3903 3924 3846 3849 4135 1845 2052 1532 4153 3279 1302 2465 4229 4224 4159 3944 3943 3926 3895 3914 3764 3815 4188 2532 2158 1314 3821 1298 1398 1979 2031 275 4089 3244 1786 1407 3860 3867 4228 4223 4253 1562 3819 4235 2467 1697 4118 4180 3927 4271 4 2 7 44 1815 3749 3942 4203 1772 1162 1355 1907 4236 2466 4261 4256 3848 4166 4273 1241 3827 4262 4270 3874 3837 4272 3845 2175 2023 4196 2717 1641 1912 3829 4124 4290 1978 1878 3920 3861 1858 1693 4284 2543 3828 4117 1315 1554 2719 4088 2558 2910 1658 3830 3900 4182 2909 2174 2041 3919 3856 4208 4321 4322 3873 4122 4123 4131 3899 2451 3853 1740 1685 1720 1572 1219 1957 4183 4240 3929 4181 4195 2516 4189 2712 2612 3852 4168 2720 4092 1668 4169 2508 2613 3043 4297 2519 4299 1767 2093 4296 2486 4254 2832 4298 4257 1417 1689 1518 3277 2424 2425 3290 2528 2576 4295 4091 4263 4318 2833 3291 3272 1789 4210 1739 4255 2598 1766 2599 3219 2591 1714 3199 3289 2503 2642 2571 1833 2523 2989 1857 1899 1862 3288 2480 1671 2710 2483 2479 1642 1790 2573 4315 2589 3206 2954 2611 2674 2890 3229 3234 3266 2488 2446 252 2889 2615 3226 490 2511 2559 2473 2567 2502 2949 2757 2616 3242 2950 2509 2743 3227 1921 2441 2679 2510 2880 1787 2666 2911 2998 2501 2897 2442 2627 2788 2992 2626 1711 2804 2672 3230 3023 1184 2673 2625 2645 3240 2831 2665 3197 3256 1896 2680 2744 2443 2722 2078 2162 4314 4316 2512 2526 3095 3243 3218 2704 3103 2566 3251 2953 3292 477 2895 1804 2896 3 2 1 43 3215 2552 2482 2481 2569 2568 2570 2606 2618 2 4 7 62 3217 2459 2593 2013 1742 2585 2555 2572 2899 1174 1640 1958 2038 4313 2474 2898 1563 1768 2039 1960 1596 4116 4319 1840 1653 2040 4264 4266 2650 2475 1672 1785 3205 2630 2602 1808 1669 1665 1754 4239 4226 3158 2518 4130 1839 1771 1545 3832 2713 2464 2450 2605 2507 1816 1915 1575 1769 1946 3833 4214 4167 2517 1517 1910 1890 2142 1670 2032 2033 2941 3162 2489 2830 2984 2834 3239 3233 2669 2855 2546 2829 2547 2946 2735 2548 2848 2542 2988 3068 2436 3115 2958 2902 3014 2560 2800 2866 3148 3224 2433 2453 2537 2745 2819 3161 2661 2881 3247 2879 2956 2963 2952 1791 2632 2653 3117 2957 2617 2945 2444 3006 3042 2900 3041 3100 2678 2883 2550 3037 2853 2823 2534 2536 2652 3021 3153 3067 2620 3005 2840 2595 2535 2795 2824 2849 2670 3022 2622 2986 2563 2706 2435 3157 3146 2699 2850 2659 3029 2638 2584 2514 2430 3025 3147 2852 2499 2515 2635 2884 2506 2948 3130 3008 2490 2658 2759 3107 3124 2497 3191 2810 3036 2597 2529 3123 2872 3126 3058 2978 3078 2564 3056 2966 2920 3187 2469 3221 2513 2629 2921 3054 2816 2639 2683 3192 2491 3209 2802 2851 3275 2892 2498 2993 3278 3231 3269 3184 2478 2987 2640 3172 2730 2769 2871 2596 2427 2715 2723 2971 2917 3156 2767 2844 2495 2610 3128 2846 2863 2667 2777 3276 2544 2870 2861 2859 2873 2721 3119 3038 2864 2843 2553 2845 3049 3118 2811 2729 3098 2842 2761 2923 2812 2906 2876 2874 2791 3127 2809 3097 2770 2763 2768 3149 3096 3099 3028 2937 3030 2928 2972 3122 2965 2675 2877 2936 2732 2914 3057 2908 2758 2505 3168 3046 3238 2822 2901 2694 2470 2538 2695 2554 2633 2682 2905 2862 2574 2687 3066 3050 2774 3195 3010 3179 3017 2878 3106 2867 2471 2979 3114 2431 3196 2438 2437 2904 2860 2858 2601 3003 2981 2806 2736 2738 2762 3120 3139 3186 3027 2731 3284 2940 3033 2875 3281 2808 2734 2520 2760 3198 2714 2619 2432 2439 2448 2907 2581 2964 3182 2522 2778 2807 3047 2931 2737 3193 2821 3142 2854 2702 2651 3108 2991 2575 2592 2686 3132 2779 2533 2935 2826 2815 2942 2891 2698 3064 2634 3154 2449 2887 3189 2970 3228 2838 2724 2857 2756 2755 3080 3177 2740 2789 2701 2790 2912 2631 2545 2885 2886 2888 3105 3039 2839 3152 3250 3194 2766 2452 2551 2663 2976 2794 2637 2685 3210 2614 2455 3026 2796 3176 2932 2561 2691 2903 2623 2462 2707 3220 3268 3143 3110 3020 2742 2494 2969 2705 3116 2590 2733 3144 2429 2711 3125 2671 3001 2803 2718 2980 3040 2463 3246 3263 2709 2690 2668 2608 3270 2681 3204 2780 2700 3079 2624 3283 3134 2708 3004 2968 2781 2773 2982 3160 2496 2933 3183 3133 3093 3203 2771 3002 3165 2628 3222 2865 3178 3035 2837 2943 3062 2825 2764 2739 3104 3019 3094 3267 3235 2772 2727 3009 3287 3265 3174 2684 2434 2841 3007 2977 3264 2765 2468 2813 3190 3111 2445 2951 3175 2655 2600 2944 2697 2962 3253 2820 3112 2847 2741 3200 2662 3121 3188 2456 3211 2527 3131 2603 2654 2487 2786 2797 2728 2587 2460 3248 2955 2525 2983 2798 2997 3109 2524 2461 3052 2696 2930 2787 3011 3232 3151 3207 2959 3013 3274 3012 2814 2967 2024 3150 3055 2485 3060 2588 2454 2609 2782 2703 2818 2990 3063 3216 2577 2504 2973 237 2793 3000 1792 2985 2856 2827 3155 2961 3208 3225 3254 3 2 3 63 3273 3258 2457 3245 2025 2947 2426 2540 2607 2578 2882 2604 3261 3260 2828 2960 2472 2493 2492 2975 2649 2676 2817 2447 2549 2641 2974 2644 2458 2893 2996 2656 2539 2646 3212 2648 3255 2677 2894 1889 1854 2428 2586 3015 2647 3259 2565 1590 1645 1793 2664 2799 2541 3170 1716 1743 1741 2835 2582 2583 3016 2643 3257 2836 2805 2521 3053 2152 2160 1985 2161 1578 1606 3059 2999 3280 3092 2915 2927 3129 3173 3249 3138 2934 3048 3141 3034 3135 3018 3045 3185 2801 2924 2913 2995 2918 2692 3065 2868 2657 2785 3241 3213 2484 3180 3159 3202 3164 2922 2693 2938 2925 3163 2783 3073 3262 3282 3181 3145 2792 2916 2636 2994 3077 2784 3090 2926 3166 3072 3140 3171 2530 2688 3051 3076 3286 2531 3070 3091 2929 3113 3044 2689 3081 3069 2919 2725 2500 3167 2579 3285 3101 2746 3271 2726 3084 2775 3089 2749 3102 3031 3061 2580 2754 3075 3032 3082 2776 3137 2753 3088 3087 2747 3136 2752 3074 2869 3071 3083 2748 3085 2750 3024 3086 3201 2751 2660 Internet A. Sasha Gutfraind Lauren A. Meyers and Ilya Safro Power grid Big SYNTHETIC Data with MUSKETEER

MOTIVATION FOR SYNTHETIC DATA 1 Networks are the central part of many systems considered by analysts, e.g. social, infrastructure, neural systems 2 We need to evaluate ideas/methods/algorithms on them 3 Challenges: 1 Difficult or Impossible to get real data 2 Insufficient: want to show robustness on 10 2 to 10 6 networks

APPLICATIONS OF SYNTHETIC DATA Synthetic data are needed to Model networked populations Simulate what-if scenarios Compensate for missing/insufficient data Anonymize data

METHODS FOR NETWORK GENERATION 1 Network model: Erdős-Rényi, Kronecker Graph, ERGM, Watts-Strogatz, Liu-Chung expected degrees, Barabási-Albert, etc. 2 Mechanistic model 3 Randomize empirical data 4 An application-specific topology generator: BRITE, INET, Tiers, GT-IGM, PLOD, GridG, GeNGe, etc. New (5.): MUSKETEER Ref: Multiscale Network Generation. Free and Open source. arxiv.org/abs/1207.4266

TRADITIONAL MULTISCALE ALGORITHM What is a multi-scale algorithm? 1 Iteratively coarsen i.e. reduce the number of variables in a problem: L 0 L 1 L k L 1 L 0 e.g. L i+1 = P T L i P 2 Solve in level k and then refine it back to level 0 Strengths: O(m) or O(m log m) performance for Polynomial or NP-hard problems Pitfalls: Enforcing constraints & Precision Very successful in large linear/nonlinear equation solvers Ref: Knepley/UC - PETSc

ARCHITECTURE OF MUSKETEER The MUSKETEER algorithm: 1 Generates a hierarchy of coarsened networks 2 Edits at any level of coarsening 3 Synthethic nodes are resampled 4 Synthetic edges preserve locality Version 1.2 (Dec): Fast editing algorithm

NETWORKS Examples Statistics Let s make some networks...

Examples Statistics PRESERVATION OF HIDDEN PROPERTIES

Examples Statistics REPLICATION WITH A RANDOM KRONECKER GRAPH

Examples Statistics EXAMPLE: COAUTHORSHIP Collaboration network (Newman): GCC 379 nodes growth rate: nodes [0, 0.3]; edges:[0, 0.1]

Examples Statistics EXAMPLE: POWER GRID Western Interconnection - a power grid with 4941 nodes edit rate: nodes [0, 0.1]; edges:[0, 0.1]

Examples Statistics EVALUATION OF RANDOM NETWORKS

Examples Statistics QUALITY OF RANDOM NETWORKS Experiment with MUSKTEER Level 0 edits: 8% nodes, 8% edges Level 1 edits: 7% nodes, 7% edges Generally, the choice of edit rates is based on the problem Colorado Springs HIV (left) and replica (right) Ref: Potterat et al.

Examples Statistics QUALITY OF RANDOM NETWORKS FIGURE: Colorado Springs Network Median of replicas num nodes num edges num comps clustering avg degree total deg*deg assortativity avg eccentricity avg distance harmonic avg distance avg between. centrality modularity 0.99 1.02 1.00 0.87 1.02 0.94 0.97 1.01 1.07 1.03 1.00 powerlaw exp 0.0 0.5 1.0 1.5 2.0 Relative to real network 1.02 Diversity: 30% of nodes and 60% of edges are new or removed

SELECT USE STORIES Examples Statistics S Leyffer, I Safro Developed an algorithm for blocking cyber attacks on large networks MUSKETEER helped discover implementation errors MUSKETEER data provide performance evaluation M Bergner, ME Lübbecke, J Witt Investigate the packed cuts problem Developed a new Branch-Price-and-Cut Algorithm MUSKETEER data provide performance evaluation

Examples Statistics SUMMARY & EVALUATION MUSKETEER: Synthetic network data Controlable: fine and global editing; size expansion Suitable for many types of networks Runs in O(m) agutfrai@uic.edu G, Meyers and Safro. Multiscale Network Generation. www.cs.clemson.edu/~isafro/musketeer THANKS DTRA, Argonne & Los Alamos LDRD program, NIH/MIDAS; I Safro & LA Meyers

ABSTRACT Examples Statistics "Multiscale network generation and modeling" Networks are widely used in science and engineering to represent relationships between entities, such as social ties, ecological links and computer infrastructure. However, the structures of real-world networks are often not known completely, and they may exhibit considerable variation so that no single network is sufficiently representative of a system. In such situations, researchers may turn to synthetic network data for modeling the system and performing simulations. In the talk I will introduce a flexible strategy for synthesizing realistic network data using ideas inspired by multigrid methods. The strategy, termed MUSKETEER, is to start from a known network dataset, perform a series of mappings that repeatedly coarsen and later repeatedly uncoarsen the network, while applying perturbations to create diversity. Using examples from several domains, I will show that MUSKETEER can generate diverse ensembles of networks including their edge and node labels. Statistical analysis shows that MUSKETEER can also achieving greater fidelity across a suite of network properties than do other commonly used network generation algorithms. Bio: A. "Sasha" Gutfraind - University of Illinois at Chicago Sasha Gutfraind received a Bachelor s and a Master s from the University of Waterloo in Applied Mathematics and a Ph.D. from Cornell University. He develops mathematical models to illuminate problems in complex networks, public health and security using methods from the theories of complex systems, mathematical optimization and dynamical systems. Prior to coming to UIC, he worked at Los Alamos National Laboratory and at the University of Texas at Austin