HIGH-DIMENSIONAL CHANGEPOINT DETECTION VIA SPARSE PROJECTION 3 6 8 11 14 16 19 22 26 28 31 33 35 39 43 47 48 52 53 56 60 63 67 71 73 77 80 83 86 88 91 93 96 98 101 105 109 113 114 118 120 121 125 126 129 133 134 136 139 140 142 146 149 152 156 158 161 163 165 168 171 174 176 180 181 185 188 190 193 196 200 202 206 210 214 218 221 222 225 229 231 234 238 239 240 241 244 246 248 252 255 259 262 265 268 269 272 275 278 279 281 284 286 289 291 294 296 300 304 308 311 314 317 320 324 326 330 334 336 339 341 345 349 351 354 356 360 364 368 369 371 372 376 380 381 385 388 391 394 396 400 402 406 410 411 415 417 420 423 424 427 429 432 435 438 440 442 446 448 451 452 456 458 461 463 464 468 469 473 476 478 480 483 485 486 490 491 495 496 498 501 504 506 508 511 512 513 517 518 522 524 528 532 534 536 540 543 545 549 552 556 560 562 564 565 569 571 572 575 579 581 582 585 589 591 594 595 599 600 603 604 606 610 612 616 618 621 625 628 629 632 634 637 639 643 647 651 655 656 659 660 662 666 667 669 671 674 677 679 683 687 689 693 696 698 702 703 707 708 709 710 714 718 721 724 727 730 733 734 738 739 742 745 747 751 755 758 762 766 768 769 773 775 779 782 784 785 789 792 794 797 800 802 805 807 811 815 816 818 821 823 827 829 832 836 838 842 843 847 850 853 855 856 859 863 866 868 871 874 878 880 881 885 887 891 892 896 897 901 902 905 909 910 912 914 915 918 921 924 928 929 933 937 939 942 943 945 948 951 954 956 959 962 966 968 969 970 972 976 978 979 981 984 988 992 994 998 1001 1002 1003 1007 1008 1009 1011 1012 1016 1017 1021 1022 1023 1027 1028 1029 1031 1035 1036 1039 1042 1045 1048 1051 1053 1057 1058 1059 1062 1065 1068 1072 1075 1077 1081 1085 1089 1093 1095 1098 1102 1104 1107 1109 1110 1111 1115 1117 1120 1124 1127 1129 1133 1137 1141 1145 1148 1150 1153 1155 1157 1161 1164 1165 1167 1171 1173 1176 1178 1179 1183 1184 1185 1189 1191 1192 1196 1197 1198 1201 1202 1206 1207 1209 1210 1212 1213 1217 1220 1224 1226 1229 1231 1233 1236 1240 1241 1245 1247 1250 1252 1256 1257 1260 1261 1265 1267 1270 1272 1273 1276 1278 1282 1286 1287 1291 1292 1294 1298 1302 1305 1308 1311 1313 1315 1316 1318 1321 1324 1326 1330 1331 1335 1339 1343 1344 1347 1350 1353 1355 1359 1363 1367 1370 1372 1375 1378 1379 1382 1385 1387 1389 1392 1395 1398 1401 1405 1406 1409 1411 1414 1416 1419 1423 1427 1431 1434 1435 1439 1441 1443 1447 1451 1455 1458 1461 1462 1463 1465 1468 1470 1474 1475 1477 1480 1482 1485 1489 1492 1494 1497 1502 1504 1505 1509 1510 1514 1516 1519 1523 1525 1529 1532 1534 1537 1540 1544 1547 1550 1552 1555 1558 1559 1562 1564 1568 1572 1576 1578 1582 1584 1587 1590 1592 1596 1598 1601 1603 1607 1611 1614 1618 1621 1625 1628 1631 1632 1636 1638 1642 1643 1647 1648 1649 1653 1656 1658 1659 1663 1664 1668 1671 1674 1676 1679 1683 1687 1689 1692 1694 1698 1700 1703 1706 1709 1712 1715 1719 1722 1724 1728 1729 1732 1733 1737 1741 1744 1748 1752 1756 1757 1761 1763 1767 1770 1772 1776 1779 1783 1785 1786 1790 1794 1796 1797 1801 1804 1807 1810 1813 1816 1819 1820 1824 1828 1831 1832 1836 1837 1840 1843 1847 1848 1849 1852 1853 1854 1855 1859 1860 1864 1868 1869 1873 1875 1877 1881 1883 1887 1888 1891 1893 1895 1899 1901 1903 1906 1908 1911 1912 1914 1916 1919 1923 1924 1927 1930 1932 1935 1938 1940 1944 1946 1949 1950 1953 1957 1960 1964 1966 1968 1972 1973 1975 1978 1982 1985 1988 1991 1994 1995 1997 0 5 10 15 20 25 30 500 500 1000 1500 1500 2000 nodes in binary segmentation algorithm peak of projected CUSUM Richard Samworth, University of Cambridge Joint work with Tengyao Wang
Tengyao November 15-2
Heterogeneity in Big Data One of the most commonly-encountered issues with Big Data is heterogeneity. Departures from traditional, stylised i.i.d. models can take many forms, e.g. missing data, correlated errors, data combined from multiple sources,... In data streams, heterogeneity is manifested through non-stationarity. Perhaps the simplest model assumes population changes occur at a finite set of time points. November 15-3
Changepoint estimation Changepoint problems have a rich history (Page, 1955). State-of-the-art univariate methods include PELT (Killick, Fearnhead and Eckley, 2012), Wild Binary Segmentation (Fryzlewicz, 2014) and SMUCE (Frick, Munk and Sieling, 2014). Some ideas extend to multivariate settings (Horváth, Kokoszka and Steinebach,1999; Ombao, Von Sachs and Guo, 2005; Aue et al., 2009; Kirch, Mushal and Ombao, 2014). Increasing interest in high-dimensional setting, possibly with a sparsity condition on coordinates of change (Aston and Kirch, 2014; Enikeeva and Harchaoui, 2014; Jirak, 2015; Cho and Fryzlewicz, 2015; Cho, 2016). November 15-4
Basic model Let X = (X 1,..., X n ) R p n have independent columns X t N p (µ t, σ 2 I p ). Assume there exist changepoints 1 z 1 < z 2 < < z ν n 1 such that µ zi +1 = = µ zi+1 =: µ (i), 0 i ν, where z 0 := 0 and z ν+1 := n. Writing θ (i) := µ (i) µ (i 1), 1 i ν, we assume k {1,..., p} s.t. θ (i) 0 k for 1 i ν. November 15-5
Further model assumptions Assume stationary run lengths satisfy 1 n min{z i+1 z i : 0 i ν} τ, and the magnitudes of mean changes are such that θ (i) 2 ϑ, 1 i ν. Let P(n, p, k, ν, ϑ, τ, σ 2 ) be the set of distributions of such X. November 15-6
Projection-based single changepoint estimation µ + W = X Let ν = 1, write z := z 1, θ := θ (1) and τ := n 1 min{z, n z}. For any a S p 1, a X t N(a µ t, σ 2 ). Hence a = θ/ θ 2 =: v maximises the magnitude of the difference in means between the two segments. November 15-7
CUSUM transformation Define CUSUM transformation T p,n : R p n R p (n 1) by [T (M)] j,t = [T p,n (M)] j,t := ( n t(n t) n r=t+1 M j,r t n t r=1 M j,r t ). T (µ) + T (W ) = T (X) A + E = T November 15-8
SVD of CUSUM transformation When ν = 1, we can compute A explicitly: t n(n t) A j,t = (n z)θ j, if t z =: (θγ ) j,t, n t nt zθ j, if t > z so the oracle projection direction is the leading left singular vector of the rank 1 matrix A. We could therefore consider estimating v by ˆv max,k argmaxṽ S p 1 (k) T ṽ 2, and indeed when n 6, with probability at least 1 4(p log n) 1/2, sin (ˆv max,k, v) 16 2σ k log(p log n). ϑ nτ November 15-9
A computationally efficient projection Computing the k-sparse leading left singular vector of a matrix is NP-hard (Tillmann and Pfetsch, 2014). However, max u S p 1 (k) u T 2 = max u S p 1 (k),w S u T w n 2 = max u S p 1,w S n 2, u 0 k uw, T = max M, T, M M where M := {M : M = 1, rk(m) = 1, nnzr(m) k}. For λ > 0, we therefore consider computing ˆM argmax M S 1 { T, M λ M 1 }, where S 1 := {M R p (n 1) : M 1}, using ADMM. We can then let ˆv be a leading left singular vector of ˆM. November 15-10
Alternative relaxation Let S 2 := {M R p (n 1) : M 2 1}. Then the simple dual formulation leads to M := soft(t, λ) soft(t, λ) 2 = argmax M S 2 { T, M λ M 1 }. Suppose ˆM argmax M S { T, M λ M 1 } for S = S1 or S = S 2 and let ˆv argmaxṽ S p 1 ˆM ṽ 2. If n 6 and λ 2σ log(p log n), then w.p. at least 1 4(p log n) 1/2, sin (ˆv, v) 32λ k τϑ n. November 15-11
Changepoint estimation after projection Input: X R p n, λ > 0. Step 1: Perform CUSUM transformation T T (X) Step 2: Find ˆM { } argmax M S T, M λ M 1 for S = S 1 or S 2 Step 3: Find ˆv argmaxṽ S p 1 ˆM ṽ 2. Step 4: Let ẑ argmax 1 t n 1 ˆv T t, where T t is the tth column of T, and set T max ˆv Tẑ Output: ẑ, T max November 15-12
Sample-splitting version performance Suppose σ > 0 is known and X P P(n, p, k, 1, ϑ, τ, σ 2 ). Let ẑ be the output of sample-splitting algorithm with input X, σ and λ := 2σ log(p log n). If n 6 is even and σ ϑτ k log(p log n) n 3 128, then with probability at least 1 4{p log(n/2)} 1/2 2/n, 1 32σ log n ẑ z n ϑ nτ. If log p = O(log n), ϑ n a, τ n b, k n c and a + b + c/2 < 1/2, then rate of convergence is o(n 1 2a b 2 +δ ) for all δ > 0. November 15-13
Multiple changepoint estimation inspect Wild binary segmentation scheme (Fryzlewicz, 2014) November 15-14
Multiple changepoint estimation inspect Wild binary segmentation scheme (Fryzlewicz, 2014) November 15-15
Multiple changepoint estimation inspect Wild binary segmentation scheme (Fryzlewicz, 2014) November 15-16
Example 0 5 10 15 20 25 30 500 1000 1500 2000 candidate changepoint location projected CUSUM statistics 3 6 8 11 14 16 19 22 26 28 31 33 35 39 43 47 48 52 53 56 60 63 67 71 73 77 80 83 86 88 91 93 96 98 101 105 109 113 114 118 120 121 125 126 129 133 134 136 139 140 142 146 149 152 156 158 161 163 165 168 171 174 176 180 181 185 188 190 193 196 200 202 206 210 214 218 221 222 225 229 231 234 238 239 240 241 244 246 248 252 255 259 262 265 268 269 272 275 278 279 281 284 286 289 291 294 296 300 304 308 311 314 317 320 324 326 330 334 336 339 341 345 349 351 354 356 360 364 368 369 371 372 376 380 381 385 388 391 394 396 400 402 406 410 411 415 417 420 423 424 427 429 432 435 438 440 442 446 448 451 452 456 458 461 463 464 468 469 473 476 478 480 483 485 486 490 491 495 496 498 501 504 506 508 511 512 513 517 518 522 524 528 532 534 536 540 543 545 549 552 556 560 562 564 565 569 571 572 575 579 581 582 585 589 591 594 595 599 600 603 604 606 610 612 616 618 621 625 628 629 632 634 637 639 643 647 651 655 656 659 660 662 666 667 669 671 674 677 679 683 687 689 693 696 698 702 703 707 708 709 710 714 718 721 724 727 730 733 734 738 739 742 745 747 751 755 758 762 766 768 769 773 775 779 782 784 785 789 792 794 797 800 802 805 807 811 815 816 818 821 823 827 829 832 836 838 842 843 847 850 853 855 856 859 863 866 868 871 874 878 880 881 885 887 891 892 896 897 901 902 905 909 910 912 914 915 918 921 924 928 929 933 937 939 942 943 945 948 951 954 956 959 962 966 968 969 970 972 976 978 979 981 984 988 992 994 998 1001 1002 1003 1007 1008 1009 1011 1012 1016 1017 1021 1022 1023 1027 1028 1029 1031 1035 1036 1039 1042 1045 1048 1051 1053 1057 1058 1059 1062 1065 1068 1072 1075 1077 1081 1085 1089 1093 1095 1098 1102 1104 1107 1109 1110 1111 1115 1117 1120 1124 1127 1129 1133 1137 1141 1145 1148 1150 1153 1155 1157 1161 1164 1165 1167 1171 1173 1176 1178 1179 1183 1184 1185 1189 1191 1192 1196 1197 1198 1201 1202 1206 1207 1209 1210 1212 1213 1217 1220 1224 1226 1229 1231 1233 1236 1240 1241 1245 1247 1250 1252 1256 1257 1260 1261 1265 1267 1270 1272 1273 1276 1278 1282 1286 1287 1291 1292 1294 1298 1302 1305 1308 1311 1313 1315 1316 1318 1321 1324 1326 1330 1331 1335 1339 1343 1344 1347 1350 1353 1355 1359 1363 1367 1370 1372 1375 1378 1379 1382 1385 1387 1389 1392 1395 1398 1401 1405 1406 1409 1411 1414 1416 1419 1423 1427 1431 1434 1435 1439 1441 1443 1447 1451 1455 1458 1461 1462 1463 1465 1468 1470 1474 1475 1477 1480 1482 1485 1489 1492 1494 1497 1502 1504 1505 1509 1510 1514 1516 1519 1523 1525 1529 1532 1534 1537 1540 1544 1547 1550 1552 1555 1558 1559 1562 1564 1568 1572 1576 1578 1582 1584 1587 1590 1592 1596 1598 1601 1603 1607 1611 1614 1618 1621 1625 1628 1631 1632 1636 1638 1642 1643 1647 1648 1649 1653 1656 1658 1659 1663 1664 1668 1671 1674 1676 1679 1683 1687 1689 1692 1694 1698 1700 1703 1706 1709 1712 1715 1719 1722 1724 1728 1729 1732 1733 1737 1741 1744 1748 1752 1756 1757 1761 1763 1767 1770 1772 1776 1779 1783 1785 1786 1790 1794 1796 1797 1801 1804 1807 1810 1813 1816 1819 1820 1824 1828 1831 1832 1836 1837 1840 1843 1847 1848 1849 1852 1853 1854 1855 1859 1860 1864 1868 1869 1873 1875 1877 1881 1883 1887 1888 1891 1893 1895 1899 1901 1903 1906 1908 1911 1912 1914 1916 1919 1923 1924 1927 1930 1932 1935 1938 1940 1944 1946 1949 1950 1953 1957 1960 1964 1966 1968 1972 1973 1975 1978 1982 1985 1988 1991 1994 1995 1997 0 5 10 15 20 25 30 500 500 1000 1500 1500 2000 nodes in binary segmentation algorithm peak of projected CUSUM November 15-17
S 1 or S 2? Angles (in degrees) between oracle projection direction v and estimated projection directions ˆv S1 (using S 1 ) and ˆv S2 (using S 2 ), for different choices of ϑ. ϑ 0.1 0.2 0.3 0.4 0.5 (ˆv S1, v) 80.3 63.1 51.6 39.4 28.6 (ˆv S2, v) 79.5 63.9 52.9 40.6 30.2 ϑ 0.6 0.7 0.8 0.9 1 (ˆv S1, v) 25.8 21.7 19.0 16.7 14.4 (ˆv S2, v) 27.3 23.4 20.4 18.0 15.6 November 15-18
Single changepoint simulations RMSE θ = (1, 2 1/2,..., k 1/2, 0,..., 0) R p. n p k z ϑ inspect dc sbs scan 1000 200 10 400 0.57 32.3 82.2 99.6 46.2 1000 200 14 400 0.41 97.2 274.5 215.7 218.1 1000 200 200 400 0.57 65.5 262.3 180.1 156.4 1000 500 10 400 0.57 48.2 125.7 181.4 106.1 1000 500 22 400 0.52 86.9 240.5 235.5 190.3 1000 500 500 400 0.89 24.5 106.4 96.8 22.5 1000 1000 10 400 0.57 48.6 118.6 185.4 149.4 1000 1000 32 400 0.62 58.7 143.9 171.4 151.3 1000 1000 1000 400 1.26 10.1 28.1 42.7 15.1 2000 200 10 800 0.35 126.3 327.5 293.9 221.1 2000 200 14 800 0.41 88.1 213.7 155.2 121.0 2000 200 200 800 0.57 57.6 221.3 155.1 60.9 2000 500 10 800 0.35 169.9 348.1 456.0 305.5 2000 500 22 800 0.33 195.2 578.4 511.8 535.9 2000 500 500 800 0.89 21.3 45.0 62.4 27.0 2000 1000 10 800 0.35 131.5 416.4 460.5 397.7 2000 1000 32 800 0.40 138.4 441.0 448.6 401.6 2000 1000 1000 800 1.26 6.7 30.8 33.7 13.8 November 15-19
Changepoint density estimates Left: (n, p, k, z, ϑ) = (2000, 1000, 32, 800, 0.40). Right: (n, p, k, z, ϑ) = (2000, 1000, 32, 800, 1.02). density 0.000 0.002 0.004 0.006 inspect dc sbs scan 0 500 1000 1500 2000 estimated changepoint location density 0.00 0.02 0.04 0.06 0.08 0.10 0.12 inspect dc sbs scan 700 750 800 850 900 950 estimated changepoint location November 15-20
Misspecified settings (n, p, k, z, ϑ) = (2000, 1000, 32, 800, 1.7). Model inspect dc sbs scan M unif 3.0 13.8 17.6 3.8 M exp 2.8 11.9 47.7 5.5 M cs,loc (0.2) 3.4 8.4 17.5 6.8 M cs,loc (0.5) 5.6 10.8 23.7 8.4 M cs (0.5) 1.5 7.5 14.2 3.5 M cs (0.9) 2.5 6.5 10.2 2.9 M temp (0.1) 4.0 16.9 96.2 10.1 M temp (0.3) 14.5 24.9 226.4 14.7 November 15-21
Multiple changepoint simulations n = 2000, p = 200, k = 40, z = (500, 1000, 1500). Writing ϑ (i) := θ (i) 2, set (ϑ (1), ϑ (2), ϑ (3) ) = ϑ(1, 1.5, 2). ˆν ϑ method Rand % best 0 1 2 3 4 5 0.63 inspect 0 0 10 73 14 3 0.91 46 dc 0 0 23 63 13 1 0.86 14 sbs 0 0 6 69 22 3 0.85 24 scan 0 0 65 33 2 0 0.79 16 0.51 inspect 0 0 23 50 22 5 0.82 52 dc 0 0 47 40 12 1 0.76 22 sbs 0 0 30 48 14 8 0.77 20 scan 0 0 94 6 0 0 0.71 7 0.38 inspect 0 0 48 42 10 0 0.77 55 dc 0 7 66 23 4 0 0.69 18 sbs 0 0 58 36 6 0 0.70 14 scan 0 11 88 1 0 0 0.68 26 November 15-22
Histograms of estimated changepoints n = 2000, p = 200, k = 40, z = (500, 1000, 1500), (ϑ (1), ϑ (2), ϑ (3) ) = (0.63, 0.95, 1.26), σ = 1. 0 0 frequency 5 10 20 30 frequency 5 10 20 30 0 500 1000 1500 2000 0 500 1000 1500 2000 changepoints estimated by inspect changepoints estimated by dc 0 0 frequency 5 10 20 30 frequency 5 10 20 30 0 500 1000 1500 2000 0 500 1000 1500 2000 changepoints estimated by sbs changepoints estimated by scan November 15-23
Summary inspect is a new method for high-dimensional changepoint estimation Convex relaxation used to find projection direction, then CUSUM and WBS to identify multiple changepoints R package InspectChangepoint available! November 15-24
References Aston, J. A. D. and Kirch, C. (2014) Change points in high dimensional settings. arxiv:1409.1771. Aue, A., Hörmann, S., Horváth, L. and Reimherr, M. (2009). Break detection in the covariance structure of multivariate time series models. Ann. Statist. 37, 4046 4087. Cho, H. (2016) Change-point detection in panel data via double CUSUM statistic. preprint. Cho, H. and Fryzlewicz, P. (2015) Multiple changepoint detection for high dimensional time series via sparsified binary segmentation. J. R. Stat. Soc. Ser. B, 77, 475 507. Enikeeva, F. and Harchaoui, Z. (2014) High-dimensional change-point detection with sparse alternatives. arxiv:1312.1900v2. Frick, K., Munk, A. and Sieling, H. (2014) Multiscale change point inference. J. R. Stat. Soc. Ser. B 76, 495 580. Fryzlewicz, P. (2014) Wild binary segmentation for multiple change-point detection. Ann. Statist., 42, 2243 2281. November 15-25
Horváth, L., Kokoszka, P. and Steinebach, J. (1999) Testing for changes in dependent observations with an application to temperature changes. J. Multi. Anal., 68, 96 199. Jirak, M. (2015) Uniform change point tests in high dimension. Ann. Statist., 43, 2451 2483. Killick, R., Fearnhead, P. and Eckley, I. A. (2012) Optimal detection of changepoints with a linear computational cost. J. Amer. Stat. Assoc. 107, 1590 1598. Kirch, C., Mushal, B. and Ombao, H. (2015) Detection of changes in multivariate time series with applications to EEG data. J. Amer. Statist. Assoc., 110, 1197 1216. Ombao, H., Von Sachs, R. and Guo, W. (2005) SLEX analysis of multivariate nonstationary time series. J. Amer. Statist. Assoc., 100, 519 531. Page, E. S. (1955) A test for a change in a parameter occurring at an unknown point. Biometrika, 42, 523 527. Tillmann, A. N. and Pfetsch M. E. (2014) The computational complexity of the restricted isometry property, the nullspace property, and related concepts in compressed sensing. IEEE Trans. Inform. Theory, 60, 1248 1259. Wang, T. and Samworth, R. J. (2016) High-dimensional changepoint estimation via sparse projection. http://arxiv.org/abs/1606.06246. November 15-26