Cover by Vincent Bumgarner

Safari, the world’s most comprehensive technology and business learning platform.

Find the exact information you need to solve a problem on the fly, or go deeper to master the technologies and skills you need to succeed

Start Free Trial

No credit card required

O'Reilly logo

Calculating top for a large time frame

One common problem is to find the top contributors out of some huge set of unique values. For instance, if you want to know what IP addresses are using the most bandwidth in a given day or week, you may have to keep track of the total of request sizes across millions of unique hosts to definitively answer this question. When using summary indexes, this means storing millions of events in the summary index, quickly defeating the point of summary indexes.

Just to illustrate, let's look at a simple set of data:

Time

1.1.1.1

2.2.2.2

3.3.3.3

4.4.4.4

5.5.5.5

6.6.6.6

12:00

99

100

100

100

  

13:00

99

 

100

100

100

 

14:00

99

100

 

101

100

 

15:00

99

 

99

100

100

 

16:00

99

100

  

100

100

Find the exact information you need to solve a problem on the fly, or go deeper to master the technologies and skills you need to succeed

Start Free Trial

No credit card required