Reduced the amount of go-routines. Now sorting
is done in the same go-routine as URLs are fetched.
Then these sorted lists are merged one by one
in a loop as they arrive.
This greatly reduces complexity.
Previously we sorted a list e.g. [3,2,1]
that we already got, then fetched another
list e.g. [6,7,8,2]. Instead of just
sorting the new list and manually merging it
we would sort the appended lists:
[1,2,3, 6,7,8,2]
Despite mergesort, this is now optimized in
a two-step process:
* ONLY sort the new list
* call a Merge function on both lists, that assumes
sorted and deduplicated lists
This should be faster.
* First, lower the MaxResponseTime which is used
in the Handler to cancel the running goroutines
and finalize the result. Since we want to stay below 500ms
we cannot pick 500ms, because the finalization steps
take some time too.
* Second, set WriteTimeout to 500ms in order to make sure
the handler never exceeds that timeframe.
Julian Ospald