Heavy mybinder.org load

[betatim]

Hi guys,

you are too popular :) This morning we noticed quite a lot of load on mybinder.org and tracked it down to nbinteract.

We track the number of binders launched (in a 10m window), which is plotted here:
https://grafana.mybinder.org/dashboard/db/1-overview?refresh=1m&orgId=1&panelId=17&fullscreen&from=1523872922006&to=1523916122006 basically all of that is nbinteract (after about 6pm Zurich time)

What seems to cause a lot of load for us isn't so much the total number of binders connected to nbinteract but the fact that you are launching a lot of them. Could you implement something that reuses a running binder when a user clicks on a different page or something?

@choldgraf @willingc

[betatim]

We just lowered the per repo limit from 200 to 100 to see if this will help us cope better with the load and increase the pod launch success rate.

Right now there are about 150 nbinteract binders running.

jupyterhub/mybinder.org-deploy#556

[cwilling]

An email of your initial "too popular" comment was sent to user cwilling as well as (or instead of) to user willingc. Maybe that was a typo (since edited), otherwise an interesting bug in the infrastructure.

[betatim]

(Yes I made a typo in the initial comment, meant to reach willingc not cwilling :) )

[betatim]

For the moment I banned nbinteract-image. Mostly to debug if this is the only source of load we have or if there are others. Tracking it in this PR: jupyterhub/mybinder.org-deploy#557

Let's see how much we find out by the time it is morning on the west coast and then re-decide enabling the repo or not.

[SamLau95]

Looks like the traffic spike went away after banning for a bit. @choldgraf and I think this might be related to jupyterhub/binderhub#317 but we haven't pinned the exact issue down.