Jesus, HN goes from zero to lynch mob faster than reddit these days.
Guy drops a zero day on a major service provider, guy gets his account suspended (temporarily, it turns out). In what possible world is disabling an account that has recently exploited your live product in a very visible way not ok? Remember, you don't have a chance to call a meeting with the C level guys and your community manager - you're one or two guys responding on a weekend.
The rest of the "oh my god the sky is falling" drivel about how terrible a bug it could have been and how they should never have had such a vulnerable bug in the first place is even worse. Security bugs are fuckups by nature - nobody sat and said well shit I was going to code this wrong but since it might allow a lot of access I won't. In terms of OH SHIT bugs this is actually rather small - I'm sure github's live infrastructure has been open to lower level remote execution vulnerabilities over the years - newsflash: we all have been. Getting user or superuser or db admin is going to almost certainly be a lot worse than an application authentication level vulnerability.
You say none of that matters because it's such an obvious bug and people have known not to do that kind of thing for years? Say hello to our old friends "buffer overflow" & "use after free" - still grabbing msft aapl & goog after all these years.
> and how they should never have had such a vulnerable bug in the first place is even worse.
Bugs happen. Even stupid oh-my-god-i-can't-believe-i-did-that bugs happen. And they happen to the best of us.
However, when someone reports a vulnerability about my code to me or I discover a problem myself, the very first thing I do is break out the grep. I grep the shit out of my code. Because I am a human being. I am a creature of habit. And if I screwed up in one place, I promise you, I did it in other places too.
The problem? Github didn't do that. At least, that's the impression from the information coming out. The guy reported the issue on Friday, they fixed that specific instance of the issue ... and it remained a problem in other places. That is unacceptable and unprofessional. They should have burned the midnight oil and made sure the same problem wasn't prevalent in other parts of the code.
Having said that, I am a loyal Github client and will remain so. Every service provider I use gets a once-a-year-screw-up credit. Github just used theirs. Switching because of one incident is premature and will be sure to cause regrets.
They should have burned the midnight oil and made sure the same problem wasn't prevalent in other parts of the code.
I understand that's the feeling here, but it's unrealistic. I've reported dozens of bugs to shops that ranged in size from 1 to borg. You simply never see a whole set of bugs fixed and pushed live over a weekend. Not even close. Exactly what company have you seen set this standard for professional? The smallest amount of time between notification and public disclosure I've seen get called responsible disclosure is a month. Many, many, many enterprises take six months or longer to push fixes (which is far too long but that's another whole discussion)
However, if such a problem was reported to LedgerSMB here is how we handle it:
1) Scope out the problem. What's affected? Are other related open source projects affected? How bad is it? This itself can take a bit of time. We do not rush this because we don't want a full disclosure when it happens to bring other problems into the fore.
2) Within a few days we let the reporter know what we have found and give them a chance to offer feedback.
3) Then we get everyone on the core team together and talk solutions. Once we implement it, we test and release a patch. Two weeks after the patch is released, we release a full disclosure along with a hat tip to the one who discovered it.
The whole process takes time. But the fact is that it's generally better to get a full fix out in two weeks than a partial fix out tomorrow.
So no, don't burn the midnight oil. More speed, less haste.
And this whole thing really doesn't look good for Github.
We didn't have the luxury of time. We were going back and forth with him over the weekend, and he pushed the commit 2 days after the original report. I admit, I was having some issues understanding his vague reports and went down a different rabbit hole early Sunday morning.
I definitely wanted to do a more careful audit and investigation, but my hands were tied.
I don't think anything will happen to Github or even Rails users. There user base will not likely decrease.It will increase infact.
More importantly people reading this forum are very scanty in the larger picture of people writing software around the world. Software and tools get used in 99% of cases for average programmers because of jobs,popularity and usability reasons.
I bet of all the Rails programmers, and Git hub users very few would have heard the news. And of those few who have heard it very few would understand the seriousness of the issue.
The only benefit that has come out of this, is a serious problem has been averted. If this vulnerability had been discovered by some body who makes his bread and butter cracking systems. He could have done a great deal of damage to a lot of systems.
Lets hope that Rails finds and fixes these sort of vulnerabilities. To avoid bigger problems in the future.
"The guy reported the issue on Friday, they fixed that specific instance of the issue ... and it remained a problem in other places."
If I have correctly understood the issue, Igor at first informed Rails, and this was the right thing to do. He was ignored by Rails and then he wanted to proof his view point by applying it to GitHub. How could GitHub inform other places? By contacting Rails, but they already downplayed the issue...
"What I want you to see in that thread I mentioned is the way the core team perceives this. You are not discovering anything unknown, we already know this stuff and we like attr protection to work the way it is."
After reading for how long he tried to bring attention to this and only got a top guy to say that kind of stuff. The guy who hacked is not right in any way but I don't even have words to describe the person who wrote the above line.
I'm not condemning anyone, but it seems that when someone points out that your framework ships Insecure By Default code with absolutely no warning in the generated code, you'd take that seriously.
Instead, the thread is full of "We've discussed this before, we like it the way it is" and "Rails is not responsible here."
Man, I've just read through this whole incident and this is the exact same conclusion I got. Fair enough, he got his account suspended, they investigated and they reinstated his account. But the comments from the devs and admin is just disgusting, acting like this is a non-issue and calling the guy a troll. This is not the kind of attitude to have. I understand the homakov probably could have acted a bit more "professional", but not surprised he acted this way after the way he was belittled. He honestly comes across very shocked he was able to do what he did, and from the comments its obvious this was made a bigger deal by the users than it was by him.
I am not a rails developer, but I am pretty sure this would have woken up a lot of people for a lot of similar issues, there will be a mad flurry of devs rechecking a whole bunch of code. Had he reported this disacreetly what are the chances of this much publicity have been generated?!?
I thought there were two different issues, though not being a rails jockey I could easily be mistaken. Even if it was disclosed four days before it was exploited I stand by my take if not my terminology.
Nope, this bug is exactly what he used for his demonstration (and there are warnings about attr_accessible going back 3 or 4 years, so it's not a "0-day" by most accounts, more of a "3-years"" vuln)
You are not getting the point about this situation.
The rants that are coming in are not about what has happened, but what could have happened.
Imagine a situation if somebody had deleted all the data or worse committed malicious code to important repos. And then used it create a bigger mess later it would have been disastrous.
If this can't be taken seriously I don't know what can be.
What exactly would you require to have happened to demonstrate that github was taking the problem "seriously" enough?
Should they have shut down github.com entirely. Tweeted in all caps every 15 minutes until they fixed the problem? Called a televised press conference? What?
I'm just pointing out the seriousness of the situation.
Punishing Egor Homakov in this case is a classic example of 'No good deed goes unpunished'. Had this vulnerability been found by somebody with evil on his mind. We would be having a very different debate now.
We all make mistakes. And I don't really blame Github for this. But we must at no cost downplay this incident.
I was on Egor's end of a similar incident back in my more impulsive years. I wasn't treated quite as well as he was but even so I eventually came around to the realization that being overly confrontational is never the right way to gain attention for security issues (even though it did result in the issue being fixed rather quickly).
The LedgerSMB project started in a similar shitstorm. I found an ability to forge credentials in SQL-Ledger. I submitted it. I went back and tried a month later on a new version (no communication from the SL author) and it was a little harder but not too hard. I exploited again, sent another email, was told to bug off.....
I tried to get the issue fixed for six months. I finally gave up and forked. When we forked we issued a security advisory publicly and stated we would offer a full disclosure soon. That's when the shitstorm started in earnest. I was accused of fearmongering. I was told I didn't understand security, that the software was plenty secure, and many choice lines that out of professionalism I will refrain from reposting to this forum.
Dozens of emails.
The end result was that Dieter fixed SQL-Ledger shortly after the fork, because those who stayed behind made him. It would not have been fixed without the fork.
After reading all this including your story,Personally I will never report any security or other incident to anybody(If I happen to find one).
Because no good deed goes unpunished. No one appreciates what good caused by your help.
It just bruises peoples egos and they violently lynch you for 'How dare you point a mistake at a genius like me, you should have tried whispering in my year'.
Its for incidents like this people refuse to help in not just security situations but also in emergency situations because you get entangled in unnecessary mess. People just let the world burn.
"Houses aren't very secure, here's a video of me picking the lock on my own front door."
"I demonstrated how insecure your house is by picking the front door lock and leaving a note on your bed."
Sometimes it can be difficult to have the empathy and perspective to see how frightening and unconscionable the 2nd action can be, but it very much is.
"But thats exactly why I left you that note. Because it frightens me just how insecure your house is. I care about you and don't want to see you hurt. I did it as a last resort, I tried to inform you but you clearly didn't take me seriously.
Empathy was casusing me pain everytime I saw you 'lock' your door with that elastic band. Attention seeking or malicious behaviour would have been to break into all the insecure doors on the street.
I broke into yours, so you would take security seriously, because I care about you and your wellbeing."
How is that so very different from, say, kidnapping someone's children and holding them hostage until they fix whatever you want fixed?
The problem here is that when you violate someone's trust you change the landscape. People get scared, they question your motives, they go into a fight or flight response. Yes, this sometimes results in the problem being fixed faster because they are very much more motivated now, but the same is true if you kidnap their family, right?
If you think someone is letting down their customers by not responding fast enough, then you go public. But violating trust is a quick way to end a professional relationship.
Surely you and every other commenter can see that breaking into houses and kidnapping children are in a completely different universe from posting comments sent from 1000 years in the future on Github right?
> Guy drops a zero day on a major service provider
This was anything but a zero-day, and that's the whole reason people are mad. Homakov very clearly made an effort to get this patched before exploiting it before taking action himself. They're not mad that it wasn't fixed immediately on Sunday; they're mad that, on Sunday, there was still a problem in the first place. Big difference.
Guy drops a zero day on a major service provider, guy gets his account suspended (temporarily, it turns out). In what possible world is disabling an account that has recently exploited your live product in a very visible way not ok? Remember, you don't have a chance to call a meeting with the C level guys and your community manager - you're one or two guys responding on a weekend.
The rest of the "oh my god the sky is falling" drivel about how terrible a bug it could have been and how they should never have had such a vulnerable bug in the first place is even worse. Security bugs are fuckups by nature - nobody sat and said well shit I was going to code this wrong but since it might allow a lot of access I won't. In terms of OH SHIT bugs this is actually rather small - I'm sure github's live infrastructure has been open to lower level remote execution vulnerabilities over the years - newsflash: we all have been. Getting user or superuser or db admin is going to almost certainly be a lot worse than an application authentication level vulnerability.
You say none of that matters because it's such an obvious bug and people have known not to do that kind of thing for years? Say hello to our old friends "buffer overflow" & "use after free" - still grabbing msft aapl & goog after all these years.
TL;DR - stop acting like children.