Log4j Vulnerability: How Two MSPs Reacted To The Massive Exploit

‘It’s one of the challenges with these kinds of vulnerabilities when they’re so critical,’ he says. ‘The community moves very rapidly to patch the open-source project but you often don’t get a perfect patch. There’s still more testing going on but it’s really important for businesses to really keep pace as these fixes come out because they’re still fairly critical,’ says Eric Walk, director of data solutions at Perficient, a St. Louis-based solution provider.

Dustin Bolander’s first thought was, “Oh God. Here we go again.”

The flaw in the Java logging library Apache Log4j sat undiscovered for years and was identified earlier this month, sparking panic among vendors and solution providers about what data could be compromised and how fast it could be stolen from them.

The open-source library used by everyone from Apache and Apple to Minecraft and Twitter gave cybercriminals an enormous attack surface to cause widespread global disruption all through a single line of text. The ripple effects of this vulnerability will be felt for months.

“This has just been a rough year for security,” said Bolander, who is the owner and CEO of Clear Guidance Partners, an Austin, Texas-based MSP. “It feels like a new one of these every couple of months.”

At first he didn’t see any impact from the vulnerability but soon realized that every software vendor that he uses more than likely had some kind of exposure. To check their own systems, his team initially shut down its own server and spent 10 days plugging in tools and testing everything to make sure everything was tightly secured.

Eric Walk, director of data solutions at Perficient, a St. Louis-based solution provider, took a similar approach.

The first step is to always work from the inside out, he said. Perficient’s internal teams started checking its internal systems and then began to patch things. They then looked outward to its project teams, identified any places where there may have been issues and then worked with its clients to patch their environments as well.

[Related: Huntress’ John Hammond: Log4j Could Have Been ‘Ransomware Armageddon’]

“Working to help clients who had the most vulnerable and most at-risk systems get patched within days, as opposed to weeks, was really a key focus last week,” Walk told CRN.

With another Log4j vulnerability discovered last Saturday, Walk said he spent all weekend making patches.

“It’s one of the challenges with these kinds of vulnerabilities when they’re so critical,” he said. “The community moves very rapidly to patch the open-source project but you often don’t get a perfect patch. There’s still more testing going on but it’s really important for businesses to really keep pace as these fixes come out, because they’re still fairly critical.”

The key takeaway, he said, is to be ready at any time and to get the patches done as quickly as possible. But it may be more difficult when some of that code is written by vendors, which could take longer.

“This is evolving rapidly and more research is being done to understand the vulnerability and identify additional problems that also need to be fixed,” Walk said.

The biggest message he can tell others is to monitor, and to never stop monitoring. The National Vulnerability Database is also a good tool, he said, as they send alerts when risk levels change and when different updates are available.

“In situations like this in particular, it was disclosed very publicly,” he said. “There wasn’t a good way for the vulnerability to be disclosed quietly to the impacted individuals to allow them time to fix it before it was made public. Because this is such a commonly used open-source library, it went public very early on and so there was a scramble.”

“The reason that the original vulnerability was risky was because it’s so easy to exploit,” he added.

When Bolander spoke to his end customers, it was clear that they were all scared.

“They’re just all panicking,” he said.

For Walk’s customers, their biggest concerns were how quickly everything could be patched and secured and how fast the vendors could get the fixes out.

“The smaller vendors move faster, the bigger vendors move slower and that‘s one of the biggest challenges that a lot of clients have been facing...staying on top of the vendors making sure they’re getting alerts,” he said.

Bolander believes many of the global vendors did a good job communicating and patching on their end.

“ConnectWise actually did a really good job communicating on this,” he told CRN. “They were sending out emails right away. We’re very much at the mercy of third-party vendors on this one.”

Going forward when this happens again, Walk said it’s critical to invest in your DevOps chain to make sure it’s all integrated and automated and there’s the ability to scan all repositories and identify what‘s vulnerable.

But the lesson he doesn’t want people to take away is that open-source software is bad.

“Open-source software is good. The community is good,” he said. “It‘s important that we nurture that community and I think it’s really important to not overreact on that front.

“Always assume the bad guy is in the building,” Walk added.

Bolander doesn’t know if it will get worse before it gets better, at the same time he doesn’t know if it can get any worse. But it’s important, he said, to communicate with everyone from vendors to clients and to take accountability if you were compromised. And also, to hold vendors accountable on their own security practices.

“MSPs can’t just be complaining on Reddit all day about it, you actually have to twist the vendor’s arm a little bit on it,” he said.