[00:00:00] Nathan Wrigley: Welcome to the Jukebox podcast from WP Tavern. My name is Nathan Wrigley.

Jukebox is a podcast which is dedicated to all things WordPress. The people, the events, the plugins, the blocks, the themes, and in this case security on the internet.

If you’d like to subscribe to the podcast, you can do that by searching for WP Tavern in your podcast player of choice, or go to WPTavern.com forward slash feed forward slash podcast. And you can copy and paste that URL into most podcast players.

If you have a topic that you’d like us to feature on the podcast, I’m keen to hear from you, and hopefully get you or your idea featured in the show. Head to WPTavern.com forward slash contact forward slash jukebox, and use the form there.

So on the podcast today we have Robert Rowley. Robert is Patchedstack’s security advocate, where his time is spent interacting with open source communities to share the word about security best practices. Given his background, the podcast today is all about internet security.

We start off with a topic which is very much in the news at the moment, the LastPass security breach.

If you’re a user of LastPass then you’ll know what their service is. But if you’re not, here’s a quick introduction. LastPass is a password manager. It will lock up your passwords and any other data for that matter, in a secure vault which can only be read if you decrypt it with the correct password.

Towards the end of 2022 LastPass announced in a series of blog posts that their customer vaults had been taken from their cloud storage. The way that this was communicated, left, many of their customers questioning their use of the service and whether they could now trust LastPass with their data.

Robert’s explains how the incident occurred and if you should be concerned. The answer is, as you might expect, it depends. There are situations in which the settings that you had in your LastPass account might mean that you need to act sooner rather than later. The length and complexity of your master password is also a key factor.

This then leads to a conversation about the broader issue of website security and the security of WordPress websites in particular. What are some of the considerations that you need to think about when protecting your website, and how can you communicate these considerations to your clients?

Towards the end of the podcast, we chat about a project that Robert’s been involved in during 2022. He’s been patching plugins which are no longer being maintained, but are still being used, so that they present less of a security threat to their users.

If you’re curious about website security, then this is a podcast for you.

If you’re interested in finding out more, you can find all of the links in the show notes by heading to WPTavern.com forward slash podcast. And you’ll find all of the other episodes there as well.

And so, without further delay, I bring you Robert Rowley.

I am joined on the podcast today by Robert Rowley. Hello Robert.

[00:03:54] Robert Rowley: Hello Nathan. How are you?

[00:03:55] Nathan Wrigley: Good, thank you. Lovely to have you on. Robert and I have actually met in person. We’ve done a variety of different podcast episodes before and, he’s here today to talk to the wheelhouse that he’s in, which is internet security.

Given that very brief introduction, Robert, I wonder if you wouldn’t mind just painting a picture for our audience of who you are, what company you currently work for, and what your background is in internet security. And if there’s a reference there to WordPress, include that as well.

[00:04:24] Robert Rowley: No problem, yeah. So I’m Robert Rowley and I am currently working as the security advocate for Patchstack. Patchstack is a WordPress plugin security company. We have a plugin and we have a bunch of services offered for the WordPress and open source communities right now.

I got started in the information security industry, I guess, in open source community, probably 20 years ago using Linux. I was using WordPress, one of the first releases. Not the first, but once it started to become popular in the early mid aughts, I guess is what they’re called. And yeah, I’ve worked professionally for hosting providers, securing and protecting the customer websites. That was my focus when I was working at hosting providers. I was doing a ton of hacked site cleanups.

And, I’ve also worked in the opposite end, instead of defending and protecting, I’ve worked in the attack arena, where I’ve worked for pen testing companies, PCI auditing companies and things like that. Where we were validating, doing security tests, to validate that our customers had a reasonable amount of security for their websites and their businesses.

So with Patchstack and my role as a security advocate, I like to blend the two. Patchstack is focused on protecting and securing the customer’s environment, and my role as an advocate, I really get to speak to people about how corporate or enterprise level security, a higher level security, really works. Which the WordPress ecosystem could really benefit from I would say. There’s a whole lot of security hygiene and best practices that are kind of skipped over or glossed over. Mostly because people aren’t asking for them. That’s basically the word that I try to spread.

[00:05:54] Nathan Wrigley: Thank you so much. We’re going to start our conversation in a somewhat unexpected and slightly time-bound way. We are recording this at the beginning of 2023. It may well air a few weeks after the recording. But over the Christmas period some news came to light, which is going to begin our conversation, and then we’ll go off in all sorts =of different tangents.

But the news is surrounding a, a very popular password manager called LastPass. And LastPass really holds whatever data you wish to throw at it. And the promise is that it will hold that data in an encrypted form, which is only readable by you. Now that’s great, except just prior to Christmas, just prior to the Christmas holidays, there was an announcement on the LastPass website, which indicated that their data had been breached.

Now, that isn’t to say that the data had been decrypted, or at least we don’t know the status of that decryption. But the blob of data which holds your encrypted information has been taken, and there’s been a real ground swell of concern around this issue. And so the conversation that we’re going to have is going to be beginning all about password sanity checking, and hygiene and all of those kind of things.

Do you just want to give us a bit of a backstory on what’s been going on over at LastPass? Obviously important to anybody managing passwords in LastPass, but it also might be interesting more broadly for audience members.

[00:07:26] Robert Rowley: Yeah, I’ll try to hit it from multiple angles. So LastPass had this breach that they announced in December, but it wasn’t initially first announced in December. It started in the summer. They had announced the first indicators that they had signs of a breach had happened to their systems, back in August or earlier than that maybe. But, I think it was somewhere in the summer of 2022.

Well, the big worry is with LastPass, what is it that they have that is of value or of risk to an individual who is a user? And the whole purpose of LastPass is that you have all your secrets, right? Your passwords, your credit card numbers, SSL certificates, all these really highly sensitive things. LastPass was offering a service that said, you can store those with us, we are a cloud service provider, and we will encrypt those using zero trust systems. Zero trust means is that they are not going to be able to decrypt it unless they know what your password is. And they’re going to store all of your stuff in an encrypted way that nobody there at LastPass should ever be able to decrypt it.

However, as we are now aware since the story began in the summer and ended in the winter, to this day, or at least to this point. That there was a breach. They had access to the source code. At first, the attackers had access to the source code. Then we learned later the attackers had access to a developer’s machine and that developer’s machine had access to these cloud storage drives. But nothing, you know, no customer data yet.

And it wasn’t until it was December 22nd, that is when LastPass updated. So just three days before Christmas, they said, oh, by the way, also all of our customers stored encrypted vaults were also exfiltrated by the attackers. I shouldn’t say all, they just said the customer vaults were exfiltrated by these attackers.

Now that’s going to be a lot of data to move, and it’s not useful to the attackers until they can get people’s master passwords. But since the attackers have access to the source code and they have access to the encrypted vaults, it’s just a matter of brute forcing these passwords.

So if you use LastPass and you have a very weak password, maybe your master password with something like the word password, uh, which is a terrible idea, that’s a really bad security hygiene. That would be brute forced in probably a matter of seconds.

If you had a strong master password, which I’m sure LastPass encouraged users to do, then it may take years for that master password to be bruteforce, to unlock the encrypted vault that contains all of your passwords. Which puts everybody at a weird position. If you’re a LastPass user, you’re now aware as of December 22nd that the encrypted vault that stored your passwords, but really a lot of LastPass user’s passwords has been leaked and could be, at any point in time the attackers could be beginning to do the work to attempt to brute force these vaults.

And as they become successful, they’ll have access. They’ll know the url, right? They’ll know your username very likely. They’ll know everything that was stored in your vault. So it puts people at a high risk. But there’s this big thing of a matter of time. And it’s going to take a good amount of time, depending on the strength of your password versus the strength of the computers the attackers can use to attempt to brute force these vaults.

[00:10:30] Nathan Wrigley: I’m just going to add a little bit of context and forgive me, Robert, if I say something which is factually incorrect, please alert me and I will backtrack. But my understanding is that LastPass in effect rolls up all of your data into one giant blob. I’m imagining it in my mind as like a football.

So all of the passwords, all of the credit card details are, if you like, poured into this football and the football is encrypted. But the point is everything is inside that one blob. And so if the attackers decrypt one thing they have decrypted all the things. So it’s not like there’s a password connected to this website over here and this credit card number over here.

As soon as they’ve bruteforced it, and figured out way to get in, every single item inside that vault is now available in plain text. Is that, for a start, is that true? Once they’ve got something they’ve got everything?

[00:11:29] Robert Rowley: I believe that’s true because that one thing that the attackers need to get is your master password. Your master password is the secret that LastPass is not aware of on their end, which they used to hold true for their marketing spiel which is zero knowledge, right? Like they can’t decrypt your passwords unless they have your master password.

So that football or that blob of encrypted data, once the attackers are able to brute force, and that blob is unique to every user, because every blob is encrypted with each user’s master password. But the only secret that needs to be gotten, you know, brute forced is that master password.

[00:12:02] Nathan Wrigley: I suppose it raises all sorts of really interesting concerns because the promise was that it was encrypted and there’s zero insight from LastPass. If you lose your master password, there’s no point in going to Lastpass support and saying, well I’ve lost my password. Can you please send it in an email, and I’ll be trouble free. If you lose that, that’s tough.

But it’s the nature of what’s inside that vault. So, if it was just a handful of passwords. If you were a user of the internet fairly infrequently, and you were just logging onto a, couple of websites, your email and what have you. Then you could quickly go around and sweep up all of those websites and change the passwords and you know that you’re fine.

But I think a lot of people using services like LastPass have gone all in. And so, as we said, credit card numbers, mortgage details, pension details, bank account details, credit card numbers, all of these things have gone in there. And so the worry now is that if that is retrieved, then all of that is available.

And the problem is you can’t go to LastPass and simply change your password. All you are doing is changing the current blob’s password. The ship has already sailed there. The hackers who’ve got this, they have it, and if they figure out your password, no matter how many times you change it, what settings you fiddle within in LastPass. If they get through your password the day it was stolen, then everything is up for grabs.

It’s just the wealth of things that must be in there. So in my case, I’m a LastPass user, I have a paid account. There’s an awful lot of things that I would really wish didn’t escape. So, financial things and so on and so forth. But imagine across the population of the, I don’t know how many users they had, let’s imagine it’s hundreds of thousands, possibly millions, I don’t know. There’s a trove of information. So there must be a giant incentive for the hackers to get to work and figure out these passwords, one at a time. Even if that’s what it takes.

[00:14:07] Robert Rowley: You’re absolutely right There’s a difference in the data that was stored in LastPass. You made a great point there where, a password can be changed. A password can be updated and it’s no longer a threat if the old password is leaked, or compromised. But information like privately or personally identifying information, privacy things that were stored in LastPass, those are going to be a much harder thing, right?

It’s hard to change your mortgage information, right? It’s hard to change your, your address. If your address is stored somewhere in there. But there will be certainly things that are extraordinarily difficult for people to get rotated or changed out. Here in the US, right a social security number, things like that. If that gets leaked, then it’s really a big pain to deal with identity theft the rest of your life.

[00:14:47] Nathan Wrigley: Yeah, and I guess to be fair, we did realize, I mean the technology was explained. There is this blob, we don’t know the password. If the blob were to be stolen then the level of complexity that that password had will be crucial to whether it’s decrypted or not. As you say, if the password is the word password, a few seconds will pass.

Do you have any insight from a technological point of view in terms of the power that computers can bring to bear trying to decrypt these? I’m guessing it’s brute force. It’s literally just trying a password. No, move on. Try another one. Trying it over and over and over again.

Let’s imagine that we had a, let’s say it was a ten digit password of just pseudo random nonsense. You know, just some characters and some strange punctuation. It’s unintelligible, it’s not a dictionary word. Can we be fairly sanguine that we’re still talking decades, possibly hundreds of years for computers to be able to brute force this, or do we need to be concerned?

[00:15:48] Robert Rowley: You should always be concerned, but not overly concerned. This isn’t an emergency. You know, unless your password was password, then it’s an emergency. But you shouldn’t be too concerned if you have a decent password hygiene, right? Do you choose decently strong passwords? They’re gibberish or they’re things that, you know, don’t use things like your name or a birth date that’s important to you, like the year numbers. Things like that.

As long as you’re not using something that’s very common and you are using a properly gibberish one, it works. And part of this thing is, and I’ll steel man LastPass a bit, because they did the technology right, they did it to the best they could do.

They tell people that your master password was hashed over 100,000 times, in newer versions after certain releases. Which means that a computer to try to guess that password would have to run this hashing algorithm over 100,000 times.

It’s 100,100 times exactly. And that just takes time. That means they did something intentionally that slows down the process if you are randomly trying to guess the password. Which buys the user’s time to rotate out their passwords and take appropriate action in response to the incident.

And as far as it goes for enter, even enterprise level security, that’s the best you can offer in relation to stored secrets that get leaked. You say you’ve provided sufficient time for the reaction to, uh, take place before the attackers are able to decrypt the vaults.

[00:17:09] Nathan Wrigley: So given Moore’s law, I should probably explain. Moore’s Law basically says computers only ever get better, they never get worse, and they get better quite rapidly. If we were to look at a computer from 20 years ago and ask it to sort of hash passwords, or rather brute force passwords. It would be able to do that at a significantly slower rate than computers of today can.

And my understanding is that things like GPUs have been repurposed, and essentially those pieces of hardware can do this work significantly quicker. So given Moore’s law, and the fact that this trove, this vault, this football is in somebody’s possession probably for the rest of time. Do you think that there is going to be concern enough that you should now be really, at breakneck speed, starting to change the passwords that you had in the LastPass vault?

Because that’s really the only mitigation here. If your blob is accessed and the, the information leaks out, and the hackers get inside. If you’ve changed all the passwords, well it didn’t matter anyway. But from what you are saying the grade of security that was applied by LastPass, at the minute, still holds up. But do you have any insight into how long we can be quite so cavalier?

[00:18:27] Robert Rowley: I’ll go back. I’ve been working in security and security related fields for, oh for about 20 years now, and I do remember 20 years ago they were talking about certain encryption algorithms being unbreakable. And they would do these mathematical calculations and they’d say if you encrypt it using this size key, using this algorithm, it will take you 20 or 50 or a hundred years, right, depending on the length of your password to decrypt the vault.

This wasn’t specific to LastPass, but this was just like back then what we were talking about. And then five years later, well, it stops being 5, 10, 100 years. It starts being 1, 3, 10 years. And then five years after that, in 2010 or so, that same algorithm with the same size key is decryptable within one year. It’s always this thing, as computers get faster, and you’re right, GPUs change the game. GPUs are able to fire off multiple concurrent threads to attempt the same brute force in quick succession. And that basically changes the whole game in certain algorithms where it will reduce over time.

So today’s numbers, and I’ve seen some posted around online, if you have like an eight character password that’s properly gibberish, it might take 20 or 30 years, but that’s just today’s numbers. And the tactics used, you mentioned GPUs. I have a GPU system at my house that I’ve turned on. Hashcat is the application. It works really quick to attack passwords, and break down passwords.

What you need to know is again, yeah, knowing the time you have. The lead time you have is useful for that moment, and it really is helpful towards how fast you should respond. So a lot of that is, in the enterprise security world, it’s the time to response, right? How much time do you have to take action in response to a threat or a compromise?

And what we’re talking about right now with LastPass, assuming you had a decent password as your master password, it’s a number of years, most likely. And also assuming that there are no, how to say, exotic computer systems that the attackers have access to that can crack these passwords much faster than what the current knowledge is.

As long as they don’t have exotic systems and as long as your password was sufficiently strong, you probably have some time to rotate your passwords. But really we should be talking about, hygiene, general hygiene. You should be rotating your passwords anyways. LastPass, like again to steel man them a bit, even though I don’t use them anymore. They have a feature within the app that will rotate the passwords for you. And you can set it on a schedule and just say, hey, rotate the password for this web app every six months. And they’ll just handle that for you. Doing this rotation of passwords is very useful.

[00:20:54] Nathan Wrigley: I’m confused by how that would work in the sense that, so let’s say for example Gmail or some sort of Google property. If it’s going to rotate the passwords for me, presumably it is mimicking my login. It goes to the website, it puts in the username and passwords, and they’ve got some mechanism for navigating to the page where the password is changed and they’ll substitute in the. How would that work, for example, if I’ve got 2FA, so let’s say I’ve got an authenticator app or something. Presumably at that point it’s going to be stifled and it won’t work.

[00:21:25] Robert Rowley: Well, you’re thinking that it’s going to log in for you, but this was a feature of the browser extension, as I remember it. It was something you’d go into the browser extension and set to change passwords for certain web apps. And I’ve said enough nice things about LastPass that I should say something bad.

This feature rarely worked correctly for me. It wouldn’t update very often. Sometimes it would update and then it would have the wrong information in LastPass. Like the password they updated in LastPass did not match what was on the website, so it would lock me out of it, and I’d have to go through my password history to find the correct one.

But the idea is there is that they knew rotation of passwords is a, it’s a security best practice. It’s a hygiene thing. You should be doing it every so often any ways. Just like not reusing passwords is something that is also very much good security hygiene, and good security best practice.

And again, the reason why, how it would work is it would basically, while your browser is currently logged into that web app, it just hijacks your browser and makes a request, to send the update password.

[00:22:17] Nathan Wrigley: Got it, okay. Given everything that we’ve talked about, the bottom line with all of this is the password hygiene. And so we’re recording this, like I said, beginning of 2022. Caveat emptor, if you listen to this in six months time or a year’s time, everything that we’re talking about could have changed.

Maybe the news has been updated. Maybe there’s been some miracle of hacking and they’ve managed to brute force all the passwords. Who knows? But given where we are now, could you just talk us through, so this is nothing to do with LastPass, this is just general website, internet password hygiene. What are your sort of recommendations in terms of how long they should be, how dictionary based they should be. Whether you’ve got a technique for coming up with passwords by appending things to the beginning or the end.

In other words, making it more memorable to you. Just lay out what your best advice is for a typical user. Not somebody who’s really obsessing about all this, all the time.

[00:23:13] Robert Rowley: I’ll try to step it up slowly. The average user, right? The person who doesn’t want to be bothered too much by security best practices, they find them annoying and difficult. I know this because I’ve interacted with a lot of these people in my career. For you, and I have somebody in my mind right now. Just use a unique password on every website that you visit. And ideally, if you’re repeating the same password like I love pancakes, right, every account you have. When one of those accounts gets compromised, then all of your accounts will end up getting compromised.

It’s not a good thing. And it should be something strong. It should not be your last name. It shouldn’t be the year you were born. It shouldn’t it be anything that’s guessable or even your address of your business or your location, because that information is semi-public and an attacker could start guessing, right? They can feed in this information into a bot that’ll try to rebuild the dictionaries against you. So it should be unique for every website and strong.

Now, if that’s a little bit too hard for you, this is what I used to say, then use a password generating tool like LastPass. That would create new passwords for you, you know, at least 12 or 16 characters long, and it’ll store the password for you. So you never need to really remember it yourself. Now, of course, LastPass is kind of confusing if we should recommend it anymore, but there are other options. There is 1Password. There is Bitwarden. There’s a bunch of options.

[00:24:37] Nathan Wrigley: There’s one called Dashlane if memory serves.

[00:24:39] Robert Rowley: Yes. Dashlane is another one. They vary in price. Some are free, some are open source. I believe Bitwarden is a great example of a free or very affordable option, which is really similar to LastPass, especially how LastPass used to be. You store your data in the cloud, so yes, the same risk is present. Whereas they could get their cloud services compromised, and then you’re going to have to go through the same, rotate all your passwords process.

Or you could go a little bit more hardcore and start storing your passwords locally. This means it’s going to be stored on your laptop or your PC, maybe on a USB drive, but you’re going to have to choose a piece of software that does that. A good option for that would be KeyPass or KeyPass SX. They’ve got a few versions of it. They all use the same underlying technology. It’s mostly an interface to access this vault. But the vault always exists on systems you own and you control. That’s the only way you can get outside of that, the realm of risk. Or you can alleviate and reduce the risk of somebody breaking into a cloud service provider and stealing all the passwords.

With all that said, that’s the basics, right? And if you’re a basics user and you’re a little, but you’re a little bit more than a basics user, right? And you got a more serious account, maybe it’s your banking account or your Amazon EC2 accounts. Well then you need to use something more like a two factor authentication, like a second factor. It could be your email or sms, like your cell phone number. Or it could be something stronger like a Fido key, which is like Yubikey. There’s a few other vendors that make these physical hardware keys that punch out random gibberish.

Or it could be what we’re really familiar with is this Google Authenticator. This is a time-based token, and it’s a one-time token for this little 30 second period of time, and it’s about a six, sometimes eight character pin. And those things, those require you to have physical access to a phone, right? That you run the Google app on or the Google Authenticator app on, or there are alternatives.

LastPass has an alternative for it. Authy is another very popular one. The big difference between, I’ll compare Authy to Google Authenticator, is that Google Authenticator, if your phone dies, there’s really no way to restore those secret tokens, right? Or if you upgrade your phone and delete, get rid of the old one. It’s really hard to upgrade and move it to the new phone. You have to do an intentional process where you do it an export first, and then you later import it.

Authy on the on the other hand, does cloud-based storage, and one of those benefits of cloud-based storage is it’s easy to share between devices. So with Authy, you’re able to set up one device or one account. And then if you lose your phone or the phone gets destroyed, you can easily reset up and get all your old two-factor authentication tokens, working easily and quickly.

[00:27:12] Nathan Wrigley: In the case of this breach that we’re talking about, the two factor authentication, if you have that enabled or quickly go and enable that, that really does put a bit of a roadblock in the hackers path. Because even if they get your password, username and all of that good stuff, they’re going to be hitting this barrier of being asked to perform another action.

So they’ll be able to successfully partially log in, but then they’ll be required to, I don’t know, either push a button on a mobile phone or press a button on a Yubikey or a Fido key or whatever. And that’s going stop them in their tracks. And also commonly with that, you get an email alert, assuming they haven’t got into your email, which is probably the first thing to shore up. You’ll get an email saying, look, something peculiar is going on. You need to be looking at your whatever, I don’t know, Dropbox or whatever account it is. So that’s another layer of security, which really would help.

[00:28:08] Robert Rowley: Yeah, absolutely, yeah. There’s a great point there on why two factor authentication may protect. And you should have two factor authentication on, even if you’re a basic user. You should have that set up for some of your more important accounts. If your password gets leaked, they’re still not going to be able to get into your, to your systems.

[00:28:23] Nathan Wrigley: In the case of the listenership to this podcast, I would imagine there’s quite a lot of people who are using password managers, and they are using it for their client websites. So I don’t know, you’ve got a hundred client websites. And all of a sudden you are facing this jeopardy that your business, not just your personal details, but your business is in some kind of danger, because the last thing you want is for the hackers to gain access to one, two, a hundred of your client websites.

Would you, if you were in the business of building WordPress websites for clients, would you rank that as a fairly good priority? Should people be going out and informing their clients that, look, I actually held this in a LastPass vault. That vault has been breached. Do you know if we have any obligations for our clients? And would you recommend that they, being circumspect essentially? Go out there and start changing these things pronto.

[00:29:21] Robert Rowley: Yeah, oh absolutely. I believe they should definitely get ahead. The best way you can react to any sort of security incident is to get ahead of anything else bad happening. You have to say the bad news, right? That starts with it. You have to tell that your customer’s, I store, I was storing your website account passwords in LastPass and as you may know, LastPass had a breach. All you have to do, if the next sentence is, I have changed the password. That’s it, and then you can say very confidently that there is no longer any risk associated with the fact that I used to store the WordPress password, WordPress access website access passwords in LastPass.

That’s how you get ahead of a security incident. And that’s a, that’s another great way to approach security as well as security hygiene thing. If you do experience a, let’s say LastPass aside, we’ll just put that as not the issue here at all, you experience a compromise on one of your customer’s websites. If you try to go in and manually clean it up yourself and you don’t know what you’re doing. And, you know, the hacks persists. If you inform the customer, hey, we saw this, it looks like it’s hacked, here’s what we did.

And it gets hacked again, say, oh, we’re going to do more this second time, right? We’re going to do more. We’re going to hire an outside party now because obviously our services didn’t meet the needs. And that’s how you get ahead of the problem. Whereas if your customer’s site experienced a hack and you try to clean it up and you don’t tell the customer anything, you just hope they never notice, and then they get hacked again. What are you going to do that second time, right? Are you going to keep trying to clean it up? You’re going to keep going through this process? Or you’re probably going to create a little lie saying, oh, you got a hacked site and now we’re going to hire this third party.

But, what I’ve seen in my experience, in my career, the sooner anybody’s ever transparent and upfront with the incident as it happens, and they are as clear as possible, including having a recourse, basically, here’s the next steps we’re going to take. That’s the clearest sign that somebody’s taking security responsibly, right?

They have a mature security model. They understand that breaches happened. These things you know, they didn’t cause it, some hacker caused it somewhere. Some nefarious person is doing something nefarious. But here’s the things that we did to address the issue. We’re aware of the issues. Here’s what we do to fix the issues in the future. You look at it as a learning experience for everybody involved. We could reflect back on LastPass and say, well, why was it that they saw the compromise start in the summer, yet it wasn’t until the very dead of winter that they announced the worst part.

The one thing that everybody was most concerned about. Had they done that at a different time, it may be different. The PR, right. How it would look to people would be different.

[00:31:48] Nathan Wrigley: The recommendation, I’m guessing, that you would have is that, you said a moment ago that 2FA ,two factor authentication, really you should be using that where it’s available. And I know that in WordPress there’s a whole slew of different ways of doing that. For example, the company that you work for, Patchstack, they offer a 2FA option, as do a whole bunch of security vendors.

But there’s also plugins which just simply do that one thing. Would you be recommending that for every username and password on any WordPress website, or are you kind of limiting this to the administrator roles and the other ones perhaps less of a concern? I’m just trying to get an idea of how judicious you think you would need to be if you were a website agency at this point informing your clients that there’s possibly a breach, and trying to guide them towards better solutions, more robust things like 2FA.

[00:32:43] Robert Rowley: Yeah, it’s a good question. I would agree administrator users make the most sense for these stronger, or higher requirements for authentication. What you can think of it is, it’s not just the administrator user necessarily. It’s any user that can upload a plugin, upload a theme, edit PHP files. Any of those key roles or capabilities within the user are what are important. Those would directly connect to compromise the website, right. If a user is compromised and they have ability to upload a plugin, that plugin they upload could just be a backdoor.

So you should start with that, understanding the capabilities, if you have unique capabilities and unique custom roles built into your WordPress website. If you don’t, then it’s easier. Yes, admin users are the ones that can upload plugins and such. So those are the ones that you need to make sure have stronger authentication requirements.

As for the remainder of the users, that’s really up to the organization of the website owners. Their ability to understand risk, right? It may not be that bad if a, well, it’s not necessarily bad at all if a subscriber account gets compromised. It’s not good, but more concerning if an author account, right?

They could start editing posts that were published by that author or things like that. But if you prevent the authors publishing new posts by having an editor role who needs to approve things, then you’ve got a good little safeguard there, right? An author getting compromised isn’t the worst thing either. However, they should have been using strong, unique passwords, because that’s the basics.

Do they need 2FA? Maybe, maybe not. And then you kind of go up, as you go up the roles and capabilities of every user group in your WordPress website, maybe you’re thinking, yeah, this person can do this thing, and that would be horrible for our business, right? Maybe you have a role that’s specific for handling your shipping items, right? Or your coupons for your WooCommerce site, or something like that. Those roles, those custom roles, would be a very high effect if they were to be compromised. So maybe on those, those accounts, right?

If they handle your customer data, shipping information, coupon codes, right? You don’t want somebody creating a 99% off coupon code. So you want to lock those accounts down too, with a higher level of requirement. And I’ll be honest, that after you get used to the process of 2FA or some of the other options, right?

Instead of a password, a pass key, or IP address limitations. People can only log in from certain zones or certain areas. Once you start doing that and just becomes part of the process of logging into the website, it really becomes not a big issue. It’s that initial, that initial adoption period that you’ll have the most pushback and then people get used to it.

And most people find that 2FA, well, it can be annoying if you can’t find your phone, or if you can’t find the physical key. Generally you remember to pack it. So you, you’d end up not ever going anywhere. You don’t take your laptop to go work on your WordPress website without also bringing your 2FA token with you, so it just becomes a habit.

[00:35:25] Nathan Wrigley: I confess in my case, I began using 2FA, almost as soon as it was an option. And I remember really disliking it to the point where I disabled it and I did another six months, and then I thought, actually, do you know what, there’s a lot of sense in this. So I switched it back on. So this is going back quite a number of years. And it really has become part of the muscle memory of logging into a site. You know, I go there, I type in the username and the password, or in my case, the password manager handles that.

And then I get this additional prompt. And all in all the whole thing is an additional possibly 10 seconds. And whilst it’s irritating, that 10 seconds probably could be better spent. On the grand scheme of things, it’s really not that amount of time. And I always thinking that if something is inconvenient, then it’s probably a good idea. With greater inconvenience, probably lies greater security.

[00:36:20] Robert Rowley: I don’t know if I’d fully agree, cause I can think of some greatly inconvenient things, but you have the right idea. I remember, yeah, many years ago, I was working at DreamHost and we rolled out 2FA for access to the DreamHost panel. So this is access authentication where somebody could take over all your websites. They could migrate your domains elsewhere, right? Like it’s your whole business.

And there was pushback. There was genuine pushback and it was an option too. It was quite funny. But, there was genuine pushback from our customers saying, I would never enable this because what a waste of time it takes to type in this code. But I think over the years people have simply adopted how to use it. I’m sure, I’m not this old yet, but I’m sure back when passwords were first created, right? The idea of having to log in with both your username and some sort of password caused some uproar at some university’s on old Unix systems. They’re like my login should be just my login. We should trust everybody. And then of course, you know, they learn that, yeah, you need to do a, some form of challenge response to verify authentication, who the user is, who they claim they are.

[00:37:23] Nathan Wrigley: It’s not only something that you know, your password and username, but it’s also something that you have, a physical possession, in this case, a phone or a Fido key or whatever it may be. It really adds that extra layer.

One of the things that we keep talking about, I guess it’s par for the course really, given the nature of the conversation, is passwords. The fact that we have to memorize a combined thing. There’s a username and a password. I don’t quite know how that came to be, the way that we logged onto more or less everything, but there’s these two fields. Username, typically an email or some kind of thing that you’ve decided to use. Might be a, a shortened version of your own name or something like that. And then there’s the password which sits alongside of it. Given that that system, should it be discovered, allows complete access to whatever is in that service, Google, Dropbox, whatever it may be.

I’m . Wondering if that system is broken. I wonder if it’s time to get away from, or slowly start to move away from, the username and password combination, which allows access to everything once successfully done. Even though it can be married with 2FA, like we’ve described. And I’m wondering if any new and emergent technologies have passed your radar that may be replacements for things like usernames and passwords. I’m sure there must be some ingenious cryptographers out there somewhere trying to get rid of this devilish thing, the username and password, but I don’t really know much about them. So I’m just going to hand it to you and see if you do.

[00:38:55] Robert Rowley: Well, I’m going to first start off saying I do not understand enough about cryptography. And so you’re absolutely right to call them genius cryptographers. They’re phenomenal at math. I’ve read their papers and I, my eyes still gloss over. But I understand their high concepts, which is why they’re truly geniuses, is that they’re able to understand the big concepts of this very convoluted math to more lay people.

It is true. What we’re dealing with is a lot to do with cryptography. This is a matter of a secret which is known to an individual, which is then stored somewhere and then verified so that we can be decrypted in a way that we confirm that the individual is the only one who we assume has the knowledge of that secret to decrypt this vault or password cache or things like that.

So what we’re dealing with is secrets and cryptography. Oh boy, I don’t even want to get into the whole cryptography thing, but like, public-private pairs. Where you can store them. How you can store a public key, and the public key is not a big deal if it’s been stored or shared publicly because it’s only your private key that can decrypt data that’s been encrypted using the public key.

Basically this two key system. When you really kind of like, understand how it all works, you’re like, oh, okay, cool, this makes sense. But really in the end of the day a password is just a key. It’s something you know. A good analogy maybe is when you were using LastPass, you and I were both using LastPass, probably for a number of years. Did you have a knowledge of your passwords?

[00:40:17] Nathan Wrigley: No. None whatsoever. I always go for a very long gibberish password. And even if you forced me to read it out, I wouldn’t be able to memorize even one of them. They were so ridiculously long, yeah.

[00:40:30] Robert Rowley: Exactly, so that’s how I was using LastPass as well. LastPass, I knew my master password, which was being used to decrypt these gibberish long passwords, which were all stored in LastPass. And I was using LastPass as this storage device for these long gibberish passwords. But they’re not words, they’re not pass phrases, right?

Let’s define a few terms. Passwords sound like a word, which is a secret word, like, open sesame, to enter a, get authentication to enter a system. Passphrase is another terminology that the security community pushed out there for a while. And this is more like, horse, battery, banana, stable. Something like that. You’re creating a phrase, a whole sentence, which makes a longer word. It’s not really a word anymore, it’s a phrase. They’re trying to encourage people to use sentences. I used an example earlier. I love pancakes, right? That’s something I encountered in my life, for bad pass phrases.

But, now we can get into a new world where we can define this as a pass key. And now a pass key is kind of what that thing that we’re using LastPass to do. We’re storing this large gibberish, basically a little blob that we don’t know, we can’t even pronounce if we wanted to. And that is the secret that is being stored and saved with the server, or basically not saved with the server, but as a challenge in authentication step within the server.

And we’ve extended what started as a password and as the common term would be password to passphrase. And now we have this new thing called a pass key. And how we were using it in LastPass is bastardized version of what a pass key should be. And there are new technologies now, being the thing that on the internet. Because web browsers make web applications accessible to the whole wide world, we’re starting to see that pass keys, this high level of entropy. This long amount of gibberish. This inability for an attacker to brute force the authentication step is what we’re needing in order to protect ourselves against attackers.

And in that case, in that sense, pass keys are actually a real thing. You don’t have to implement them using LastPass, using long gibberish things that you can never remember yourself. But you can use them by storing them locally, or having a system that can unlock that key only when you basically, like we were using with LastPass. It could unlock the key, which then is being sent to the web server, web application to pass that challenge for authentication.

There are plugins for WordPress, which are already available, and they will utilize a system that’s more of a pass key system. They’re not always how we were explaining with LastPass. Some of them will use your phone, it’ll scan a QR code, and when your phone can decrypt the QR code correctly, it will pass the challenge, the authentication challenge that the plugin, or the web application, has presented. So it verifies you as authenticated. I hope I’ve explained that right?

[00:43:08] Nathan Wrigley: Yeah, let me just outline whether or not I’m confused about that. So with passwords and pass phrases, essentially both parties need to know what they are. So the website, let’s take the example of Google, Google needs to know what my password is. I need to know what my password is. And in order to keep that secure, I encrypt it inside my password vault and Google encrypt it on their servers with whatever technology they have available. Hopefully, Decent and strong.

But the point is there’s two secrets held in two different locations. The same would be true for pass phrases because it’s just another, it’s just a really, it’s more or less exactly the same thing. But is a pass key in any way different to that? Is it being stored in both locations. Do I need to store a copy of the pass key and does the website need to store a copy of the pass key? Or is there something going on which is slightly different where only one of us knows? That’s the bit that I haven’t quite worked out.

[00:44:05] Robert Rowley: What you were explaining is what’s called a two-way or symmetrical encryption. Both parties know a secret and they both use that same secret to confirm a identity, right.

[00:44:15] Nathan Wrigley: Yeah, thank you.

[00:44:16] Robert Rowley: If the bouncer on the other side of the door knows the secret password to enter the club is open sesame, then they wait for people to say it on the outside. Both parties need to know this. There’s another way with encryption. This is why cryptologists are geniuses, which is called one way encryption. One way says that the bouncer on the other side of the door actually does not know what the passphrase is.

What they know though is for your user, some mathematical equation, right? I’m going to simplify this. That will embarrass myself, because I’m going to go to junior high level maths. And this is a terrible example, but like one plus x is equal to seven. Solve four x, right. It’s that sort of thing.

But they’re doing mathematics, which are like multiplication charts, an elliptical curve. They go way out there. So it’s very hard to do this, you can’t do ’em in your head. But they do, and actually they’re using prime, I believe. They’re using prime numbers, which are hard to, it’s hard to calculate in the reverse direction.

It’s that sort of idea is that the bouncer on the other side knows the maths to do, right? They know the algorithm, or not the algorithm, they know the equation. And on the other side, you just simply say the word six, and then the bouncer on the other side puts six into this math equation. They run this math equation, which depending on the speed of the bouncer’s CPU in his head, he has one plus six is equal to seven, is that true? And that’s how they work.

So the bouncer does not know at any point in time what the secret is until you give it to him. And then that, basically then he uses the math behind the algorithm to verify that the secret is true. Does that make sense? That’s a one way.

So you have a application, we’ll see this with GPG or PGP, which is a public-private key system, where your private key is what you need to keep secret. And then the public key is what’s shared publicly. So anybody could know your public key and then they can compare it. And then you would basically, they would use the public key to encrypt data to you, and then you would use your private key to decrypt the data or vice versa.

[00:46:06] Nathan Wrigley: Yeah, I understand. I think I’m hoping for an era in which the knowledge that I have doesn’t need to be known at all by them. So I could display something to a website or a SaaS app or whatever it may be, and the mere fact that I possess it combined with something that they possess. But the two never need to collide, if you know what I mean.

I can constantly keep my thing secret. They can keep their bits and pieces secret. And I believe there are endeavors to do things like that. I think in my case that the LastPass data breach has made me realize that having trust for all the things in a third party service, that’s been shaken a little bit for me, over the last few weeks. And I would like to hope that things, I don’t know, inside the browser or inside the Mac or inside the iPhone or whatever it may be, will make this easier over time.

[00:46:59] Robert Rowley: Yeah, definitely. It’s a thing that’s always going to change, but we have to remember, we’re using computers and they’re reliant on math. And unfortunately it’s going to be up to some really, really, and this is why I’m so nice to cryptographers, I think they are all geniuses. It’s going to take some really smart guys and girls as cryptographers, to figure out the algorithms that are going to work and be resilient against attacks like bruteforce attacks.

That’s what LastPass was doing, was hashing your password. So one over 100,000 times because if you hash, that makes it 100,000 times harder, or slower, for the process of hashing, right? So hashing it once might take a millisecond, but hashing it a hundred thousand times, now that takes a second or so. And that’s what they did based on the technology and the cryptography as we understand it today, the applied cryptography, I should say. That was their best option to slow things down.

And doing that, choosing the algorithms they chose, choosing the bit sizes for the keys that they chose, choosing the number of iterations of hashing that they chose, all gave us some time. And that’s the time that we needed to update and rotate out our passwords and our secrets that unfortunately were lost.

[00:48:08] Nathan Wrigley: Well that was a good segue. You mentioned time and time is slowly running out for us. But I just want to give you an opportunity to mention a couple of the things that you are doing in the WordPress space, which are nothing to do with LastPass, even though the name of it may be, with retrospect, may collide very closely.

Tell us about Last Patch, which is a project that you’ve been involved in, don’t know for how long, but tell us what you’re doing over there. It’s wonderful actually. It’s such a nice, almost philanthropic thing.

[00:48:37] Robert Rowley: Just a bit, yeah. Unrelated to LastPass, I’ve been writing a series of blog posts that I’ve been calling Last Patch. The concept here is that, I wanted to write about vulnerabilities and exploits on WordPress plugins for some time, but I’ve found it, it’s not very nice to talk accidents and mistakes other peoples have made, especially when they’re still writing actively to the project, right?

Like, I don’t want to take a, somebody patch this bug, this security bug, which is a more sensitive bug than a normal one. And I don’t want to just put ’em on blast saying, hey everybody, here’s how you attack this bug in case nobody updated yet. So instead, what I did is I found out, and this happened last year, we were writing a white paper, which basically was explaining about a 2021’s, a year in WordPress security retrospective.

And we found out that a good handful, I think it was seven or eight plugins were disabled and they had, out of 30 critical vulnerabilities that were reported in WordPress plugins in 2021, about seven or eight plugins received no patch. So a critical vulnerability received no patch, and sites were still simply running an insecure version of these plugins.

And that’s not good either. So a solution oriented towards fixing things instead of just pointing out mistakes, was that I wanted to start writing patches for these abandoned plugins that had security bugs in them. So in quarter four of 2022, I was given some time and I wrote up a few blog posts about six in total explaining for six, each blog post is its own plugin that has its own vulnerability in it, and none of those plugins got patched in 2022. So what I went through and I went ahead and just wrote the patches. I explained how the vulnerabilities worked. The target audience for this could be really anybody.

If you’re a site owner, and you’re running one of these plugins and you want to patch it yourself, I don’t recommend running my patches, because these are abandoned projects. If it’s an abandoned plugin, I recommend you find a new alternative. But if you absolutely need to run it, yeah, you can use the example that I’ve given you.

But it’s even better for a developer. If you’re a WordPress developer or even a new developer, and you’re kind of curious how about how security bugs work and what to do when you encounter them, these series of blog posts will walk you through how these security bugs work and how to basically write a patch.

Most security bug patches are pretty rudimentary. You’re going to be looking at writing an allow list, verifying authorization, or sanitizing or escaping data correctly. So that’s the series of blog posts that I’ve released, and they’re all available on the, LastPass, last sorry, Patchstack blog.

[00:51:03] Nathan Wrigley: Too many patches and lasts in this episode. Pass, patch, last. There’s lots going on. Yeah. That’s amazing. What a nice endeavor. Do you intend, funding and time permitting, is this something that you would wish to continue? Are you going to try to do this through 2023?

[00:51:20] Robert Rowley: I would hope so. I genuinely had fun writing these patches. Reviewing the code. I’m an awkward person in the head, I guess because I like looking at other people’s source code, figuring out what went wrong, and adding, because I, I’ve done that in my career for the past 15, 20 years.

I’ve reported security vulnerabilities to developers. I’ve become empathetic enough to understand their position of not wanting to see the report, not enjoying that process. But I genuinely enjoyed this process and I like sharing with other people, especially developers. The idea that a mistake, mistakes can be made, right?

Mistakes were made. It’s okay. It doesn’t matter what happened. What matters is how you respond to it. And you should be responding to security breaches like LastPass. Or you should be responding to security bugs, like open source developers. The majority of them, I’ll have to share here, the majority open source developers are very receptive to security bug reports.

It should be considered a contribution to the project. It’s a way to make the project better, more secure. And as a developer for the developers, it’s a way to improve your skill sets. You know how to identify, like if you take it seriously, you’ll learn how to identify security bugs, how to program defensively so that security bugs don’t affect your application, and so on and so forth.

[00:52:34] Nathan Wrigley: What a great endeavor, yeah. Thank you for doing that on everybody’s behalf. That’s really wonderful. Robert, because we’re close, closing in on an hour, I’m going to knock it on the head. But before that, I’m going to ask you to tell us where we can find you online. If anybody’s listened and wants to reach out. Do you have any publicly available Twitter handles or email addresses or contact forms that you want to mention?

[00:52:57] Robert Rowley: Sure. A great way to follow me online nowadays is Mastodon. There’s been this wonderful guy who created a wpbuilds.social Mastodon account. I’m on that as well as rawrly, r a w r l y. I am apologizing for such a weird name, but that’s also my wordpress.org username.

So if you follow me there, you can see what I’ve done on wordpress.org and you can find me on the wpbuilds.social Mastodon account. You can also, if you want to keep up to date and you don’t need to talk with me, but you just want to hear more about security topics and information, you can go to the Patchstack blog. I write articles there on occasion.

And every week I do a Patchstack Weekly, I think it’s episode 53 right now. And for the beginning of this year, all I’m going to be talking about a security hygiene best practices. All those things that you maybe should be doing about once a year. That’s my New Year’s resolution, to get a handful of these things shared with the public so that they can take, especially the WordPress public, this community can take security more seriously. Just knowing what to do is really what most people need. So again, Patchstack.com, or wpbuilds.social.

[00:53:59] Nathan Wrigley: Robert Rowley, thank you for chatting to us today on the podcast. I really appreciate it.

[00:54:03] Robert Rowley: Thank you.