Verify updates on air gapped servers

I'm looking for a way to verify that all security patches have been applied to a set of air gapped servers. Currently, I have a script that is run on each server during patching that generates a list of installed RPMs after update. I'm trying to verify that list of packages against the RedHat Security Data API. The code I'm using is here: https://gitlab.com/devans10/check-rhel-sec But I'm not getting results I expect.

I'm wondering if there is a better way to accomplish this.

Edit: A bit more information.... These systems are patched via a disconnected Satellite. We had an instance where an update was missed because it was not in the repository. We can only speculate on what happened, because it wasn't discovered until a future patch cycle when we saw an older patch being applied. So we are looking for a way to audit the patches outside the repositories.

Posted Aug 18, 2018, 5:17 PM UTC · edited

5218
plastikman187•

Build a spacewalk, satellite, or Foreman setup. You can use that system as a proxy for patches and updates.

You can pull in the errata and see what patches are required.

16
devans67OP•

Sorry, I guess I should have been a bit more specific. These systems are patched via a disconnected Satellite. We had an instance where an update was missed because it was not in the repository. We can only speculate on what happened, because it wasn't discovered until a future patch cycle when we saw an older patch being applied. So we are looking for a way to audit the patches outside the repositories.

8
evilgentoo•

Nessus can audit systems for missing patches. I'm not sure how complete it will be but it should catch the big ones.

9
burning1rr•

There is not much you can do to avoid the risk of missing and update if you are reliant on a single system to manage those updates. Best bet is to run additional security auditing, and maybe download the RedHat CVE data separately to audit.

5
wossack•

I’d have thought it easier to audit the repo’s then each patched server?

7
devans67OP•

That is a good idea. Any ideas on how to do that?

Are there any tools that mirrors use, or is it possible to checksum a repo?

4
SuperQue•

RedHat checksums all their repos, and I think signs those files.

I would look at the RedHat official docs on the subject.

5
DigitalPlumberNZ•

"Easier to audit the repo's then each patched server"? Or "easier to audit the repo's than each patched server"?

Not trying to be a dick, but the latter sounds like way less work and I cannot deduce your intention from the sentence.

-1
wossack•

My bad - not one then the other, but instead of the other

3
moofishies•

I cannot deduce your intention from the sentence

I mean, yes you can. You are just choosing not to.

0
DigitalPlumberNZ•

No, I really cannot. Not with certainty. OP presented a complex scenario where auditing the server then the clients may well have been easier than other options.

0
devans67OP•

So I'm now looking at oscap with OVAL content for this. I had always looked at openscap as security compliance monitoring, but with the OVAL content downloaded from https://www.redhat.com/security/data/oval/com.redhat.rhsa-all.xml.bz2 that coordinates with my repository sync. I can scan the servers immediately after patching, and generate a report that will "hopefully" show the server being fully patched.

3
bdniner•

Nessus is a great tools for doing this and you can load SCAP and STIG content for compliance scans as well.

2
[deleted]•

[deleted]

1
three18ti•

This is exactly the situation InSpec was designed for. It allows you to define a standard and in a consistent and repeatable way, test your entire infrastructure.

You can run it remotely, aka "agentless" or via some agent. We use Chef to initiate the inspec process, but all controls are hosted remotely and downloaded each run. (The profiles themselves are kb so the "download and verify controls" each run is trivial) In some instances, we're leveraging habitat to run our InSpec profiles and it works a dream.

Our workflow is basically:

  • add updates to new channel
  • update channel via Chef Env
  • next Chef run node updates
  • at the end of that chef run, runs InSpec profiles to validate state
  • nodes that don't check in with new versions are flagged for review

We generally roll out in stages, e.g. the lab/pre-dev has internet access. Dev has no internet access but access to our repo proxy (we don't use Satellite, But close enough). Test has dedicated near prod-like (might have broken packages that don't make it to prod) repo server, prod has dedicated repo server.

And when package updates come, we don't dread it.

Happy to talk more. But ultimately, I believe you need a tool to manage the update process, and, for lack of a better term, an "independent third party" to validate those changes.

1
AfterSpencer•

I have been able to audit for CVE impacted packages with Hubblestack in an airgapped environment.

1
BeerJunky•

It’s air gapped, fuck patches. YOLO!

-4
HildartheDorf•

I assume you are joking but...

You still need updates for exploits where an active attacker is on your air-gapped network or has some form of physical access. Your surface area may be massively reduced without the internet, but it's still not invulerable.

2
BeerJunky•

It’s a joke of course. But this of course is why physical access is the most important security concern IMO.

0